Assessing and Monitoring Literacy, Numeracy and Life Skills ...

PACIFIC ISLANDS FORUM SECRETARIATPIFS(10)FEDMM.13(a)FORUM EDUCATION MINISTERS’ MEETINGCrown Plaza Hotel, Port Moresby, Papua New Guinea13-14 October 2010SESSION TWELVEASSESSING AND MONITORING LITERACY, NUMERACYAND LIFE SKILLS – PROGRESS REPORTThe paper provides an update and further recommendations on the work currently underwayby the regional and international organisations (SPBEA/UNESCO/UNICEF) on theBenchmarking of Literacy, Numeracy and Life Skills and will be presented by SPBEA.

PACIFIC ISLANDS FORUM SECRETARIATPIFS(10)FEDMM.13(a)FORUM EDUCATION MINISTERS’ MEETINGCrown Plaza Hotel, Port Moresby, Papua New Guinea13-14 October 2010ASSESSING AND MONITORING LITERACY, NUMERACYAND LIFE SKILLS – PROGRESS REPORTA new role for assessment has however emerged recently. Assessment is being usedaround the developed world to „gear up‟ education systems in response to global changes.Economics drive the rhetoric: technological developments demand better educated, morethoughtful and flexible workers across the labour market, to strengthen the country‟stechnological base and to foster a spirit of enterprise and initiative. The apparentmismatch between the output of the schools and the needs of the labour market in the1980s, as indicated by the number of unqualified school leavers and by the number ofyoung unemployed in various countries suggested that education had departed from the„real world‟ of work. The result has been to seek to re-couple education with economy(Neave 1988). A key component in this role for assessment is the use of national andinternational testing programmes to monitor performance – and competitiveness.(Stobbart & Gipps 1997)PurposeThis paper provides an update to the status of the monitoring of literacy, numeracyand life skills in the region that was presented to the Education Ministers‟ Meeting held inTonga in March, 2009. The 2009 paper was part of the status report on the Monitoring andEvaluation initiatives, which is included as Attachment 1.Introduction2. Competency in literacy, numeracy and life skills need to be developed from an earlyage. These are essential if pupils are to secure the maximum benefit from a broad andbalanced school curriculum. Proficiency in literacy and numeracy is also essential if youngpeople are to fulfill their potential in their careers and as adult members of the community.The link between low basic skills and employment is well researched. Tertiary institutionsand employers alike, regularly bemoan the standards of literacy, numeracy and life skillsshown by students emerging from secondary level schooling to either enter the workplace orto pursue further study at tertiary level. Admittedly this is anecdotal evidence, but it isrepeated so often that there is likely to be more than a grain of truth in the claim. Clearly it is2

in the interests of all countries to maximize the potential of its citizens, and raising literacyand numeracy standards is one way of contributing towards that maximization.3. Within the Pacific context, with each country having its own language or set oflanguages, literacy should focus on both the development of skills in the vernacular and inEnglish which is perceived as the medium for regional communication.4. At a regional workshop in 2006, participants from the Pacific Island countries cameup with the following definitions for literacy, numeracy and life skills.“Knowledge and skills necessary to empower a person to communicate through any form oflanguage of their society, with respect to everyday life”(Pacific definition of Literacy)5. A person is considered to be functionally literate if she/he has acquired the necessaryknowledge and skills to be able to communicate effectively through any form of language oftheir society, with respect to everyday life.6. The literacy status of a person between the ages of 6 to 14 years will be determinednationally and regionally (if required) by referencing his/her literacy skills to the indicatoroutlined in the regional benchmarks. However, a person is considered to be functionallyliterate if he/she has completed four years of formal education and has met the indicatoroutlined for Year 4.“Knowledge and skills necessary to empower a person to be able to use numbers inmathematical processes, as well as the language of mathematics, for a variety of purposes,with respect to every day life”(Pacific definition of numeracy)7. A person considered to be functionally numerate is therefore someone who hasacquired the necessary knowledge and skills to be able to use numbers effectively inmathematical processes, as well as the language of mathematics, for a variety of purposes ineveryday life not only within the society he/she lives but beyond.8. The numeracy status of a person between the ages of 6 to 14 years will be determinednationally and regionally (if necessary) by referencing his/her numeracy skills to thebenchmarks. However, a person is considered to be functionally numerate if he/she hascompleted four years of formal education and has met the numeracy benchmark outlined forYear 4.“Practical knowledge, skills and behaviours which in conjunction with culturally rootedknowledge, attitudes and values, empower an individual to live a happy, healthy andproductive life in his or her community and beyond”(Pacific definition of Life skills)9. The Forum Leaders in their communiqué affirmed the 2010-2013 Pacific Planpriorities at the 41 st Pacific Islands Forum Meeting held in Port Vila from the 4 th – 5 thAugust, 2010. It is noteworthy that at this meeting, trends in literacy and numeracy werehighlighted by CROP executives „one of the five key issues in the Pacific Plan priorities; to3

focus education efforts on increasing literacy and numeracy rates in selected Pacific Islandcountries‟.10. Over the past several years, SPBEA has been monitoring literacy and numeracystandards for use principally at two year groups, namely year 4 and year 6. This has beendone in Kiribati, Vanuatu, Tuvalu, Solomon Islands and to a lesser extent in Nauru andTonga. Part of the process of constructing a curriculum based standardized test is to identifya cross section of key skills that can be considered representative of the particular subject.Once the key skills are clearly identified, a statement or a series of statements known as„learning outcomes‟ are carefully framed in clear and specific language. Through themonitoring of standards, and subsequent intervention, whose construction reflects the needsexposed by the data from the monitoring process, classroom standards of literacy andnumeracy can be improved.Current Statusa) National Level11. Reports of baseline achievements in literacy and numeracy have been completed forfive countries (Kiribati, Solomon Islands, Tonga, Tuvalu and Vanuatu) for both Year 4 andYear 6 students. The CEO of the Ministry of Education in each of these countries hasreceived their specific reports. Each report contains comprehensive data, relating toachievements, at National, Provincial and Island levels, broken down by gender, and by keystrands and key skills. The reports also contain a set of recommendations for consideration.12. Each report presents sets of baseline achievements, against which data from futureadministrations of the tests can be compared. Although country reports do differ from eachother in many details, it is noticeable that the Region faces similar problems in theachievement of both literacy and numeracy skills.13. Three of these countries have conducted a second administration of the tests fromwhich results are compared against the baseline achievements. The reports for two of thesecountries have already been completed and disseminated, while the remaining report is inprogress. The fourth country will run its second administration of the standardised teststowards the end of 2010.Early Grade Reading Assessment (EGRA)14. The World Bank has also been engaging with a couple of countries in carrying outearly grades reading assessment (EGRA) using an adaptable ready-made toolkit to testreading skills. Tonga administered EGRA in 17% of classes 1-3 primary school students inlate 2009, following an earlier pilot test. The Tongan EGRA report, to have been prepared bythe World Bank, is not yet available. Vanuatu prioritised EGRA activities in its 2010 workplan, and carried out its pilot administration in the last few months. Both Tonga and Vanuatuhad the EGRA tests translated into their preferred local languages. The Solomon Islands,however has decided to shift any EGRA work to a later time.15. It should be noted that development of standardised tests for Life Skills is pioneeringwork in the region and lags behind those for Literacy and Numeracy. Since there is no Life4

Skills curriculum per se in the countries, the identification of life skills within the nationalcurricula needed to be done in the first instance. Such content is then used as a basis fordeveloping country-specific assessment instruments on life skills.16. SPBEA and UNICEF have been providing assistance to Kiribati and Vanuatu incurriculum mapping and development of assessment instruments for life skills since 2009.Thus far, Vanuatu has already developed and trialed her life skills instruments for Years 4and 8 in July this year, while draft instruments have already been developed in Kiribati. Workis gaining momentum in the next few months, and it is expected that a full administration inthese two countries can be conducted towards the end of the year. The experience gainedfrom these pioneering countries will be used to support the development of similarinstruments in the other countries.Literacy and Numeracy Project17. A major breakthrough in efforts to strengthen the institutional and technical capacityof SPBEA to meet the increasing demands on its services, especially in the delivery ofnational assessments of literacy and numeracy will be achieved through the newly approvedAusAID funded project, Improving Literacy and Numeracy Assessment. The project willsupport SPBEA activities in four broad components: institutional capacity building, staffdevelopment, development of early grade assessments, and strategic planning.18. Specifically, the proposed activities of the project will:b) Regional Levelprovide technical support to Curriculum Development Units and AssessmentUnits in the region in carrying out literacy and numeracy assessments at year2, 4, 6 and 8;design and conduct the third round of national monitoring of literacy andnumeracy at years 4 and 6 for five countries in the region, and establishbaseline data through national monitoring tests of literacy and numeracy infour (or more) countries;begin the development of early grade assessments;analyse, report and disseminate the results of assessments to policy makers, aswell as recommending intervention strategies; andprepare longer term strategies for development of SPBEA to meet the needs ofan expanded clientele through organizational and human resourcedevelopment planning and development of additional assessments of learningachievement for use in the region.19. Most countries are ensuring that the skills defined by the Regional benchmarks areappropriately incorporated within the curriculum at the indicated year level. This opens upthe opportunity for Regional versions of the standardised tests to be developed, which focuson skills common to all countries and which are aligned to the regional benchmarks. Thebenchmarking work presents an excellent platform for regional collaboration in enhancingquality education and thereby meeting PEDF goals.5

20. SPBEA, together with other development partners, UNESCO and UNICEF arecontinuing to collaborate under the proposed extension of their partnership agreement to2013. It is expected that the extended partnership agreement will continue with the existingarrangements: for SPBEA to look after national level monitoring, UNESCO and SPBEA tocollaborate on regional monitoring, while SPBEA and UNICEF work together on life skills.As previously agreed, the regional assessment for literacy and numeracy will be conducted atYears 2, 4, 6 and 8 while life skills will be tested only at Years 4 and 8.21. Preliminary discussions on the development of the regional assessment of literacy,numeracy and life skills were held between UNESCO and SPBEA, and a tentative timelineidentified to progress the work in the latter months of 2010, and early 2011. It isrecommended from these discussions that the pilot administration of the regional tests are tobe carried out in Fiji, Kiribati, Palau, PNG and Samoa, in consideration of country size andsub regional area representation.22. A tentative implementation schedule, indicating responsible organisation has beendrawn up and is shown in the table below:TimeOutputs/activitiesAugust/Sept 2010 Initiative is communicated to countries (SPBEA) Regional Assessment Framework design is finalised (UNESCO) Country samples identified down to school level and communicated toparticipating countries and schools (SPBEA)End of September Draft assessment instruments for literacy (SPBEA) and numeracy2010(UNESCO) are ready.13/14 October 2010 Presentation on the initiative at FEdMM (SPBEA)November 2010 Trialling of instruments for Year 6 (SPBEA)March 2011 Trialling of instruments for Year 5 (end of Year 4) (SPBEA)23. At the adult level, Development Partners are working towards implementing a literacysurvey throughout the countries in an effort to obtain concrete data on the level of adultliteracy in the Pacific and to set baselines for future use. In this regard, UNESCO is currentlyworking with Papua New Guinea in an effort to collect data on actual adult literacy as well asdata on the literacy environment in three provinces in an effort to get information on the adultliteracy situation in the country. This involves the development of an instrument to assess theactual adult literacy level as well as one to assess the conduciveness of literacy environment.Once completed (late 2010) the instruments will be adapted for use in other countries in thePacific.Regional Workshop24. Along with the work of developing assessment instruments and training of assessmentunit staff on the conceptual basis and processes involved in monitoring standards, SPBEA hasalso developed an in-house statistical tool called STALLIAN 1 to process the data from eachtesting administration. Eventually, assessment units are expected to take on the fulladministration of their assessment and monitoring cycle.1 STALLIAN – Statistical Tool for Analysis of Literacy, Life Skills and Numeracy.6

25. SPBEA ran a regional Data Analysis workshop from 24 th – 28 th May, 2010 forprofessional and IT officers of the Education Assessment units in its member countries. Twodays were spent on STALLIAN work to provide participants with an overview of the purposeand capacity of STALLIAN and to give them practical experience with its usage. A keyoutput from these sessions was the production of a report that can inform their ministry atnational, district and school levels on issues related to literacy and numeracy.Challenges26. Dissemination of the reports is a critical aspect of assessment and monitoring work.Unfortunately, this step is often overlooked, resulting in key stakeholders being left out of theinformation loop.27. Increasingly, the countries are recognizing the importance of their literacy andnumeracy monitoring reports for planning and other purposes. Understanding the rationale,purposes, processes and limitations of standards monitoring is a challenge which may resultin misinterpretation and misuse of the reports.28. Training of Assessment unit staff, teachers and other key persons to be able to handlemonitoring work. Standards monitoring requires a change of mindset and different skillsfrom those used in the regular examinations approach.29. Competing priorities within ministries of education often pose challenge to theimplementation of standards monitoring. Standards monitoring requires staffing and fundingcommitment to a regular testing schedule if the comparison of standards are to be valid.30. The sustainability of standards monitoring in the medium to long term is a challengefor any system. For monitoring to be effective and meaningful, its existence has to beincluded in curriculum and/or assessment frameworks and national education work plans.31. At the adult literacy level, the key challenge is the lack of any formal structure fornon-formal education in countries thus making it difficult to define a “literate person”. Thismeans that while instruments are being developed, there has to be agreement as to theequivalency of literacy at the formal and non-formal sector. This requires consideration ofissue of equivalency between the formal and non-formal sector even if structure for nonformalsector may not be in place.Recommendations32. Ministers are invited to:(a) task their officials to ensure that assessment reports are disseminated by theirMinistry in a timely manner to appropriate sections of the ministry and keystakeholders for their information and appropriate interventions or actions;(b) encourage key education stakeholders to spend time to study, discuss andunderstand the monitoring reports, and are able to use the report findings toinform decisions related to the standards of literacy and numeracy;7

(c) support strengthening Assessment units within their ministries in terms of bothnumbers and skills to be able to deal with the work demand and intricacies ofstandards monitoring;(d) support the inclusion of standards monitoring in national assessment frameworksor policies, and its integration into education action plans as a core activity of thesystem;(e) encourage the alignment of National monitoring of standards in their Ministriesto the regional benchmarking framework; and(f) support the initiative by development partners (UNESO, SPBEA, UNICEF) toassess and monitor levels of literacy at the adult level (year 15+ and out of school)in an effort to set baseline both at the country and at the regional level.Pacific Islands Forum Secretariat, Suva21 September 20108

Attachment 1Status Report on Monitoring and EvaluationB. Monitoring of Literacy, Numeracy and Life Skills componentsPurposeThe purpose of this paper is to inform the Pacific Forum Education Ministers on thebackground and current status of work being done in the area of monitoring national literacy,numeracy and life skills. This work comprises one aspect of the wider monitoring andevaluation initiatives being delivered by partners including UNESCO, UNICEF and SPBEAas endorsed by the Forum Education Ministers in 2006 on the Regional benchmarks forliteracy, numeracy and life skills.Background“Literacy is a foundation of learning. While schooling is the principal route for acquiringreading, writing and numeracy skills, an exclusive focus on formal education for childrenignores stark realities: first, too many pupils leave school without acquiring minimumliteracy skills, second, one-fifth of the world‟s adult population – 771 million adults – livewithout the basic learning tools to make informed decisions and participate fully in thedevelopment of their societies.”EFA Global Monitoring Report 2006, UNESCO2. Achievement in literacy and numeracy has been shown to be a key determinant ofeducational outcomes. For example, students who achieve high levels of literacy andnumeracy during the compulsory years of schooling are more likely than lower-achievingstudents to stay on to complete Year 12 and enter higher education (Marks, et al., 2000), andto obtain higher tertiary entrance scores (Marks, et al., 2001).3. Achievement in literacy and numeracy has also been linked to a range of labourmarket outcomes. Making a successful transition from school to full-time employment, thetype of occupation obtained, and earnings are positively related to literacy and numeracy.Conversely, persons with lower literacy and numeracy levels are more likely to be outside thelabour force or unemployed, and to experience longer periods of unemployment. Literacy andnumeracy are essential ingredients for effective communication and participation in adult life.They have been linked to social outcomes such as community participation, engagement inlifelong learning, and health (OECD & Statistics Canada, 2000; Roberts & Fawcett, 1998).4. The central roles of literacy and numeracy are also reasons these and othereducational outcomes are monitored regularly by countries and groups of countries.American students‟ academic achievement is monitored periodically through the NationalAssessment of Educational Progress (NAEP), which comprises tests in reading, mathematicsand science administered to samples of nine, thirteen and seventeen year-olds in each state.Student progress in writing, history, geography and other fields is less frequently monitored.Two major international studies of student achievement are currently in progress, with morethan 30 countries participating in the Trends in International Mathematics and Science Study9

(TIMSS) in 1995, 1999 and 2003, and 32 countries in the Programme for InternationalStudent Assessment (PISA) in 2000, 2003 and 2006. TIMSS concentrates on achievement inmathematics and science, while PISA focuses on three types of „literacy‟ reading,mathematics and science. 25. In the Pacific Island Countries, the monitoring of literacy and numeracy was a majorfocus of the BELS programme operating in the decade of the 1990s. All participatingcountries used the same test instruments for both literacy (English) and numeracy. Theseinstruments were not constructed by establishing direct links with the curricula operating atthe time. The countries did not take the results seriously as they were not part of thedevelopment and the fact that BELS had its roots outside. In 2002 work started in a numberof countries on developing standardised tests based upon the specific curriculum operatingwithin each country. Since there were curriculum differences between countries, it was seenthat each standardised test should be specific to a particular country. During the years 2004 to2008, SPBEA worked with the curriculum and assessment units in six Pacific Islandcountries to develop, trial, modify and administer these tests. In each country the tests havebeen developed for use at the end of the primary years 4 and 6.6. There have been hiccups in the process. These have included; establishing confidencein the representative nature of samples (particularly at island or provincial levels);modifications to curricula upon which tests are constructed; developing expertise of markersin thinking „achievement‟ as opposed to „marks‟ and developing understanding of thedifference in purpose of these standardized tests and high stakes examinations.7. Nevertheless, valuable information is beginning to emerge in the areas of literacy andnumeracy; information that can assist education ministries in identifying priorities relating toprovision of resources and training.8. In 2006, a separate regional initiative saw the development, and acceptance of sets ofbenchmarks for literacy, numeracy and life skills. These were defined for the primary years 2,4, 6 and 8 for literacy and numeracy and years 4 and 8 for life skills. These benchmarks werepresented to the Ministers of Education at their 2006 meeting in Nadi, and were subsequentlyaccepted as regional indicators.9. Published literacy figures for Pacific Island Countries (PIC) are relatively high and ismisleading as it does not reflect the true situation in each country. However, the true level ofliteracy and numeracy depends on what each country considers as being literate and numerate.There is a general understanding that despite the high level of literacy published, there is ahigh level of hidden illiteracy resulting in a significant proportion of children completingschool but still lacking the basic literacy and numeracy skills to be able to survive in ourrapidly changing society.2 Future references to the Trends in International Mathematics and Science Study (TIMSS) willinclude the year of the study, using the Northern Hemisphere year (1995, 1999, 2003). TIMSS 1995was originally referred to as the Third International Mathematics and Science Study, and TIMSS 1999as the Third International Maths and Science Study–Repeat (or TIMSS-R). Six countries in theSouthern Hemisphere, including Australia, test between October and December of the previous year.All references to the Programme for International Student Assessment (PISA), which had its first cycleof testing in 2000, will also include the year. Participating countries in the Southern Hemisphere testtheir students early in the same calendar year as countries in the Northern Hemisphere.10

Current Status10. Reports for literacy and numeracy in five countries have been completed for bothYear 4 and Year 6 students. The CEO of the Ministry of Education in each of these countrieshas received their specific reports. It should be noted that development of standardised testsfor Life Skills is lagging behind those for Literacy and Numeracy.11. The main reason for this relates to the delayed identification of life skills within thenational curricula. Some countries are now in a position to proceed with development ofstandardised tests for life skills. SPBEA and UNICEF expect to provide assistance tocountries in this area during 2009.12. Each report contains comprehensive data, relating to achievements, at National,Provincial and Island levels, broken down by gender, and by key strands and key skills. Thereports also contain a set of recommendations for consideration.13. Each of the reports presents sets of baseline achievements, against which data fromfuture administrations of the tests can be compared. Although country reports do differ fromeach other in many details, it is noticeable that the Region does face similar problems in theachievement of both literacy and numeracy skills.14. Most countries are ensuring that the skills defined by the Regional benchmarks areappropriately incorporated within the curriculum at the indicated year level. This opens upthe opportunity for Regional versions of the standardised tests to be developed, which focuson skills common to all countries and which are aligned to the regional benchmarks.15. SPBEA together with other development partners, UNESCO and UNICEF are nowworking on developing the regional instruments based on the common items from thecountries. It is hoped that these will be trialed before the end of the year.11