CXC Examiner May 2012 - Caribbean Examinations Council

More documents

Recommendations

Info

The Caribbean ExaminerCXC Examination SystemTest Developmentin Public Examinations –CXC Stands TallBy John Andor, PhDINTRODUCTIONThe reputation of any public examinationbody hinges on three key pillars: the technicalquality of its examination papers, the structureof its assessment and the sanctity of its testadministration process.The Technical Quality of the ExaminationPapersHere I mean the extent to which the testitems, and by extension the examination papers,possess the two key psychometric propertiesof validity and reliability. Reliability indicesare usually available to the examination boardsthemselves. Their internal systems are designedto generate them. They can be made available onrequest.At the Caribbean Examinations Council(CXC), reliability indices are produced for multiplechoice items and papers, paper components andentire subjects for every examination. CXCboasts of very high reliability indices year afteryear for all its examinations. In furtheranceof its avowed aim of enhancing quality in itsexaminations, an internal recommendation hasbeen made to produce indices to show the extentto which the entire examination (in each subjectand components) for two successive years maponto each other – a form of concurrent validity.This will tell CXC how fair it has been to the twocohorts comparatively using their performancesas a measure. That done, CXC will be able tomake definitive statements on that aspect of itsquality assurance process – I do not know of anyexamination board that produces this information– CXC may be setting the pace.The Structure of the AssessmentBy this I mean the various ways in whichthe examination is designed to capture theoverall learning outcomes of students such thatthe ultimate certification can reflect the extent towhich the assessment mirrors the content domainas laid out in the various syllabuses.The CXC has a unique triangular model.Almost every subject examined at both theCSEC and CAPE levels has a multiple-choicecomponent, a written component and a schoolbasedassessment (SBA) component. With thisstructure, CXC is able to assess a high percentageof the expected learning outcomes in both depthand breadth. Many examination boards, that Iknow of, do not use multiple-choice items fortesting at the advanced level; doing so is a bolddecision and CXC needs to be congratulated fordoing so because it demonstrates an acceptance ofmeasurement experts’ belief that multiple choicecan be used for testing even at the highest level.Until recently, the inclusion of SBA inexternal certification had not been a feature ofmost public examinations. The old GCE ‘O’and ‘A’ levels bequeathed to many developingcountries in Africa by the British did not have SBAcomponents. Even though some of these boardshave now introduced SBAs, they do not come closeto what CXC does in terms of giving guidance andin the training of teachers. Some boards simplyallocate a fixed percentage of the total mark persubject to the teacher (irrespective of subject).On receipt of the scores from schools, statisticalmoderation is used to bring the scores into lineusing the external scores as a moderator.The Sanctity of the Test Administration ProcessTests and test items can be valid and reliable;the assessment structure can be comprehensivebut if the test administration procedure iscompromised, the entire examination isworthless.The key issues in test administration aresecurity, logistics, effective candidate supervision(to avoid examination malpractices) and atransparent marking process.Every public examination body’s worstnightmare is to find out that its papers for anexamination yet to be taken or even in somecases already taken, are out there in the marketor were out there before the examination waswritten verbatim et literatim on a massive scale.This can be damaging to the board’s reputationas reprinting and re-administration is costly. Theusual panacea adopted by the boards in paperleakage situations is to fix new dates for the leakedpapers to be rewritten. There is an undocumentedcost to candidates, whose examination papers havebeen leaked and who have to rewrite, (which noexamination board has ever computed, at leastnot that I know of) in terms of the inconveniencecaused and the psychological trauma thatthese innocent candidates go through. In suchsituations, affected candidates believe that thereplacement papers are more difficult than theoriginal ones. This is especially true for thosewhose expected grades fall below their originalexpectations.In the Caribbean, examination leakage isvery rare compared to sub-Saharan Africa. Forthis, credit must first go to CXC for designingand adhering to security regulations that havemade security breaches a rare occurrence in itsexaminations. What is even more fascinatinghow tightly CXC is able to uphold its securityprocedures across the many separate islands. CXC’sagents in the various participating territories, theLocal Registrars, play a major role in enforcingand monitoring security procedures relating tothe administration of the examinations in theirrespective countries.Credit must also go to stakeholders ineducation in the Caribbean as a whole forupholding high levels of moral discipline.26 MAY 2012 www.cxc.org
The Caribbean ExaminerTest Development in Public ExaminationsElsewhere, parents, teachers and students areknown to go shopping for leaked papers prior tothe start and even during the examinations.Examination leakage (foreknowledge)remains the single most embarrassing headachefor most countries in Africa.TEST DEVELOPMENT AT CXC AND OTHERBOARDS/COUNTRIES COMPAREDFrom the above accounts, it can be seenclearly that CXC stands tall when compared withother boards in Africa and the UK under all threefactors that determine the credibility of publicexamination boards. I say this, not as an act ofpatronage, but out of personal knowledge andexperience that I have acquired over the yearshaving dealt with examination boards in threecontinents dating back to 1981.My interaction with colleague educationists,both within and outside this subregion makesme believe that my observations about CXC arenot frivolous. In fact, I know of an internationalassessment expert who dealt with CXC manyyears ago, and who was only recently contractedto set up a credible examination board in a verybig country in Asia – naturally he will apply someof what he knows CXC does.What many stakeholders do not know is howCXC sets its examination papers as compared toother international boards. This is another crucialarea where comparisons can be made, lessons canbe learnt and best practices adopted.I do not know of any public examinationboard that has ever put its test developmentprocedures out to the public; but in the contextof the quest for transparency and accountabilityin the use of public resources as well as the adventof the information superhighway, nothing can beconsidered a secret anymore; large empires havefallen, the Cold War is over and access to the mosthidden information has become a matter of justa tap on the keyboard. CXC will set the pace inthis disclosure.SO WHO SETS THE EXAMINATION PAPERSFOR PUBLIC EXAMINATION BOARDS?Public examinations that are domain-basedoriginate from content experts who are preferablypeople who have been trained to teach, thosecurrently teaching (although for security reasonsnot the year group for whom the test is meant) orwho have some experience in teaching at the levelfor which the test is meant. Measurement officersof public examination bodies facilitate the papersettingprocess and fine-tune the questions toensure that they meet the psychometric standardsrequired.The single justification for using people withpedagogical training to write test items is thatassessment experts agree that assessment must bealigned to instruction (Gipps 1994, Tucker 2009)and so it stands to reason that teachers who aretrained to deliver classroom instruction and whodeliver this instruction on a daily basis are thosewho should set test items.It must be emphasised that the ability towrite a test item that will suit the learning styleof students is a skill that must be developed andnurtured. No one in my opinion can boast ofthis skill other than teachers who set test itemsin their classrooms to diagnose student learningdifficulties, to evaluate their teaching strategiesand determine student mastery of content.It is widely recognised that teachers arethose who are in the best position to determinethe suitability of psychological measurementinstruments (whether tests or attitudinal scales)meant for pupils who they teach and whoselearning styles they know. It is therefore notstrange that Logan and Medford (2012) submittedtheir questionnaires to classroom teachers toevaluate their suitability for the age range ofchildren in Years 3, 4, 5 and 6 in their study.According to Abbas (2008), the newlyestablished Aga Khan University ExaminationsBoard in Pakistan “has several ways to supportteachers and expose them to the demands of thenew style of examination” (n.p.n). These includeconducting one-day workshops for teacherswhere the examination syllabuses are explainedto them; material development workshops to helpteachers focus their teaching, “while workshopson item writing prepare teachers to offer qualityassessments to their students in class”.There is considerable research evidence insupport of the efficacy of the use of assessmentto support teaching and learning. In fact, Gipps(1994) in Rehmani (2003) has suggested that “themajor purpose of assessment is to support theteaching and learning process” (p.2).Tucker (2009), in his paper on howtechnology-enabled assessment data can be usedto improve instruction, has stated that:“These [test] data could then be used to adaptinstruction by creating a better understandingabout students’ knowledge, and their conceptualunderstanding and cognitive development, whichwould lead not only to better assessment butto significant improvements in instruction andlearning (2009 p.2).”Similarly, Bennettt (n.d) in Tucker (2009)“has envisioned testing merged with instruction,which would allow teachers and students touse feedback from testing to adjust teaching toimprove student achievement” (op. cit).What I have tried to do in this section is toadduce research evidence to support the use ofcontent experts, especially those in the classroom,to generate test items. It is a world-wide practicewhich, apart from helping teachers to refine andimprove their teaching, is a fulfillment of therequirement to align assessment to teaching; whatis more, it contributes in no small measure to thedevelopment of teacher professionalism.HOW ARE THE TEST PAPERS IN CXCEXAMINATIONS SET VIS-À-VIS OTHERBOARDS?The ‘bible’ on which domain-dependentpublic examinations are based is the syllabus. Thesyllabus is more or less the contract documentbetween the boards on one hand, and students,teachers, parents and employers on the other.In some jurisdictions, there is a clear distinctionbetween the teaching syllabus which is designedfor teaching in schools by the national educationauthorities and the examination syllabus whichis an extraction from the teaching syllabus of thetestable topics. The use of examination syllabusesconcurrently as teaching syllabuses is not new.Until the early 1980s, most African countrieswhich wrote the London GCE ‘O’ and ‘A’ levelsused those syllabuses as their national teachingsyllabuses.Currently, almost all territories in thesubregion use CXC’s syllabuses as their teachingsyllabuses. There is nothing wrong with thispractice, provided that these syllabuses reflectnational and regional aspirations. I havescrutinised CXC documentation and I am morethan convinced that CXC is a driver of educationalassessment excellence in the subregion; theboundaries of its own educational aspirationsand those of the territories are virtually coterminous.This article will not give the details of theprocedure for question paper setting step by step,but an attempt will be made to show how CXC,like any other credible examination bodies, goesthrough all the necessary steps to ensure that itdesigns tests that are not only valid and reliable butalso suitable for the level they are meant to test.In line with internationally accepted practice,CXC identifies content experts with the requiredexperience and gives them the required trainingin test construction. As mentioned elsewhere inthis article, these content experts may be peoplewho are trained to deliver instruction, currentlydelivering instruction (just above the level forwhich the test will be used) or who have for manyyears delivered classroom instruction and are nowretired. With this policy, CXC meets the demandof ensuring that assessment, whether high stakesor low stakes, must be aligned to instruction.The Initial DraftBased on identified criteria and therelevant syllabuses, trained subject expertsare commissioned to write draft examinationquestions and submit for consideration. In somejurisdictions, one person can be commissionedto write a whole paper, the advantage of which isthat the test writer has a holistic view of the entirewww.cxc.org MAY 2012 27
Page 5: The Caribbean ExaminerCPEA Testimon
Page 11: The Caribbean ExaminerCXC in the Cl
Page 15: The Caribbean ExaminerThe CVQ: Cert
Page 18 and 19: The Caribbean ExaminerCXC Examinati
Page 20 and 21: The Caribbean ExaminerUnderstanding
Page 23 and 24: The Caribbean ExaminerSo, what IS s
Page 25: The Caribbean ExaminerCriterion-Ref
Page 29 and 30: The Caribbean ExaminerTest Developm
Page 31 and 32: The Caribbean ExaminerWhat institui
Page 38 and 39: The Caribbean ExaminerRECOGNITION O
Page 41 and 42: The Caribbean ExaminerOjeda Vanterp
Page 44: The Caribbean ExaminerCaribbean Pri
Page 49: The Caribbean ExaminerCPEA Testimon
Page 55: The Caribbean ExaminerFuture Caribb
Page 58 and 59: The Caribbean ExaminerCXC NEWSCXC T
Page 61: The Caribbean ExaminerShastri RamI
Page 66: The Caribbean ExaminerCXC NEWSBusin

CXC Examiner May 2012 - Caribbean Examinations Council

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?