CXC Examiner May 2012 - Caribbean Examinations Council

The Caribbean ExaminerCXC Examination Systempaper in terms of balance and difficulty level inits draft stage. The problem with this approach,however, is that it can be very taxing and the papermay be affected by the personal idiosyncrasies ofthat single writer.In other jurisdictions, identified writers arecommissioned to write test items for sectionsof each paper only. The advantages are, firstly,it is less tiresome and secondly, when differentitems for different sections of the paper fromdifferent writers are put together, the completepaper becomes richer because it benefits fromthe perspectives of not just one person. This isCXC’s approach.The Paper-setting MeetingThe next step in the paper-setting processis to bring together a number of content expertsto a roundtable to work on the drafts that havebeen produced. The Chairperson of this team isthe person who has overall responsibility for thequestion paper and is usually designated the ChiefExaminer, appointed beforehand. The job of thissmall group of experts is to use their professionalexpertise to look at each item that will eventuallyconstitute the paper to ensure that it is capable ofteasing out of the candidate the type of behaviouror learning outcome that the learning objectivesstated in the syllabus seek to deliver. In otherwords, the item must fall within the syllabus, relateto a specific measurable skill/objective, be suitablefor the specific age level (Piaget c.1968), and mustnot measure triviality. The item must also satisfygrammatical rules and be free from ambiguity.Essentially, this group of experts moderates theoriginal draft questions to bring them to thelevel appropriate for those they are meant totest; hence in some jurisdictions this meeting iscalled a moderating meeting rather than a papersettingmeeting. It must be emphasised thatpapers can also be set and moderated at one andthe same meeting, (from where the term papersettingmeeting was coined). The disadvantage,however, is that time pressure can lead to qualitybeing sacrificed and/or an incomplete papersettingexercise. The items from this meeting areeventually compiled into a test paper but that isnot the end of the process.The next stage of the process is for the contentexperts also to consider the expected answers tothe questions – which may have typically beendrafted at the time the paper was originally set.These answers to the questions are very importantbecause they are supposed to be used as a commonguide for markers who will be recruited to markcandidate scripts after the test has been taken.Together with the question paper, the agreedanswers to the questions (usually called a markscheme with mark allocations) can be likened to ameasuring instrument such as a ruler (or tape) orscale used to measure length and determine weightrespectively in everyday life. What we are doingactually in testing is measuring student learningoutcomes, hence the title measurement (MED)officers formerly used for designated staff in CXC.The Measurement Officer is the originator and thefacilitator of the entire process from paper settingthrough to marking and grading.Pre- and Peer EditingThe technical quality of a test paper dependson the quality assurance measures to which itis subjected. Mindful of this, therefore, CXCfirst subjects the draft paper to a pre- and peeredit after typesetting. At this stage the entirepaper and its mark scheme are scrutinised by aprofessional in-house editor and then by colleaguemeasurement officers to ensure that they meetpreliminary identified criteria —conform tohouse-style, are free from errors and ambiguity,satisfy the information put out in the syllabus andare appropriate for the level being tested.Experience has shown that this step isextremely useful, as very vital issues that mayhave eluded the compositors and content expertshave in many cases been identified. However,this practice is not universal because it has thesingle disadvantage of expanding the populationthat has foreknowledge (albeit authorised) of thecontents of the paper. In Africa and elsewherewhere question paper leakage is endemic, peerediting is not practised; the subject officer him/herself has the added responsibility of checkingthe word-processed paper.External/Single/Anonymous ModeratorAfter peer edit and amendment, the testpaper is taken through another crucial stage. Thepaper, the mark scheme, the relevant syllabus andother specifications of the paper are forwardedto an independent content expert called themoderator. His/her role is critical and usually his/her identity is not disclosed.The job of the external moderator, which ispart of the quality assurance process, is to considerall the papers that make up the examination of thesubject at the specific level, and based on his/herexpertise and experience submit an independentprofessional opinion on any aspect of the paperin terms of the criteria itemized above. Themoderator is expected to put himself/herself inthe position of the candidate who will be takingthe paper and make any observation that he/shethinks fit. In fact, the moderator is supposed towork through the paper and time him/herself tobe able to comment meaningfully on the adequacyof the time allotted to the paper.The use of the moderator in the paperdevelopment process is a key feature of the UKpublic examination system and was inherited bysome African examining boards. Unfortunately,here too, because of rampant leakages, thisstage has been scrapped to limit the extent offoreknowledge. CXC uses it to positive effect.The Chief Examiner’s ResponseThis is another quality assurance measure.The comments of the moderator are forwarded tothe Chief Examiner for consideration. The ChiefExaminer examines the moderator’s commentsand where he/she agrees, makes any requiredchange and sends the paper back to CXC. Themeasurement officer then continues the qualityassurance process. The Chief Examiner is alsofree to disagree with the moderator; in eithercase reasons must be given and documented. Themeasurement officer then makes the necessaryamendments in the light of the Chief Examiner’sresponses and puts the paper into near finishedform. Examination bodies that do not use themoderator interventions will definitely lose outon the extra quality contribution that this stagemakes to the process.Technical EditingThis is about the last stage of the formalquality assurance process. It is an in-houseactivity. This stage appears unique to CXC in myexperience. Most examination boards do not useit for security reasons but in a society like ourswhere we can boast of staff honesty and loyalty,there appears to be no risk. Here, the completepaper with the marking scheme and the syllabusand other specifications are submitted to twoidentified measurement officers for furtherindependent scrutiny. Their written commentson any aspect of the paper are re-submitted to theofficer concerned, who then looks at the meritsof each comment and decides on what to acceptor reject.Chief Examiner’s Final SignatureBefore the paper is sent to print, the ChiefExaminer is given another opportunity toscrutinise it and give his/her approval by signingit off. Even this is not final. On receipt of thesigned paper from the Chief Examiner, themeasurement officer still has a responsibility tore-check to ensure that the paper is error-freebefore authorising printing.FinalisationThe paper is then finalised and readied forprint. At this point the paper is supposed to be freefrom any form of error as far as humanly possible.Before the paper goes to print however, theHead of Measurement who has overall oversightresponsibility for the quality of every questionpaper also scrutinises it and brings any concernsto the attention of the measurement officer forresolution. In most examination boards in Africa,the Head only has supervisory authority but doesnot see the live papers.28 MAY 2012 www.cxc.org