Blank Doc with SSC Logo and RDMS Doc Number - State Services ...

87 Standard-setting meetings using a sample of initial scripts were held. The papers were
trial marked to see ‘what the candidates had done to the paper’ and to detect any rogue or
ambiguous questions not detected earlier in the paper setting exercise. The draft-marking
schedule was then adjusted in light of what the markers found and debated. This is
standard practice and it was thoroughly conducted. However, the fact remains that the
unusual results that appeared, showing wide variations between subjects, did not set off
an alarm in NZQA.
88 The explanation for this seems to be that NZQA was aware that there would be variation,
but was not aware that the nature of this variation would be a surprise to nearly everyone
else. The 2004 approach can be contrasted with NZQA’s previous experience with unscaled
School Certificate, where marks from the first 10% of papers were entered on to a
database and the potential mean found. A decision was then made whether the aggregate
results were within ‘professional public tolerances’. Intervention took place from time to
time in the subjects with significant numbers of entries, though some with a small
number of entries would have been left alone.
89 As already mentioned, the business processes that NZQA employed were highly
efficient. These were driven by very intense pressures to deliver the results as rapidly as
possible. The Examination Centre Managers coordinated the collection of the candidate’s
scripts and forwarded them to regional hubs, from where they were sent to markers.
Markers then sent in results and returned the scripts to candidates via a hub. The drive
and tempo behind these processes meant that an analytical pause, a step back to interpret
the wider impact and meaning of the results, did not occur. In the end, the overall picture
from all the subjects was not apparent until results were in the pipeline for processing
and mailing to candidates. By then it was too late.
90 Major time pressure at two points is difficult for NZQA to manage. On the one hand, the
schools exert pressure for the examinations to happen as late as possible in the school
year, to maximise teaching time and the discipline that impending assessment applies to
the students. On the other hand, schools, tertiary institutions, parents and the public want
results to be out as soon as possible. Added to this is the operational ‘given’ that
marking has to be over by Christmas, because markers are less reliably available
afterwards.
91 This matter will have to be tightly managed in 2005, because the Scholarship
examinations are scheduled to be split into two halves, some before the NCEA
assessments and some after. Security arrangements will need to be increased for papers
to remain secure in the Examination Centres for a month; alternatively, the papers will
have to be sent out twice over that period. In addition the period of time between the last
examination and Christmas is shortened by a minimum of four days. NZQA are
exploring an arrangement whereby the Examination Centre Managers send the scripts
directly to markers rather than to the regional hubs for distribution. This makes good
business sense, but a note of caution is that business processes may take precedence over
sound and measured assessment practices. The Review Team emphasises that more
time, not less, needs to be available to NZQA to manage its responsibilities and to ensure
fair results.
Patterns of Entry
92 The numbers and characteristics of the candidates were unknown for the new 2004
Scholarship. Publicity stressed the elite nature of the examination, but the combination
of monetary awards and entry requirements that did not cost students any more than the
15