Blank Doc with SSC Logo and RDMS Doc Number - State Services ...

REPORT ON THE 2004 SCHOLARSHIP
TO THE DEPUTY STATE SERVICES COMMISSIONER
BY THE
REVIEW TEAM
LED BY DOUG MARTIN
29 April 2005

CROWN COPYRIGHT
ISBN 0-478-24491-6

TABLE OF CONTENTS
Background ................................................................................................................ 1
Review Process and Methodology............................................................................ 2
Context ........................................................................................................................ 2
Overview of Findings ................................................................................................. 3
Discussion of Findings.............................................................................................. 4
1.1 Whether the roles and responsibilities of the various agencies involved were
clearly defined.................................................................................................................. 4
1.2 Whether the identification and addressing of issues arising in designing and
implementing the new system were adequate.................................................................. 6
1.3 Whether the processes to set and moderate the standards for the qualification
were adequate................................................................................................................... 8
Coordination of roles............................................................................................... 8
Setting the standards ............................................................................................. 10
Pre-Moderation (Understanding the Standards).................................................... 11
Professional Development .................................................................................... 12
Monetary Awards.................................................................................................. 13
1.4 Whether the administration of the examinations against those standards,
including review of the processes for setting, moderating, and marking the
examination was adequate.............................................................................................. 13
Paper Setting ......................................................................................................... 13
Marking and Moderation (of marking and marks)................................................ 14
Patterns of Entry.................................................................................................... 15
1.5 Whether communication and guidance to teachers and students concerning New
Zealand Scholarship were adequate ............................................................................... 16
1.6 Any other matter which the Review Team considers relevant to the foregoing
questions......................................................................................................................... 17
Recommendations ................................................................................................... 18
Annex 1: Terms of Reference for Review of the New Zealand Qualifications Authority
by the State Services Commissioner ........................................................................... 21
Annex 2: Organisations consulted during the course of this Review ........................................ 23

Background
1 In February 2005 the Associate Minister of Education, Hon David Benson-Pope, asked
the State Services Commissioner to review the adequacy of the setting and management
of the 2004 Scholarship and, more widely, the performance of New Zealand
Qualifications Authority (NZQA) in respect of the school sector qualifications [CAB
Min (05) 5/14]. The Associate Minister of Education requested the review under Section
11 of the State Sector Act 1988. This request was in response to surprise at the degree of
variability of results between subjects for the 2004 Scholarship and the fact that
relatively few Scholarships were awarded.
2 The Deputy State Services Commissioner, Tony Hartevelt, on behalf of the State
Services Commissioner, asked Doug Martin of Martin Jenkins and Associates to lead the
Review. Dr Alan Barker, who is an independent education and public management
consultant based in Hong Kong, assisted Mr Martin. They were supported by staff of the
State Services Commission. The team has approached the Review in two parts: the
implementation of the 2004 Scholarship and the performance of NZQA in relation to
secondary school qualifications. This report covers the implementation of the 2004
Scholarship only.
3 Concurrent with this Review, the Associate Minister of Education, Hon David Benson-
Pope, convened a group of education practitioners and experts to consider the shape of
the 2005 Scholarship. This group (the Scholarship Reference Group 1 ) reported to the
Associate Minister in mid-March 2005 and Cabinet subsequently considered and adopted
25 of their 26 recommendations on 29 March 2005. The Ministry of Education (the
Ministry) remains responsible for setting the Scholarship standards, NZQA remains
responsible for operating the external assessment and awarding the Scholarships, and the
Ministry of Social Development distributes the monetary awards.
4 This report, by focussing on the public sector management aspects of Scholarship,
supports the revised policy settings arising from the work of the Scholarship Reference
Group.
5 Part one of the Terms of Reference ask for a review of the 2004 Scholarship with
particular regard to:
1.1 whether the roles and responsibilities of the various agencies involved were clearly
defined
1.2 whether the identification and addressing of issues arising in designing and
implementing the new system was adequate
1.3 whether the processes to set and moderate the standards for the qualification were
adequate
1.4 whether the administration of the examinations against those standards, including
review of the processes for setting, moderating, and marking the examination was
adequate
1.5 whether communication and guidance to teachers and students concerning New
Zealand Scholarship were adequate, and
1.6 any other matter which the Review Team considers relevant to the foregoing
questions.
1 To be distinguished from the earlier Scholarship Reference Group which met in 2001 & 2002. See paragraphs 36 & 37
1

6 The full terms of reference are attached as Annex 1. This Review, which has
concentrated principally on implementation matters, makes recommendations under each
of the relevant Terms of Reference (TOR). General observations by the Review Team
are included in the overview of findings and in discussion under TOR 1.6 below.
Review Process and Methodology
7 The approach taken by the Review Team has involved identifying the principal
weaknesses in the implementation of the 2004 Scholarship, the lessons that can be drawn
from these, and the changes necessary to ensure that the 2005 Scholarship is
implemented smoothly.
8 In order to understand and analyse the implementation process for the 2004 Scholarship
the Review Team interviewed a wide range of external stakeholders as well as a number
of internal managers at both NZQA and the Ministry of Education. The Review Team
also met with the Education Review Office. The Team developed structured questions
that were made available to those organisations it met (a list of the organisations is
attached as Annex 2).
9 The Ministry and NZQA provided a large amount of written material to the Review
Team. In addition, the Review Team invited comment from students, teachers and
members of the public. Advertisements seeking comment were placed in the Education
Gazette and the Education Review on 11 and 21 March 2005 respectively. Teachers sent
in 22 individual or collective (whole school) comments. Six stakeholder groups sent in
comments. The Chief Executive of NZQA wrote to all 4,500 students who entered the
2004 Scholarship, on behalf of the Review Team, inviting comment on their experience.
220 emails were received from students in reply.
10 On 31 March 2005 the Review Team ran a workshop to test preliminary findings with
key stakeholders who the Team had previously interviewed. Members of the team met
separately with some stakeholder groups who were not able to attend, to ensure that there
was an opportunity for them to provide feedback on the preliminary findings. The
Review Team has also engaged with the NZQA Board, presenting its initial findings on 7
April 2005. The feedback on the initial findings indicated to the Review Team that they
had captured the essence of the problem. This report builds on those findings and makes
appropriate recommendations to address the issues for the 2005 Scholarship.
Context
11 Whilst, as the report notes, there have been significant difficulties with the 2004
Scholarship, it is important to view these within the context of the much wider reforms to
teaching, learning and assessment in New Zealand’s senior secondary schools which
have been progressively implemented over the previous decade or more. These reforms,
which reflect changes in the nature and role of the senior secondary school since the
1970s, are directed at providing all students with the opportunity to succeed and to make
successful transitions into tertiary study. The National Certificate of Educational
Achievement (NCEA), as it was set out in the 1998 Cabinet paper was a starting point,
representing a huge and unprecedented change in teaching, learning and assessment
outcomes.
12 The development of the new approach to senior secondary school qualifications was
contentious, and the policy that was agreed by Cabinet in 1998 represented a compromise
between widely polarized positions on senior school assessment and qualifications (see
2

discussion under TOR 1.6). NCEA provides for external and internal assessment at three
levels, as well as the externally assessed Scholarship at Level 4 (in 2004). In total this
amounts to more than 150,000 candidates and approximately 1.9 million individual
examination booklets, a massive undertaking. In short, the implementation of NCEA
required a major, complex change management process. This included three years of
professional learning and development managed by the Ministry, the instituting of
significant new business processes by NZQA, and extensive relationship management
with schools by NZQA.
13 Furthermore, the implementation of NCEA and Scholarship, commencing in 2002, has
been undertaken over a very short timeframe, necessary in order to provide a coherent
pathway for students through senior secondary school. The implementation of Level 1
NCEA was particularly difficult, with a major industrial dispute with secondary school
teachers complicating communication and discussion on Level 1 implementation in the
early months of 2002. In contrast, the implementation of Level 2 NCEA was relatively
smooth, although there were some technical difficulties with the posting of results on the
NZQA website.
14 When it came to implementing Level 3 NCEA and the 2004 Scholarship in the same
year, there was insufficient appreciation of the need to differentiate between the two
assessments. In the event, implementation of Level 3 NCEA, which has a much wider
impact on students than Scholarship, was the main focus of attention, with Scholarship
essentially being implemented in its slipstream. The strategic risks, both policy and
implementation, associated with Scholarship received insufficient attention.
15 It is important to appreciate that the implementation of NCEA is in its early stages and
will continue to evolve. The magnitude of the change is such that the transition to better
learning, teaching and assessment will continue over at least the next five years. The
focus of attention and debate should not merely be on approaches to assessment. The
Review Team notes that to this point the change has been implemented reasonably well
overall.
Overview of Findings
16 The overall finding is that the difficulties experienced with the 2004 Scholarship resulted
from inadequacies in both policy advice and implementation.
17 The Government received inadequate advice on the policy risks associated with the 2004
Scholarship. The principles on which Scholarship was to be based were established in
1998, with the more detailed aspects of policy being developed through consultation with
key stakeholders over the succeeding five years. This was a long process during which
officials failed to adequately take stock of the policy settings and provide the government
with an explicit analysis of the implications for the outcome of the 2004 Scholarship. A
stocktake and analysis might well have led to a further refinement in the policy settings
for the 2004 Scholarship, along the lines of those recently approved by the government.
Instead, there was a ‘drift’ into implementation without an adequate analysis of the
strategic policy risks.
18 The result was that the expectations of Ministers were not aligned with those of officials
in the Ministry of Education and NZQA, and the outcome of the 2004 Scholarships came
as a surprise to both Ministers and the wider public. For example, officials saw the
variability of results between subjects as a consequence of the approach to Scholarship,
rather than as a significant risk that could undermine the credibility of the examinations.
3

19 In similar vein, the government was not provided with adequate advice on the strategic
risks associated with implementing both the high stakes Level 3 NCEA and Scholarship
in the same year, 2004. Officials were focused on operational risks and lost sight of the
higher level implementation risks which impacted on outcomes. Strategies to mitigate
these risks were not identified and put into effect in the approach to the 2004 Scholarship
to ensure a fair result for students.
20 As it transpired, the risks were significant. For example, Scholarship was still being
implemented part way through the normal examination cycle; the registered standards
were not available until December 2003, delaying the writing of exemplars; there was
only one exemplar produced for each subject; new subjects were being examined with no
previous examination history at Scholarship level; and there was no dedicated
professional development for teachers on Scholarship, notwithstanding the different
nature of a one-off, externally assessed examination which needed to identify top
students. These risks were not adequately taken into account in the approach to the 2004
Scholarship. The consequence was that teachers and some students were not well
prepared for the 2004 Scholarship and the outcome was unfair to those students. This
may also have affected the entry of students into some University courses and their
eligibility for subsequent award opportunities.
Discussion of Findings
21 The detailed findings of the Review Team are discussed below under each of the Terms
of Reference (TOR) 1.1 – 1.6.
1.1 Whether the roles and responsibilities of the various agencies involved
were clearly defined
22 In a formal sense the roles and responsibilities of the Ministry of Education (the
Ministry) and NZQA are clearly defined, with the Ministry being responsible for the
provision of policy advice, the development of standards and professional development,
and NZQA for operational matters including external assessment. Nonetheless, external
stakeholders commented consistently to the Review Team that they were unclear as to
who was leading the change and at what point the focus shifted from policy to
implementation. A compounding factor was that some significant policy issues were
addressed during the implementation phase, with NZQA taking the lead on these
sometimes without Ministry involvement (see paragraph 32). This lack of clarity added
to the perception, notably by teachers and schools, that they “were being shuffled from
one agency to the other, and neither agency was taking responsibility for the issues they
raised.”
23 The Ministry established a dedicated NCEA project team, the Qualifications
Development Group (QDG), in early 1999. This group, which was based in the Ministry
and drew on considerable expertise from both agencies, wrote the standards, produced
resource material (e.g. exemplars) for Levels 1, 2 and 3, undertook initial planning for
implementation including addressing emerging issues. The QDG also managed
communications and professional development, which was largely contracted out to
Colleges of Education which in turn ran several ‘Jumbo’ days involving all secondary
teachers for years 11, 12, and 13. The group was disestablished in 2001 when tagged
implementation funding came to an end.
24 One key stakeholder told the Review Team that the disestablishment of the
Qualifications Development Group was a significant loss to implementation of NCEA, as
4

teachers no longer had a single point of entry into the system where they could take their
concerns. Instead responsibility was split between the two agencies. This resulted in a
noticeable communication problem for the 2004 Scholarship. The Review Team also
believes that the disestablishment of the QDG weakened the implementation focus for
Scholarship from 2001.
25 Following the disestablishment of the Qualifications Development Group, the
responsibility for NCEA implementation oversight (but not actual implementation), fell
primarily to the Joint Overview Group (JOG) of senior and mid-level managers from the
Ministry and NZQA. However, JOG’s mandate was fluid, involving principally
information exchange and discussion of operational policy matters. Throughout 2004,
the focus of JOG was on broader matters associated with NCEA and a number of wider
issues like the draft Secondary Education Work Programme. Scholarship received little
attention.
26 At the explicit request of Ministers in 2002 the two agencies adopted a number of
measures to reduce role confusion. 2 Again, the focus on Scholarship was weak and
confusion over roles remained for external stakeholders. The lack of well-focused
implementation for Scholarship drawing on expertise from both agencies contributed to
the drift into implementation without adequate analysis of the strategic implementation
risks.
27 The work of the Qualifications Development Group, the Joint Overview Group, and the
other initiatives noted in paragraph 26 did, however, significantly improved the climate
of cooperation and collaboration between the Ministry and NZQA.
28 The Review Team concludes that a dedicated implementation team should be established
by NZQA for the 2005 Scholarship, and that this team should include people with
academic expertise in the area, senior practitioners, and key representatives from the
Ministry of Education.
29 Given the magnitude of the change from the old Scholarship arrangements to the new,
which involved introducing a separate standards-based examination at Level 4 for 27
subjects, some of which had never been tested at Scholarship level before, the Review
Team would have expected there to have been:
• a formal handover of completed policy specifications from the Ministry to NZQA,
with supporting documentation that clearly explained the policy to be
implemented and the expected outcomes,
• a comprehensive implementation plan for Scholarship developed by NZQA, with
an analysis of implementation risks and strategies to deal with them, a
communications plan and a plan to monitor progress.
30 Neither eventuated. The absence of this documentation meant that the identification and
addressing of issues in the design and implementation of Scholarship was inadequate (see
1.2 below), and the confusion over which agency was leading the change continued for
external stakeholders.
31 The Review Team concludes that for 2005 and beyond the policy settings should be
formally communicated to the NZQA Board in a letter from the Associate Minister of
Education, together with an expression of the expected policy outcomes. NZQA should,
in turn, develop a detailed implementation plan that is formally approved by the
2 These included joint publication of NCEA Update, enhancement of the integrated NCEA Website, and joint chairing of
the Leaders Forum from May 2003
5

Associate Minister. At the same time, both agencies should aim for a seamless
implementation process whereby NZQA is involved in work that the Ministry leads and
the Ministry is involved in work that NZQA leads, so that the value of the combined
expertise and different perspectives is maximised. This seamlessness will require better
feedback loops between officials and the sector and a preparedness to take on board
sector concerns.
32 As noted above, a number of important policy issues surfaced during the implementation
phase, such as the number of subjects to be examined; how to fairly manage language
examinations for native and non-native speakers of languages; and the allocation of
monetary awards. There appears to have been a degree of role confusion in settling these
matters. NZQA saw some of these issues as operational in nature and did not always
involve the Ministry in their resolution. Some were in fact substantive policy issues,
albeit with important operational dimensions, and both agencies should have been
involved in the process of advising the Minister. The protracted consideration of these
issues during the implementation phase also created uncertainty in the sector about the
nature of the new Scholarship.
33 The degree to which NZQA had dedicated operational policy capacity to help resolve
these policy issues is also relevant. NZQA originally had substantial policy capacity
when it was established. This capacity was progressively reduced from the mid 1990s
because it created tensions with the Ministry’s overarching responsibility for policy. The
result was that NZQA was left primarily as an operational agency. The Review Team
believes that this situation, coupled with the loss of staff experienced in dealing with the
policy and operational risks of running competitive examinations, contributed to the drift
into implementation, and the insufficient analysis and communication of potential risks.
34 The Review Team notes that the Ministry has the infrastructure and institutional
capability to set up and run effective professional development. This should be utilised to
provide professional development for the 2005 Scholarship. NZQA needs to provide
input and expertise to the materials to be produced for students and teachers.
35 In light of the above discussion, the Review Team has also come to the view that there is
an immediate need for the Ministry to provide active leadership to help coordinate and
integrate the various parts of the sector. In particular, the Ministry needs to ensure that
for the 2005 Scholarship, the Leaders’ Forum, the Ministry (policy and monitoring roles)
and the NZQA are all part of a coherent whole that leads to a consistent approach and
results.
1.2 Whether the identification and addressing of issues arising in designing
and implementing the new system were adequate
36 Cabinet agreed in 1998 to the principles of a new National Certificate of Educational
Achievement (NCEA) including a new standards-based Scholarship. Details of the new
2004 Scholarship were worked out over the next five years principally through a range of
sector-based consultative groups. These included:
• Scholarship Working Group reporting to the Leaders’ Forum in July 2000
• Scholarship Reference Group 3 – met from February 2001 to November 2002
• Secondary Principals’ and Leaders’ Forum – re-convened in May 2003 with a
much wider brief
3 This the first of two different groups referred to as “the Scholarship Reference Group” See paragraph 3
6

37 The Scholarship Reference Group (SRG) comprised representatives drawn from the
sector including members of the New Zealand Education and Scholarship Trust
(NZEST), which ran a private and independent scholarship examination from 1992 to
2003. The NZEST representatives withdrew from the SRG in November 2002 and the
SRG did not meet again after that. This withdrawal removed the main voice of
opposition to standards-based assessment and weakened the capacity of the consultative
process to confront and resolve some critical differences.
38 At the conclusion of the consultation process, the Government was not provided with a
stocktake of the policy settings by the Ministry, including an explicit analysis of the
implications of those policy settings for the outcome of the 2004 Scholarship. Given the
extended period over which the policy evolved, and with the benefit of hindsight, a
stocktake of this nature was warranted, and might well have resulted in a further
refinement of the policy settings along the lines approved in March 2005 by the
Government for the 2005 Scholarship.
39 The absence of a stocktake meant that Ministers, the public, and students were
unpleasantly surprised by the results of the 2004 Scholarship. The surprise occurred both
in terms of the relatively few scholarships awarded (leading to a decision to subsequently
create a new category of awards known as Distinction Awards) and the extent of the
variability between subjects that was acceptable.
40 It was clear that in 2004 the variability of results would be more transparent than ever
before, because the results of standards-based assessment are not scaled to achieve
comparability across subjects and across years. This was a very new way of assessing
and reporting results, and NZQA and the Ministry should have given far greater attention
to preparing the education community and the public to understand it.
41 Variation in the results of subjects from year to year has a number of causes, only one of
which is due to annual changes to examination questions. Other causes lie, for instance,
in the curriculum, how it is interpreted, and whether it is relevant; the standard of
teaching (across the system and also at a particular school); candidate selection; and
whether parts of the curriculum receive attention but not others. The results of the new
2004 Scholarship could not of course be compared with results from previous years
because the standard was new.
42 Variation is also affected by the success of moderation employed before the marking
(such as exemplars and teacher professional development) which aim to establish
common understanding for teachers, examiners and markers; and moderation during
marking when schedules and marking practices are adjusted to modify unexpected
outcomes. The unknown composition of the candidature of the 2004 Scholarship also
contributed to variation and is discussed below in paragraph 92. There are also other
seemingly minor but nevertheless influential factors, such as whether papers are
timetabled in close succession for some students while other students have time between
papers to refresh and do further preparation.
43 The combination of these factors may outweigh the variation caused by the changes in
questions from year to year, although the latter is normally the most prominent source of
variation.
44 The public, however, has a reasonable expectation that the assessment system will
deliver fairness. Most people interviewed thought that it was unreasonable that students
should be disadvantaged for matters beyond their control, such as inconsistency in
preparing students, extra-hard questions, an inadequate curriculum, or an overly
compressed examination timetable.
7

45 The Government was not provided with adequate advice on the strategic risks, including
variability, associated with implementing high stakes Level 3 NCEA and the new
Scholarship in the same year, 2004. There was no appreciation of the need to
differentiate between Level 3 NCEA and Scholarship. The focus was on detailed
operational issues that were common to both, rather than the high-level strategic risks
around implementation of Scholarship.
46 The outcome of that attention was that the business processes developed to implement
Level 3 NCEA and Scholarship in 2004 were comprehensive and efficient. They
adequately addressed the detailed operational risks associated with managing external
assessment for that cohort of students.
47 As it turned out, however, the strategic implementation risks were significant. For
example:
• the registered standards were not available until December 2003, even though the
Ministry made available ‘final form’ standards from mid 2003, effectively
delaying the production of exemplars
• for NZQA the effective implementation of Scholarship commenced six months
into the normal examination cycle, resulting in very compressed timeframes
• only one exemplar per subject was produced
• new subjects were added to Scholarship without any previous history of
examination
• there was no dedicated professional development for teachers of Scholarship, and
• the first year of significant change is more difficult for the teachers and students
than succeeding years, even with exhaustive preparation (which was not possible
in 2004).
48 Teachers and students therefore could not possibly, or uniformly, have been as well
prepared for the 2004 Scholarship as was desirable and this lack of preparedness was not
adequately taken into account by officials. The result was that the outcome was unfair on
some students because they did not perform as well as they might otherwise have done.
1.3 Whether the processes to set and moderate the standards for the
qualification were adequate
Coordination of roles
49 As discussed under TOR 1.1 the Ministry was responsible for setting the standards for
the 2004 Scholarship. NZQA participated in the development of the standards and
contributed expertise and judgement from an assessment perspective. Once the standards
were written and signed off by the Ministry, NZQA played a lead role in registering the
standards which involved further checks on the clarity and assessability of the standards.
This reflected firm demarcations between these two agencies, separating the role of
standards-setting from the role of assessing students against those standards.
50 Separating these two roles is common for many standards-based qualifications.
Professions, technologies and industries typically set their standards and appropriate
assessment instruments are subsequently devised. This division of roles is not so clear
for a traditional ranking examination for school-aged students, because the assessment
instruments (the examination questions and marking schedule) are, to a degree,
8

inseparable from them. The standards are embedded in the examination. This holds true
even for the new 2004 Scholarship which aimed to set an explicit, stand-alone standard
for each subject.
51 Setting standards for school students, based on national curriculum statements, which are
deliberately developmental in nature, presents a different challenge from setting
standards in industry, the professions and most tertiary courses. School students are
developing and growing, and the standards must acknowledge a wide base of
participating talent and developing achievement. They require the interpretation of
experienced teachers.
52 The Ministry and NZQA recognised this complexity, and did not ignore links between
curriculum and assessment. The standards for the 2004 Scholarship subjects, the
subsequent examinations papers, marking schedules and the marking were in the hands
of a reasonably tight core of people in the secondary and tertiary education community.
These dedicated people, on whom the assessment system relies, were frequently past
examiners and markers of the University Bursary examinations or the NZEST
examinations. A number of them served on the Ministry’s standards-setting panels and
brought their experience and knowledge to the task.
53 They did however need firm guidance and direction. A degree of inflexibility in the roles
of the two agencies seems to have impeded the coordination and resolution of issues
needed to implement a new examination with new standards, and a new way of assessing
and ranking. This inflexibility manifested itself most obviously at the working level and
is likely to be rooted in history and is difficult to untangle. While differences in view
amongst professionals are to be expected, the lesson to be drawn is that the issues
debated in both organisations need to be referred to senior managers for resolution rather
than languishing at lower levels. The leaders of the respective organisations have not
been fully successful in ensuring that substantive issues are escalated for resolution and
this requires urgent and sustained attention.
54 The National Assessment Facilitators in NZQA should have been fully involved in
setting the 2004 Scholarship standards instead of being at the margins, if only to reduce
the time it took later to refine the standards and make the examinations operationally
possible. More realistic knowledge of the length and demands of assessment operations
would have informed the content of the standards, kept them realistic for a one-off
competitive assessment, and kept the standards-setting on target so that support material
could be generated in time to meet the school year. Equally, the Ministry should have
been kept informed and involved in the assessment details instead of being kept at a
distance by NZQA.
55 One point where the relationship broke down was professional development. It seems
that at the time of the Scholarship examination standards being written, the writers
developed early exemplars. The Ministry felt the quality of these exemplars was uneven
but they were a resource and were given to NZQA. Further, the Ministry felt that the
Jumbo days offered the chance to stage-manage some of this exploratory and
developmental material. However, NZQA was reluctant to use the material because it
wished to have accurate assessment exemplars based on firmly agreed subject standards,
knowing that teachers rely closely on past experiences and examples when preparing
students for competitive examinations. NZQA subsequently contracted the development
of new exemplars. The two agencies did not seem able to resolve their differences. The
outcome was that the booklets for the professional development Jumbo days contained
little material to help embed the Scholarship examination standards.
9

56 Most important of all, better coordination between the Ministry and NZQA would have
surfaced and brought to resolution the different understandings and interpretations of
standards-based assessment that identifies top students which were held by different
parties. As it was, these differences continued unresolved throughout implementation.
What should have been a seamless and integrated implementation process became
fragmented, and as we have noted under TOR 1.1, teachers and interested parties were
uncertain which agency they should be dealing with.
57 The fact that the Ministry did not hand over to NZQA a clear, documented policy
specification for the 2004 Scholarship meant that subsequently there was no benchmark
for monitoring implementation. The subject standards were delivered to NZQA without
a shared understanding of what they really represented and how they would be applied in
high stakes, competitive examination. Equally, NZQA did not seek to clarify
uncertainties, or challenge decisions which committed it to unrealistic timelines.
Setting the standards
58 The standards for the 2004 Scholarship in each subject were based on a broad, generic
descriptor established for all subjects by the Ministry’s Scholarship Working Group in
2002 4 :
• Learners who have been awarded Scholarship in a subject will have demonstrated:
- High level critical thinking, abstraction and generalisation; and
- The ability to integrate, synthesise and apply knowledge, skills,
understanding and ideas to complex situations.
• Depending on the learning area, a range of the following will have also been
displayed:
- comprehensive content knowledge (breadth & depth)
- effective communication
- original or sophisticated solutions, performances or approaches
- critical evaluation
- flexible thinking in unfamiliar/unexpected contexts.
Assessment is restricted to the content of the 1evel 3 achievement standards for that
subject, derived from Level 8 of New Zealand Curriculum or their equivalent, but the
skills and understanding required will meet the Scholarship criteria.
59 In 2003 this generic descriptor guided the panels which set the specific standards for each
2004 Scholarship subject. Notes were added to each subject standard, clarifying the
scope of application of the standard within the subject but these notes and guidance
varied from subject to subject. For instance, Accounting provided some detail on the
body of law, regulations and conventions the standard covered. This set a boundary for
the Accounting examination standard. On the other hand, Biology had less specific
guidance for its standard. In part, the difference between the two subjects reflects their
4 In March 2005 it was confirmed by the 2005 Scholarship Reference Group and agreed by Cabinet in March 2005 when
it was renamed more accurately an ‘Outcome Statement’
10

nature – Biology is a more open-ended discipline than Accounting. But these differences
possibly contributed to the uneven performance of students in the new 2004 Scholarship.
60 The different boundaries for subjects may also have posed particular issues for the
sciences. Whereas recent University Bursary examinations emphasised the calculation of
correct answers, the new 2004 Scholarship asked additionally for commentary and
explanations of those answers.
61 This experience highlighted two related problems. These were the level at which the
standard was pitched and how realistic that level was. Setting an examination paper
requires clarity about what is expected of students at a certain age (or a certain level of
learning). The issue is whether the standards address a level that the best students can
realistically reach, or whether they are aspirational and aim at a level that the ideal
student ought to reach. It is not clear which approach the chief examiners and markers
put in practice in the 2004 Scholarship, and whether each subject had the same notion of
the standard.
62 With the benefit of hindsight, the Ministry and NZQA should have confirmed a common
understanding of what was involved in the new Scholarship examination. NZQA staff
should then have been more flexible in their application of standards-based assessment.
Less reliance should have been placed on the subject standard, and more guidelines
provided to the examiners and markers to assist them to interpret the standard.
63 The Review Team supports the recommendation of the Scholarship Reference Group
report that student performance needs to be assessed against an agreed assessment
schedule that allows markers to discriminate between students.
64 The preparation for developing the standards for the new 2004 Scholarship required
special attention. The addition of new subjects with no supporting case history, wider
curricula than before and a new approach to high stakes assessment all contributed to
implementation risk.
65 The standards for the individual subjects were not registered until December of 2003, and
the assessment exemplars were consequently delayed until the next year, 2004, when
teaching and the preparation of students was already underway. This squeezed the
implementation as far as teachers and students were concerned.
66 It is noticeable from the minutes of the Joint Overview Group (JOG) of the Ministry and
NZQA, that NZQA was not comfortable with the logistics of running four parallel sets of
examinations. While the minutes record debate, no formal advice was forwarded to the
Minister.
67 The 2004 Scholarship was one of four cycles of external assessment (along with internal
assessment in the three NCEA levels), amounting to more than 150,000 candidates and
1.9 million external assessment booklets. This was a huge undertaking and the fact that
the great majority of it went well deserves praise.
Pre-Moderation (Understanding the Standards)
68 There are two stages of moderation. Pre-moderation occurs when the teaching
community works collectively to understand the standard and achieve a common
interpretation (this process is sometimes referred to as the socialisation of assessors). It
happens through a range of activities, in particular formal professional development
occasions where teachers exchange assessments, and informal accessing of websites or
item banks for common assessment tasks and marked examples.
11

69 Pre-moderation contrasts with post-moderation, where the marking of assessments
generates common understanding, and adjustments are made to marking on the basis of
that shared understanding. Standards-based assessment places a lot of emphasis on premoderation,
whereas norm-referenced assessment places more emphasis on postmoderation
(see paragraphs 99 to 101). In the event, pre-moderation activities for the
2004 Scholarship proved insufficient.
70 The generic descriptor and the standard for each subject were designed to avoid
overloading the teaching of NCEA with new content. They were also designed to fit
with the model of standards-based assessment. However, both the general descriptor and
the subject standards were very general and provided little prescription or definition for
the teachers and students.
71 Traditional examination systems normally revolve around the prescription for each
subject, a document which sets out which elements of the curriculum (the broader body
of knowledge) will be assessed and broadly how this assessment will occur. The
prescription is a key document for examinations because teachers teach to it, and the
students learn from it – which is one reason why the educational benefit of examinations
is questioned. However, in the case of the new 2004 Scholarship teachers and students
had little prescription, and the interpretation of the subject standard was more open and
tentative than before. They had little to pin down the standards and to prepare for a
competitive examination.
72 This open interpretation may be more faithful to learning than preparing students for an
examination event, but it is not realistic for a competitive examination which has
substantial money involved and where the reputation of some schools is felt to be at
stake. It departed too far from past practices for it to be understood and accepted by
teachers without very careful preparation and communications.
73 In the absence of an examination prescription, teachers and students were even more
dependent on the examinations exemplars. However, these were few in number (one for
each subject), and they were completed and posted on the website late in the
examinations cycle because the subject standards were not completed until late in 2003.
Some website exemplars were posted in 2003 but others were not posted until 2004,
when the teaching year had begun. The caution was understandable to the extent that if
exemplars had preceded the standards, NZQA would have repeated the implementation
mistake of Level 1 NCEA, where teachers were irate that the early exemplars differed
from the external assessments that were finally set. The 2004 Scholarship involved
higher stakes. Unfortunately, this situation left little time for teachers and students.
Professional Development
74 Professional development is an important part of pre-moderation, particularly for
standards-based assessment and the Ministry invested substantially in running several
Jumbo days. However, everyone agrees that the professional development in the Jumbo
days focused on the NCEA and the first year of Level 3. The aim was to build a common
understanding of the internally assessed standards. There was no focus on externally
assessed standards, and the 2004 Scholarship was an external assessment.
75 In most situations, this focus on NCEA for professional development would be rational.
NCEA is the main qualification for the great majority of senior secondary students, and
this was the first year of Level 3. In comparison, the 2004 Scholarship was a one-off
competitive examination for a small minority of students. But the reality was that the
Scholarship had iconic status to some of the education community because it challenged
12

the most gifted students, attracted large monetary rewards and was a high priority for
many elite schools which jostle to be winners. The education community collectively
took its eye off the ball.
76 The Review Team stresses ‘collectively’ because the responsibility for professional
development is a shared one. The schools and some of their representative bodies have
much of the management control. Although Colleges of Education are funded to provide
professional development, it seems that few approaches were made by schools to the
Colleges to request 2004 Scholarship training. This is understandable as schools could
reasonably expect to receive advice from the Ministry either directly or through sector
forums as to the availability of professional development. In spite of this, it appears from
the Scholarship results that some schools were able to prepare their students well from
the material provided.
77 The assessment exemplars posted were marking schedules without marked (mock or
real) student work. NZQA acknowledges that this was a thin resource, but felt the
demands for more could not be met from its limited physical resources and in the
available timeframe. The exemplars would also be developed by the same small group
of overburdened chief examiners and markers who were setting the examinations for the
first time. No item bank or previous examples could be called upon. Unfortunately,
neither the Ministry nor NZQA formally drew attention to the implementation risk this
posed.
78 As noted in TOR 1.1, ownership at management level of professional development for
the 2004 Scholarship was unclear. Whilst the Ministry has formal responsibility for
professional development, this work in the context of a competitive examination is unlike
development for the standards-based NCEA. It focuses on the examination itself, not the
curriculum or teaching. Standards are set in practice, not by documentation and therefore
NZQA should have been deeply involved in the professional development for
Scholarship. The agencies understood this but it was not translated into action.
Monetary Awards
79 The policy decision to attach substantial monetary rewards to performance in the 2004
Scholarship was finalised in a Cabinet paper in late 2003, with the Minister announcing
the awards in December 2003. The introduction of monetary awards raised the stakes for
the 2004 Scholarship and may have prompted some candidates to enter without a realistic
chance of achieving Scholarship. The Cabinet paper included, for costing purposes,
assumptions about the number of Scholarships, but the policy parameters meant that
NZQA felt unable to give guidance about the desired success rates.
1.4 Whether the administration of the examinations against those standards,
including review of the processes for setting, moderating, and marking the
examination was adequate
Paper Setting
80 To write the papers NZQA advertised for Chief Examiners and used established practices
to ensure the papers were considered by external experts. Because of the new nature of
the 2004 Scholarship, two assistants were employed to provide assessment materials to
the Chief Examiner. Otherwise the processes followed the well-oiled precedents of the
University Bursary and School Certificate examinations.
13

81 An independent checker (sometimes two) sat the examination as a candidate and
provided feedback on time, language and instructions. The National Assessment
Facilitators (NAF) in NZQA considered the checkers’ comments and provided their own
feedback. The NZQA editors also checked the papers for examination layout, language
and terminology. Pre-testing of some elements of the examination questions was used in
some limited circumstances. There were, on average, three further editorial checks, one
further NAF check, and two further examiner checks before final sign off. This was a
thorough process.
82 Past resources were not particularly useful for paper setting because the 2004 Scholarship
was a new hybrid form of assessment. However, many of the Chief Examiners were
University Bursary examiners and their broad experience came with them. In some
cases, such as English, the NZEST examinations proved useful. In other cases, such as
the sciences, they were not. For example, NZEST did not have a requirement to provide
written explanations of abstract, underlying concepts.
83 There may have been some influence from the similarity of Level 3 NCEA and the 2004
Scholarship. For the four Level 3 NCEA externally assessed English standards, students
had to write three essays and ten short answers. For Scholarship, they had to write three
essays. The Scholarship essays were framed around very similar questions and identical
source material – the literature studied as for Level 3. Overall, the Scholarship paper
appeared similar to the Level 3 paper and students were expected to answer in similar
ways to very similar starter quotes. This would have provided some comfort and
familiarity.
84 However, the four Level 3 externally assessed Biology standards required students to
write 39 reasonably short answers in response to 12 questions. For Scholarship, they had
to write two essays and three short answers to three questions. The Scholarship answers
had to respond to six pages of unseen resources, each two solid pages of text and
diagrams. For Level 3, the unseen resources were in bite-sized portions, most about a
third of a page in length. This brief analysis suggests that Scholarship was an unexpected
challenge for Biology students and their respective teachers.
Marking and Moderation (of marking and marks)
85 Panel Leaders and Markers were recruited from both the tertiary and secondary sectors.
Most subjects had one Panel Leader and one Panel Marker. Those with larger numbers
of candidate entries had up to three additional markers appointed. 17 of the 52 Panel
Leaders and Panel Markers were from the tertiary sector, the rest were from the
secondary sector. Every panel had tertiary input through markers, material developers,
material critiquers or independent checkers.
86 Little guidance was given, however, on how to conduct and rank the assessments and
each subject took its own approach. There were variations in the process whereby
Scholarships and Outstanding Scholarships were awarded which may have exceeded the
natural differences between subjects. For example, English had three essay questions.
The marking panel applied a minimum yardstick of two Scholarship standards plus one
Not Achieved to award an overall Scholarship. By contrast, Media Studies, which also
had three essay questions, required the Scholarship standard in all three questions before
it gave the overall Scholarship award. This can be explained as acceptable subject
variation, but in our view a more standardised approach would have improved
professional and public confidence.
14

87 Standard-setting meetings using a sample of initial scripts were held. The papers were
trial marked to see ‘what the candidates had done to the paper’ and to detect any rogue or
ambiguous questions not detected earlier in the paper setting exercise. The draft-marking
schedule was then adjusted in light of what the markers found and debated. This is
standard practice and it was thoroughly conducted. However, the fact remains that the
unusual results that appeared, showing wide variations between subjects, did not set off
an alarm in NZQA.
88 The explanation for this seems to be that NZQA was aware that there would be variation,
but was not aware that the nature of this variation would be a surprise to nearly everyone
else. The 2004 approach can be contrasted with NZQA’s previous experience with unscaled
School Certificate, where marks from the first 10% of papers were entered on to a
database and the potential mean found. A decision was then made whether the aggregate
results were within ‘professional public tolerances’. Intervention took place from time to
time in the subjects with significant numbers of entries, though some with a small
number of entries would have been left alone.
89 As already mentioned, the business processes that NZQA employed were highly
efficient. These were driven by very intense pressures to deliver the results as rapidly as
possible. The Examination Centre Managers coordinated the collection of the candidate’s
scripts and forwarded them to regional hubs, from where they were sent to markers.
Markers then sent in results and returned the scripts to candidates via a hub. The drive
and tempo behind these processes meant that an analytical pause, a step back to interpret
the wider impact and meaning of the results, did not occur. In the end, the overall picture
from all the subjects was not apparent until results were in the pipeline for processing
and mailing to candidates. By then it was too late.
90 Major time pressure at two points is difficult for NZQA to manage. On the one hand, the
schools exert pressure for the examinations to happen as late as possible in the school
year, to maximise teaching time and the discipline that impending assessment applies to
the students. On the other hand, schools, tertiary institutions, parents and the public want
results to be out as soon as possible. Added to this is the operational ‘given’ that
marking has to be over by Christmas, because markers are less reliably available
afterwards.
91 This matter will have to be tightly managed in 2005, because the Scholarship
examinations are scheduled to be split into two halves, some before the NCEA
assessments and some after. Security arrangements will need to be increased for papers
to remain secure in the Examination Centres for a month; alternatively, the papers will
have to be sent out twice over that period. In addition the period of time between the last
examination and Christmas is shortened by a minimum of four days. NZQA are
exploring an arrangement whereby the Examination Centre Managers send the scripts
directly to markers rather than to the regional hubs for distribution. This makes good
business sense, but a note of caution is that business processes may take precedence over
sound and measured assessment practices. The Review Team emphasises that more
time, not less, needs to be available to NZQA to manage its responsibilities and to ensure
fair results.
Patterns of Entry
92 The numbers and characteristics of the candidates were unknown for the new 2004
Scholarship. Publicity stressed the elite nature of the examination, but the combination
of monetary awards and entry requirements that did not cost students any more than the
15

NCEA fees may have contributed to higher than expected numbers of clearly unsuitable
candidates in some subjects. In other subjects there was a contrasting pattern of low
percentages of entries from the eligible cohort. The reasonably high correlation between
NCEA results which rated ‘Excellent’ and the 2004 Scholarship performance (which
would be expected) did not happen uniformly. There is also some evidence that not all
the best students entered the 2004 Scholarship, whereas students who in the past entered
the University Bursary examinations were automatically eligible for scholarship grades
(higher than an ‘A’ grade) and possible monetary awards which were given to top
category students. 5 (Note that the NZEST scholarship examination remained separate
and independent and was not government funded – see paragraph 37). This pattern of
elective entry further unbalanced the results.
1.5 Whether communication and guidance to teachers and students
concerning New Zealand Scholarship were adequate
93 The Review Team specifically invited comment from teachers and from students on their
views about how well they felt prepared for the 2004 Scholarship (see Review Process
and Methodology). The universal response was they did not feel well prepared.
94 For teachers this was caused by a combination of the deficiencies in professional
development, and the compressed timeframes already discussed. The Post Primary
Teachers’ Association, in a submission to the Review Team, stated, “there appears to
have been a general unawareness in the profession of what the Scholarship exams would
require.”
95 A summary of comments from students highlights the following points:
• The main focus in 2004 was on preparation for the introduction of Level 3 NCEA.
• The difference between the old scholarship and the new was not clear to parents,
students and the public, e.g. the requirement to write well especially in the
sciences.
• The information teachers received about Scholarship was patchy due to the lack of
professional development, early problems for some schools in accessing the
exemplars on the website, not receiving what little material was available or not
being sure how to interpret what they did receive.
• The students were dependent on their teachers for guidance on whether or not to
enter Scholarship. This led to some top students not entering and some students
who were less capable entering, spurred by the motivation of monetary award,
resulting in a variable quality of entrants. This contrasted with previous practice
where scholarship was awarded on the basis of University Bursary examination
performance. In other words entry was automatic.
• The results that were graded as Outstanding, Scholarship or Not Achieved were
not understood. Students wanted more transparency. This was compounded by
students receiving their exam scripts back with “unintelligible marking” on them.
96 Despite the above, many students commented that the level of scholarship exams was
about right, even those who did not achieve a Scholarship. There were some outliers like
Calculus which students found very difficult. These comments are consistent with the
Review Team’s findings as discussed in TOR 1.1 - 1.4.
5 Up until 2003 the term ‘Scholarship’ was used to denote both subject grades and monetary awards
16

97 The way any implementation agency approaches communication about change is critical
to building confidence with key stakeholders. In terms of communicating this significant
change to the sector, as one stakeholder put it, “you need to understand what it is you are
trying to do. You need to agree key messages, which are simple and straightforward, and
you need to keep saying them.” The Review Team agrees.
1.6 Any other matter which the Review Team considers relevant to the
foregoing questions
98 There are three matters that the Review Team believes it is worth commenting on briefly.
They need to be seen in the spirit of moving forward and implementing the 2005
Scholarship successfully. They are:
• Continuous improvement - implementation needs to be a process whereby the
implementation team makes constant adjustments to the way it does things, to
ensure a successful outcome. Any rigid adherence to an ideal (if this did indeed
occur) will get in the way of a successful outcome.
• Feedback loops – sector representatives who participate in the Leaders’ Forum
need to be kept informed of policy and implementation decisions and issues in a
continuous way. The Ministry and NZQA need to dedicate resource specifically
to keeping the sector and its key stakeholders consistently well informed as well
as involved in its consultative forums.
• Building sector capacity in external assessment – the Scholarship and the wider
NCEA call constantly on the same and small number of experienced teachers in
the tertiary and secondary sectors. The Review Team believes that there is an
urgent need to build up this capacity and a related need to review the conditions
under which they are engaged to facilitate development.
99 A final word on the debate about assessment. Norm-referenced assessment is at one end
of the assessment spectrum and standards-based (or criterion-referenced) assessment lies
at the other. Norm-referencing usually applies norms at the stage of assessment results
and reporting, using an inherent understanding of standards to set the mean scores
appropriate for an age group or cohort. Standards-based approaches tend to apply norms
at an earlier, pre-assessment stage, e.g. when setting students’ achievement standards.
100 These positions are seen as opposing positions, though in fact it is more constructive to
see them as complementary, not exclusive, and differing largely at the point where the
norms are applied. Norms and standards in fact apply continuously throughout the whole
education and assessment process. When teachers use exemplars they apply both norms
and standards. So too do markers when they adjust schedules after assessing an initial
selection of student’s scripts. There has been an international movement to focus more
on standards-based assessment because of the balance of advantage it offers to teaching
and learning. The intention of standards-based assessment is that teachers have better
knowledge of a student’s success or failure before it is too late to do anything about it,
students have improved knowledge of their targets, teachers improve their professional
assessment skills, and deficiencies in the wider educational system are known.
101 While both assessment approaches belong to the same spectrum, the reality is that the
new 2004 Scholarship and NCEA were introduced in a climate of polarisation. This has
affected the way in which the sector and the public perceive this major transition in
secondary school education. These two approaches need not and should not be opposing
and it is incumbent on both agencies but notably the Ministry to provide leadership in
17

ensuring that the new approach to teaching, learning and assessment is implemented in a
way that works for New Zealanders. For both agencies and particularly NZQA, there
needs to be a strong internal culture which welcomes different opinions about assessment
so that robust policies and practices are implemented.
Recommendations
The Review Team recommends that:
In respect of 1.1:
1 The Ministry takes the lead in ensuring that all the activities (involving government
agencies as well as sector consultation groups) associated with the 2005 Scholarship are
properly coordinated and deliver a consistent approach.
2 The NZQA Board receives clear specifications and expectations relating to Cabinet’s
decisions on the Scholarship Reference Group report [CAB Min (05) 11/2], in a letter
from the Associate Minister of Education.
3 The NZQA Board engages with the Minister and the Ministry on any concerns or lack of
clarity related to leadership of implementation of the 2005 Scholarship.
4 NZQA develops a comprehensive implementation plan for the 2005 Scholarship and
beyond in response to the letter from the Associate Minister of Education. The plan
should be approved by the Associate Minister. This plan should cover, but not be limited
to:
• resources required
• timelines and milestones
• an analysis of risks and risk mitigation strategies
• IT integration matters
• a professional development strategy including exemplar development
• a single communication strategy (see recommendations 17 and 20)
• progress reporting including exception reporting for milestones not met
• plans for establishing the two independent external groups recommended by the
SRG and approved by Cabinet on 29 March 2005.
5 NZQA sets up as a matter of urgency a dedicated implementation team for the 2005
Scholarship, with representation from the Ministry of Education, which has strong
feedback loops with the sector and which is responsive to concerns raised.
6 Operational policy matters and risks relating to the 2005 Scholarship and beyond be
clearly identified by NZQA in consultation with the Ministry and conveyed to the
Associate Minister of Education. NZQA, in consultation with the Ministry, should lead a
process for resolution of these matters. This should include the timing of the
examinations and release of the results (see recommendation 16).
In respect of 1.2:
7 The Ministry Education provides the Associate Minister with an analysis of the risks
associated with the revised policy settings for the 2005 Scholarship and strategies to
mitigate those risks, prior to the Associate Minister writing to the NZQA Board to
18

communicate the government’s expectations around the implementation of the 2005
Scholarship.
8 The NZQA and the Ministry of Education develop a programme to build capability in the
secondary sector to widen the already stretched pool of examiners and markers.
In respect of 1.3:
9 NZQA establishes a small team of National Assessment Facilitators (NAFs) specifically
for the 2005 and 2006 Scholarships, or ensures that NAFs have an explicit component of
their work dedicated to the Scholarships.
10 The Ministry of Education involves NZQA and sector expertise actively in the continued
revision of the Scholarship standards.
11 NZQA, with input from the Ministry of Education, revises the 2005 papers to ensure the
questions can adequately distinguish a range of student performance, and establish
marking schemes that allow transparent differentiation of achievement and transparent
ranking.
12 NZQA, with input from the Ministry of Education, establishes conventions that provide
consistent guidance to marking panels of different subjects on how to award
Scholarships.
13 NZQA ensures the development of multiple exemplar resources for teachers to use in the
2005 school year including some with marked students’ scripts 6 .
14 The Ministry of Education conducts two professional development days in 2005 and two
in 2006, tailored specifically to the needs of Scholarship.
In respect of 1.4:
15 The implications of the boundaries of tolerance set for each subject in the 2005
Scholarship, as agreed by Cabinet following the SRG report 7 should be clearly
communicated to Ministers.
16 NZQA provides urgent advice to the Associate Minister of Education about the potential
for an earlier start to the examinations and a later release of results than occurred for the
2004 Scholarship. Timing and resources should allow for the start and completion of
script marking to reasonably identify issues, and if necessary, for expert advice to be
obtained and consultation with Ministers to occur. A delay should be possible in the
release of results and the return of examination scripts to students until issues are
satisfactorily resolved.
17 A concerted information campaign, as part of the communications strategy, be conducted
in 2005 to set realistic professional and public expectations for the release of results and
to provide reassurance that the issues that emerged in 2004 have been adequately
understood and addressed.
18 NZQA ensures a conservative, risk-averse approach to the 2005 Scholarship, so that
people and systems are prepared for the outcomes and can implement appropriate
management strategies.
6 It may be possible to use the guinea pig scripts that were retained from 2004
7 Cabinet has agreed to a range of 2-3% for each subject, with an annual variation of plus +/- 1%. For small subjects the
variation can extend to +/-1-5 candidates
19

19 An examination “hot-line” between agencies and the office of the Associate Minister of
Education be maintained over the Christmas holiday period.
In respect of 1.5:
20 NZQA, with the support of the Ministry of Education, develops a communication
strategy based on a shared understanding of the implementation. Key messages need to
be simple, direct, and clear and ensure effective and consistent communication to
multiple stakeholders.
In respect of 1.6:
21 The Ministry of Education and NZQA together encourage an active internal and external
debate on assessment that engages and earns the respect of the education community.
This debate should encompass the full spectrum of assessment expertise and wellresearched
international models.
20

Annex 1
New Zealand Qualifications Authority
Terms of Reference for Review by the State Services Commissioner
Introduction
The New Zealand Qualifications Authority ("NZQA") is a Crown entity established under
Part XX of the Education Act 1989 "to establish a consistent approach to the recognition of
qualifications in academic and vocational areas"(s247).
NZQA's functions include the administration of the National Certificate of Educational
Achievement (NCEA) and other school, trade and vocational assessment. In 2004, NZQA
instituted a new Level 4 qualification, New Zealand Scholarship, to be the basis for the
allocation of university scholarship awards.
Background to Review
New Zealand Scholarship was introduced at the same time as NCEA Level 3. The new Level 4
stand-alone qualification was designed to enable the award of scholarships on the basis of
examination. Previously, scholarships had been awarded to the top 3 or 4 percent in University
Bursary examinations.
Students sat the New Zealand Scholarship examinations in November 2004, and results were
mailed to them in January 2005, and later made available on the NZQA website. The results
showed a much smaller number of scholarships awarded this year than expected and widely
differing achievement rates in different subject areas. The results gave rise to public concern
about the administration of the new qualification, and its fairness to otherwise high achieving
students.
The Associate Minister of Education has asked the State Services Commissioner to review the
adequacy of the setting and management of the 2004 Scholarship examinations and, more
widely, the performance of NZQA. The Minister's request is made under section 11 of the State
Sector Act 1988.
Terms of reference
Scholarship examinations
1 To review the 2004 New Zealand Scholarship examinations with particular regard to:
1.1 whether the roles and responsibilities of the various agencies involved were clearly
defined;
1.2 whether the identification and addressing of issues arising in designing and implementing
the new system was adequate;
1.3 whether the processes to set and moderate the standards for the qualification were
adequate;
1.4 whether the administration of the examinations against those standards, including review
of the processes for setting, moderating, and marking the examination were adequate;
21

1.5 whether communication and guidance to teachers and students concerning New Zealand
Scholarship were adequate;
1.6 any other matter which the reviewer considers relevant to the foregoing questions; and
2 To make recommendations concerning these matters.
NZQA’s Performance
3 Given the purpose and role of the NZQA to provide quality assured qualifications, to
review how well the NZQA is undertaking its role in respect of the school sector
qualifications, with particular attention to:
3.1 the governance, monitoring and reporting of NZQA performance;
3.2 the adequacy of NZQA capability including planning, systems and processes to
undertake its role:
3.3 the nature and extent of the involvement of, and responsiveness to, key stakeholders in
the decision-making of the NZQA;
3.4 whether and if so how the NZQA is providing professional leadership in examination
processes;
3.5 any other matter which the reviewer considers relevant to the foregoing question.
4 To make recommendations concerning these matters.
Reviewer
The State Services Commissioner will appoint a suitable person or persons to carry out the
review and to report to him, pursuant to section 25 of the State Sector Act.
Under section 25 of the State Sector Act, the Commissioner and persons he appoints under
section 25(2) of the State Sector Act have the same powers to summon witnesses and to
receive evidence as are conferred on a Commission of Inquiry by the Commissions of Inquiry
Act 1908.
Timing
The reviewer will report and make recommendations on:
• the questions concerning the introduction of the New Zealand Scholarship
qualification by 29 April 2005, and
• the performance of the NZQA by 31 July 2005.
The two parts of the review have separate reporting dates to enable any findings and
recommendations concerning the New Zealand Scholarship examinations to be taken into
account, as far as will be possible, in the administration of the 2005 examinations.
VA (Tony) Hartevelt
Deputy State Services Commissioner
22

Annex 2
Organisations consulted during the course of this Review
Ministry of Education
New Zealand Qualifications Authority
Education Review Office
Independent Schools of New Zealand (Tim Oughton)
Institutes of Technology and Polytechnics of New Zealand (Jim Doyle)
New Zealand Catholic Education Office (Pat Lynch)
New Zealand School Trustees’ Association (Ray Newport)
New Zealand Vice Chancellors’ Committee (Professor Roger Field)
Post Primary Teachers’ Association (Debbie Te Whaiti, Kate Gainsford)
Post Primary Teachers’ Association Principals’ Council (Don McLeod)
School Principals’ Association of New Zealand (Graham Young)
A selection of Secondary School Principals
Chief Examiners
University Assessment Academics
Auckland University
Manukau Institute of Technology
23