An investigation of the process of writing IELTS Academic Reading ...

Anthony Green and Roger Hawkey
She found that in comparison to novices, more expert item writers, those producing more positively
evaluated texts and items that met the requirements of the test developers (UK examining boards
offering tests of English as a Foreign Language):
! are more aware of the test specifications and are quickly able to recognise texts that
show potential as test material. Where novices tended to devise a listening script
from a source text first and then to write the questions, experts were more inclined
to start from the questions and then to build a script to fit with these
! are more aware of the needs of candidates for clear contextual information and are
better able to provide accessible contextualising information in the form of short,
accessible rubrics and co-text
! explore a range of possible task ideas rather than committing immediately to one
that might later prove to unworkable
! use many more learned rules or ruses than non-experts including, for example:
! exchanging words in the text and in the question so that the hypernym appears in
the text
! adding additional text to the script to introduce distraction and reduce the
susceptibility of the questions to guessing strategies
Although more experienced item writers tended to outperform the recently trained, expertise was not
simply a function of experience. One writer with no previous experience of test item writing
performed better in the judgement of a review panel than two item writers with extensive experience
(Salisbury 2005). Salisbury also concludes that expertise in Listening test item writing is collective in
nature. Individual writers rarely have sufficient capability to meet institutional requirements at the first
attempt and need the feedback they receive from their colleagues to achieve a successful outcome. It
might be added that item writer expertise itself is not sufficient to guarantee test quality. Even where
items are subject to rigorous review, piloting usually reveals further deficiencies of measurement.
The Cambridge ESOL approach to test development is described in detail by Saville (2003) and by
Khalifa and Weir (2009). The IELTS test production process for the reading and listening papers is
outlined in a document available from the IELTS website, www.ielts.org. The goal of this test
production process is that ‘each test [will be] suitable for the test purpose in terms of topics, focus,
level of language, length, style and technical measurement properties’ (IELTS 2007, 1).
IELTS test material is written by freelance item writers externally commissioned by Cambridge ESOL
in a process centrally managed from Cambridge and carried out according to confidential test
specifications or item writer guidelines laid down by the test developers (although see Clapham 1996a,
1996b for an account of the role of externally commissioned item writing teams in developing the
IELTS academic reading module). These guidelines, periodically modified to reflect feedback from
item writers and other stakeholders, detail the characteristics of the IELTS modules (speaking,
listening and academic or general training reading and writing), set out the requirements for
commissions and guide writers in how to approach the item writing process. The guidelines cover the
steps of selecting appropriate material, developing suitable items and submitting material. However, a
good deal of the responsibility for test content is devolved to the externally commissioned workers
including the item writers and their team leaders or chairs for each of the modules. Khalifa and Weir
(2009) describe the chair as having responsibility for the technical aspects of item writing and for
ensuring that item writers on their team are fully equipped to generate material of the highest quality.
IELTS Research Reports Volume 11 www.ielts.org 6