An investigation of the process of writing IELTS Academic Reading ...

More documents

Recommendations

Info

Anthony Green and Roger Hawkey 1 AIMS This research report describes a study of reading, test text selection, item writing and editing processes, areas of test production that have rarely been transparent to those outside testing organisations. Based on retrospective reports, direct observation and analyses of the texts produced, the report compares how trained and untrained item writers select and edit reading texts to make them suitable for a task-based test of reading and how they generate the accompanying items. Both individual and collective editing processes are investigated. The analyses in the study are expected to inform future high-stakes reading test setting and assessment procedures, in particular for examination providers. 2 BACKGROUND AND RELATED RESEARCH 2.1 A socio-cognitive test validation framework The research is informed by the socio-cognitive test validation framework (Weir 2005), which underpins test design at Cambridge ESOL (Khalifa and ffrench, 2008). The framework, further developed at the Centre for Research in Language Learning and Assessment (CRELLA) at the University of Bedfordshire, is so named because it gives attention both to context and to cognition in relating language test tasks to the target language use domain. As outlined in Khalifa and Weir (2009) and Weir et al (2009a and 2009b), in the socio-cognitive approach difficulty in reading is seen to be a function of 1) the complexity of text and 2) the level of processing required to fulfil the reading purpose. In Weir et al (2009a) IELTS texts were analysed against 12 criteria derived from the L2 reading comprehension literature (Freedle and Kostin 1993, Bachman et al 1995, Fortus et al 1998, Enright et al 2000, Alderson et al, 2004 and Khalifa and Weir 2009a) These criteria included: Vocabulary, Grammar, Readability, Cohesion, Rhetorical organisation, Genre, Rhetorical task, Pattern of exposition, Subject area, Subject specificity, Cultural specificity and Text abstractness. In the current study, we again employ such criteria to consider the texts produced by item writers and to analyse the decisions they made in shaping their texts. In Weir et al (2009b) the cognitive processes employed by text takers in responding to IELTS reading tasks are analysed, with a particular focus on how test takers might select between expeditious and careful reading and between local and global reading in tackling test tasks. Local reading involves decoding (word recognition, lexical access and syntactic parsing) and establishing explicit propositional meaning at the phrase, clause and sentence levels while global reading involves the identification of the main idea(s) in a text through reconstruction of its macrostructure in the mind of the reader. Careful reading involves extracting complete meanings from text, whether at the local or global level. This is based on slow, deliberate, incremental reading for comprehension. Expeditious reading, in contrast, involves quick, selective and efficient reading to access relevant information in a text. The current study was expected to throw light on how the item writers might take account of the processes engaged by the reader/ test taker in responding to the test tasks and how item writers’ conceptions of these processes might relate to reading for academic study. IELTS Research Reports Volume 11 www.ielts.org 4
An investigation of the process of writing IELTS Academic Reading test items 2.2 Item writing Item writing has long been seen as a creative art (Ebel 1951, Wesman 1971) requiring mentoring and the flexible interpretation of guidelines. This has been a source of frustration to psychometricians, who would prefer to exert tighter control and to achieve a clearer relationship between item design characteristics and measurement properties. Bormuth (1970) called for scientifically grounded, algorithmic laws of item writing to counter traditional guidelines that allowed for variation in interpretation. Attempts at standardisation have continued with empirical research into the validity of item writing rules (Haladyna and Downing 1989a and 1989b); the development of item shells – generic items with elements that can be substituted with new facts, concepts or principles to create large numbers of additional items (Haladyna 1999); and efforts to automate item generation (Irvine and Kyllonen 2002). Numerous studies have addressed the effects of item format on difficulty and discrimination (see Haldyna and Downing 1989a, Haladyna, Downing and Rodriguez 2002) and guidelines have been developed to steer test design and to help item writers and editors to identify common pitfalls (Haladyna and Downing, 1989a, Haladyna 1999). For all this, Haladyna, Downing and Rodriguez (2002) conclude that item writing remains essentially creative as many of the guidelines they describe remain tentative, partial or both. Yet stakeholder expectations of evidence-based, transparently shared validation for high-stakes language exams are increasingly the order of the era (see Bachman, 2005, and Chalhoub-Deville, Chapelle, and Duff (eds), 2006) often specified through codes of practice (eg, ALTE, 1994). Rigour is increasingly expected of item-writer guidelines in the communicative language skills testing sector. The new Pearson Test of English (PTE), due in 2009, aims, like IELTS, to provide language proficiency scores, including reading measures for colleges, universities, professional and government bodies requiring academic-level English. de Jong (2008) proposes an analysis, for PTE item writer training purposes, of item types (14 potentially applicable to the testing of reading) and a schema for item writer training structured around a general guide, item specific instructions, reference materials, codes of practice, an item writer literature review and the Common European Framework of Reference (CEFR). Cambridge ESOL’s own framework for the training and development of item writers is referenced in some detail below. A number of handbooks include guidance on item design and quality assurance issues in language tests (eg, Valette 1967, Carroll and Hall 1985, Heaton 1990, Weir 1993, Norris et al 1998, Davidson and Lynch 2002, Hughes 2003). These provide advice on the strengths and weaknesses of various item formats and stress the need for item review and piloting. It is generally taken as axiomatic that trained test item writers are superior to the untrained (Downing and Haladyna 1997). While the focus of research has been on the characteristics of items, very little attention has been given to the processes that item writers go through in creating test items and the contributions that these may make to the quality of test material. In a rare piece of research focusing on this area, Salisbury (2005) uses verbal protocol methodology and a framework drawn from the study of expertise to explore how text-based tests of listening comprehension are produced by item writers Salisbury (2005, p 75) describes three phases in the work of the item writer: ! Exploratory Phase: ‘searching through possible texts, or, possibly, contexts’ ! Concerted Phase: ‘working in an intensive and concentrated way to prepare text and items for first submission’ ! Refining Phase: ‘after either self-, peer- or editor-review, polishing/improving the test paper in an effort to make it conform more closely to domain requirements’ IELTS Research Reports Volume 11 www.ielts.org 5
Page 1 and 2: 5 An empirical investigation of the
Page 3: An investigation of the process of
Page 7 and 8: An investigation of the process of
Page 55 and 56:
An investigation of the process of
Page 57 and 58:
Page 59 and 60:
Page 61 and 62:
Page 63 and 64:
Page 65 and 66:
Page 67 and 68:
Page 69 and 70:
Page 71 and 72:
Page 73 and 74:
Page 75 and 76:
Page 77 and 78:
Page 79 and 80:
Page 81 and 82:
Page 83 and 84:
Page 85 and 86:
Page 87 and 88:
Page 89 and 90:
Page 91 and 92:
Page 93 and 94:
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
show all

An investigation of the process of writing IELTS Academic Reading ...

Create successful ePaper yourself

Delete template?

Save as template?