An investigation of the process of writing IELTS Academic Reading ...

More documents

Recommendations

Info

Anthony Green and Roger Hawkey 7 COMPARISONS BETWEEN GROUPS The approach adopted for this study involved asking both experienced and inexperienced writers about their practices based around Salisbury’s (2005) phases of the item construction process. The study collected both written (flowchart) and oral (interview and focus group) data on item writer processes and products (draft and edited texts and items) and incorporated both deductive and inductive approaches to analysis. This approach has proved useful in identifying differences across the item writer groups and between individuals within the groups. These differences highlight both the importance of item writer training and guidelines and suggest changes that might be advisable. In the context of recent research undertaken by the University of Bedfordshire into the IELTS Academic Reading test, the current study can help to explain some of the characteristics of IELTS texts identified by Weir et al (2009a) and the types of reading employed by IELTS test takers (Weir et al 2009b). 7.1 Item writing processes Both the experienced and non-experienced item writers seem to pass through similar steps in constructing their items. They typically begin from a topic, locate texts related to the topic, identify and evaluate potential IELTS texts before selecting one that seems appropriate – this is clearly Salisbury’s (2005) exploratory phase. Both groups reported that they found this the most timeconsuming stage in the item writing process. With the exception of Jane, the experienced writers all included more steps in their item writing flowcharts than their non-experienced counterparts. The flow charts include similar attention to text editing in both groups, but there is greater attention to task development among the experienced group: this being broken down into number of steps including revision and re-editing of the text following or in conjunction with item writing. In the next phase – recognisable as Salisbury’s (2005) concerted phase – all of the writers carried out an iterative process of editing the text and developing the items. Unlike the writers in Salisbury’s (2005) study, who were devising scripts for tests of listening comprehension, these writers could not be said to have started from their items in writing their texts. However, as observed by Salisbury (2005) in her study, the experienced writers seemed to have a repertoire of gambits for efficiently exploiting their source texts and paid attention to task type in text selection. They also paid attention to potential items during the initial exploratory phase – highlighting or making notes on testable material. While the untrained writers selected material that was already close to the appropriate length, trained writer texts chose much longer pieces then progressively cut out passages that seemed to repeat information or that included elements that would not be tested. The extent of editing and the desire to avoid repetition perhaps explains why the texts analysed in Weir et al (2009a) displayed relatively high type: token ratios in comparison with undergraduate textbooks (indicative of a wide range of vocabulary use and rapid progression of ideas). As a first step in what Salisbury (2005) calls the refining phase, the experienced group favoured attempting the task themselves after an intervening period (although deadlines sometimes limited the opportunities for this). The non-experienced writers also reported attempting their own tasks, but Mary and Victoria additionally asked friends to respond to their tasks and so were able to obtain some further feedback on how well the items were working before coming to the editing session. IELTS Research Reports Volume 11 www.ielts.org 50
An investigation of the process of writing IELTS Academic Reading test items 7.2 The texts The non-experienced writers drew on very similar sources to their experienced counterparts. Both Mary and Elizabeth chose articles from New Scientist articles while both Mary and Jane selected texts concerning robot technology. Victoria’s text was an article from a popular science magazine concerning dreams while Anne’s was an article from a popular science magazine concerning sleep. Readability statistics for the two groups were also very similar. The easiest and most difficult texts according to the Flesch Kincaid and Coh-Metrix measures were both produced by experienced writers (Jane and Elizabeth respectively). Both groups expressed a concern that the selection of topics in the test may be rather narrow. Where the non-experienced group saw this as a constraint imposed by the need to produce IELTS-like texts, the experienced group saw it as a by-product of the need for accessibility and cultural neutrality: arts texts tend to assume or require background knowledge in a way that popular psychology or technology-based texts do not. Members of both groups edited their (magazine) texts to make them more ‘academic’ in style and tone and less journalistic. All of the texts involved plausibly academic topics presented for the general reader. All writers in both groups edited to eliminate (some) vocabulary on the grounds that it was either too technical for the general reader, too colloquial to be appropriate in an academic text or too infrequent and so difficult for IELTS candidates. Both groups included factual texts (Mathilda’s text on ‘cities’ and Jane’s ‘wildlife cameras’ text) and opinion texts (William’s essay on literature, Elizabeth’s ‘sleep’ text, Anne’s’ laughter test’ from the experienced group; Mary’s ‘robots’ text and Victoria’s ‘dreams’ text from the untrained group). Members of both groups also sought to avoid potentially divisive or offensive issues and to eliminate culturally specific knowledge from their texts. Mary removed a paragraph from her text concerning war. The experienced group was concerned to avoid religious issues in William’s text. The trained writers seemed more ready to edit their texts; reshaping them if necessary to meet the requirements of the items. Of the untrained writers Mary seemed to have the strongest objections to revising her text, but in fact made the most substantial changes of this group. These changes included moving material between paragraphs to square her text with the items she wanted to use. In sum, the effect of editing for both groups, apparent in the analysis of the submitted texts and from the discussions in the editing meetings, was to increase the coherence and information density of the texts and to make them more accessible to readers from non-English speaking backgrounds. The changes also served to reduce technical and cultural specificity, colloquialism, journalistic touches (such as sensationalism, personal engagement of writer etc) and, particularly in the case of the experienced group’s texts, to reduce the repetition of ideas. In devising their items, both groups made use of a range of item types. The True/ False/ Not Given (T/F/NG) item type was chosen most often across groups, but no clear differences in item type selection could be seen from the small sample submitted. As was to be expected, the experienced item writers submitted items of better quality and were better able to correct the problems that they found. A greater number of shortcomings that would breach the IELTS item writer guidelines could be identified in the untrained writers’ submissions. For some untrained writers, items within sets did not consistently follow the order of information in the text where this would usually be expected (as in Mary’s MCQ items: 15 and 17 concern the first paragraph, 16 is associated with the sixth paragraph and the necessary information for item 18 is distributed throughout the text). Items within a set were sometimes isolated from each other: Mathilda’s item 17, for example, relates to her ninth paragraph while the rest of her T/F/NG items are associated with the first four paragraphs of her text. IELTS Research Reports Volume 11 www.ielts.org 51
Page 1 and 2: 5 An empirical investigation of the
Page 3 and 4: An investigation of the process of
Page 49: An investigation of the process of

An investigation of the process of writing IELTS Academic Reading ...

Create successful ePaper yourself

Delete template?

Save as template?