An investigation of the process of writing IELTS Academic Reading ...

More documents

Recommendations

Info

Anthony Green and Roger Hawkey creative All the writing is creative, even though we are starting with something – rather like putting on a play (William) Editing problem solving can be creative, but not satisfactory when you seem to be doing another item writer’s work for them (William) Proof reading Techniques for writing enough items – ‘in summaries you’ve got to go for the nouns, which you didn’t know when you first started’ (Anne) programmatic challenging Finding the texts and shaping them (Anne) Understand a subject you may not be familiar with (William) Creating the items once a suitable text has been chosen (Elizabeth) straightforward frustrating Feedback that you don’t agree with (William) ‘There are times when you have to have a quick walk round the garden’ (Anne) Losing a submission altogether (rejection) Disagreement about issues of bias – William finds Business papers less sensitive: others find Cambridge Main Suite papers more sensitive. satisfying supported Editing and preediting is supportive on the whole (Anne) Colleagues are generally helpful and supportive. Rejection of tasks comes when topic not checked in advance (William) You can ask for elaboration of pre-editing feedback. (Elizabeth) I don’t’ think I have ever disagreed with pre-editing feedback (Jane) Some texts accepted when I could answer on basis of topic knowledge, others rejected when answers did not seem guessable to me (William) The whole issue of how guessable items are is difficult (Anne) A collocation can be guessable to a native speaker, but not to NNS (William) but part of reading is the ability to predict(Elizabeth) Looking for a text is unsupported (Anne) unsupported Table 8: Summary of experienced participant focus group comments and ratings on semantic differential scales 4.2.2 Participant focus group discussions The experienced group participated next in a focus group discussion structured around a set of nine semantic differential continua (Osgood, 1957, using the unlabelled scale format compared with other formats by Garland, 1996) and as seen in Table 8. In the table, summaries of the comments made by the participants in their 20 minutes of unmediated discussion are placed in their approximate location on the continua for the nine scales. The adjectives for the continua were selected by the researchers. Again, points made by participants in the focus group discussion served to triangulate views expressed in the preceding interview activity concerning IELTS text search and treatment and item development: flowcharts and discussions already reported. Following discussion of the semantic differentials, the research team pursued emerging issues with the group. The experienced group, like the non-experienced, expressed uncertainty about candidates’ level of English language proficiency. The four discussed the need to keep the candidates in mind when writing items, but agreed that it was challenging to do this, given the ‘the variety of the situation and [the candidates’] levels of English’. Each participant had their own points of reference for these. Anne IELTS Research Reports Volume 11 www.ielts.org 26
An investigation of the process of writing IELTS Academic Reading test items also worked as an examiner for the speaking paper and so met many candidates while both William and Elizabeth had experience of preparing students for the test. However, Elizabeth reminded the group that the candidates they met in the UK would not be representative of the full range of candidates taking the test – especially those from relatively underprivileged backgrounds. Item writers also received information about candidates from IELTS. An annual report on demographic data is provided by Cambridge ESOL and ‘common wrong answers’ to open response items are discussed at pretest review meetings. What Anne described as the ‘off the wall’ nature of some of these wrong answers and the observation that ‘some people have been accepted at universities, where I thought their English was totally inadequate’ led William to the conclusion that ‘you can do reasonably well on IELTS, I think. And still have what seems to be a low level of English’. Elizabeth also questioned whether IELTS candidates would need to arrive at a full understanding of the text in order to succeed on the questions, suspecting that in IELTS ‘half the time the candidates don’t read the text from beginning to end because they don’t have to’ because local details in the text were being tested by the items rather than the overall meaning. However, Anne wondered whether William’s concern could be justified as success on the test would require adequate levels of performance on the direct speaking and writing papers as well as reading and listening. There was discussion of how the participants had developed their item writing expertise. For Jane this was not easy to explain: ‘It’s difficult to say sometimes exactly what you’re doing and how you’re doing it’. Anne agreed, observing that ‘the processes you go through aren’t necessarily conscious’. However, there were item writing skills that could be learnt. Anne had come to appreciate the importance of ‘working the task’: attempting it as a candidate would. Jane agreed that this was helpful, but admitted she rarely did this prior to submission because of the pressure of deadlines. Elizabeth had found very helpful the advice given to her at her initial training session to focus on what she felt to be the key points of the text, finding that this could help her when she was ‘stuck on something’. Anne felt that her items had improved ‘over years of seeing other peoples’ and having to mend your own’. William pointed to the value of attending editing meetings to obtain insights and Elizabeth felt that feedback at editing meetings had been one of her main sources of learning about item writing especially where a the chair of the meeting, as an experienced and successful item writer, had been effective at showing how a text or item could be improved. William spoke of having learnt how to devise plausible distractors for multiple choice items. However, there were limits to how far this could be learnt as an item writing skill and he wondered about the role of background knowledge in eliminating incorrect options: ‘I think there’s a risk with IELTS because if it’s a scientific text, I may not know nearly enough to know what would be a plausible distractor. What seems plausible to me could be instantly rejected by somebody who knows a little more about the subject.’ Testing implicit information was seen to be problematic. There were cases of disagreement between the item writers and their colleagues carrying out pre-editing reviews about ‘whether [a point] is implicit, but strongly enough there to be tested or not’ (William). For Jane, testing the writer’s interpretation against others’ was a further argument in favour of the pre-editing and editing processes: ‘fresh eyes are invaluable when it comes to evaluating a task’. Although Jane reported that she tried to keep the level of language in mind as she wrote, the group agreed that the difficulty of items was not easy to predict. None of the writers seemed to have a clear sense of the proportion of items associated with a text that a successful IELTS candidate at band 6.0 or 6.5 might be expected to answer correctly. Pretesting results often revealed items to be easier or more difficult than expected. IELTS Research Reports Volume 11 www.ielts.org 27
Page 1 and 2: 5 An empirical investigation of the
Page 3 and 4: An investigation of the process of
Page 25: An investigation of the process of
Page 77 and 78:
An investigation of the process of
Page 79 and 80:
Page 81 and 82:
Page 83 and 84:
Page 85 and 86:
Page 87 and 88:
Page 89 and 90:
Page 91 and 92:
Page 93 and 94:
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
show all

An investigation of the process of writing IELTS Academic Reading ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?