Computer-based and paper-based writing assessment: a ...

2 | CAMBRIDGE ESOL : RESEARCH NOTES : ISSUE 34 / NOVEMBER 2008of mode of composition, by giving students the choice ofmode in which to compose their essay, increases the effectof mode of presentation. Raters may be influenced, bothpositively and negatively, by the appearance of essays inhandwritten versus typed text. Studies by Powers, Fowles,Farnum and Ramsey (1994) and Russell and Tao (2004a)found that essays presented in handwritten form receivehigher scores than the same essays presented as computertext. The authors suggested a number of hypotheses toaccount for this: increased visibility of errors in thecomputer text; higher expectations for computer texts;handwriting enabling the reader to feel closer to the writer;handwritten responses being given the benefit of the doubtwhen hard to read; and handwritten responses appearinglonger and the result of greater effort.Examiner training and standardisation can help ensurethat any presentation effects are minimised as far aspossible. In a small follow-up study Russell and Tao(2004b) were able to demonstrate that the presentationeffect could be eliminated with supplementary training.In order to train raters in the presentation effect and howto avoid it, we need to look at differences in appearancebetween the texts and how they arise.MethodologyAuthentic test data was obtained in order that the resultscould be directly applied to an assessment context; thus aPET part three writing task common to both a live PB and alive CB examination session was chosen. This task is alonger piece of continuous writing (100 words) in responseto an extract from an informal letter written by a friend. Thisextract provides the topic candidates must write about, witha couple of questions included to focus their ideas. Thetask chosen was on the topic of sport.The study focused on whether the written output from anassessment task was comparable in terms of linguistic andtext features when produced via a paper- or computerbasedadministration mode. The following features werestudied:• Lexical resources: text length, standardised type-tokenratio, lexical sophistication, comparison of candidateoutput and PET wordlists• Organisation in terms of sentences and paragraphs• Surface features: punctuation, capitalisation and use oftext/email conventions• Frequency of typographical/spelling errors that do anddo not impede comprehension.The methodology was finalised after a pilot study (notdescribed here). It is important to note that the CBparticipants had no access to word-processing tools suchas spell- or grammar checkers. Analysis was conductedusing Wordsmith Tools (Scott 1998), Range (Nation andHeatley 1996) and SPSS. The linguistic features studiedwere based on sub-sections of the markscheme althoughthe range and accuracy of grammar were not studied here.Scripts were collected from the two exam sessions andcandidates selected so that they were matched on examcentre and language proficiency. This meant that sampledcandidates would have had the choice of administrationmode thus selecting the mode most suitable to them,a condition suggested by Russell and Plati (2002:21).Thus any possible impact of computer familiarity or anxietyon the test output is likely to be reduced. Candidates fromfour examination centres were selected: one Columbian,two Italian, and one Swiss centre. The total number in thePB group was 86 and the total number in the CB groupwas 82.ResultsWe present the results in four sections below, relating tolexical resources, sentences and paragraphs, surfacefeatures and spelling errors.Lexical resourcesAnalysis of the two groups using Wordsmith Tools revealedthat PB texts were on average 4 words longer; this groupalso had a greater standard deviation (19.4), suggestinggreater variability between the candidates (see Table 1).Table 1: Lexical output and variation by administration modeAdministration mode PB CB————————— —————————Mean SD Mean SDTokens 108 19.4 104 15.4Types 69 11.9 69 8.7Standardised Type/Token Ratio 77 5.4 79 5.3Average Word Length 4 0.2 4 0.3A t-test revealed no significant difference in the meannumber of words written between the two groups (t=-1.294,df=154, p=0.197). Thus there is insufficient evidence tosuggest a difference in text length between theadministration modes.Many of the studies conducted on direct writing showthat participants writing in CB mode generally write more(Goldberg et al 2000, Li 2006, MacCann et al 2002). Theseare contrary to the findings from this study. It must beremembered that this task does have a word limit:candidates are asked to write 100 words. This and the factthat it is a timed situation, will impact on the numbers ofwords written. In addition, the CB candidates have a wordcount available to them so they can be more accurate inachieving this target; which may explain the lower standarddeviation for CB texts.Interestingly, the mean number of types was the same forboth groups (69), although there was greater variation inthe PB scripts (SD 11.9) (see Table 1). This would suggestthat candidates were using a similar range of vocabulary.However the PB candidates used more words; this wouldimply that in the PB test candidates were recycling somewords rather than using the ‘extra words’ to add morebreadth to the composition. The standardised type-tokenratio confirms this; the mean ratio for the CB group is higher(79 compared with 77). A t-test revealed a significantdifference in the mean standardised type-token ratiobetween the two groups (t=2.4, df=165, p=0.019). Thus©UCLES 2008 – The contents of this publication may not be reproduced without the written permission of the copyright holder.