Computer-based and paper-based writing assessment: a ...

6 | CAMBRIDGE ESOL : RESEARCH NOTES : ISSUE 34 / NOVEMBER 2008• Errors: lexical error rates in texts produced in both modeswere similar although the nature of the errors differed.This study reveals that informal letter texts composed byhand or by computer under timed assessment conditionsdo show some variation in linguistic and text features.The findings from this study provide useful insights intothe comparability of PB and CB written output within anassessment context. The use of live data show that findingsare not an artefact of an artificial or trial situation and thusresults have an immediate and direct relevance to testing.It should be noted that the samples used in the studywere limited by those that took the CB test, which currentlyhas a smaller candidature than the PB test. As a result onlycandidates from three countries were studied. The main L1sused in these countries use a roman alphabet, thus thefindings may not necessarily be the same for those using anon-roman alphabet.In addition the study involved two separate samples ofcandidates and while the two groups were matched onlanguage ability it could be that the results were affected bythe nature of the samples themselves. The availability ofmore candidate information such as age, gender and L1would have lent weight to the matching of the sample.The results from this study have a number of implicationsfor teachers, testers and raters. For teachers, issues ofcapitalisation, punctuation and paragraphing can behighlighted. If teachers are made aware of areas in whichcandidates are having problems then these can beaddressed at classroom level. For language testers thedifferences found between the texts written in differentmodes can be built into rater training. This is both for theinterpretation of mark schemes and for dealing with thepresentation effect. For example if text organisation interms of paragraphing is to be assessed then there needsto be an awareness that this may differ depending on themode in which the text was produced.This study can be considered a starting point intoresearch into the text differences between PB and CBwritten output. It would be interesting to explore the effectfound with different L1 groups, those with different writingsystems (e.g. Cyrillic, Arabic) and those at differentproficiency levels. An exploration of how these featuresdiffer across modes in other genres would also beworthwhile. Other text features could also be analysed,for example structure, cohesion and larger lexical chunks.More in-depth exploration of the differences found wouldgive further insights enabling us to see how, for example,the lexical breadth differs between groups, i.e. what typesof words are used differently.Investigation of the writing process using protocol studieswould add insight into the composition process andperhaps shed light on why differences and similaritiesbetween the texts are found. There would also be value inexploring how the differences found affect the ratingprocess, for example how examiners respond to textspresented differently in terms of paragraphing or thenumber and length of sentences. An understanding of anyeffect could feed into rater training to enhance the fairnessand reliability of written assessment.References and further readingAlderson, J C (2000) Assessing Reading, Cambridge: CambridgeUniversity Press.Bachman, L F and Palmer, A S (1996) Language Testing in Practice,Oxford: Oxford University Press.Goldberg, A, Russell, M and Cook, A (2003) The effect of computerson student writing: a meta-analysis of studies from 1992–2002,Journal of Technology, Learning and Assessment, 2/1, retrievedfrom www.jtla.orgHorkay, N, Bennett, R E, Allen, N, Kaplan, B and Yan, F (2006) Does itmatter if I take my writing test on computer? An empirical study ofmode effects in NAEP, Journal of Technology, Learning andAssessment, 5/2, retrieved from www.jtla.orgLee, H K (2004) A comparative study of ESL writers’ performance in apaper-based and a computer-delivered writing test, AssessingWriting, 9/1, 4–26.Li, J (2006) The mediation of technology in ESL writing and itsimplications for writing assessment, Assessing Writing, 11/1, 5–21.MacCann, R, Eastment, B and Pickering S (2002) Responding to freeresponse examination questions: computer versus pen and paper,British Journal of Educational Technology, 33/2, 173–188.Nation, P and Heatley, A (1996) Range, Wellington: School ofLinguistics and Applied Language Studies, Victoria University ofWellington, retrieved from www.vuw.ac.nz/lals/staff/paulnation/nation.aspxPowers, D, Fowles, M, Farnum, M, and Ramsey, P (1994) Will theythink less of my handwritten essay if others word process theirs?Effects on essay scores of intermingling handwritten and wordprocessedessays, Journal of Educational Measurement, 31/3,220–233.Russell, M and Plati, T (2002) Does it matter what I write? Comparingperformance on paper, computer and portable writing devices,Current Issues in Education, 5/4, retrieved fromhttp://cie.ed.asu.edu/volume5/number4Russell, M and Tao, W (2004a) Effects of handwriting and computerprinton composition scores: a follow-up to Powers, Fowles,Farnum and Ramsey, Practical Assessment and Research Evaluation9/1, retrieved from http://pareonline.net/getvn.asp?v=9&n=1— (2004b) The influence of computer-print on rater scores, PracticalAssessment and Research Evaluation 9/10, retrieved fromhttp://pareonline.net/getvn.asp?v=9&n=10Scott, M (1998) WordSmith Tools, Version 3.0, Oxford: OxfordUniversity Press.Warschauer, M (2003) Technology and second language writing:Researching a moving target, in Kei Matsuda, P, Canagarajah, A S,Harklau, L, Hyland, K and Warschauer, M, Changing currents insecond language writing research: A colloquium, in Journal ofSecond Language Writing, 12/2, 151–179.Weigle, S C (2002) Assessing Writing, Cambridge: CambridgeUniversity Press.Yates, S J (1996) Oral and written linguistic aspects of computerconferencing: A corpus-based study, in Herring, S C (Ed.)Computer-mediated communication: Linguistic, social andcross-cultural perspectives, Philadelphia: John BenjaminsPublishing Co.This is an extract from Research Notes, published quarterly by University of Cambridge ESOL Examinations.Previous issues are available online from www.CambridgeESOL.org©UCLES 2008 – The contents of this publication may not be reproduced without the written permission of the copyright holder.