Reviewing the FCE and CAE Speaking tests

Research NotesOffprint from Research Notes ISSUE 30 / NOVEMBER 2007 / PAGES 24–28 30/07Reviewing the FCE and CAE Speaking testsCLARE HARRISON ASSESSMENT AND OPERATIONS GROUPIntroductionWork began on the FCE and CAE Review Project in 2004. Aswith previous examination reviews, such as the CPE revisionin 2002 (see Roger Hawkey’s article on page 2), this Projectfollowed Cambridge ESOL’s standard procedures forreviewing examinations, which includes the following keystages:• large-scale consultation with external stakeholders, suchas candidates, teachers, school owners, examiners• internal evaluation of existing tasks• creation of draft specifications based on feedback andproposals from the previous two stages• trialling of new tasks and analysis of findings• communication of trial findings and resulting proposalsto external stakeholders and elicitation of furtherfeedback• definition of final specifications.This process is iterative and cyclical, allowing fullconsultation and analysis before final decisions are reached(cf. Figure 2 on page 4).This article reports on some of the work undertaken tomodify the FCE and CAE Speaking papers within the contextof the FCE and CAE Review Project, concentratingspecifically on Parts 2 and 3 of the tests.Consultation phase 1Two activities were carried out simultaneously to elicitopinion on the FCE and CAE Speaking tests, Chairs’ reportswere commissioned and candidate and centre surveys wereundertaken.Chairs’ reportsOne of these activities involved the Chairs of both FCE andCAE Speaking in producing a report summarising whatthey saw as the strengths of each paper and areas theyfelt could benefit from improvement. Overall, the Chairsconcluded that the Speaking tests were effective intesting the constructs underpinning them since they elicita variety of lexico-grammatical forms and provideopportunities for a variety of interaction types and patterns,e.g. individual, uninterrupted long turns and discussion.However, they considered that certain improvements couldbe made to ensure that the tests were accessible to allcandidates.An internal meeting was held to allow discussion of thecontents of the Chairs’ reports. Some of the key areas fromthis discussion are outlined below.Part 2: individual long turnBoth papers currently contain similar Part 2 tasks. In FCE,each candidate is given a minute to compare and contrasttwo visuals while addressing a task, e.g.Compare and contrast these photographs, and say which of thepeople will remember this moment the longest.CAE is broadly similar although the number of visualsmay vary between two and five, and the task has two partsto it beyond the basic ‘compare and contrast’ aspect, e.g.Compare and contrast two or three of these pictures, saying howthe atmosphere is different in each situation, and what thebenefits of each method of learning might be.For Part 2, the Chairs focused their attention on thedifficulties that some FCE and CAE candidates have inaddressing the task set. While all candidates fulfil the firstpart of the task in which they are required to ‘compare andcontrast’, not all go on to satisfy the second part of the task.This was considered to be partly attributable to the fact thatthe task instructions are given orally, making them harder forsome types of learner to process and remember. Anotherreason suggested was that instructions are given after thecandidates receive the visual material, meaning that somecandidates may not be paying full attention to theinstructions because they are concentrating on the visuals.One improvement to address this which was suggestedby the Chairs was the inclusion of the task in written formon the picture sheet to enable visual learner types to see aswell as hear it.Part 3: collaborative taskPart 3 is also similar for both papers. Candidates areinvolved in a collaborative task based around pictureprompts in which they are asked to discuss somethingtogether and to come to a negotiated conclusion, e.g.First, talk to each other about how interesting these different typesof film would be. Then decide which two would be the best forstudents to discuss. (FCE)©UCLES 2007 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

2 | CAMBRIDGE ESOL : RESEARCH NOTES : ISSUE 30 / NOVEMBER 2007Talk to each other about what message these picturescommunicate about the dictionary, and then decide which picturewould be the best in appealing to people worldwide. (CAE)The Chairs felt that the task type works well and providesappropriate opportunity for assessment of the intendedtesting focus (exchanging ideas, agreeing and disagreeing,interactive communication etc.), but as with Part 2,discussion centred on the difficulties some candidateshave in addressing each part of the task.Proposed improvements for Part 3 were: the inclusion ofthe task in written form on the picture sheet as for Part 2;and the creation of a ‘split task’ in which the Part 3 task isseparated into two parts, allowing candidates two minutesto focus on the first part of the task and a further minute tofocus on the second, e.g. for FCE:First, talk to each other about how interesting these different filmswould be. (2 minutes)Now, decide which two would be the best for students to discuss.(1 minute)Candidate and centre surveysAt the same time as discussions were being held internally,opinions were also being canvassed externally by meansof a large-scale centre and candidate survey. Feedbackreceived in response to this focused on similar issues tothose raised during the internal meeting. However,additional feedback was also received on the use ofvisuals as discussion prompts. Some respondents feltpicture-based tasks may not elicit a sample of languageappropriate to FCE/CAE level, and suggested that ratherthan having visuals as prompts, FCE and CAE Part 2 taskscould be based on a question and word prompts, suchas are found in CPE, BEC, IELTS or Skills for LifeSpeaking 1 .Outcomes of the consultationThe two consultation activities indicated that it would bevaluable to explore a number of avenues for Parts 2 and 3of the Speaking tests. For example, it was suggested that,for Part 2, written task prompts could be added to thepicture sheets. Trialling would then be carried out to see ifcandidates find it easier to address the task set when thereis a written reminder of it for them to refer to. If the abovewere found to be effective, further trialling would beundertaken to determine the most appropriate form for thetask prompt to take. Two forms were suggested: directquestions based on the task set or taken verbatim from theinterlocutor’s task. Trialling would also be undertaken todetermine the most appropriate position on the picturesheet for the written-prompt to appear in.Although the use of written-prompts, such as appear inCPE, would result in a change of testing focus for FCE andCAE Part 2 in that written-prompt tasks would not elicitlanguage for comparing or describing, it was decided thatItem Writers should be commissioned to produce suchtasks in order to explore the possibility of their use with FCEand CAE candidates. Research should then be undertakencomparing the language elicited by the current picture tasksand the new written-prompt tasks.It was decided that for Part 3, as for Part 2, written taskprompts should be added to the picture sheet, and theeffectiveness of this approach trialled as suggested above.Secondly, the Part 3 task should be trialled as a ‘split task’with the two parts of the task separated to allow candidatesto focus on one part at a time.Task prompts on picture sheetsPicture sheets for Parts 2 and 3 were prepared to include awritten replication of the task instructions. Trialling wasundertaken with students preparing to take FCE and CAE,who carried out one current task, with instructions givensolely orally, and one updated task, with written taskprompts on the picture sheets alongside oral instructions.Candidates’ performance was recorded for the purpose ofanalysis, and their opinions on the experience of doing thetwo tasks were elicited.It was found that including written task prompts on thepicture sheets increased candidates’ confidence whenundertaking the tasks as they felt that, if necessary, theycould refer to the written-prompt if they were not able toremember the task that had been addressed to them.Although many candidates did not refer to the prompts,knowing they could do this was considered to bereassuring. Being able to refer back to the task wording wasshown to help candidates focus fully on the task.Follow-up activity 1Once it had been established that written task prompts onthe picture sheets helped candidates to address the taskmore fully, further trials were carried out to determine themost appropriate wording of these. Two forms of taskprompt were trialled and compared. These were:A) Task prompts consisting of an exact replication of thewording used by the interlocutor, which is delivered inthe form of an indirect question. E.g. the CAE interlocutorsays:I’d like you to compare and contrast two or three of these pictures,saying how the atmosphere is different in each situation and whatthe benefits of each method of learning might be.and this is rendered as follows on the picture sheet:• how the atmosphere is different in each situation• what the benefits of each method of learning might beB) Task prompts consisting of the interlocutor’s wordsphrased as direct questions, e.g.:• How is the atmosphere different in each situation?• What might the benefits of each method of learning be?Candidates referring back to the task prompts found iteasier to do so when these were phrased as directquestions.Follow-up activity 2Task prompts were trialled in three positions on the picturesheet: at the top; in the middle; at the bottom. It was found©UCLES 2007 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

CAMBRIDGE ESOL : RESEARCH NOTES : ISSUE 30 / NOVEMBER 2007 | 3that centrally placed prompts were distracting forcandidates, and those located at the bottom of the pagewere easily overlooked. However, those placed at the top ofthe page proved to be positioned where candidates couldsee and refer to them without being obtrusive.Part 2 written-prompt tasks: a new tasktype for FCE/CAE?In response to feedback from some stakeholders whichhighlighted concerns that the Part 2 picture tasks do notelicit an appropriate level of language for FCE and CAE, itwas decided that an alternative task type, based on thatused for the long turn in CPE, should be trialled. Writersproduced tasks for FCE and CAE which were based on astatement of opinion, a question and three short prompts,for example (at FCE level):It’s difficult to keep fit and healthy nowadays. Do you agree?• eating healthy food• getting exercise• getting enough sleepThese were trialled with students preparing for FCE andCAE, who carried out both a current Part 2 picture task anda written-prompt task to allow comparison. The candidateswere recorded and opinions elicited as before. In addition,two post-trial research studies were undertaken,summarised below.Part 2 written-prompt vs. picture task: study 1Members of the Inter-Varietal Applied Corpus Studies(IVACS) research centre at the University of Limerick,Ireland, were invited to undertake in-depth analysis of thedata collected from the trialling in order to comparecandidate output when doing written-prompt tasks in Part 2with output when doing the current picture-based task.Recordings of candidates doing written-prompt andpicture-based tasks were transcribed by researchers. Ineach case, the same candidate carried out both a currentpicture-based task and an updated written-prompt task.Four transcripts of the written-prompt and four of thecurrent picture-based tasks were analysed at each level.The same four students took part in each case, to allowcomparability. Researchers were asked to subjecttranscriptions of candidate long turns to conversationanalysis in order to answer the following research question:In the case of the Part 2 long turn, is there more evidence ofcoherence and cohesion in the candidate output when carrying outthe written-prompt task than when carrying out the picture-basedtask?The findings indicated that for both FCE and CAE Part 2the written-prompt tasks encouraged candidates to producemore varied, coherent and complex language while thepicture tasks seemed to produce less complex languageand more hesitation. However, it should be noted that thepicture-based tasks did not include the written questionprompts on the picture sheets.Part 2 written-prompt vs. picture task: study 2Independent of the study described above, a markingproject, based on the trial recordings, was also undertaken.This study was managed by Cambridge ESOL’s Researchand Validation Group and aimed to investigate the scoresof eight candidates responding to the FCE and eight to theCAE Part 2 tasks with picture prompts and with writtenprompts.Each candidate was rated by 12 experienced OralExaminers and scores recorded for Grammar andVocabulary, Discourse Management and Pronunciationcriteria (Interactive Communication was not consideredrelevant to Part 2 as it involves no interaction between thecandidates). In addition, four of the examiners completedfunctional checklists to record the language functionselicited by the two task types. The research questions forthis study were:• Is there any difference in aspects of performance on each test asreflected in the marks awarded?• Is there any difference in aspects of performance on each test asreflected in the functions elicited?It was found that the CAE candidates in the study achieveda higher score on the picture-prompt tasks than they did onthe written-prompt tasks while the FCE candidates’ scoreswere fairly similar for both task types. The CAE candidates’better scores may have been due to their familiarity with thepicture task, having very likely also prepared for it at FCElevel.In terms of functions, comparisons and descriptions weresignificantly more common with the picture-prompt tasks atboth FCE and CAE levels and resulted in more speculationat CAE level. The CAE written-prompt tasks, on the otherhand, elicited significantly more elaboration of opinionsand reasons for assertions.Part 3 ‘split’ task vs. current taskAt the start of the current Part 3 collaborative task,candidates are set two different conversational goals. Theyare expected to structure the interaction in order to addressboth of these. One of the proposals made by the Chairs inan attempt to help candidates address fully both parts ofthe Part 3 task was to split the task and its instructions intotwo parts. In the ‘split’ task, the two conversational goalsare separated and sequenced by the interlocutor so thatcandidates receive only one task at the beginning of Part 3which is then followed by the second task after they havediscussed the first.The IVACS research team was again invited to undertakeresearch to compare candidate output when theinstructions for Part 3 were given in two stages (the socalled‘split’ task described above) with output on Part 3 inthe current format. The research question which formed thebasisofthisstudywas:In the Part 3 collaborative task, are there differences in thelanguage produced when candidates are given instructions intwo stages as compared to when using the current instructionformat?The analysis of transcripts of candidates doing both tasksshowed that although there was little difference in terms ofthe linguistic complexity of candidate output, there were©UCLES 2007 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

4 | CAMBRIDGE ESOL : RESEARCH NOTES : ISSUE 30 / NOVEMBER 2007differences in goal orientation and task completion.Researchers found that candidates in the sample either didnot address both parts of the current Part 3 task, or did soonly partially. It was found that candidates worked together,but they often did not achieve task completion. In the ‘split’task, on the other hand, candidates tended to achieve bothcollaboration and task completion and it was found thatseparating the rubric into two parts provided candidateswith two manageable tasks to concentrate on one at a time,allowing their discussion to be more task-focused. It shouldbe noted, however, that task achievement per se is notdirectly assessed in the FCE and CAE Speaking tests, sothese findings would not affect the assessment of this task,and that, as in the Part 2 study, the tasks were of thecurrent type which do not include written question promptson the picture sheets.Consultation phase 2Throughout the trialling phase, consultation with externalstakeholders continued. As trialling findings were knownand further proposals were made as a result, thisinformation was presented at consultation meetings withkey external stakeholders so that their opinion could besought. With regard to the Speaking tests, some Chairsagreed that the issue of some candidates only partiallyaddressing the Part 2 and Part 3 tasks was the one thatthey would most like to see addressed.The option of basing the long turn in Part 2 on writtenpromptselicited mixed opinions from stakeholders. Somefelt strongly that the current picture tasks work very welland should be retained, maintaining that candidates likethem and that they provide support for younger, or lesssophisticated candidates, who may find it daunting to givea topical presentation such as is elicited by the writtenprompttask. In addition, some of those consulted pointedout that apart from YLE Speaking tests, the picture-basedtasks are unique to Main Suite within Cambridge ESOL andthat for this reason it would be worth retaining them.However, those in favour of the written-prompt tasks feltthat these were more authentic than the picture tasks andwould allow candidates to produce more complex andcoherent language.The proposal to separate the task instructions for Part 3also elicited a great deal of discussion. However, there wasa minority of supporters for the ‘split’ task, who believedthat in testing as in teaching, it is unwise to set two tasks atthe same time since it is hard to focus on both at once. Themajority of stakeholders were not keen on the ‘split’ task asthey felt that it would make Part 3 disjointed, with theexaminer intervening in the middle of the interaction tosupply the second task. In addition, some of thoseconsulted felt that knowing the goal of the whole task,which is only apparent on hearing the second part of theinstruction, is essential to give a reason for doing the firstpart of the task. They felt that candidates could not carryout the discussion without knowing from the outset whatthe final goal is.Most stakeholders agreed that providing task prompts onthe picture sheets in both Part 2 and Part 3 would be a wayto support candidates in fully addressing the tasks.Final decisionsAt the end of the consultation and research process, all thefeedback and findings were taken into consideration tomake final decisions about the content and format of theSpeaking tests from December 2008.Despite some of the findings from the small-scaleresearch conducted by the University of Limerick into thewritten-prompt Part 2 task, it was finally decided thatpictures should be retained at both FCE and CAE levels. Therationale for this decision takes into account the functionsand topics of this task. The functions which form the basisof the testing focus of the long turn, i.e. comparing,describing, and in the case of CAE, speculating, areeffectively elicited by the current picture task. Thesefunctions are not so effectively elicited by the writtenprompts.Thus, replacing the picture tasks with writtenpromptsin Part 2 would have involved a considerablechange to the underlying construct of one part of theSpeaking test.The topics that could be used to form the basis of thewritten-prompts are similar to those that are already in usein the Part 3 collaborative task, e.g. lifestyle, the worldaround us, our working lives. Had the written-promptsbeen adopted for Part 2, candidates could have foundthemselves talking about similar topics in two parts of thetest and also making use of similar lexis and languagefunctions to do this. Such linguistic and topic overlapwould have reduced opportunities to elicit a variedsample of language for assessment. It would thereforehave been necessary to re-think the focus and design ofthe collaborative task to avoid this potential overlap.In the final analysis, introducing written-prompts for Part 2appeared to necessitate a much greater change to theSpeaking tests than the review process had shown to benecessary or desirable. Similarly, for Part 3, whilst there wasevidence from the research conducted by the University ofLimerick that indicated that the proposed ‘split’ task hadsome benefits over the current format in terms of goalorientation and encouraging candidates to address bothparts of the task, there were other more compelling reasonsto retain the current Part 3 format in the Speaking test. Theproposed change to a ‘split’ task would have resulted in agreater focus on task achievement than is desirable sincetask achievement is not assessed in the Speaking tests and,furthermore, some stakeholder and internal feedbackindicated strong resistance to the idea of separating thetask into two parts.In the face of these conflicting messages regardingPart 3, decision makers returned to the original issue raisedby the Chairs of the papers and also by centre andcandidate survey respondents, which focused on enablingcandidates to fully address the tasks set, and determinedon a course of action that would both address the issue andtake account of stakeholders’ views.It was therefore decided that as the picture tasks wereessentially popular and working well, they should beretained in both Parts 2 and 3. However, to address theissue of task focus in both parts, picture sheets shouldinclude task prompts, which candidates could refer to,allowing them to deal fully with the task set. Trialling had©UCLES 2007 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

CAMBRIDGE ESOL : RESEARCH NOTES : ISSUE 30 / NOVEMBER 2007 | 5shown this approach to be successful and feedback fromstakeholders regarding this proposal had been positive.ConclusionAlthough this review has resulted in only slight changes tothe Speaking tests, the research carried out and opinionsgathered on the subject have provided Cambridge ESOLwith a wealth of valuable information. Clearly, monitoringand reviewing our tests is an iterative process and theinsights afforded by the current review will act as aspringboard for review projects of the future.This is an extract from Research Notes, published quarterly by University of Cambridge ESOL Examinations.Previous issues are available online from www.CambridgeESOL.org©UCLES 2007 – The contents of this publication may not be reproduced without the written permission of the copyright holder.