Assessing communicative competence - ALTE

ALTE 4 th International Conference, Kraków
8 th July 2011
Assessing communicative competence:
focussing on participatory engagement
Constant Leung
King’s College London
Jo Lewkowicz
University of Warsaw

Validating curriculum & assessment
frameworks: an insider view
Framework
selfaffirmation
through award
Learning &
assessment of
learning
Curriculum
&
assessment
frameworks
Programmes
& pedagogy
Professional
teacher
know-how

A critique of reasoning
Assumption: successful frameworks
embody models of proficiency that
reflect actual language use
Question: Do they?

Impact of curriculum & assessment
frameworks
Rubrics &
scales
Descriptors
Additional
factors
Language outcomes

Spoken interaction – CEFR & IELTS
We refer to the CEFR because:
• It claims to be built on communicative
competence (Davies, 2008)
• It’s influential in Europe and beyond
• Textbooks, curriculum frameworks and
assessments claim affiliation (e.g. IELTS)

Contextualising data
Extracts 1 & 2
1 st year university Biology – literacy for
science class
Linguistically diverse students – L1-
L2/English as lingua franca interactions
Biology professor teaching literacy in
specially designed seminar for all 1 st year
students
Compulsory 10-week course
6

Data - ESRC RES-062-23-1666 Modelling for Diversity: Academic Language and Literacies in School and University (Feb
2009- Jan 2011).
Extract 1
01 T: … science papers are not novels(.)they not something that you can read
02 while in the bath or going to sleep or something(.)they require intense
03 concentration.they are hard work(.)there’s no getting away from
04 it(.)no, to read this (an article from Nature) and understand
05 it requires several read and several hours of work=
06 S: =it need to=
07 T: =that’s (quite ) for me
08 S: yeah, the scientific method that he was using to find the Fox2P2
09 tended to let you [inaudible] it was hard(.)couldn’t do it
7

Best-fitting descriptors?
B2 Formal discussion and meetings (Council
of Europe, 2001:78):
Can keep up with an animated discussion,
identifying accurately arguments supporting and
opposing points of view. (B2.2)
Can express his/her ideas and opinions with
precision, present and respond to complex lines
of argument convincingly. (B2.2)
8

Data - ESRC RES-062-23-1666 Modelling for Diversity: Academic Language and Literacies in School and University (Feb 2009- Jan 2011).
• Extract 2
• 01 S: where is mine?
• 02 T: where is yours(.)it’s a very good question
• 03 S: I gave it in
• 04 T: I(.)in that box over there(.)was it? where did you give it
• 05 S: in the box downstairs
• 06 T: ah, that’s why I haven’t got it
• 07 S: why?
• 08 T: because it should have been in the metal box here(.)you give it to me
• 09 again(.)it may even be lost
• 10 S: because if they have any(.)I have your name on my paper [inaudible]
• 11 T: yes but it’s probably got misplaced somewhere because they are not
• 12 expecting this work=
• 13 S: =OK
• 14 T: it should have been in the metal box there [pointing to the direction of an adjacent room]
• 15 S: huh
• 16 T: this time this time I’ll let you off(.)if it happens again I can’t let you off
• 17 S: OK
• 18 T: all tutorial work is in the box [pointing to the direction of an adjacent
• 19 room again]
• 20 S: in this floor
• 21 T: yes, on the end here
• 22 S: OK
• 23 T: OK [turning away from S] right our next job
• 24 S: ahh can I give you
• 25 T: yes, when you just give it to me as soon as you can
• 26 S :OK 9

Best-fitting descriptor?
C1 Formal discussions and meetings
(Council of Europe, 2001:78)
Can argue a formal position convincingly,
responding to questions and comments
and answering complex lines of counter
argument fluently, spontaneously and
appropriately.
10

Contextualising data
Extract 3
1 st Year university Language, Communication &
Society class in London
20 students (approx) in class; a majority from
ESL background
Teacher has just given a series of min-lectures
on sociolinguistic concepts such as ‘dialect’,
‘standard’ …

Data - ESRC RES-062-23-1666 Modelling for Diversity: Academic Language and Literacies in School and University (Feb 2009- Jan 2011).
Extract 3
01 S1: you know the covert prestige thing(.)it’s kind of contradictory to what
02 it really means(.)difficult to know what it means(.)because the overt
03 one is privileged open language(.)but aah people in you know in high
04 places you know such as musicians(.)obviously in high places(.)you
05 know(.)are using this other type of language which is in a way privilege
06 because they can use it(.)and like a majority of people are using
07 the same language(.)so it kind of contradicts the meaning of the covert
08 prestige in a way
09 T: yeah I mean these are(.)these are just terms that(.)what happens aah
10 someone in the field let’s say sociolinguistics will use aah will
11 coin the usage of a term to order to help them understand you know a
12 certain context or a certain aspect of language use(.)and that will gain
13 currency ...
12

Best-fitting descriptors?
C1 Formal discussion and meetings (Council
of Europe, 2001:78):
Can easily keep up with the debate, even on
abstract, complex, unfamiliar topics
Can argue a formal position convincingly,
responding to questions and comments and
answering complex lines of counter argument
fluently, spontaneously and appropriately
13

IELTS Spoken Language
Assumptions:
• Discourse interaction highly predictable
• Content unpredictable
However, in actual interaction as shown in data:
• Course of interaction is unpredictable
• Content is predictable (because of teaching
programme)

IELTS rubrics
• Candidates expected to display features articulated in
rubrics
• No mention of interactional features or turn taking
(examiners follow interlocutor frame which test-takers will
be familiar with)
• No recognition of co-construction of meaning; lack of
Interactional Authenticity (Bachman, 1990)
• Substantive content under-specified (e.g. ‘to discuss a
variety of topics’ – Band 7; ‘talk about familiar and
unfamiliar topics ‘ – Band 5)
• Rewards well-organised ‘presentation’ (cf Extract 3,
substantive content meaning important, even if not well
organised)

Broadening: Participatory engagement
• Communicative competence frameworks
don’t account for certain aspects of actual
language use (which we consider important)
• Not looking to discard frameworks per se (not
returning to values-based criteria
• But: any benchmarking and assessment
framework should be empirically interrogated
and theoretically critiqued regularly and
periodically updated

Broadening: Participatory engagement
Issues:
Native-speaker norms/practices may be being
de-stabilized in many L2-L1 or lingua franca
contexts
Substantive content (not just language content)
(cf Extract 3)
Agentive participatory engagement in discourse
interaction – ‘the will and the skill to interact
with others’ (Byram, 1997:10; also Kramsch,
2010)

Participatory engagement
Lexicogrammatical
resources
Sociocultural
knowledge
Pragmatic moves

Ground-clearing questions
How might participation involvement be calibrated in
relation to each of the components in the current
competence models? Or would it be sufficient to
recognise it as an added dimension?
What methodology might be required to convert the
notion of participation into a rating scale? What
normative values and participant perspectives might
come into play when determining what counts as
communicatively oriented participation?
Would it be possible to talk in terms of a universal
participatory engagement, or would an empirical
orientation that takes account of situated practices
foreclose such a possibility?