ong>Theong> ong>Effectong> ong>ofong> ong>Dynamicong> ong>Assessmentong> on
Students' Performance in Oral EFL Tests
Work in Progress
ALTE 2011, Krakov
Tel Aviv University
ong>Theong> study is conducted under the guidance ong>ofong>
Prong>ofong>. Elana Shohamy, , Tel Aviv University
• Initial findings
ong>Dynamicong> ong>Assessmentong> (DA)
• ong>Theong> inclusion ong>ofong> the learning episode in the
assessment procedure (Sternberg & Grigorenko,
• Derived from Vygotsky’s (1934/1986)
sociocultural theory (SCT) that instruction leads
to development when pedagogical activities are
redefined and tied to assessment.
• DA incorporates the results ong>ofong> an intervention
into the assessment procedure in which learning
can be observed in 'real time'.
• ong>Theong> students take advantage ong>ofong> teacher-tester
'mediation' ong>ofong>fering more efficient learning
strategies - optimizing match between students
(Haywood & Lidz, , 2007).
• DA is hypothesized to provide information
about learning potential rather than about
student performance (Gillam, et al., 1999;
, et al., 1999; Tzuriel,
• DA - derived from Vygotsky’s (1934/1986)
ZPD and Feuerstein's structural cognitive
modifiability (Feuerstein et. al., 1979).
• Both concepts consider human abilities as
flexible and 'modifiable' as opposed to the
'fixed' attitude ong>ofong> the psychometric
Much ong>ofong> the research on DA has
demonstrated its usefulness mostly for
1. students with special needs, in general
cognition (e.g., Feuerstein et al., 1979; 1980; 1988; 2003),
2. L1 (e.g., Guterman, , 2002; Peña et al., 2006; Spector, , 1992)
3. L2 learning (e.g., Kosulin & Garb, 2002;
& Garb, 2002; Poehner, , 2005).
DA vs. other forms ong>ofong> assessment
• Teaching and ong>Assessmentong> viewed as
• Overlap between them is ong>ofong>ten
regarded negatively (washback, teaching
to the test, narrowing the curriculum).
• Closer interface b/w assessment and
teaching only for specific aims (Bachman
& Cohen 1996) e.g. diagnostic assessment
(Alderson 2006); interactive assessment
(forthcoming in HK -Davison, Hamp-Lyons)
Poehner, , 2009
Testing & Classroom ong>Assessmentong>:
• Discrete activity vs. ongoing opportunities to gather
various kinds ong>ofong> info.
• Other kinds ong>ofong> information vs. “scientific” tests
• Aim ong>ofong> assessment – stable trait vs. emerging through
• ong>Assessmentong> outcomes – reduction ong>ofong> performance
complexity (percentile ranking, score) vs. sharing ong>ofong>
work (products & processes).
Moss, 2003; Poehner, , 2009
From ong>Assessmentong> as Measurement to
ong>Assessmentong> as Inquiry
• Move from educational practice ong>ofong> assessment
where we a priori define what we are looking
for to better understand & transform
• Move from state ong>ofong> having knowledge to
desire for action involving participation,
transaction, & transformation.
ong>Assessmentong> as Inquiry (cont.)
• Current assessment practices:
What do students know?
• Scores represent amount or level ong>ofong> knowledge
• Early theories ong>ofong> learning (e.g. behaviorism)
• Different theoretical perspective:
not only What students know?
but also How do they accomplish this task?
Poehner, , 2009
DA & Formative ong>Assessmentong> (FA)
DA aligns conceptually with FA relating to the use ong>ofong>
test data to influence the teaching/learning process
to adapt to a variety ong>ofong> student learning needs (e.g.,
ong>Assessmentong> Reform Group, 2002; Black & William, 1998; Leung,
2004, 2005; Rea-Dickins
Dickins, , 2001, 2006; William et al., 2004).
• collecting information during the course ong>ofong> studies
• engaging students in self-reflective processes (Black,
et al., 2003)
• aiding the planning and managing ong>ofong> lessons
• generating evidence to evaluate teaching
• providing evidence ong>ofong> student learning
DA & ong>Assessmentong> for Learning
• A vehicle for improvement ong>ofong> teaching and
learning processes (Leung, 2007) for directing and
driving student learning by alignment between
instruction and assessment.
, 1997- Individual assessment
preferences overshadow disciplinary group
differences and impact student learning strategies.
• Showing the need for a dialogue between
instructors and students to structure expectations
regarding learning needs.
• Eg. Birenbuam, 1997
Integrating ong>Assessmentong> & Teaching?
• Can a single activity be used for both
evaluative and instructional purposes?
• Is the focus on the “teaching event” or on the
• How can we design tasks that perform dual
• How does one approach interactions with
• Affixing DA to OLP fits Vygotsky's theories that there is
a gap between actual and potential student performance.
• ong>Theong> analysis ong>ofong> this gap, known as the Zone ong>ofong> Proximal
Development (ZPD), allows interpreting what the
student can do in interaction with a competent adult.
• Types ong>ofong> mediation in DA differ in method, but
incorporate a training period between pre-and post-tests
and apply self-regulation models.
Lantolf & Throne, 2006; Guterman, , 2002
DA and oral tests
• Controlled interaction or a 'collaborative
dialogue' (Swain, 2000) through prompting
test-takers takers and/or applying metacognitive
tools, may amplify oral language learning
and result in improved oral performance
within the context ong>ofong> a foreign language
• Vygotskian perspective suggests occurrence ong>ofong>
learning when dialogic communication between
two or more individuals takes place (Ahmed,
• Dialogic nature ong>ofong> OLP tests make them suitable
for tester prompts relayed as mediation -
providing specific means to enhance learning.
Rationale: Current study
• Linking DA with academic achievements:
highly active area ong>ofong> development and
• If DA is to be useful in educational settings,
it must contribute to academic
1. To what extent will DA mediation affect results ong>ofong>
an OLP test setting?
2. If OLP scores improve as a result ong>ofong> mediation,
which can it be attributed to: external mediation
provided by a group or self-monitored mediation?
(Both supported by a teacher tester)
3. If changes in scores occur as a result ong>ofong> DA
mediation, what areas ong>ofong> OLP are affected most?
Pilot: video-recorded observations
• Aim: to identify the content and format ong>ofong>
OLP that is most suitable for effective
mediation and design the research plan.
• to examine OLP characteristics, while
validating Israeli Ministry ong>ofong> Education oral
rubrics through expert analysis.
• 73: 11-grade students; from comprehensive HS sharing
assessment practices and SES features.
• Pre- and post- OLP tests. (recordings)
• Treatment group 1: group-mediation guided by a teacher-
• Treatment group 2: 'self-mediation' relating to the same
elements and criteria as in the group- mediation.
• Control group 3: No mediation.
• To avoid pre-teaching towards post-test, test, teachers and
students were randomly mixed when re-tested.
Identical mediation groups
• Quantitative analysis ong>ofong> scores comparing pre
and post-test test results.
• Audio recordings ong>ofong> mediation sessions were
analyzed qualitatively to gain in-depth depth insights
ong>ofong> the learning process between tests and as a
result ong>ofong> mediation.
• Systemic-Cognitive Instructional Approach
to help learners internalize scientific
concepts. (1) the units ong>ofong> instruction, (2)
materialized, and (3) verbalized both
externally and internally by the learner to
a) a diagnostic pre-test
b) content coverage ong>ofong> the OLP test applying
c) taking advantage ong>ofong> instructional quality
and time using the SCOBA to create student
Mediation: SCOBA (cont.)
1. Introducing the (scientific) concept ong>ofong> the OLP and
personal orientation ong>ofong> goal-setting to be conferred by
examining the videotape ong>ofong> students’ performance.
2. Applying the tool within a treatment group through
3. Casual communication and interaction about their
performance as examined in their own videotape.
4. Verbalizing and summarizing the points learned.
Figure 1: Comparison ong>ofong>
CA score (mean) between
pre- and post- tests
• Significant differences were found
between mediation groups for both
measures ong>ofong> Communicative Ability
• Improvement in the
two mediation groups
for both measures ong>ofong>
OLP scores in
Figure 2: Comparison ong>ofong>
Accuracy score (mean)
between pre- and post- tests.
A follow-up test
• Two weeks later-
Follow -up test
program retain its
Follow -up test
RQ 3: 3 Structured observations ong>ofong> oral
interaction in mediation: sample categories
1.1. Seeking clarification
1.2. Giving clarification
1.3. Expressing miscomprehension
1.4 Trying to understand
1.5 Seeking confirmation
2.1. Expressing agreement
2.3. Incorporating ideas & elaborating
2.5. Responding to prompt/Eliciting a
2.6. Eliciting an elaboration
RQ 3: 3 Cluster categories ong>ofong> oral
performance qualitative data (Cont.)
Importance ong>ofong> category in testee statements in %
Relating to others’ ideas
Taking charge ong>ofong>
Modifiability scale and learner strategies’
checklist (Pena, 2000; Brooks et. Al., 2009),
adapted to examine student interactions.
Student interaction: Findings
Significant differences in all measures besides
Attention between individual and group-mediation.
Results (RQ 3):
• Group: : more supportive ong>ofong> learning
than individual self-reflective learning.
• Individual students: while reflecting are
more critical towards themselves and
what is lacking. Very few expressed
even limited satisfaction.
Implications and plans for further data
• Interviewer active as partner in dialogue
prompting, elaborating, initiating.
• Number ong>ofong> exchanges in intervention groups.
• Students spoke about speaking mostly using L1.
L1 is used to think about and promote L2.
• Interviewer and participant's roles in dialogue
get blurred as they all contribute to the growth
ong>ofong> meaning thus creating a GZPD.
Thank you for your attention