The Effect of Dynamic Assessment on Students - ALTE

<strong>The</strong> <strong>Effect</strong> <strong>of</strong> <strong>Dynamic</strong> <strong>Assessment</strong> on
Students' Performance in Oral EFL Tests
Work in Progress
ALTE 2011, Krakov
Tziona Levi
Tel Aviv University
<strong>The</strong> study is conducted under the guidance <strong>of</strong>
Pr<strong>of</strong>. Elana Shohamy, , Tel Aviv University

Plan
• Literature
• Methodology
• Initial findings

<strong>Dynamic</strong> <strong>Assessment</strong> (DA)
• <strong>The</strong> inclusion <strong>of</strong> the learning episode in the
assessment procedure (Sternberg & Grigorenko,
2002)
• Derived from Vygotsky’s (1934/1986)
sociocultural theory (SCT) that instruction leads
to development when pedagogical activities are
redefined and tied to assessment.

• DA incorporates the results <strong>of</strong> an intervention
into the assessment procedure in which learning
can be observed in 'real time'.
• <strong>The</strong> students take advantage <strong>of</strong> teacher-tester
tester
'mediation' <strong>of</strong>fering more efficient learning
strategies - optimizing match between students
and tasks.
(Haywood & Lidz, , 2007).

• DA is hypothesized to provide information
about learning potential rather than about
student performance (Gillam, et al., 1999;
2000).
, et al., 1999; Tzuriel,
• DA - derived from Vygotsky’s (1934/1986)
ZPD and Feuerstein's structural cognitive
modifiability (Feuerstein et. al., 1979).
• Both concepts consider human abilities as
flexible and 'modifiable' as opposed to the
'fixed' attitude <strong>of</strong> the psychometric
approach.

Much <strong>of</strong> the research on DA has
demonstrated its usefulness mostly for
1. students with special needs, in general
cognition
cognition (e.g., Feuerstein et al., 1979; 1980; 1988; 2003),
2. L1 (e.g., Guterman, , 2002; Peña et al., 2006; Spector, , 1992)
3. L2 learning (e.g., Kosulin & Garb, 2002;
& Garb, 2002; Poehner, , 2005).

DA vs. other forms <strong>of</strong> assessment
• Teaching and <strong>Assessment</strong> viewed as
separate issues.
• Overlap between them is <strong>of</strong>ten
regarded negatively (washback, teaching
to the test, narrowing the curriculum).
• Closer interface b/w assessment and
teaching only for specific aims (Bachman
& Cohen 1996) e.g. diagnostic assessment
(Alderson 2006); interactive assessment
(forthcoming in HK -Davison, Hamp-Lyons)
<strong>Assessment</strong>
Teaching
Poehner, , 2009

Testing & Classroom <strong>Assessment</strong>:
Fundamental Differences
• Discrete activity vs. ongoing opportunities to gather
various kinds <strong>of</strong> info.
• Other kinds <strong>of</strong> information vs. “scientific” tests
• Aim <strong>of</strong> assessment – stable trait vs. emerging through
collaborative performance.
• <strong>Assessment</strong> outcomes – reduction <strong>of</strong> performance
complexity (percentile ranking, score) vs. sharing <strong>of</strong>
work (products & processes).
Moss, 2003; Poehner, , 2009

From <strong>Assessment</strong> as Measurement to
<strong>Assessment</strong> as Inquiry
• Move from educational practice <strong>of</strong> assessment
where we a priori define what we are looking
for to better understand & transform
information.
• Move from state <strong>of</strong> having knowledge to
desire for action involving participation,
transaction, & transformation.
Delandshere, 2002:1475-1480

<strong>Assessment</strong> as Inquiry (cont.)
• Current assessment practices:
What do students know?
• Scores represent amount or level <strong>of</strong> knowledge
• Early theories <strong>of</strong> learning (e.g. behaviorism)
• Different theoretical perspective:
not only What students know?
but also How do they accomplish this task?
Poehner, , 2009

DA & Formative <strong>Assessment</strong> (FA)
DA aligns conceptually with FA relating to the use <strong>of</strong>
test data to influence the teaching/learning process
to adapt to a variety <strong>of</strong> student learning needs (e.g.,
<strong>Assessment</strong> Reform Group, 2002; Black & William, 1998; Leung,
2004, 2005; Rea-Dickins
Dickins, , 2001, 2006; William et al., 2004).
Such as:
• collecting information during the course <strong>of</strong> studies
• engaging students in self-reflective processes (Black,
et al., 2003)
• aiding the planning and managing <strong>of</strong> lessons
• generating evidence to evaluate teaching
• providing evidence <strong>of</strong> student learning

DA & <strong>Assessment</strong> for Learning
• A vehicle for improvement <strong>of</strong> teaching and
learning processes (Leung, 2007) for directing and
driving student learning by alignment between
instruction and assessment.
, 1997- Individual assessment
preferences overshadow disciplinary group
differences and impact student learning strategies.
• Showing the need for a dialogue between
instructors and students to structure expectations
regarding learning needs.
• Eg. Birenbuam, 1997

Integrating <strong>Assessment</strong> & Teaching?
• Can a single activity be used for both
evaluative and instructional purposes?
• Is the focus on the “teaching event” or on the
“assessment event”?
• How can we design tasks that perform dual
function?
• How does one approach interactions with
learners?
<strong>Assessment</strong>-Teaching

Current study
• Affixing DA to OLP fits Vygotsky's theories that there is
a gap between actual and potential student performance.
• <strong>The</strong> analysis <strong>of</strong> this gap, known as the Zone <strong>of</strong> Proximal
Development (ZPD), allows interpreting what the
student can do in interaction with a competent adult.
• Types <strong>of</strong> mediation in DA differ in method, but
incorporate a training period between pre-and post-tests
tests
and apply self-regulation models.
Lantolf & Throne, 2006; Guterman, , 2002

DA and oral tests
• Controlled interaction or a 'collaborative
dialogue' (Swain, 2000) through prompting
test-takers takers and/or applying metacognitive
tools, may amplify oral language learning
and result in improved oral performance
within the context <strong>of</strong> a foreign language
oral test.

Studying OLP
• Vygotskian perspective suggests occurrence <strong>of</strong>
learning when dialogic communication between
two or more individuals takes place (Ahmed,
1994)
• Dialogic nature <strong>of</strong> OLP tests make them suitable
for tester prompts relayed as mediation -
providing specific means to enhance learning.

Rationale: Current study
• Linking DA with academic achievements:
highly active area <strong>of</strong> development and
research.
• If DA is to be useful in educational settings,
it must contribute to academic
achievements.
DA
Oral achievements

Research questions
1. To what extent will DA mediation affect results <strong>of</strong>
an OLP test setting?
2. If OLP scores improve as a result <strong>of</strong> mediation,
which can it be attributed to: external mediation
provided by a group or self-monitored mediation?
(Both supported by a teacher tester)
3. If changes in scores occur as a result <strong>of</strong> DA
mediation, what areas <strong>of</strong> OLP are affected most?

Pilot: video-recorded observations
• Aim: to identify the content and format <strong>of</strong>
OLP that is most suitable for effective
mediation and design the research plan.
• to examine OLP characteristics, while
validating Israeli Ministry <strong>of</strong> Education oral
rubrics through expert analysis.

Research design
• 73: 11-grade students; from comprehensive HS sharing
assessment practices and SES features.
• Pre- and post- OLP tests. (recordings)
• Treatment group 1: group-mediation guided by a teacher-
tester.
• Treatment group 2: 'self-mediation' relating to the same
elements and criteria as in the group- mediation.
• Control group 3: No mediation.
• To avoid pre-teaching towards post-test, test, teachers and
students were randomly mixed when re-tested.
Identical mediation groups

Design (cont.)
• Quantitative analysis <strong>of</strong> scores comparing pre
and post-test test results.
• Audio recordings <strong>of</strong> mediation sessions were
analyzed qualitatively to gain in-depth depth insights
<strong>of</strong> the learning process between tests and as a
result <strong>of</strong> mediation.

Mediation:
SCOBA
and
Rubrics
• Systemic-Cognitive Instructional Approach
to help learners internalize scientific
concepts. (1) the units <strong>of</strong> instruction, (2)
materialized, and (3) verbalized both
externally and internally by the learner to
promote internalization.
a) a diagnostic pre-test
b) content coverage <strong>of</strong> the OLP test applying
the rubrics
c) taking advantage <strong>of</strong> instructional quality
and time using the SCOBA to create student
engagement.
.

Mediation: SCOBA (cont.)
1. Introducing the (scientific) concept <strong>of</strong> the OLP and
personal orientation <strong>of</strong> goal-setting to be conferred by
examining the videotape <strong>of</strong> students’ performance.
2. Applying the tool within a treatment group through
self-monitoring.
3. Casual communication and interaction about their
performance as examined in their own videotape.
4. Verbalizing and summarizing the points learned.

Results:
(RQ 1,2)
CA
18
16
Figure 1: Comparison <strong>of</strong>
CA score (mean) between
pre- and post- tests
Group-Mediation
Self-Mediation
Control Group
14
12
10
8
Pretest
Posttest
6
• Significant differences were found
between mediation groups for both
measures <strong>of</strong> Communicative Ability
and Accuracy.

Results:
(RQ 1,2)
• Improvement in the
two mediation groups
for both measures <strong>of</strong>
OLP scores in
comparison to
the control
group.
Group-Mediation
Self-Mediation
Control Group
Figure 2: Comparison <strong>of</strong>
Accuracy score (mean)
between pre- and post- tests.
Accuracy
16
14
12
10
Pretest
Posttest
8
6

CA
A follow-up test
20
18
Self-Mediation
• Two weeks later-
Group-Mediation
Control Group
16
14
Pretest
Posttest
Follow -up test
12
Did mediation
program retain its
effect?
Self-Mediation
Accuracy
20
18
Group-Mediation
16
Control Group
14
Prestest
Posttest
Follow -up test
12

RQ 3: 3 Structured observations <strong>of</strong> oral
interaction in mediation: sample categories
Category clusters
Classification
Relating to
others’ ideas
Categories
1.1. Seeking clarification
1.2. Giving clarification
1.3. Expressing miscomprehension
1.4 Trying to understand
1.5 Seeking confirmation
2.1. Expressing agreement
2.2. Disagreeing
2.3. Incorporating ideas & elaborating
2.4. Approving
2.5. Responding to prompt/Eliciting a
response
2.6. Eliciting an elaboration
2.7 Restricting

RQ 3: 3 Cluster categories <strong>of</strong> oral
performance qualitative data (Cont.)
Importance <strong>of</strong> category in testee statements in %
Clarification
14%
12%
Relating to others’ ideas
Expressing emotions
Process-related
commentary
21%
22%
Typically teacher-related
categories
Taking charge <strong>of</strong>
interaction
26%
6%

Student interactions
Modifiability scale and learner strategies’
checklist (Pena, 2000; Brooks et. Al., 2009),
adapted to examine student interactions.

Student interaction: Findings
Significant differences in all measures besides
Attention between individual and group-mediation.
Group-
Mediation
Self-Mediation
M
SD
M
SD
F(1,42)
Eta 2
Attention
1.58
.32
1.62
.48
.11
.00
Planning
1.53
.25
.73
.56
38.52***
.48
Regulation
1.14
.40
.77
.52
13.35*
.15
Transfer
1.42
.39
.26
.39
93.64***
.69
Motivation
1.30
.48
.60
.39
26.77***
.39
Dialoguing
1.54
.12
.66
.44
87.14***
.68
Languaging
.70
.47
.28
.13
10.44**
.26

Results (RQ 3):
• Group: : more supportive <strong>of</strong> learning
than individual self-reflective learning.
• Individual students: while reflecting are
more critical towards themselves and
what is lacking. Very few expressed
even limited satisfaction.

Implications and plans for further data
analysis
• Interviewer active as partner in dialogue
prompting, elaborating, initiating.
• Number <strong>of</strong> exchanges in intervention groups.
• Students spoke about speaking mostly using L1.
L1 is used to think about and promote L2.
• Interviewer and participant's roles in dialogue
get blurred as they all contribute to the growth
<strong>of</strong> meaning thus creating a GZPD.

Thank you for your attention
z_levi@netvision.net.il