The Effect of Dynamic Assessment on Students - ALTE

alte.org

The Effect of Dynamic Assessment on Students - ALTE

ong>Theong> ong>Effectong> ong>ofong> ong>Dynamicong> ong>Assessmentong> on

Students' Performance in Oral EFL Tests

Work in Progress

ALTE 2011, Krakov

Tziona Levi

Tel Aviv University

ong>Theong> study is conducted under the guidance ong>ofong>

Prong>ofong>. Elana Shohamy, , Tel Aviv University


Plan

• Literature

• Methodology

• Initial findings


ong>Dynamicong> ong>Assessmentong> (DA)

ong>Theong> inclusion ong>ofong> the learning episode in the

assessment procedure (Sternberg & Grigorenko,

2002)

• Derived from Vygotsky’s (1934/1986)

sociocultural theory (SCT) that instruction leads

to development when pedagogical activities are

redefined and tied to assessment.


• DA incorporates the results ong>ofong> an intervention

into the assessment procedure in which learning

can be observed in 'real time'.

ong>Theong> students take advantage ong>ofong> teacher-tester

tester

'mediation' ong>ofong>fering more efficient learning

strategies - optimizing match between students

and tasks.

(Haywood & Lidz, , 2007).


• DA is hypothesized to provide information

about learning potential rather than about

student performance (Gillam, et al., 1999;

2000).

, et al., 1999; Tzuriel,

• DA - derived from Vygotsky’s (1934/1986)

ZPD and Feuerstein's structural cognitive

modifiability (Feuerstein et. al., 1979).

• Both concepts consider human abilities as

flexible and 'modifiable' as opposed to the

'fixed' attitude ong>ofong> the psychometric

approach.


Much ong>ofong> the research on DA has

demonstrated its usefulness mostly for

1. students with special needs, in general

cognition

cognition (e.g., Feuerstein et al., 1979; 1980; 1988; 2003),

2. L1 (e.g., Guterman, , 2002; Peña et al., 2006; Spector, , 1992)

3. L2 learning (e.g., Kosulin & Garb, 2002;

& Garb, 2002; Poehner, , 2005).


DA vs. other forms ong>ofong> assessment

• Teaching and ong>Assessmentong> viewed as

separate issues.

• Overlap between them is ong>ofong>ten

regarded negatively (washback, teaching

to the test, narrowing the curriculum).

• Closer interface b/w assessment and

teaching only for specific aims (Bachman

& Cohen 1996) e.g. diagnostic assessment

(Alderson 2006); interactive assessment

(forthcoming in HK -Davison, Hamp-Lyons)

ong>Assessmentong>

Teaching

Poehner, , 2009


Testing & Classroom ong>Assessmentong>:

Fundamental Differences

• Discrete activity vs. ongoing opportunities to gather

various kinds ong>ofong> info.

• Other kinds ong>ofong> information vs. “scientific” tests

• Aim ong>ofong> assessment – stable trait vs. emerging through

collaborative performance.

ong>Assessmentong> outcomes – reduction ong>ofong> performance

complexity (percentile ranking, score) vs. sharing ong>ofong>

work (products & processes).

Moss, 2003; Poehner, , 2009


From ong>Assessmentong> as Measurement to

ong>Assessmentong> as Inquiry

• Move from educational practice ong>ofong> assessment

where we a priori define what we are looking

for to better understand & transform

information.

• Move from state ong>ofong> having knowledge to

desire for action involving participation,

transaction, & transformation.

Delandshere, 2002:1475-1480


ong>Assessmentong> as Inquiry (cont.)

• Current assessment practices:

What do students know?

• Scores represent amount or level ong>ofong> knowledge

• Early theories ong>ofong> learning (e.g. behaviorism)

• Different theoretical perspective:

not only What students know?

but also How do they accomplish this task?

Poehner, , 2009


DA & Formative ong>Assessmentong> (FA)

DA aligns conceptually with FA relating to the use ong>ofong>

test data to influence the teaching/learning process

to adapt to a variety ong>ofong> student learning needs (e.g.,

ong>Assessmentong> Reform Group, 2002; Black & William, 1998; Leung,

2004, 2005; Rea-Dickins

Dickins, , 2001, 2006; William et al., 2004).

Such as:

• collecting information during the course ong>ofong> studies

• engaging students in self-reflective processes (Black,

et al., 2003)

• aiding the planning and managing ong>ofong> lessons

• generating evidence to evaluate teaching

• providing evidence ong>ofong> student learning


DA & ong>Assessmentong> for Learning

• A vehicle for improvement ong>ofong> teaching and

learning processes (Leung, 2007) for directing and

driving student learning by alignment between

instruction and assessment.

, 1997- Individual assessment

preferences overshadow disciplinary group

differences and impact student learning strategies.

• Showing the need for a dialogue between

instructors and students to structure expectations

regarding learning needs.

• Eg. Birenbuam, 1997


Integrating ong>Assessmentong> & Teaching?

• Can a single activity be used for both

evaluative and instructional purposes?

• Is the focus on the “teaching event” or on the

“assessment event”?

• How can we design tasks that perform dual

function?

• How does one approach interactions with

learners?

ong>Assessmentong>-Teaching


Current study

• Affixing DA to OLP fits Vygotsky's theories that there is

a gap between actual and potential student performance.

ong>Theong> analysis ong>ofong> this gap, known as the Zone ong>ofong> Proximal

Development (ZPD), allows interpreting what the

student can do in interaction with a competent adult.

• Types ong>ofong> mediation in DA differ in method, but

incorporate a training period between pre-and post-tests

tests

and apply self-regulation models.

Lantolf & Throne, 2006; Guterman, , 2002


DA and oral tests

• Controlled interaction or a 'collaborative

dialogue' (Swain, 2000) through prompting

test-takers takers and/or applying metacognitive

tools, may amplify oral language learning

and result in improved oral performance

within the context ong>ofong> a foreign language

oral test.


Studying OLP

• Vygotskian perspective suggests occurrence ong>ofong>

learning when dialogic communication between

two or more individuals takes place (Ahmed,

1994)

• Dialogic nature ong>ofong> OLP tests make them suitable

for tester prompts relayed as mediation -

providing specific means to enhance learning.


Rationale: Current study

• Linking DA with academic achievements:

highly active area ong>ofong> development and

research.

• If DA is to be useful in educational settings,

it must contribute to academic

achievements.

DA

Oral achievements


Research questions

1. To what extent will DA mediation affect results ong>ofong>

an OLP test setting?

2. If OLP scores improve as a result ong>ofong> mediation,

which can it be attributed to: external mediation

provided by a group or self-monitored mediation?

(Both supported by a teacher tester)

3. If changes in scores occur as a result ong>ofong> DA

mediation, what areas ong>ofong> OLP are affected most?


Pilot: video-recorded observations

• Aim: to identify the content and format ong>ofong>

OLP that is most suitable for effective

mediation and design the research plan.

• to examine OLP characteristics, while

validating Israeli Ministry ong>ofong> Education oral

rubrics through expert analysis.


Research design

• 73: 11-grade students; from comprehensive HS sharing

assessment practices and SES features.

• Pre- and post- OLP tests. (recordings)

• Treatment group 1: group-mediation guided by a teacher-

tester.

• Treatment group 2: 'self-mediation' relating to the same

elements and criteria as in the group- mediation.

• Control group 3: No mediation.

• To avoid pre-teaching towards post-test, test, teachers and

students were randomly mixed when re-tested.

Identical mediation groups


Design (cont.)

• Quantitative analysis ong>ofong> scores comparing pre

and post-test test results.

• Audio recordings ong>ofong> mediation sessions were

analyzed qualitatively to gain in-depth depth insights

ong>ofong> the learning process between tests and as a

result ong>ofong> mediation.


Mediation:

SCOBA

and

Rubrics

• Systemic-Cognitive Instructional Approach

to help learners internalize scientific

concepts. (1) the units ong>ofong> instruction, (2)

materialized, and (3) verbalized both

externally and internally by the learner to

promote internalization.

a) a diagnostic pre-test

b) content coverage ong>ofong> the OLP test applying

the rubrics

c) taking advantage ong>ofong> instructional quality

and time using the SCOBA to create student

engagement.

.


Mediation: SCOBA (cont.)

1. Introducing the (scientific) concept ong>ofong> the OLP and

personal orientation ong>ofong> goal-setting to be conferred by

examining the videotape ong>ofong> students’ performance.

2. Applying the tool within a treatment group through

self-monitoring.

3. Casual communication and interaction about their

performance as examined in their own videotape.

4. Verbalizing and summarizing the points learned.


Results:

(RQ 1,2)

CA

18

16

Figure 1: Comparison ong>ofong>

CA score (mean) between

pre- and post- tests

Group-Mediation

Self-Mediation

Control Group

14

12

10

8

Pretest

Posttest

6

• Significant differences were found

between mediation groups for both

measures ong>ofong> Communicative Ability

and Accuracy.


Results:

(RQ 1,2)

• Improvement in the

two mediation groups

for both measures ong>ofong>

OLP scores in

comparison to

the control

group.

Group-Mediation

Self-Mediation

Control Group

Figure 2: Comparison ong>ofong>

Accuracy score (mean)

between pre- and post- tests.

Accuracy

16

14

12

10

Pretest

Posttest

8

6


CA

A follow-up test

20

18

Self-Mediation

• Two weeks later-

Group-Mediation

Control Group

16

14

Pretest

Posttest

Follow -up test

12

Did mediation

program retain its

effect?

Self-Mediation

Accuracy

20

18

Group-Mediation

16

Control Group

14

Prestest

Posttest

Follow -up test

12


RQ 3: 3 Structured observations ong>ofong> oral

interaction in mediation: sample categories

Category clusters

Classification

Relating to

others’ ideas

Categories

1.1. Seeking clarification

1.2. Giving clarification

1.3. Expressing miscomprehension

1.4 Trying to understand

1.5 Seeking confirmation

2.1. Expressing agreement

2.2. Disagreeing

2.3. Incorporating ideas & elaborating

2.4. Approving

2.5. Responding to prompt/Eliciting a

response

2.6. Eliciting an elaboration

2.7 Restricting


RQ 3: 3 Cluster categories ong>ofong> oral

performance qualitative data (Cont.)

Importance ong>ofong> category in testee statements in %

Clarification

14%

12%

Relating to others’ ideas

Expressing emotions

Process-related

commentary

21%

22%

Typically teacher-related

categories

Taking charge ong>ofong>

interaction

26%

6%


Student interactions

Modifiability scale and learner strategies’

checklist (Pena, 2000; Brooks et. Al., 2009),

adapted to examine student interactions.


Student interaction: Findings

Significant differences in all measures besides

Attention between individual and group-mediation.

Group-

Mediation

Self-Mediation

M

SD

M

SD

F(1,42)

Eta 2

Attention

1.58

.32

1.62

.48

.11

.00

Planning

1.53

.25

.73

.56

38.52***

.48

Regulation

1.14

.40

.77

.52

13.35*

.15

Transfer

1.42

.39

.26

.39

93.64***

.69

Motivation

1.30

.48

.60

.39

26.77***

.39

Dialoguing

1.54

.12

.66

.44

87.14***

.68

Languaging

.70

.47

.28

.13

10.44**

.26


Results (RQ 3):

• Group: : more supportive ong>ofong> learning

than individual self-reflective learning.

• Individual students: while reflecting are

more critical towards themselves and

what is lacking. Very few expressed

even limited satisfaction.


Implications and plans for further data

analysis

• Interviewer active as partner in dialogue

prompting, elaborating, initiating.

• Number ong>ofong> exchanges in intervention groups.

Students spoke about speaking mostly using L1.

L1 is used to think about and promote L2.

• Interviewer and participant's roles in dialogue

get blurred as they all contribute to the growth

ong>ofong> meaning thus creating a GZPD.


Thank you for your attention

z_levi@netvision.net.il

More magazines by this user
Similar magazines