A Brief Introduction to Evidence-Centered Design CSE Report 632 ...

More documents

Recommendations

Info

Project 3.6 Study Group Activity on Cognitive ValidityRobert J. Mislevy, Project Director, CRESST/University of Maryland, College ParkCopyright © 2004 The Regents of the University of CaliforniaThe work reported herein was supported under the Educational Research and Development CentersProgram, PR/Award Number R305B960002, as administered by the Office of Educational Research andImprovement, U.S. Department of Education.The findings and opinions expressed in this report do not reflect the positions or policies of theNational Institute on Student Achievement, Curriculum, and Assessment, the Office of EducationalResearch and Improvement, or the U.S. Department of Education.
A Brief Introduction to Evidence-Centered DesignRobert J. Mislevy, CRESST/University of MarylandRussell G. Almond & Janice F. Lukas, Educational Testing ServiceAbstractEvidence-centered assessment design (ECD) is an approach to constructing educationalassessments in terms of evidentiary arguments. This paper provides an introduction to thebasic ideas of ECD, including some of the terminology and models that have beendeveloped to implement the approach. In particular, it presents the high-level models ofthe Conceptual Assessment Framework and the Four-Process Architecture for assessmentdelivery systems. Special attention is given to the roles of probability-based reasoning inaccumulating evidence across task performances, in terms of belief about unobservablevariables that characterize the knowledge, skills, and/or abilities of students. This is therole traditionally associated with psychometric models, such as those of item responsetheory and latent class models. To unify the ideas and to provide a foundation for extendingprobability-based reasoning in assessment applications more broadly, however, a moregeneral expression in terms of graphical models is indicated. This brief overview of ECDprovides the reader with a feel for where and how graphical models fit into the largerenterprise of educational and psychological assessment. A simple example based onfamiliar large-scale standardized tests such as the GRE is used to fix ideas.OverviewWhat all educational assessments have in common is the desire to reasonfrom particular things students say, do, or make, to inferences about what theyknow or can do more broadly. Over the past century a number of assessmentmethods have evolved for addressing this problem in a principled and systematicmanner. The measurement models of classical test theory and, more recently, itemresponse theory (IRT) and latent class analysis, have proved quite satisfactory for thelarge scale tests and classroom quizzes with which every reader is by now quitefamiliar.1
Page 1: A Brief Introduction to Evidence-Ce
Page 5 and 6: models, this primer places such mod
Page 7 and 8: As powerful as it is in organizing
Page 9 and 10: ActivitySelection ProcessAdministra
Page 11 and 12: evidence in the form of behavior in
Page 13 and 14: θXjFigure 5: The measurement model
Page 15 and 16: constraints describe how tasks must
Page 17 and 18: concerns administering pre-assemble
Page 19 and 20: are all being carried out as testin
Page 21 and 22: ReferencesAlmond, R.G. Steinberg, L
Page 23 and 24: Appendix AFurther Readings about th
Page 25 and 26: Mislevy, R.J., Steinberg, L.S., & A
Page 27 and 28: Mislevy, R.J. (1994). Evidence and
Page 29 and 30: Appendix BA Glossary of Evidence-Ce
Page 31 and 32: Four Processes. Any assessment must
Page 33: Task. A Task is a unit of work requ

A Brief Introduction to Evidence-Centered Design CSE Report 632 ...

Create successful ePaper yourself

Delete template?

Save as template?