A Brief Introduction to Evidence-Centered Design CSE Report 632 ...

A Brief Introduction to Evidence-Centered DesignRobert J. Mislevy, CRESST/University of MarylandRussell G. Almond & Janice F. Lukas, Educational Testing ServiceAbstractEvidence-centered assessment design (ECD) is an approach to constructing educationalassessments in terms of evidentiary arguments. This paper provides an introduction to thebasic ideas of ECD, including some of the terminology and models that have beendeveloped to implement the approach. In particular, it presents the high-level models ofthe Conceptual Assessment Framework and the Four-Process Architecture for assessmentdelivery systems. Special attention is given to the roles of probability-based reasoning inaccumulating evidence across task performances, in terms of belief about unobservablevariables that characterize the knowledge, skills, and/or abilities of students. This is therole traditionally associated with psychometric models, such as those of item responsetheory and latent class models. To unify the ideas and to provide a foundation for extendingprobability-based reasoning in assessment applications more broadly, however, a moregeneral expression in terms of graphical models is indicated. This brief overview of ECDprovides the reader with a feel for where and how graphical models fit into the largerenterprise of educational and psychological assessment. A simple example based onfamiliar large-scale standardized tests such as the GRE is used to fix ideas.OverviewWhat all educational assessments have in common is the desire to reasonfrom particular things students say, do, or make, to inferences about what theyknow or can do more broadly. Over the past century a number of assessmentmethods have evolved for addressing this problem in a principled and systematicmanner. The measurement models of classical test theory and, more recently, itemresponse theory (IRT) and latent class analysis, have proved quite satisfactory for thelarge scale tests and classroom quizzes with which every reader is by now quitefamiliar.1