A Brief Introduction to Evidence-Centered Design CSE Report 632 ...

More documents

Recommendations

Info

Presentation Material. Presentation Material is material that is presented to aparticipant as part of a task (including stimulus, rubric, prompt, possible optionsfor multiple choice).Presentation Process. The Presentation Process is the part of the Assessment Cyclethat displays the task to the examinee and captures the results (or WorkProducts) when the examinee performs the task.Presentation Material Specification. Presentation Material Specifications are acollection of specifications that describe material that will be presented to theexaminee as part of a stimulus, prompt, or instructional program.Reporting Rules. Reporting Rules describe how Student Model Variables should becombined or sampled to produce scores, and how those scores should beinterpreted.Response. See Work Product.Response Processing. Response Processing is the part of the Assessment Cycle thatidentifies the essential features of the examinee’s response and records these as aseries of Observations. At one time referred to as the “Evidence IdentificationProcess,” it emphasizes the key observations in the Work Product that provideevidence.Response Processing Data. See Evidence Rule Data.Strategy. Strategy refers to the overall method that will be used to select tasks in theAssembly Model.Student Model. The Student Model is a collection of variables representing theknowledge, skills, and abilities of an examinee about which inferences will bemade. A Student Model is comprised of the following types of information: (1)Student Model Variables that correspond to aspects of proficiency the assessmentis meant to measure; (2) Model Type that describes the mathematical form of theStudent Model (e.g., univariate IRT, multivariate IRT, or discrete BayesianNetwork); (3) Reporting Rules that explain how the Student Model Variablesshould be combined or sampled to produce scores.Summary Scoring Process. The Summary Scoring Process is the part of theAssessment Cycle that updates the scoring based on the input it receives fromResponse Processing. At one time referred to as the “Evidence AccumulationProcess,” the Summary Scoring Process plays an important role in accumulatingevidence.30
Task. A Task is a unit of work requested from an examinee during the course of anassessment. In ECD, a task is a specific instance of a Task ModelTask/Evidence Composite Library. The Task/Evidence Composite Library is adatabase of task objects along with all the information necessary to select andscore them. For each such Task/Evidence Composite, the library stores (1)descriptive properties that are used to ensure content coverage and preventoverlap among tasks; (2) specific values of, or references to, Presentation Materialand other environmental parameters that are used for delivering the task; (3)specific data that are used to extract the salient characteristics of Work Products;and (4) Weights of Evidence that are used to update the scoring fromperformances on this task, specifically, scoring weights, conditional probabilities,or parameters in a psychometric model.Task Model. A Task Model is a generic description of a family of tasks that contains(1) a list of variables that are used to describe key features of the tasks; (2) acollection of Presentation Material Specifications that describe material that willbe presented to the examinee as part of a stimulus, prompt, or instructionalprogram; and (3) a collection of Work Product Specifications that describe thematerial that the task will be return to the scoring process.Task Model Variables. Task Model Variables describe features of the task that areimportant for designing, calibrating, selecting, executing, and scoring it. Thesevariables describe features of the task that are important descriptors of the taskitself, such as substance, interactivity, size, and complexity, or are descriptors ofthe task performance environment, such as tools, help, and scaffolding.Weights of Evidence. Weights of Evidence are parameters that provide informationabout the size and direction of the contribution an Observable Variable makes inupdating beliefs about the state of its Student Model parent(s). The Weights ofEvidence provide a way of predicting the performance of an examinee with agiven state of the Student Model Variables on a given task. Examples are scoringweights in number-right scoring and item parameters in item response theorymodels.Work Product. A Work Product is the Examinee’s response to a task from a giventask model. This could be expressed as a transcript of examinee actions, an artifactcreated by the examinee, and/or other appropriate information. The WorkProduct provides an important bridge between the Task Model and the EvidenceModel. In particular, Work Products are the input to the Evidence Rules.31
Page 1 and 2: A Brief Introduction to Evidence-Ce
Page 3 and 4: A Brief Introduction to Evidence-Ce
Page 5 and 6: models, this primer places such mod
Page 7 and 8: As powerful as it is in organizing
Page 9 and 10: ActivitySelection ProcessAdministra
Page 11 and 12: evidence in the form of behavior in
Page 13 and 14: θXjFigure 5: The measurement model
Page 15 and 16: constraints describe how tasks must
Page 17 and 18: concerns administering pre-assemble
Page 19 and 20: are all being carried out as testin
Page 21 and 22: ReferencesAlmond, R.G. Steinberg, L
Page 23 and 24: Appendix AFurther Readings about th
Page 25 and 26: Mislevy, R.J., Steinberg, L.S., & A
Page 27 and 28: Mislevy, R.J. (1994). Evidence and
Page 29 and 30: Appendix BA Glossary of Evidence-Ce
Page 31: Four Processes. Any assessment must

A Brief Introduction to Evidence-Centered Design CSE Report 632 ...

Create successful ePaper yourself

Delete template?

Save as template?