Assessing Local Dependence in Educational Performance ...

Reins, the level of dependence may be more visible in a free-response modc than in thcmore commonly used forced-choice mode.MSPAP is pan of a larger school reform effort initiated by the Maryland StateDepartment of Education. Performance on the MSPAP is used to evaluate schoolperformance and to provide information to guide school improvement efforts. Thc 1991MSPAP included assessments of widely distributed learning outcomes in reading, writing,language usage and mathematics in grades 3.5, and 8. All tasks in thc MSPAP rcquircbrief and extended open-ended responses to performance tasks: Examinees write, diagram,and sketch responses to tasks that focus on thcir ability to construct and extend meaningfrom what they read, construct and cxlcnd meaning through writing, and solve multi-stepmathematics problcms. Tasks arc designed to clicit examinees' thoughtful application ofknowledge, skills, and thinking processes. Examinee responses are ~ored by trainedteachers using scoring activity-specific keys for brief correct and incorrect responses;generic ~les for longer responses which are scored as correct/adequate, partiallycorrect/adequate, or incorrect/inadequate; and rubri~ for essays and other extendedresponses. Although each 1991 MSPAP test form was administered to a randomlyassigned group of at least 3,000 examinees, samples of approximately 1,400 examineeswere used in this study.Task developers were trained to minimize response dependence in the assessmentactivities they wrote for MSPAP assessment tasks. After the development phase, taskswere submitted to both judgmental and statistical analyses. One such judgmental reviewfocused on response dependence. The goals of this review were to (a) identify clusters ofinterdependent responses, and (b) treating interdependent clusters as testlets, determine ifan entire assessment form contained adequate numbers of independent responses to allowfor IRT scaling. (Details on scaling procedures are available in C'TB Macmillan/McGraw-Hill, 1992.) To identify interdependent responses within item clusters, reviewers read eachitem that was designed to elicit an examinee response and answered the questions, "Can