Technical Manual - Renaissance Learning

Psychometric Research Supporting STAR EarlyLiteracy EnterpriseSince 2000, there have been three major phases of research leading to thedevelopment and publication of STAR Early Literacy in 2001, and subsequently tothe publication of STAR Early Literacy Enterprise in 2012. These are referred to asthe 2000 Calibration Study, the 2001 Validation Study, and the 2012 STAR EarlyLiteracy Enterprise Research Study.The 2000 Calibration Study is described in detail in the section on Item Calibrationbelow. The 2001 Validation Study is described in the Validity section, beginning onpage 55. The 2012 STAR Early Literacy Enterprise Research Study is describedfollowing the Validation Study, beginning on page 86.Item CalibrationBackgroundIn the course of developing the item bank for the initial version of STAR EarlyLiteracy, item writers wrote almost 3,000 test items to measure early literacy skills.This section describes the process by which those items were calibrated on acommon scale of difficulty. A later section will describe the process that has beenused subsequently to calibrate new items; we call that process “dynamiccalibration.” Subject matter experts and editors reviewed the content of everyitem and recommended retaining some and rejecting others. After this itemcontent review, 2,929 items, measuring seven broad literacy areas and 41literacy-related skills, remained as candidates for inclusion in the item bank.In order to use the test items for computer-adaptive testing, every item had to beplaced on a continuous scale of difficulty—the same scale used to select itemsadaptively and to score the adaptive tests. The procedures of IRT were chosen asthe basis for scaling STAR Early Literacy item difficulty, a process called“calibration.”IRT calibration is based on statistical analysis of response data—it requireshundreds of responses to every test item. To obtain these data, RenaissanceLearning conducted a major item Calibration Study in late 2000. For theCalibration Study, 246 test forms were designed, and the 2,929 STAR Early Literacyitems were distributed among these forms. Every form contained ten mousetraining items, ten practice items, and forty test items (the keyboard was not usedto enter answers for this study). The forms were graded as to developmental level:STAR Early LiteracyTechnical Manual37