Technical Manual - Renaissance Learning

Psychometric Research Supporting STAR Early Literacy EnterpriseStatistical Analysis: Fitting the Rasch IRT Model to the Calibration DataFor reliable IRT scale linking, it is important for anchor items to be representativeof the content of the tests they are used to anchor. To that end, the distribution ofanchor items was approximately proportional to the distribution of items amongthe domains and skills summarized in “Content and Item Development” onpage 15.To accomplish the second objective of the Calibration Study, many of theparticipating students were asked to take two STAR Early Literacy tests so that thecorrelation of their scores on two occasions could be used to evaluate the retestreliability of STAR Early Literacy tests over a short time interval. Topics related toreliability are described in “Reliability and Measurement Precision” on page 44.To accomplish the third objective, a subsample of the grade 1, 2 and 3 studentsalso took a computer-adaptive STAR Reading 2.x assessment to provide a basis forevaluating the degree of correlation between STAR Early Literacy and readingability. Statistical results are presented in “Validity” on page 55.Statistical Analysis: Fitting the Rasch IRT Model to the Calibration DataWith the response data from the Calibration Study in hand, the first order ofbusiness was to calibrate the items and score the students’ tests. This was doneusing the “Rasch model,” an IRT model that expresses the probability of a correctanswer as a function of the difference between the locations of the item and thestudent on a common scale. Rasch model analysis was used to determine thevalue of a “difficulty parameter” for every item, and to assign a score to everystudent. In the analysis, a number of statistical measures of item quality andmodel fit were calculated for each item.Item parameter estimation and IRT scoring were accomplished using WINSTEPS, acommercially available Rasch model analysis software package. WINSTEPS iscapable of Rasch analysis of multiple test forms simultaneously. Using thiscapability, three item parameter estimation analyses were conducted. All Level Btest forms were analyzed first, and the resulting scale was used as the referencescale for the other forms. Following that, separate analyses were conducted of theLevel A and Level C forms. In each of the last two analyses, the parameters ofanchor items common to Level B were held fixed at the values obtained from theLevel B analysis. This had the effect of placing all Level A and Level C itemparameters on the Level B scale. 11. All 246 test forms contained a number of anchor items. At each of the three levels, a small set ofitems specific to that level was common to all of the forms; these “horizontal anchors” served to linkall forms at a given level to a common scale. Additionally, every form contained some items incommon with forms from adjacent levels; these “vertical anchors” served to link the scales of LevelsA and C to the reference scale based on Level B.STAR Early LiteracyTechnical Manual41