Technical Manual - Renaissance Learning

Psychometric Research Supporting STAR Early Literacy EnterpriseSelection of Items from the Calibration Item BankThe principal end products of the item calibration process were the IRT itemparameter estimates themselves, along with traditional indices of item difficulty(sample proportion correct) and item discriminating power (correlationcoefficients between item score and the Rasch ability score).Selection of Items from the Calibration Item BankOnce the calibration analysis was complete, a psychometric review took place.The review evaluated both the IRT-based results and the traditional item analysisresults, such as proportion correct and item-total correlations.Reviewers evaluated each item’s difficulty, discriminating power, model fitindices, statistical properties and content to identify any items that appearedunsuitable for inclusion in the adaptive testing item bank. The review work wasaided by the use of interactive psychometric review software developedspecifically for STAR Early Literacy. This software displays, one item at a time, theSTAR Early Literacy question (including audio and graphics) and its correctanswer, along with a variety of item statistics. The statistics include Rasch modelfit indices, traditional proportion correct and biserial statistics to assess difficultyand discriminating power, an analysis of each response alternative, and the Raschitem difficulty parameter.There are check boxes for the reviewer to record disqualifying properties and torecommend acceptance, and an area for the reviewer to use to record notes aboutthe item. All reviewers’ recommendations and notes were compiled into apermanent database of the psychometric history of all test items developed foruse in STAR Early Literacy.Following completion of the psychometric reviews by individual reviewers, asecond review of the database was conducted. In that review, differences inreviewer recommendations were reconciled, and final decisions were made aboutretention or rejection of each item. Of the 2,929 items in the calibration item bank,2,485 were accepted by the psychometric review team for use in the adaptiveversion of STAR Early Literacy. Of these, 18 were reserved for use as practice items;another 30 items designed specifically as mouse training items were reserved forthat purpose. Prior to release of the publication version of STAR Early Literacy, anumber of other items were deleted in response to independent reviewers’suggestions. The final version of the STAR Early Literacy item bank thereforecontains 2,350 items; 2 of these, 2,332 are available for use in adaptive testing andthe other 18 are used as practice items.2. The item count stood at 2,369 until it was reduced to 2,350 with the release of STAR Early LiteracyRP version 3.3.STAR Early LiteracyTechnical Manual42