Technical Manual - Renaissance Learning

Reliability and Measurement PrecisionSplit-Half Reliabilitywhere σ 2 error is the variance of the errors of measurement, and σ2 total is thevariance of the test scores. In STAR Early Literacy the variance of the test scores iseasily calculated from Scaled Score data. The variance of the errors ofmeasurement may be estimated from the conditional standard error ofmeasurement (CSEM) statistics that accompany each of the IRT-based test scores,including the Scaled Scores, as depicted below.σ 2 error=1nΣ nCSEM 2iwhere the summation is over the squared values of the reported CSEM for studentsi = 1 to n. In each STAR Early Literacy 3.x and higher test, CSEM is calculated along withthe IRT ability estimate and Scaled Score. Squaring and summing the CSEM valuesyields an estimate of total squared error; dividing by the number of observations yieldsan estimate of mean squared error, which in this case is tantamount to error variance.“Generic” reliability is then estimated by calculating the ratio of error variance toScaled Score variance, and subtracting that ratio from 1.Using this technique with the STAR Early Literacy 2.0 norming data resulted in thegeneric reliability estimates shown in the rightmost column of Table 13 onpage 49. Because this method is not susceptible to error variance introduced byrepeated testing, multiple occasions, and alternate forms, the resulting estimatesof reliability are generally higher than the more conservative alternate formsreliability coefficients. These generic reliability coefficients are, therefore,plausible upper bound estimates of the internal consistency reliability of the STAREarly Literacy versions prior to STAR Early Literacy Enterprise.While generic reliability does provide a plausible estimate of measurementprecision, it is a theoretical estimate, as opposed to traditional reliabilitycoefficients, which are more firmly based on item response data. Traditionalinternal consistency reliability coefficients such as Cronbach’s alpha andKuder-Richardson Formula 20 (KR-20) cannot be calculated for adaptive tests.However, another estimate of internal consistency reliability can be calculatedusing the split-half method. This is discussed in the next section.Split-Half ReliabilityIn classical test theory, before the advent of digital computers automated thecalculation of internal consistency reliability measures such as Cronbach’s alpha,approximations such as the split-half method were sometimes used. A split-halfreliability coefficient is calculated in three steps. First, the test is divided into twohalves, and scores are calculated for each half. Second, the correlation betweenthe two resulting sets of scores is calculated; this correlation is an estimate of thereliability of a half-length test. Third, the resulting reliability value is adjusted,STAR Early LiteracyTechnical Manual45