Pathway to Proficiency: Linking the STAR Reading and STAR Math ...

Additional reports automatically generated by the STAR tests help educators screen for laterdifficulties and progress monitor students’ responsiveness to interventions.Sources of DataSTAR Reading and STAR Math data were gathered from schools that use those assessments onRenaissance Learning’s Real Time platform. 2 Performance-level distributions from the ARMT+Reading and Mathematics were retrieved from the Alabama Department of Education.ARMT+ uses four performance categories:Level I, Level II, Level III, and Level IV.Students scoring in the Level III or Level IVrange would be counted as meetingproficiency standards for state and federalperformance-level reportingThis study uses STAR Reading, STAR Math, and ARMT+ data from the 2011–12 school year.MethodologyARMT+ Performance Levels:• Level I• Level II• Level III• Level IVMany of the ways to link scores between two tests require that the scores from each test beavailable at a student level. Obtaining a sufficient sample of student-level data can be a lengthyand difficult process. However, there is an alternative technique that produces similar resultswithout requiring us to know each individual student’s ARMT+ score and STAR scaled score.The alternative involves using school-level data to determine the STAR scaled scores thatcorrespond to each ARMT+ performance level cutscore. School level ARMT+ data arepublically available, allowing us to streamline the linking process and complete linking studiesmore rapidly.The STAR scores used in this analysis were “projected” scaled scores. Each observed STARscore was projected to the mid-point of the ARMT+ administration window using STARReading and STAR Math decile-based growth norms. The growth norms are both grade- andsubject-specific and are based on the growth patterns of more than one million students usingSTAR assessments over a three-year period. They provide typical growth rates for students basedon their starting STAR test score, making predictions much more accurate than a “one-size-fitsall”growth rate.For each observed score, the number of weeks between the STAR test administration date andthe mid-point of the ARMT+ window was calculated. To get the total expected growth from thedate of the STAR test to the ARMT+ , the number of weeks between the two tests was multipliedby the student’s expected weekly scaled score growth (from our decile-based growth norms,which take into account grade and starting observed score). The total expected growth was thenadded to the observed scaled score to determine their projected score at the time of the ARMT+ .If a student took multiple STAR tests during the school year, all their projected scores wereaveraged.2 Renaissance Place Real Time is a service that involves “hosting” schools’ data from the STAR tests and otherproducts. For more information about Real Time, see http://www.renlearn.com/RPRT/default.aspxRenaissance Learning Linking STAR assessments to the ARMT+ Page 4

Sample Description: ReadingA total of 409 unique schools across grades 3 through 8 met the sample requirements for reading(explained in Sample Selection). Racial/ethnic characteristics for each grade of the sample arepresented along with statewide averages in Table 1 and suggest that White students were slightlyover-represented.Table 2 displays by-grade test summaries for the reading sample. It includes counts of studentstaking STAR Reading and the ARMT+ Reading. It also includes percentages of students in eachperformance level, both for the sample and statewide. Despite White students beingoverrepresented, students in the reading sample had similar ARMT+ performance to thestatewide population.Table 1. Characteristics of reading sample: Racial/ethnic statisticsGradeNumber ofschoolsAmer.IndianPercent of students by racial/ethnic categoryAsian Black Hispanic White Mult.3 116 0.5% 0.5% 30.1% 2.7% 66.2% 0.0%4 145 0.3% 0.6% 29.3% 1.7% 68.1% 0.0%5 159 0.4% 0.6% 33.3% 1.3% 64.4% 0.0%6 134 0.6% 0.5% 38.2% 1.5% 59.2% 0.0%7 99 0.5% 0.8% 35.6% 1.0% 62.1% 0.0%8 92 0.5% 1.1% 38.2% 0.7% 59.5% 0.0%Statewide 0.9% 1.4% 34.0% 4.6% 58.3% 0.8%Table 2. Characteristics of reading sample: Performance on STAR Reading and theARMT+ for ReadingGradeSTARReadingStudentsARMT+ReadingStudentsLevel I Level II Level III Level IVSample Statewide Sample Statewide Sample Statewide Sample Statewide3 8,891 8,685 1% 1% 11% 11% 32% 34% 56% 54%4 12,079 11,798 0% 0% 10% 12% 30% 32% 60% 56%5 14,126 13,786 0% 0% 11% 11% 27% 29% 62% 60%6 13,015 12,753 1% 1% 11% 11% 22% 22% 66% 66%7 12,951 12,806 0% 1% 12% 13% 30% 31% 58% 55%8 11,872 11,674 1% 1% 18% 20% 38% 39% 43% 40%Renaissance Learning Linking STAR assessments to the ARMT+ Page 6

Sample Description: MathA total of 297 unique schools across grades 3 through 8 met the sample requirements for math(explained in Sample Selection). Tables 3 and 4 present demographic and achievement data fromthe math sample, along with comparisons to state averages. Like reading, White students wereover-represented in the math sample. Despite White students being overrepresented, the mathsample was similar to the statewide student population in terms of ARMT+ performance.Table 3. Characteristics of math sample: Racial/ethnic statisticsGradeNumber ofschoolsAmer.IndianPercent of students by racial/ethnic categoryAsian Black Hispanic White Mult.3 105 0.1% 0.7% 31.9% 1.5% 65.8% 0.0%4 123 0.7% 0.5% 31.3% 1.5% 66.0% 0.0%5 114 0.6% 0.8% 31.8% 0.8% 66.0% 0.0%6 101 0.7% 0.9% 27.9% 1.4% 69.0% 0.1%7 72 0.8% 1.5% 33.0% 1.3% 63.4% 0.0%8 71 1.1% 1.3% 31.3% 1.1% 65.2% 0.0%Statewide 0.9% 1.4% 34.0% 4.6% 58.3% 0.8%Table 4. Characteristics of math sample: Performance on STAR Math and the ARMT+ forMathematicsGradeSTARMathStudentsARMT+MathStudentsLevel I Level II Level III Level IVSample Statewide Sample Statewide Sample Statewide Sample Statewide3 8,488 8,287 3% 3% 12% 12% 24% 24% 61% 61%4 10,055 9,838 2% 2% 12% 12% 26% 27% 60% 59%5 10,025 9,785 0% 0% 8% 8% 28% 29% 64% 63%6 10,078 9,853 0% 0% 19% 22% 41% 43% 40% 35%7 9,659 9,549 0% 1% 26% 27% 39% 38% 35% 34%8 9,428 9,362 0% 0% 18% 21% 45% 48% 37% 31%Renaissance Learning Linking STAR assessments to the ARMT+ Page 7

AnalysisFirst, we aggregated the sample of schools for each subject to grade level. Next, we calculatedthe percentage of students scoring in each ARMT+ performance level for each grade. Finally, weordered STAR scores and analyzed the distribution to determine the scaled score at the samepercentile as the ARMT+ Level III level. For example, in our third grade reading sample, 1% ofstudents were Level I, 11% were Level II, 32% were Level III, and 56% were Level IV.Therefore, the cutscores for proficiency levels in the third grade are at the 1 st percentile for LevelII, the 12 th percentile for Level III, and the 44 th percentile for Level IV.Figure 1. Illustration of linked STAR Reading and the third-grade ARMT+ for ReadingscaleRenaissance Learning Linking STAR assessments to the ARMT+ Page 8

Results and ReportingTable 5 presents estimates of equivalent scores on the STAR Reading score scale and theARMT+ for Reading. Table 6 presents estimates of equivalent scores on the STAR Math scorescale and the ARMT+ for Mathematics. These results will be incorporated into STARPerformance Reports (see Sample Reports, pp. 12–15) that can be used to help educatorsdetermine early and periodically which students are on track to reach the Level III status orhigher and to make instructional decisions accordingly.Table 5. Estimated STAR Reading cutscores for the ARMT+ for Reading performancelevelsGradeLevel I Level II Level III Level IVCut score Cut score Percentile Cut score Percentile Cut score Percentile3 < 129 129 1 279 12 423 444 < 149 149 1 333 10 489 405 < 161 161 1 382 11 536 386 < 173 173 1 427 12 563 347 < 197 197 1 457 12 671 428 < 226 226 1 563 19 893 57Table 6. Estimated STAR Math cutscores for the ARMT+ for Mathematics performancelevelsGradeLevel I Level II Level III Level IVCut score Cut score Percentile Cut score Percentile Cut score Percentile3 < 414 414 3 534 15 602 394 < 451 451 2 586 14 662 405 < 452 452 1 587 8 703 366 < 466 466 1 689 19 797 607 < 467 467 1 712 26 820 658 < 479 479 1 713 18 848 63Renaissance Learning Linking STAR assessments to the ARMT+ Page 9

References and Additional InformationBandeira de Mello, V., Blankenship, C., & McLaughlin, D. H. (2009). Mapping state proficiencystandards onto NAEP scales: 2005–2007 (NCES 2010-456). Washington, DC: U.S.Department of Education, Institute of Education Sciences, National Center for EducationStatistics. Retrieved February 2010 fromhttp://www.schooldata.org/Portals/0/uploads/Reports/LSAC_2003_handout.pptCronin, J., Kingsbury, G. G., Dahlin, M., & Bowe, B. (2007, April). Alternate methodologies forestimating state standards on a widely used computer adaptive test. Paper presented atthe American Educational Research Association, Chicago, IL.McGlaughlin, D. H., & V. Bandeira de Mello. (2002, April). Comparison of state elementaryschool mathematics achievement standards using NAEP 2000. Paper presented at theannual meeting of the American Educational Research Association, New Orleans, LA.McLaughlin, D., & Bandeira de Mello, V. (2003, June). Comparing state reading and mathperformance standards using NAEP. Paper presented at the National Conference onLarge-Scale Assessment, San Antonio, TX.McLaughlin, D., & Bandeira de Mello, V. (2006). How to compare NAEP and state assessmentresults. Paper presented at the annual National Conference on Large-Scale Assessment.Retrieved February 2010 from http://www.naepreports.org/task 1.1/LSAC_20050618.pptMcLaughlin, D. H., Bandeira de Mello, V., Blankenship, C., Chaney, K., Esra, P., Hikawa, H., etal. (2008a). Comparison between NAEP and state reading assessment results: 2003(NCES 2008-474). Washington, DC: U.S. Department of Education, Institute ofEducation Sciences, National Center for Education Statistics.McLaughlin, D.H., Bandeira de Mello, V., Blankenship, C., Chaney, K., Esra, P., Hikawa, H., etal. (2008b). Comparison between NAEP and state mathematics assessment results: 2003(NCES 2008-475). Washington, DC: U.S. Department of Education, Institute ofEducation Sciences, National Center for Education Statistics.Perie, M., Marion, S., Gong, B., & Wurtzel, J. (2007). The role of interim assessments in acomprehensive assessment system. Aspen, CO: Aspen Institute.Renaissance Learning. (2010). The foundation of the STAR assessments. Wisconsin Rapids, WI:Author. Available online from http://doc.renlearn.com/KMNet/R003957507GG2170.pdfRenaissance Learning. (2012a). STAR Math: Technical manual. Wisconsin Rapids, WI: Author.Available from Renaissance Learning by request to research@renlearn.comRenaissance Learning. (2012b). STAR Reading: Technical manual. Wisconsin Rapids, WI:Author. Available from Renaissance Learning by request to research@renlearn.comRenaissance Learning Linking STAR assessments to the ARMT+ Page 10

Independent technical reviews of STAR Reading and STAR MathU.S. Department of Education: National Center on Intensive Intervention. (2012). Review ofprogress monitoring tools [Review of STAR Math]. Washington, DC: Author. Availableonline from http://www.intensiveintervention.org/chart/progress-monitoringU.S. Department of Education: National Center on Intensive Intervention. (2012). Review ofprogress monitoring tools [Review of STAR Reading]. Washington, DC: Author.Available online from http://www.intensiveintervention.org/chart/progress-monitoringU.S. Department of Education: National Center on Response to Intervention. (2010). Review ofprogress-monitoring tools [Review of STAR Math]. Washington, DC: Author. Availableonline from http://www.rti4success.org/ProgressMonitoringToolsU.S. Department of Education: National Center on Response to Intervention. (2010). Review ofprogress-monitoring tools [Review of STAR Reading]. Washington, DC: Author.Available online from http://www.rti4success.org/ProgressMonitoringToolsU.S. Department of Education: National Center on Response to Intervention. (2011). Review ofscreening tools [Review of STAR Math]. Washington, DC: Author. Available onlinefrom http://www.rti4success.org/ScreeningToolsU.S. Department of Education: National Center on Response to Intervention. (2011). Review ofscreening tools [Review of STAR Reading]. Washington, DC: Author. Available onlinefrom http://www.rti4success.org/ScreeningToolsRenaissance Learning Linking STAR assessments to the ARMT+ Page 11

Sample Reports 3Sample STAR Performance Reports focusing on the Pathway to Proficiency. This reportwill be available to Alabama schools using STAR Reading Enterprise or STAR Math Enterprise.The report graphs the student’s STAR Reading or STAR Math scores and trend line (projectedgrowth) for easy comparison with the pathway to proficiency on the ARMT+ .3 Reports are regularly reviewed and may vary from those shown as enhancements are made.Renaissance Learning Linking STAR assessments to the ARMT+ Page 12

Sample Growth Proficiency Chart. Using the classroom GPC, school administrators and teachers can better identify best practices that arehaving a significant educational impact on student growth. Displayed on an interactive, web-based growth proficiency chart, STARAssessments’ Student Growth Percentiles and expected State Assessment performance are viewable by district, school, grade, or class. Inaddition to Student Growth Percentiles, the Growth Report displays other key growth indicators such as grade equivalency, percentile rank,and instructional reading level.Renaissance Learning Linking STAR assessments to the ARMT+ Page 14

AcknowledgmentsThe following experts have advised Renaissance Learning in the development of the STAR assessments.Thomas P. Hogan, Ph.D., is a professor of psychology and a Distinguished University Fellow at theUniversity of Scranton. He has more than 40 years of experience conducting reviews of mathematicscurricular content, principally in connection with the preparation of a wide variety of educational tests,including the Stanford Diagnostic Mathematics Test, Stanford Modern Mathematics Test, and theMetropolitan Achievement Test. Hogan has published articles in the Journal for Researchin Mathematics Education and Mathematical Thinking and Learning, and he has authored two textbooksand more than 100 scholarly publications in the areas of measurement and evaluation. He has also served as consultant to awide variety of school systems, states, and other organizations on matters of educational assessment, program evaluation, andresearch design.James R. McBride, Ph.D., is vice president and chief psychometrician for Renaissance Learning. Hewas a leader of the pioneering work related to computerized adaptive testing (CAT) conducted by theDepartment of Defense. McBride has been instrumental in the practical application of item responsetheory (IRT) and since 1976 has conducted test development and personnel research for a variety oforganizations. At Renaissance Learning, he has contributed to the psychometric research anddevelopment of STAR Math, STAR Reading, and STAR Early Literacy. McBride is co-editor of aleading book on the development of CAT and has authored numerous journal articles, professionalpapers, book chapters, and technical reports.Michael Milone, Ph.D., is a research psychologist and award-winning educational writer and consultantto publishers and school districts. He earned a Ph.D. in 1978 from The Ohio State University and hasserved in an adjunct capacity at Ohio State, the University of Arizona, Gallaudet University, and NewMexico State University. He has taught in regular and special education programs at all levels, holds aMaster of Arts degree from Gallaudet University, and is fluent in American Sign Language. Miloneserved on the board of directors of the Association of Educational Publishers and was a member of theLiteracy Assessment Committee and a past chair of the Technology and Literacy Committee of theInternational Reading Association. He has contributed to both readingonline.org and Technology & Learning magazine on aregular basis. Over the past 30 years, he has been involved in a broad range of publishing projects, including the SRA readingseries, assessments developed for Academic Therapy Publications, and software published by The Learning Company andLeapFrog. He has completed 34 marathons and 2 Ironman races.R44719.130215