Pearson Successmaker Math Efficacy Study 2009-10 Final Report

SuccessMaker Math RCT Gatti Evaluation Inc. 9-15-10EXECUTIVE SUMMARYPearson partnered with Gatti Evaluation to conduct rigorous research to support the assertion thatthe SuccessMaker Math computer based learning program effectively increases studentmathematics achievement and attitudes. The program was evaluated in sixty-three diverseelementary and middle grade classrooms from ten schools in seven different states (i.e., AZ, AR,CA, IN, KS, NY, PA) during the 2009-10 school year. Students in classrooms randomlyassigned to use SuccessMaker made regular use of the program while students in comparisonclassrooms received supplemental instruction from non-computerized supplemental mathematicsprograms. Four widely-used classroom mathematics programs were utilized by the sites at 3 rdand 5 th grade, and three different programs were utilized at 7 th grade.The study schools come from public school districts located in large cities or suburbs of largecities. The study schools show considerable variation in ethnicity, students eligible for reducedpriced lunch, as well as a wide range of ability with respect to mathematics and readingachievement. The evaluation team sought out diversity in the study sample to ensure theprogram would be used by learners of all abilities and backgrounds, thus reflecting the realitythat is today’s elementary classrooms. Five schools began the study in the first month, threebegan in the third month, one in the fourth and the last in the fifth month of the school year. Thefinal study sample was large, consisting of 505 3 rd grade (i.e., SuccessMaker = 282, comparison= 223), 408 5 th grade (i.e., SuccessMaker = 224, comparison = 184) and 273 7 th grade (i.e.,SuccessMaker = 136, comparison = 137) students.A challenging assessment battery was group administered to students at baseline and again at theend of the school year. The assessment battery consisted of the Group Mathematics Assessmentand Diagnostic Evaluation (GMADE), and the mathematics attitude survey developed by theprincipal investigator where students respond to self-report questions regarding general mathattitude, confidence, motivation, and self-perceived aptitude. Comparisons on assessmentoutcomes were made between study groups using model adjusted end-of-year raw score groupmean differences. Adjusted group mean differences are calculated holding the effects ofconfounding variables constant for both groups. The equating of confounding variables and themaintaining of consistent implementation ensures the outcomes may more confidently beattributed to the study conditions randomly assigned to these groups.Results were broken out and analyzed separately for each GMADE subtest (i.e., Concepts andCommunication, Operations and Computation, and Process and Applications). Results were alsobroken out and analyzed for separate levels of five key student populations (i.e., Englishproficiency, ethnicity, gender, meal status, math ability). Further, the performance for thecomparison group was compared to four blocks of program usage (i.e., block 1 = 1 to 9 hours,block 2 = 10 to 19 hours, block 3 = 20 to 29 hours, block 4 = 30 or more hours).RQ: How did teachers and students react to the SuccessMaker Math program?Focus groups were conducted at each school during site visits between April and early June.These sessions provided the evaluators with insights into teacher and student experiences withthe program. Teachers and students became quickly comfortable with the SuccessMakerprogram, and felt the program was a good educational investment. The teacher response to the- 1 -