Pearson Successmaker Math Efficacy Study 2009-10 Final Report

SuccessMaker Math RCT Gatti Evaluation Inc. 9-15-10Operations and Computation 0.85Process and Applications 0.77Math Academic Attitude Survey 0.761. Sample estimated coefficient alpha intraclass reliability.Teacher MeasuresThe research team also collected data through teacher logs and classroom observations, as wellas teacher interviews and focus groups. The teacher and classroom data increased the validity ofthe research findings related to achievement outcomes by verifying results through multiple datacollection methods, adding context for results through the perspectives of various participants,and by collecting data at various time points during the study.The research team collected achievement, attitudinal, as well as, observationaland self-report data making the study both quantitative and qualitative in nature.In addition to the assessment battery, qualitative data collection methods were also employed.The sources of qualitative data included; program reports, teacher surveys, daily lesson logs,classroom observations, as well as, teacher notes from electronic correspondences. Teacherswere routinely asked for their opinions throughout the school year. Weekly mathematics lessonnotes were collected for both SuccessMaker and comparison classes (i.e., 10-15 minutescompletion time per week). Cumulative usage reports and program implementation logs wereregularly collected from SuccessMaker users. All study classrooms were observed twice duringthe school year teaching routine mathematics lessons and SuccessMaker teachers were furtherinterviewed as to their opinions regarding the program. All this data was compiled and contentanalyzed to determine teacher attitudes and performance, as well as to illuminate the variousways teachers and students interact with the program.Weekly Teacher LogsAll study teachers were required to complete weekly logs in which they describe theirmathematics lessons. Information from the weekly logs was important for two reasons. The firstis to guarantee SuccessMaker teachers fully and regularly utilized all key components ofSuccessMaker Math to provide adequate opportunity for the program to positively influencestudent achievement. The second reason was to document the instructional model for all studyteachers, including; classroom environment, teaching style, pacing and mathematics content andmethods.Teachers were asked not to spend more than 15 minutes per week completing the logs. It is clearseveral teachers spent more time, however, as many of the logs were returned with detailedcomments. Teachers often shared candid weekly experiences with the Gatti Evaluation projectmanager and were typically happy to provide documentation describing weekly instruction andlearning experiences related to the program. SuccessMaker and comparison group teacherssummarized daily classroom mathematics instruction time, topics, and methods. Dailysummaries also included the amount of time spent on these activities. In addition, SuccessMaker- 9 -