CMT-CAPT Skills Checklist Technical Manual. - NAAC

ReliabilityReliability questions relate to the extent to which an assessment instrument producesconsistent results over time. With an instrument such as the CMT/CAPT Skills Checklistwhich requires teachers to rate student performance in different settings over a period oftime, a critical consideration is the extent to which raters apply scoring criteria similarlyand consistently over time. As was described previously (See “Preparing Raters,” Page28) extensive ongoing training has been provided to teachers who use the CMT/CAPTSkills Checklist to ensure consistent administration of the instrument. Discussions arecurrently underway to determine how best to supplement planned 2007-2008 trainingactivities, including training to criterion, with a study to determine inner-rater reliabilitycoefficients for individuals who are using the Checklist.Internal ConsistencyInternal consistency (alpha) reliabilities were computed for all components of thechecklist at each grade level. These are shown on the following tables. The reliabilityindices are uniformly high for the Access Skills as well as for the three content areasassessed; Reading, Communications and Mathematics. For the total population, thereliability at each grade ranged from .96 to .99 for each subtest. Analyses for thedisaggregated student population based on race, gender and Free Lunch status yield allreliability estimates to equal or exceed .93 with most of the estimates in the range from.97 to 1.0. Although the CSDE does not report scores from the checklist at the strandlevel, strand level reliability estimates are provided here as they are used for further testdevelopment purposes. These estimates were also generally high at .90 or higher, withmost being from .94 to .99. In only three cases (Basic Literacy in grade 5 and ExpressiveCommunication and Basic Spatial Relations in grade 6) did the reliability estimate dropbelow .90. These areas will be reviewed during the next phase of test development.These results, as presented in Tables 4 through 6, suggest that the checklist has beenapplied in a highly consistent manner in each of the grades assessed."14