MAKING VALID AND RELIABLE DECISIONS IN ... - CCSSO projects

that definitions of variables and procedures for collecting and editing that had seemed adequate forroutine management and administrative purposes are usually found to be inadequate to ensure theintegrity of the results for accountability purposes.4. Developing theModel: Designingthe Rules forMaking Inferencesand Decisionsabout SchoolSuccess.This may be the most important component; it surely is one of the mostpowerful determinants of the ratings and classifications of the schools.The model and its procedures follow philosophically from the statementof purposes and goals of the system, and are responsible forimplementing the vision for the system, including the particular focus ofthe system in driving a particular kind of change for particular types ofschools and students. It dictates how the indicators are to be weightedand used to make decisions about different types of schools, howdifferent types of measurement and sampling error are to be dealt with, and how the integrity ofthe data and the process is to be ensured in the face of the natural forces that mitigate them. Itincludes such specifics as the relative emphasis on status or change; the relative focus on differenttypes of students; the processes of combining the results to form a judgment, includingcompensatory and conjunctive approaches; the metric to be used, how small schools andsubgroups are handled, and the combining of data across years and grades (and any other topicsdiscussed in Chapter 3).5. Implementingthe Decisions.In this phase, the decisions are being executed. Schools are beingclassified and specified reform strategies are being implemented. Thekey issues here revolve around fairness and fidelity of the process. Itincludes the strategies for tailoring the reforms to the nature of the schools and their studentpopulations, and perhaps, at least for some states, to the degree to which the schools failed to meetadequate yearly progress (e.g., the number of subgroups that failed to meet AYP).6. Evaluating theEffects.The accountability system has its own evaluation component. This isdifferent from the evaluative function that the system fulfills for theeducational system; this component looks specifically at the impact ofthe accountability system itself. Is it having the right kind of effects? Is it being implementedproperly? Which components are problematic? Where does it seem to be working best? Anapproach to conducting this evaluation is presented in the next section.Part 3. A definition of validity for AccountabilitySystemsThere is almost no literature on the validity of accountability systems. 13 Therefore, in this section,the definitions of validity as used in the worlds of science, business, medicine, and psychologicaland educational testing are applied to accountability systems. The following definition would seemto be a proper application:An accountability system can be said to have validity when the evidence is judged to be strongenough to support the inferences that• The components of the system are aligned to the purposes, and are working inharmony to help the system accomplish those purposes; and13 The thought-provoking set of proposed standards for accountability systems developed by CRESST is one of the veryfew resources (2001).38 Making Valid and Reliable Decisions in Determining AYP