MAKING VALID AND RELIABLE DECISIONS IN ... - CCSSO projects

More documents

Recommendations

Info

20 Making Valid and Reliable Decisions in Determining AYP
Chapter2Improving the Validity and Reliability of StateAccountability Systems“Validation was once a priestly mystery, a ritual performed behind the scenes, withthe professional elite as witness and judge. Today it is a public spectacle, combiningthe attractions of chess and mud-wrestling” (Cronbach, 1988).The No Child Left Behind Act of 2001 requires States to develop valid and reliable Stateaccountability systems. Although the references to validity and reliability are ubiquitous inthe law 4 , our focus is on three specific areas: assessment systems 5 , accountability systems,and definitions of Adequate Yearly Progress. The law specifies the following:Both the assessments and the other indicators that a State chooses to use are to be “validand reliable, and…consistent with relevant, nationally recognized professional andtechnical standards.”Each State’s definition of Adequate Yearly Progress must be “statistically valid andreliable.” Although each State is required to disaggregate the results for specified groups of students,it specifically relaxes this requirement in cases where “the number of students in a categoryis insufficient to yield statistically reliable information…” [Section 1111(b)(I)(ii)].The goal of this chapter is to present a general description and analysis of the concepts of validityand reliability as they apply to State educational accountability systems. This will entail adescription of State accountability systems (of which assessment systems are a component), and ananalysis of their relationship to States’ overall educational/instructional systems. It will deal withaccountability systems in a general way, rather than focusing specifically on the requirements of4 In fact, the law uses the words in this phrase 59 times. In 21 of those references, the law requires something to be“reliable” and to have “validity” 18 times. Another 20 times an assessment or process is required to be either “valid andreliable” or “reliable and valid.”5 The treatment of validity and reliability of assessment systems is only touched upon tangentially since they are viewed asrelatively separate systems. Furthermore, the U. S. Department of Education has set forth separate requirements forassessment systems under pending regulations.Making Valid and Reliable Decisions in Determining AYP 21
Page 3 and 4: CHIEFSTATESCHOOLofCOUNCILOFFICERSMA
Page 6 and 7: INTER-COMPONENT PROBLEMS: MAL-ALIGN
Page 8 and 9: percentage increases over time. Man
Page 10 and 11: The information provided in this pa
Page 12 and 13: AYP MeasuresRewards and SanctionsEn
Page 14 and 15: making AYP will be included in the
Page 16 and 17: performance of subgroups of student
Page 18 and 19: An aligned system of academic conte
Page 20 and 21: How to include small schools and sm
Page 22 and 23: Further, the law, in Section 1116,
Page 24 and 25: The extent to which LEAs and SEAs w
Page 28 and 29: the NCLB Act. This is not because t
Page 30 and 31: whether of a student or of a school
Page 32 and 33: The solution is not simple. It is a
Page 34 and 35: How does the measurement correspond
Page 36 and 37: The information that an accountabil
Page 38 and 39: Overall GoalsAccountability systems
Page 40 and 41: Ineffective schools should be ident
Page 42 and 43: FIGURE 5. A SIMPLIFIED THEORY OF AC
Page 44 and 45: that definitions of variables and p
Page 46 and 47: no satisfactory way to compute a co
Page 48 and 49: classification rules—across schoo
Page 50 and 51: Occasionally, however, the errors c
Page 52 and 53: more rudimentary skills. (Or, is th
Page 54 and 55: Intent of the accountability modela
Page 56 and 57: ReferencesAmerican Educational Rese
Page 58 and 59: Hill, R. (2001). Issues related to
Page 60 and 61: 54 Making Valid and Reliable Decisi
Page 62 and 63: Finally, AYP has proven difficult s
Page 64 and 65: accountability model proposed. To t
Page 66 and 67: However, defining a school is not a
Page 68 and 69: FIGURE 9. FACTORS TO CONSIDER AS ST
Page 70 and 71: school population (Crane, 2002). As
Page 72 and 73: If one wants to establish a measure
Page 74 and 75: Using this approach, a State could
Page 76 and 77:
strategy on reliability grounds 23
Page 78 and 79:
SummaryA school’s results for the
Page 80 and 81:
567). Los Angeles, CA: Center for t
Page 82 and 83:
Is the accountability system leadin
Page 84 and 85:
not the very next year, with no cha
Page 86 and 87:
Minimum “n”Minimum “n” is i
Page 88 and 89:
District and State officials may fi
Page 90 and 91:
2. Rolling averages Decision: How c
Page 92 and 93:
approach illuminates these issues,
Page 94 and 95:
What are the unintended consequence
Page 96 and 97:
Further guidance is provided in reg
Page 98 and 99:
Accountability Requirement for Engl
Page 100 and 101:
“Safe Harbor”Safe harbor” pro
Page 102 and 103:
This review process provides “one
Page 104 and 105:
will likely have significant impact
Page 106 and 107:
Simulation studies to evaluate pote
Page 108 and 109:
Assessment: Any systematic method o
Page 110 and 111:
Reliability of Assessment Results:
show all

MAKING VALID AND RELIABLE DECISIONS IN ... - CCSSO projects

Create successful ePaper yourself

Delete template?

Save as template?