Teacher – Full Teacher Evaluation Information - North Dakota ...

quality of every classroom or a global estimate of the quality of a given program or locality? Are there
high stakes for CLASS scores? How will you use your data to improve teacher-child interactions on a
classroom-by-classroom basis? Answers to these questions should guide your ultimate data collection
plan.
Do we need to send more than one observer to each classroom? One of the
best ways to improve the reliability of CLASS scores is to have multiple observers make ratings of the
same classroom. Although the associated expense of “double coding” often is prohibitive, we still
recommend that at least a portion (between 5 to 15%) of classroom observations be double coded to
assess reliability. Having this data will help you communicate to stakeholders about the fairness of the
tool in practice. For example, Virginia has reported obtaining approximately 90% reliability (scores
within one point) in the field.
How do we decide how many classrooms to observe and how long each
should be observed? The answer to this question depends greatly on the goals you have for
data collection. For example, just as survey researchers figure out how best to collect a random
sampling of data on which to base judgments on election polling numbers, complex analyses exist
that can help you develop a sampling plan to match your goals. While it is beyond the scope of this
overview document to provide a detailed answer, general guidance and a few examples are provided
below. They highlight trade-offs that have to be considered when making decisions about the number
of classrooms to be observed and for what length of time.
General Principles to Consider:
1. The more ratings you are able to obtain and aggregate, the more stable your estimates of typical
classroom interactions will be.
2. In most cases, we find that a two-hour observation (4 CLASS cycles) provides a reliable estimate
of the overall status of teacher-child interactions in a classroom.
3. There typically is more variance in CLASS scores within an organization (program, school,
grantee, etc) than there is between organizations. This means you have to assess a fair number of
classrooms within any one organization to get a reliable estimate of that organization.
4. Even if all observers are CLASS certified, there will be small, systematic differences between their
scoring. Some observers may tend to give slightly higher scores, while others may tend to be
slightly more critical. Although slight differences fall within our threshold for “reliability,” collectively,
they can produce skewed results. The best way to minimize any potential “observer effects” is to
randomly assign observers to classrooms within any organization (program, school, grantee, etc.).
As an example of the trade-offs embedded in these decisions, consider two examples – the Office of
Head Start (OHS) and the City of Chicago.
OHS conducts triennial reviews of all Head Start grantees. Each grantee typically oversees multiple
Head Start programs. Monitoring visits are designed to provide feedback at the grantee level. OHS
was interested in including the CLASS as part of this review process. OHS did not intend to share or
analyze data at the program or classroom level.
CLASS Implementation Guide 39