A COMPENDIUM OF SCALES for use in the SCHOLARSHIP OF TEACHING AND LEARNING

whether reports of reliability are adequate. For example, most researchers use a cutoff for
alpha at .70 or higher as adequate (Hogan et al., 2000; Schmitt, 1996); however, others note .80
or higher to be the recommended levels for alpha (Henson, 2001; Lance, Butts, Michels, 2006).
However, higher is not always better as an overly high alpha might be the result of redundancy
in the items (Ryan & Wilson, 2014; Streiner, 2003; Tavakol & Dennick, 2011). Ryan and Wilson’s
(2014) brief version of the Professor-Student Rapport Scale with a Cronbach’s alpha of .83 is a
good example of acceptable internal consistency that is not so high as to suggest issues with
redundancy.
Some of the most common methods of assessing reliability include inter-observer agreement,
test-retest reliability, and measures of internal consistency. What follows is a brief discussion of
these common ways of assessing reliability as well as additional resources faculty can examine
to learn more about these topics. It is important to keep in mind that this discussion is a brief
introduction into concepts related to reliability; entire books are devoted to this topic (e.g., see
Fink & Litwin, 1995; Litwin, 2003).
Inter-Observer
Assessing inter-observer agreement comes into play when researchers want to look at the
reliability of responses given by two or more raters or judges (Warner, 2013). Depending on the
goal of the researcher, the level of agreement between raters could be allowed to vary or not.
For example, in a psychology statistics class, two teachers should be in perfect agreement
regarding whether a student correctly computed a specific statistical analysis. In contrast, a
group of teaching assistants might be asked to come into a classroom and judge students’
poster projects for the best designed poster to examine if a certain teaching intervention was
effective. In this case, perfect score agreement might not be needed or beneficial. Instead, we
might look at how the judges ranked the posters from best to worst. Researchers can also
examine percentage of agreement between raters by counting the number of times the raters
were in agreement and dividing by the total number of judgments made. However, this method
of assessment does not take into account chance levels of agreement between raters and tends
to work best when the rated variable is objective rather than subjective. To take into account
chance agreements, one would need to calculate Cohen’s kappa (ks coefficient (Viera &
Garrett, 2005), with coefficients below .40 being poor, between .40 and .59 fair, .60 and .74
good, and.75 and 1.00 excellent (Cicchetti, 1994). Sources of measurement error to be mindful
of include judges’ background and training regarding the variable of interest and the tool being
used as well as prevalence of the finding (DeVillis, 2012; Guggenmoos-Holzmann, 1996; Viera &
Garrett, 2005).
Test-Retest
Reliable instruments should provide constant scores if used to assess a variable at one time
point and then again at another time point (so long as theoretically that variable should remain
constant across time); however, the error of measurement each time will vary. A correlation
could then be computed to assess consistency in scores for the two assessment points.
36