A COMPENDIUM OF SCALES for use in the SCHOLARSHIP OF TEACHING AND LEARNING

eflect overall teacher behaviors or separated into two subscales (or two factors) representing
caring/supportive teaching behaviors and professional competency/communication skills.
Researchers also use factor analysis for reduction purposes, including limiting the number of
items in an instrument as well as the number of factors included in an instrument (Yong &
Pearce, 2013). For example, another scale discussed further in Chapter 11 of this e-book is the
Satisfaction with Life Scale (SWLS) (Diener, Emmons, Larsen, & Griffin, 1985; see Layous,
Nelson, & Legg, 2015). These authors reviewed the literature, found preexisting scales of life
satisfaction to be inadequate, and therefore, began the process of developing a new
instrument. In the first phase, they developed 48 items related to satisfaction with one’s life,
which was decreased to 5 items by using factor analysis. Procedures such as these help
researchers to find ways to more effectively measure a construct, while also reducing
participant fatigue with answering numerous items and increasing the likelihood of replicating
findings (Henson & Roberts, 2006).
It is also important to note that statistical programs will only show where and how items load; it
is up to the researcher to interpret the findings using the literature and/or a theoretical
framework (Henson & Roberts, 2006; Yong & Pearce, 2013). For example, deleting items as
suggested by SPSS does not always increase reliability (Cho & Kim, 2015). Depending on the
findings of the factor analysis, researchers might go back to the literature and develop
additional items to pilot or determine the instrument’s structure and assess its reliability and
validity (Warner, 2013).
Reliability
Regardless of whether an instrument being used for SoTL is preexisting, an adapted measure, or
newly developed, it is important to assess its reliability. Generally speaking, reliability refers to
how consistently an instrument measures a certain construct or variable of interest (John &
Benet-Martinez, 2000). Scores on the instrument should not change unless there is a change in
the variable. It is not possible to calculate reliability exactly, as there is inevitable measurement
error. Therefore, researchers have to estimate reliability. In doing so, all attempts to remove or
control sources of error should be made so as to obtain a truer score of the latent variable
(DeVellis, 2012).
The question then becomes, how do we know if an instrument is reliable? The methods for
assessing reliability can vary depending on type of data and information sought (John & Benet
Martinez, 2000); however, they are all based on the idea that reliability is the proportion of
variance of the observed score that can be attributed to the true score of the latent variable
being examined (DeVillis, 2012). Many researchers will report a single coefficient alpha
regarding reliability (Hogan, Benjamin, & Brezinski, 2000), but that is a limited picture of
reliability and not always the best indicator (Cho & Kim, 2015; Cronbach & Shavelson, 2004;
John & Benet-Martinez, 2000). For example, Cortina (1993) demonstrated how alpha can
increase with the number of items in an instrument despite low average item intercorrelations.
Therefore, it is important to examine whether additional reports of item relationships and/or
reliability are given (Schmitt, 1996). There is also somewhat of a challenge in determining
35