A COMPENDIUM OF SCALES for use in the SCHOLARSHIP OF TEACHING AND LEARNING

Content Validity
Although there are some differences in how content validity is defined in the literature, “there
is general agreement in these definitions that content validity concerns the degree to which a
sample of items, taken together, constitute an adequate operational definition of a construct”
(Polit & Beck, 2006, p. 490). However, content validity is often left to the judgment of the
researcher and/or a panel of experts to ensure only relevant items related to the construct are
included (DeVellis, 2012). One issue with stopping at this step is that it is a subjective judgment
of whether an instrument has adequate content validity. Some researchers suggest the
computation of a content validity index (CVI) to better judge the psychometrics of an
instrument (Polit & Beck, 2006; Wynd, Schmidt, & Schaefer, 2003). Generically speaking, this
involves having a panel of raters review each item in the measure, rate how relevant each item
is to the construct being measured, and then the researcher(s) assess agreement between
raters.
Construct Validity
Construct validity is based on a theoretical framework for how variables should relate.
Specifically, “it is the extent to which a measure ‘behaves’ the way that the construct it
purports to measure should behave with regard to established measures of other constructs”
(DeVellis, 2012, p. 64). Some deem construct validity as one of the most important concepts in
psychology (John & Benet-Martinez, 2000; Westen & Rosenthal, 2003). However, there is no
agreed upon or simple way to assess construct validity (Bagozzi, Yi, & Phillips, 1991; Westen &
Rosenthal, 2003). Instead, many researchers use correlations to show relationships between
variables as suggested by the literature, often referred to as convergent and discriminant
validity (Messick, 1995). For example, Ryan and Wilson (2014) examined the validity of a brief
version of the Professor-Student Rapport Scale by comparing the brief measure to scales that
were similar (e.g., Immediacy Scale, to which it should positively relate) and dissimilar (i.e.,
Verbal Aggressiveness Scale, to which it should negatively relate). There are also computations
that can be performed to demonstrate levels of construct validity including confirmatory factor
analysis, effect size correlations, and structural equation modeling (Bagozzi et al., 1991; Westen
& Rosenthal, 2003).
Criterion-Related Validity
Criterion-related validity is mainly concerned with practicality versus theoretical underpinnings
like construct validity; “it is not concerned with understanding a process but merely with
predicting it” (DeVellis, 2012, p. 61). However, many confuse construct validity and criterionrelated
validity because the same example listed above can be used for either type of validity;
the difference is in the intent of the researcher (e.g., to explain or explore variable relationships
versus simply identifying and predicting) (DeVellis, 2012). In the case of the latter, there are
multiple types of correlations that can be computed to identify specific relationships among
variables and in turn, different types of validity. For example, researchers can examine whether
the scores on their instrument predict future behaviors (i.e., predictive validity), correlate with
scores on another instrument shown to measure the same construct (i.e., convergent validity),
38