A COMPENDIUM OF SCALES for use in the SCHOLARSHIP OF TEACHING AND LEARNING

Assessing and Reporting Validity
In addition to forms of reliability, another important component of the psychometric properties
of scales is validity. As is noted earlier in this e-book (see Hussey & Lehan, 2015), there are
several forms of validity (e.g., face, construct, criterion) which constitute ways in which
researchers test if variables are actually measuring the constructs they are intended to
measure. Although reliability was consistently assessed and reported in SoTL research using
scales, validity was far less likely to be assessed and reported. Only eight of 47 (17%) published
studies using scales to measure SoTL outcomes in the past ten Teaching of Psychology issues
(not including scale development-focused articles) assessed any form of validity and reported
those results. Two of those eight studies reported previously published validity information
rather than validity from the present study (e.g., Wielkiewicz & Meuwissen, 2014). One
example of a study that assessed and reported validity in the current study is Buckelew, Byrd,
Key, Thornton and Merwin’s (2013) study of perception of luck or effort leading to good grades.
The researchers adapted a scale of attributions for good grades from a previous study, noting
that no published validity information was available for the original scale. Researchers then
established validity through a pilot test where they correlated values of their adapted scale to a
previously established criterion and found significant positive correlations with the total scale
and subscales. This example of establishing criterion validity through a pilot test is one way to
ensure valid measures before collecting actual study data. However, it should be noted that
validity is truly established over time and through many studies, providing a compelling reason
why routinely reporting validity is vitally important despite the dearth of studies assessing it.
Validity in its several forms (e.g., content, construct, and criterion) are crucial to researchers’
confidence that the data collected are measuring the constructs they were intended to
measure. SoTL researchers are urged to incorporate validity testing and reporting into the work
they do.
The Development and Validation of Scales
There have been some great advances in the ways in which scales are developed and validated.
The practice of using both exploratory analyses (e.g., principal factor analysis) and then
confirmatory factor analysis has greatly enhanced researchers’ confidence in the psychometric
properties of the scales used to measure outcomes (Noar, 2003). The increasingly common use
of structural equation modeling has enhanced scale development by providing easily accessible
ways to conduct confirmatory factor analysis and latent variable modeling (Noar, 2003).
There were five studies out of 141 articles published in the last ten issues of Teaching of
Psychology devoted to the development and validation of a scale. These studies employed
some of the most thorough methodology in scale development and validation. One particularly
good example of such scale validation is Renken, McMahan, and Nitkova’s (2015) initial
validation of the Psychology-Specific Epistemological Belief Scale (Psych-SEBS). In this two-part
study, researchers first drafted and then refined an item pool using exploratory factor analysis,
followed by confirmatory factor analysis and an assessment of internal consistency, test-retest
reliability, and convergent validity of the Psych-SEBS. In the second study, researchers assessed
the criterion validity by comparing the Psych-SEBS with an established criterion and also tested
47