A COMPENDIUM OF SCALES for use in the SCHOLARSHIP OF TEACHING AND LEARNING

found their 40-item measure could be reduced to 25 items to decrease redundancy, which can
impact reliability as discussed below. Researchers might also use cognitive interview techniques
with participants after they complete a newly developed instrument to garner feedback about
the instrument as well as have participants think aloud while answering each survey question
(Ryan, Gannon-Slater, & Culbertson, 2012). These findings should inform modifications to the
instrument. Such processes are especially important when special populations are involved
(e.g., children, non-native speakers, etc.) (Presser et al., 2004).
Instrument Administration
Delivery Options
Once the instrument is complete, it is ready to be fully administered. However, it is also
important to consider the many practical and logistical issues related to administering tests
(Nosek, Banaji, & Greenwald, 2002; Wright, 2005). For example, a researcher might wonder
whether it is more appropriate to have student participants complete instruments in-person or
online (Miller et al., 2002). Many scholars have opted to collect data online due to benefits
such as lower costs and increased access to participants (Miller et al., 2002; Wright, 2005). In
addition, researchers have found comparable reliability and validity of instruments
administered online versus in-person with paper and pencil (Meyerson & Tryon, 2003; Miller et
al., 2002). Furthermore, researchers have found online samples to approximate the
characteristics of traditional samples (Gosling, Vazire, Srivastava, & John, 2004).
Survey Design
Researchers also need to be cognizant in how the design of the survey can potentially impact
participant responses (e.g., completion time and valid responses) (Christian & Dillman, 2004;
Couper, Traugott, & Lamias, 2001). For example, Mahon-Haft and Dillman (2010) showed that
the design of online surveys, including aesthetics and type of open response format, can
negatively impact the quality of data collected (e.g., omissions and shorter responses). This
issue is crucially important to reliability and validity, both of which are discussed further below.
Sample Size
Additional concerns related to instrument administration is adequate sample size and the
characteristics of the sample. It is worth noting that although there are varying beliefs in what
constitutes an adequate sample size when developing an instrument (e.g., at least 150, 300,
varying variable ratios such as 20:1), it is safe to say the larger the sample size, the better
(Schmitt, 2011; Williams, Onsman, & Brown, 2010; Yong & Pearce, 2013). Sample size and
reliability become critically important when considering their impact on power to detect an
effect and the ability to reproduce observed effects (Fraley & Vazire, 2014; Henson, 2001;
Simons, 2014). In regard to scale development, sample size is also important in that smaller
sample sizes can limit the type of analyses that can be performed (Noar, 2009). It is also
important for researchers to consider the demographic characteristics of the participants who
33