A COMPENDIUM OF SCALES for use in the SCHOLARSHIP OF TEACHING AND LEARNING

Recommendations

Info

Content Validity Although there are some differences in how content validity is defined in the literature, “there is general agreement in these definitions that content validity concerns the degree to which a sample of items, taken together, constitute an adequate operational definition of a construct” (Polit & Beck, 2006, p. 490). However, content validity is often left to the judgment of the researcher and/or a panel of experts to ensure only relevant items related to the construct are included (DeVellis, 2012). One issue with stopping at this step is that it is a subjective judgment of whether an instrument has adequate content validity. Some researchers suggest the computation of a content validity index (CVI) to better judge the psychometrics of an instrument (Polit & Beck, 2006; Wynd, Schmidt, & Schaefer, 2003). Generically speaking, this involves having a panel of raters review each item in the measure, rate how relevant each item is to the construct being measured, and then the researcher(s) assess agreement between raters. Construct Validity Construct validity is based on a theoretical framework for how variables should relate. Specifically, “it is the extent to which a measure ‘behaves’ the way that the construct it purports to measure should behave with regard to established measures of other constructs” (DeVellis, 2012, p. 64). Some deem construct validity as one of the most important concepts in psychology (John & Benet-Martinez, 2000; Westen & Rosenthal, 2003). However, there is no agreed upon or simple way to assess construct validity (Bagozzi, Yi, & Phillips, 1991; Westen & Rosenthal, 2003). Instead, many researchers use correlations to show relationships between variables as suggested by the literature, often referred to as convergent and discriminant validity (Messick, 1995). For example, Ryan and Wilson (2014) examined the validity of a brief version of the Professor-Student Rapport Scale by comparing the brief measure to scales that were similar (e.g., Immediacy Scale, to which it should positively relate) and dissimilar (i.e., Verbal Aggressiveness Scale, to which it should negatively relate). There are also computations that can be performed to demonstrate levels of construct validity including confirmatory factor analysis, effect size correlations, and structural equation modeling (Bagozzi et al., 1991; Westen & Rosenthal, 2003). Criterion-Related Validity Criterion-related validity is mainly concerned with practicality versus theoretical underpinnings like construct validity; “it is not concerned with understanding a process but merely with predicting it” (DeVellis, 2012, p. 61). However, many confuse construct validity and criterionrelated validity because the same example listed above can be used for either type of validity; the difference is in the intent of the researcher (e.g., to explain or explore variable relationships versus simply identifying and predicting) (DeVellis, 2012). In the case of the latter, there are multiple types of correlations that can be computed to identify specific relationships among variables and in turn, different types of validity. For example, researchers can examine whether the scores on their instrument predict future behaviors (i.e., predictive validity), correlate with scores on another instrument shown to measure the same construct (i.e., convergent validity), 38
and/or do not correlate with instruments that are unrelated to the construct of interest (i.e., discriminant validity; Warner, 2013). In many cases, researchers will want both construct and criterion-related validity due to basing their survey development in theory and striving toward the ability to predict an outcome. Conclusions In sum, scale development is an iterative, rather than a linear process. It is likely that researchers might have to return to earlier steps in the scale-development process at times. In addition, the process should also be deliberate and be guided by a theoretical foundation where appropriate. Only when such a process is employed can a psychometrically sound measure be developed that yields meaningful data. As the saying goes, “a little knowledge is a dangerous thing,” yet this is exactly what was provided in this chapter. The purpose was to familiarize readers with some of the main concepts and processes that researchers must consider when developing or modifying a scale. The hope is that readers might use the easily digestible sources cited in this chapter as a springboard for further skill building related to scale development. 39
Page 1 and 2: A COMPENDIUM OF SCALES for use in t
Page 3 and 4: Table of Contents Chapter 1: Introd
Page 5 and 6: Chapter 1: Introduction Rajiv S. Jh
Page 7 and 8: and boredom; whereas the former inc
Page 9 and 10: References Bishop-Clark, C. & Dietz
Page 11 and 12: Chapter 2: Best Practices in Scale
Page 13 and 14: o Participants respond to items suc
Page 15 and 16: - Be aware of whether the scale is
Page 17 and 18: References References marked with a
Page 19 and 20: *Wielkiewicz, R. M. & Meuwissen, A.
Page 21 and 22: example, McCabe (2015) examined the
Page 23 and 24: psychometrically sound tool. Howeve
Page 25 and 26: a history of the authors’ work do
Page 27 and 28: *Oleson, K. C., Poehlmann, K. M., Y
Page 29 and 30: Chapter 4: A Primer on Scale Develo
Page 31 and 32: scales or develop new ones. In eith
Page 33 and 34: found their 40-item measure could b
Page 35 and 36: eflect overall teacher behaviors or
Page 37: Although there is some variability
Page 41 and 42: Fraley, R. C., & Vazire, S. (2014).
Page 43 and 44: Streiner, D. L. (2003). Starting at
Page 45 and 46: Psychology, 36 of the 47 articles (
Page 47 and 48: Assessing and Reporting Validity In
Page 49 and 50: Grades Another common source of alr
Page 53 and 54: model of collaborative undergraduat
Page 55 and 56: Chapter 6: Measuring Learning and S
Page 57 and 58: joins a new team to teach them abou
Page 59 and 60: Summative Assessments Summative Rub
Page 61 and 62: programs. In a national survey of p
Page 63 and 64: Course Outcomes Scale Centra and Ga
Page 65 and 66: discuss The Motivated Strategies fo
Page 67 and 68: The Motivated Strategies for Learni
Page 71 and 72: Hacker, D. J., Bol, L., & Bahbabani
Page 73 and 74: Stoloff, M. L., & Feeney, K. J. (20
Page 75 and 76: interpretation of analogies is an a
Page 77 and 78: Similar difficulties in accurately
Page 79 and 80: How do we balance the need for effi
Page 83 and 84: *Halpern, D. F. (2010). Halpern Cri
Page 85 and 86: Appendix: A Compendium of General C
Page 87 and 88: Chapter 8: Student Engagement Towar
Page 89 and 90:
particular items. Indeed, a point o
Page 91 and 92:
employing the SCEQ have shown other
Page 93 and 94:
one of eight achievement emotions m
Page 95 and 96:
Recent efforts (Saddler, Sonnert, H
Page 97 and 98:
References References marked with a
Page 99 and 100:
64, 847-860. doi:10.1037/0022-3514.
Page 101 and 102:
*Schaufeli, W.B., Salanova, M., Gon
Page 103 and 104:
course. Pretest and posttest survey
Page 105 and 106:
Perceptions 7 Problem-Solving 1 Sat
Page 107 and 108:
Culture and Issues Knowledge of Cur
Page 109 and 110:
Civic Attitudes and Skills Question
Page 111 and 112:
Scale of Service-Learning Involveme
Page 113 and 114:
engagement and psychological measur
Page 115 and 116:
Giles, D. E., & Eyler, J. (2013). T
Page 117 and 118:
Appendix A: Research Measures for M
Page 119 and 120:
Robby, M.A. -CASQ -SAT/9 Scores -¹
Page 121 and 122:
Steinberg, K. S., Hatcher, J. A., B
Page 123 and 124:
Chapter 10: Measuring Individual Di
Page 125 and 126:
either unnecessary or is referenced
Page 127 and 128:
Park, 2005). The merit of the instr
Page 129 and 130:
Similar factor structures were prod
Page 131 and 132:
Page 133 and 134:
Chapter 11: Measuring Well-Being in
Page 135 and 136:
items, and the PANAS 44 items, and
Page 137 and 138:
Meaning in Life The Meaning in Life
Page 139 and 140:
future,” using a 5-point scale (1
Page 141 and 142:
Domain-Specific Stress and Anxiety
Page 143 and 144:
demonstrate peoples' varying levels
Page 145 and 146:
Gehlbach, H. (2015). Seven survey s
Page 147 and 148:
Ritchie, R. A., Meca, A., Madrazo,
Page 149 and 150:
Chapter 12: Assessing Professor-Stu
Page 151 and 152:
ends with some unanswered questions
Page 153 and 154:
positively about their professors
Page 155 and 156:
The PSRS underwent further validati
Page 157 and 158:
help or hinder the graduate student
Page 159 and 160:
Komarraju, M. (2013). Ideal teacher
Page 161 and 162:
Chapter 13: Effective Tools for Ass
Page 163 and 164:
offer guidance based on their exper
Page 165 and 166:
The Original TBC Study Our approach
Page 167 and 168:
student/faculty rating with critica
Page 169 and 170:
eveals that five qualities (knowled
Page 171 and 172:
Page 173 and 174:
*Wann, P.D. (2001, January). Facult
Page 175 and 176:
Nessie the Mythical Loch Ness Monst
Page 177 and 178:
components of knowledge of cognitio
Page 179 and 180:
a cognitive activity (Cacioppo et a
Page 181 and 182:
teacher? In other words, how do we
Page 183 and 184:
Christopher (2015) also suggests th
Page 185 and 186:
Kirk, C., Busler, J., Keeley, J., &
show all

A COMPENDIUM OF SCALES for use in the SCHOLARSHIP OF TEACHING AND LEARNING

Create successful ePaper yourself

Delete template?

Save as template?