2008 - Marketing Educators' Association

ankings, alumni ratings, and the rating of trained
observers (Aleamoni, 1999; Cashin, 1995; Kulik,
2001). However, even here, problems emerge.
Finkelstein (1995) reviewed some literature to
suggest that the instruments will not lead to
instructional improvement unless skilled consultants
help the faculty interpret the ratings. The literature
seems to be suggesting that the instruments are
concurrently valid, but that the concurrent
experience that faculty members or students have
with these instruments, without expert supervision, is
invalid.
Predictive Validity
The predictive validity of a measurement instrument
relates to its usefulness as a predictor of some other
characteristic or behavior. The research base here
is generally lacking and contradictory. These
contradictions can be shown by two examples.
“There is no evidence that the use of teaching
ratings improves learning in the long run”
(Armstrong, 1998). But Overall and Marsh (1979)
stated, “Students of instructors who got student
feedback scored higher on achievement test and
assessments of motivation for learning than students
of instructors who got no feedback.” Many writers
maintain, without reservation, that the evaluation
instruments can be utilized to improve instruction
(Aleamoni, 1999; Kulik, 2001; Theall & Franklin,
2001), even though Kulik (2001, p. 10) begins his
arguments by stating, “The catch is that no one
knows what measure to use as the criterion of
teaching effectiveness.” The reported need for
consultants in this process (Finkelstein, 1995)
weakens the argument.
Construct Evaluations: Convergent Validity
Constructs are complex, necessitating several
methods of validation. One of them is to establish
that the measurement of the construct is associated
with independent measures that, according to the
underlying structural hypothesis, should be
associated with the construct. A major problem with
academic measures is highlighted by Garson (2006,
p. 2):
“A good construct has a theoretical basis
which is translated through clear operational
definitions involving measurable indicators. A
poor construct may be characterized by lack
of theoretical agreement on its content, or by
flawed operationalization such that its
indicators may be construed as measuring
one thing by one researcher and another
thing by another researcher.”
9
According to this definition, many researchers would
insist that academic measures would intrinsically
lack construct validity. A validity measure can,
however, be approximated. For example, although
the definition of “good” teaching can vary, most
educators would generally agree that students will
learn more from a “good” teacher than from a “bad”
teacher. In addition, most observers would conclude
that effort and learning are related. Numerous
attempts have been made to establish that SET has
these relationships with mixed results. Many
researchers have reported finding a positive
relationship between learning and student ratings of
instructors (Dowell & Neal, 1982; Marlin & Niss,
1980; Lundsten, 1986; Baird, 1987), many others
have found either no, or even a negative,
relationship (Attiyeh & Lumsden, 1972; Jake, 1998;
Johnson, 2003; Robin & Robin, 1972, Weinberg,
Fleisher & Hashimoto, 2007; Yunker & Yunker,
2003). The same pattern with rigor has been found
with many researchers finding a strong relationship
between rigor and SET (Chacko, 1983; Cashin,
1995; Sixbury & Cashin, 1995), while others have
found no, or a negative, relationship (Clayson &
Haley, 1990; Greenwald & Gillmore, 1997a; Marks,
2000).
The number of articles on this topic is a testament to
the difficulty of establishing convergent validity with
SET. It also shows why honest researchers can
come to such widely different conclusions in the
debate about the convergent validity of SET.
Construct Evaluations: Discriminant Validity
The evaluative measures of a construct should not
be related to criterion unrelated to the construct.
This is the area that has created the most heated
debate with SET.
For example, an instructor should not be able to
“buy” good evaluations by giving good grades.
Numerous studies have found that grades are not
unjustifiably related to the evaluations (Cashin,
1995; Marsh & Dunkin, 1992; Marsh & Roche, 1999,
2000; Kaplan, Mets, & Cook, 2000; see Marsh &
Roche (2000) for extensive reviews). Another large
group of studies have found that grades are related
to the evaluations (Clayson, 2004; Gillmore &
Greenwald, 1999; Johnson, 2003; Weinberg,
Fleisher, & Hashimoto, 2007; see Clayson, Frost, &
Sheffet (2005) for extensive reviews).
Personality unrelated to learning, the so-called “Dr.
Fox Effect,” should not be related to SET. Many
studies and reviews have found personality to not be
related to the evaluations (Aleamoni, 1999;