Eric Grosch, Letter to Dr. Morgenstern on LOR - Semmelweis ...

divisions. (18)
Kingsbury enumerated those allegedly ideal conditions:
Ratings, <strong>to</strong> be reliable, necessitate (1) averaging three independent ratings, each made on an
objective scale; (2) these scales must be comparable and equivalent, made in conference under
expert supervision; (3) the three raters must be competent <strong>to</strong> rate.[18]
Paterson joined, with slightly different wording, in affirming Kingsbury's ideal conditions (2) and
(3) and in thus implicitly alluding <strong>to</strong> pitfalls of the method:
. . . Ratings should be accepted and filed for use only from those who have proved themselves
capable of accurately judging human qualities. . . a rating scheme will not work au<strong>to</strong>matically. It
must be closely supervised preferably by trained personnel research workers who must continually
subject the ratings <strong>to</strong> critical analysis and assist in training executives in proper use of the method.
There is no escape from this requirement.[19]
Paterson and Kingsbury omitted mention of what specifics the training they proposed for the
personnel-research workers should comprise and accomplish but they presumably intended,
among other things, that the trained supervisors should somehow ensure separate evaluation of
labeled traits <strong>to</strong> exclude Thorndike's halo-effect; then, by averaging, fine-tuning, adjustment and
manipulation of the scores from at least three raters, all of whom knew how <strong>to</strong> provide accurate
ratings (presumably assessed by the raters' mutual agreement on each candidate's score on each
criterion) obtain a set of ratings consistent with the aggregate global impression each candidate
made on the raters (the candidate's halo). The circularity of the rationale seems inescapable.
Prior <strong>to</strong> receiving requests <strong>to</strong> fill out forms consisting of Likert-scale ratings on others'
performance, I have never received any of the extensive training or testing <strong>to</strong> prove myself
“capable of accurately judging human qualities,” nor, I daresay, has any appraiser of my
performance received such training and testing, <strong>to</strong> my knowledge. The origina<strong>to</strong>rs of such forms
seemed <strong>to</strong> assume that the rating scemes would “work au<strong>to</strong>matically,” contrary <strong>to</strong> Kingsbury's
admonition.
Rugg may have had more insight:
. . . The unordered -- yes, the chaotic -- character of the judgments appears, irrespective of what
traits are considered or of what kinds of scales are compared. I now believe that the evidence
establishes the futility of obtaining single “ratings” on point scales of such dynamic qualities as
“intelligence,” “personal qualities,” “general work,” and the like.[20]
Paterson cautioned and predicted:
These rating methods should not be looked upon as perfect or final. Further research is necessary,
and industry will profit . . . as progressive, experimentally minded executives realize the scope of
the problem and engage in the necessary research . . . <strong>to</strong> develop newer and more reliable methods