Technical Report - International Military Testing Association

.
.
and performed .the duties 01 the Primary NOS. The ‘mean co-vorker rating
given by 3 En for each of the men in the particular validation sample
serves as the criterion. Since a sumnary of buddy ratings in military
research by Hollander (1954) emphasized the relevant values of peer
ratings, this information vae instrumental in the USAEEC research
decision to apply the job performance criterion.
An element of anxiety may surround the peer rating mthod in aom
quarterr. It has been found wantfng because of ineffestfve application
in most instances. For example, young recruits could not generally be
expected to rate leadership in the military setting because they do not
have adequate knowledge of the role and no degree of experience in making
such a judgment. The fault lies not fn the racing process but tn the
Ineffective preparation for fts use. The comparison most apt may be
that of trying to ahoot a bull’s eye vfth a defective weapon. A stand wss
made in our program to develop a rating form which would minimize rating
format influences and insufficient knowledges of co-vorkers about each
other. The rem1 t being, that peer ratings of a relatively structured
style, are postulated with a sense of confidence toward obtatnfng a
realistic estimate of job performance on an eleven-point scale.
The administration of rating wa8 prefaced by special instructions
to induce the raters’ acceptance of the task in a mre Informed and
responsible may. A research psychologist conducted the rating session,
while the teat control officer (‘X0) at esch installation we requested
to schedule all of the available racers who were qualified to rate @I
in the specified HOS. Croups of about 20 to 40 men were assigned to meet
in suitable place8 for the rb;ing sessions, which were usually completed
in about 20 minutes.
Three phases of analysis compose the substance of the test valfdation
procedure: (1) analysis of the total evaluation test; (2) analysis
of the valid portion of the evaluation test; and (3) providtng rccomnencled
numbers of item by evaluation test outline. These phases art organized
to assist in recuring the desired evaluation test speclflcations through
te8 t rev16 ion.
Initially, in the first phase the relationships between item
statistics and test statistics are thoroughly delineated. Results of
aoalyeia in tabular form show the total test, technical test and Broad
Subject-Hatter Areas (B!SMA’a) by number of items with the rtapective
means, standard deviations, RR-20 reliability coefficients, validity
coefficienta, beta weights, the multipltR,and coirected R after shrinkage.
men follow in another table the correlation coefficients betveen each of
the BStU’a, the BSHA’s and the criterion, and total evaluation test and
criteriorr. In u third suxnary table the rerults reported for items
include item p-vaiues for the total HOS population, p-values computed
107
:. .