I__. - International Military Testing Association

the behavioral statements for guiding interview ratings, and orient the
interviewer to use the questions, the worksheet, and the interview rating
form.
&lot Testinq the Structured Interview
The interview protocol and rating form were pilot tested in two wave:
with a total of 31 officer interviewers and 93 applicants in seven
different locations. Means and standard deviations of the ratings provid.
data on their spread and overall distribution.
As part of the pilot testing, an interrater reliability study was
conducted. One way to assess the quality of data emerging from the ne:q
structured ,interview is to determine how closely two interviewers agree i
their independent ratings of the same interviewees. Thus, we initiated a
interrater reliability study with 10 officers interviewing a total of 24
applicants. All interviewers were trained to do the structured intervie,.+
and to use the rating form. Each applicant was interviewed by two office
recruiters.
After each interview session, the interviewer completed the rating
form and provided a copy to the researcher. Officers interviewing the s:
applicant never discussed that applicant before making their ratings, so
the interview judgments were generated totally independently. Intraclaorcorrelations
were computed for each dimension separately and for the sum
the dimension ratings. This provides an estimate of the across-intervie-consistency
of ratings made using the new interview protocol and rating
form.
RESULTS
Means and standard deviations for Wave 2 interview ratings, gathered
after major revisions of the interview protocol (after Wave 1 pilot
testing), appear in Table 1. For these 59 applicants, means are close TV
4.0 (on a 5-point scale) and standard deviations are approximately 1.0.
Further, these means compare favorably with data for the previous inter:;;
(M=4.68 in 1985). Of course, this is not a very fair comparison because
ratings on the new format were gathered for research, whereas the 4.68 r:t;
is based on operational ratings. Nonetheless, applicant ratings using th
new protocol appear to provide reasonable spread for the interview rati.n!:l.
of these typically high quality NROTC applicants.
Table 1 also contains the interrater reliability coefficients for
ratings made of the 24 applicants evaluated by two independent
interviewers. These are very high reliabilities, with considerable
agreement shown on the part of the interviewers.
495