I__. - International Military Testing Association

tive by the mean effectiveness rating for that response, and then averaging these item level scores to
obtain an overall effectiveness score based on choices for the least effective response alternative. This
score will be referred to as L-Effectiveness, and the parallel score based on choices for the most effective
responses (described previously) will be referred to as M-Effectiveness.
Finally, a scoring procedure that involved combining the choices for the most and the least effective
response alternative into one overall score was also explored. For each item, the mean effectiveness of
the response alternative each soldier chose as the least effective was subtracted from the mean effectiveness
of the response alternative they chose as the most effective. Because it is actually better if
respondents indicate that less effective response alternatives are the least effective, this score can be seen
as a sum or composite of the two effectiveness scores described previously (i.e., subtracting a negative
number from a positive number is the same as adding the absolute values of the two numbers). These
item level scores were then averaged together for each soldier to generate yet another score, and this
score will be referred to as M-L Effectiveness.
Descriotive Statistics. Descriptive statistics and internal consistency reliability estimates (RR-20)
were computed for each of the five scoring procedures. Intercorrelations were also computed among the
five scores generated by the five different scoring procedures.
Preliminarv Information Concerning Construct Validity
The data from this field test were also used to obtain preliminary information concerning the construct
validity of the SJT as a criterion measure supervisory job knowledge. As mentioned previously, collecting
the field test data for the SJT was a part of a larger data collection effort. Several other job performance
measures were administered concurrently with the SJT, including job knowledge tests, a self-report
administrative information survey, and supervisory simulation exercises (involving training a subordinate,
disciplinary counseling, and personal counseling). Performance ratings were also collected from
peers and supervisors using behavior-based rating scales. If the SJT is a valid measure of supervisory job
knowledge, certain relationships would be expected with these other measures. For example, it should
have at least moderate correlations with the scores on the supervisory simulations and performance ratings
on supervisory dimensions. Correlations of SJT scores with several of these other job performance
measures were examined.
Another type of information that was used to assess the construct validity of the SJT was the extent
to which the knowledges assessed by the SJT am learned on the job. If the SJT is a valid measure of job
knowledge, soldiers who have more experience or training would be expected, on average, to obtain
higher scores than soldiers with less experience or training. Self report information was collected from
the soldiers in this field test sample concerning whether or not they had attended any supervisory training
and how regularly they were required to supervise other soldiers, Mean SJT scores for soldiers with
different levels of training and experience were also examined.
Field Test Results
Results
Table 1 presents the mean score for each of the five scoring procedures. The maximum possible for
the M-Correct scoring procedure is 35 (i.e., all 35 items answered correctly), but the mean score obtained
by soldiers in this sample was only 16.25. The maximum score obtained was only 27. The mean number
of least effective response alternatives correctly identified by this group was only 14.86. Clearly the SJT
was difficult for this group of soldiers.
Table 1 also presents the standard deviation for each of the five scoring procedures, and all of the
scoring procedures resulted in a reasonable amount of variability in scores obtained by the soldiers in this
sample. Table 1 also shows that the internal consistency reliabilities for all of these scoring procedures
are quite high. The most reliable score is M-L Effectiveness, probably because this score contains more
information than the other scores (i.e., choices for both the most and least effective response).
.~ .,
f---~..-wz.-.-.. _ .,._ _ 2 7 0