I__. - International Military Testing Association

The average estimate of reliability (Cronbach's alpha) of the ratings across
the six job families was .99 for the psychologists and .84 for the incumbents.
There was very high agreement between the psychologists and the job
incumbents in terms of the relative importance of the abilities to the jobs in
each job family. The product-moment correlations among the mean ability ratings
for the two groups of raters ranged from .96 to .98 and rank order correlations
ranged from .89 to .96.
To investigate whether or not the psychologists and the job incumbents
agreed in terms of the absolute importance ratings given to the abilities, tests
of the significance of the difference between the means for the two groups of
raters were performed. In the majority of cases, the pairs of mearls were found
to be significantly different. It should be borne in mind, however, that due
to the large numbers in the incumbent group, even very small absolute differences
will be statistically significant.
When the mean ratings for both groups were dichotomized into those
determined to be important (equal to, or greater than, 3.0--"Important" on the
five-point scale) and those determined not to be important (less than 3.0 on the
five-point scale), the two groups of raters were found to be in perfect
agreement.
STUDY 2
Wernimont (1988) has indicated that governmental guidelines on employee
selection still emphasize the necessity of focussing on job tasks and duties in
job analysis, followed by documentation and justification for the inferences made
about needed abilities. Perhaps this is a function of the fact that, as Schmidt
(1988) points out, the empirical data upon which these governmental guidelines
are based are inadequate in many areas, particularly job analysis.
Our job-analytic research provided an opportunity to add to the empirical
database by determining if job-ability linkage results obtdined in a "holistic"
manner (i.e., by having job incumbents rate the importance of an ability to
overall job success) were comparable to the job-ability linkages results obtained
by requiring incumbents to rate the importance of abilities for each duty they
perform. If the results obtained from the two methods were found to be similar,
significant reductions in the cost, as well as the intrusiveness, of the job
analysis process for test development could be possible.
Data Collection: As indicated earlier, in the job analysis inventory the
incumbents were asked to rate each of the seven abilities, using a five-point
rating scale, for their importance to overall job performance (i.e., the holistic
approach). Average ability importance ratings were then computed for each job
family.
After rating the importance of the ability to the overall job, these sama
incumbents were asked to rate the importance of each of the seven abilities to
the performance of each individual job duty they had previously indicated they
pet-formed (i.e., the traditional fine-grained, duty-ability linkage approach).
Using this traditional approach, a mean ability rating was determined by summing
each incumbent's ability rating for each duty performed and then dividing by the
number of duties that the incumbent performed. Averages were then computed fcr
each job family for each ability.
RESULTS
The average ratings for each ability across duties performed, for the same
job families, are presented in Table 2. For ease of comparison, the mean overall