I__. - International Military Testing Association

intraclass correlations for many scales were zero
or negative. A standardization transformation to
remove scale use differences among raters was
needed. For these scales, a zero rating has
absolute meaning--that a task requires no skills
or knowledges related to a particular scale. A
standardization transformation should not change
zero ratings. For this reason, the following
standardization was applied to the ratings:
Y ,,k = X,,, / [sum(k= 1,57) Xijk ] [Equation 41
where yi,k = standardized rating for rater i on
scale j and task k, and
X Ilk = raw rating for rater i on scale j and
task k.
Figure 2 presents interrater agreement
statistics for the 26 rating scales after this
standardization. The ratings have acceptable
interrater agreement. Three of the scales,
Medical-Patient Care, Medical-Equipment
Oriented, and Medical Procedures had no
non-zero ratings for tasks in this occupation.
Thus, meaningful intraclass correlation statistics
could not be computed for these scales although
all raters agreed on all ratings for these scales
(zero).
For modeling purposes, we augmented the
training-time data file from the TDS R&D with
scores on the 26 skill and knowledge scales. We
used mean standardized ratings across raters for
each task and scale. If the 26 scales are a useful
basis for estimating training-time models, then
equation 3, which uses scores on the scales along
with training times to predict proficiency, should
account for most of the proficiency variation
accounted for by equation 2, which includes
actual task identities.
Results
Scale ( Omega21 Rkk
1. Clencal .28 .66
2. Computatronal .13 .44
3. Office Equipment Operatton 34 .72
4. Mechanical .13 A3
5. Simple Mechanrcaf .06 .23
EquipmentlSystems
Operatron
6. Complex Mecnanicaf .Ol .06
Equipment/Systems
Operation
7. Mecnanical-Electrical .15 .47
8. Mechanfcai-Electronrc .20 .56 -.
9. Elecnrcal .ll .38
10. Eiewomc .20 .56
11. E!ectncaf-MechanrcaJ .22 .58
12. Eiectncaf-Eiectronc .13 A3
13. Eiectronic-Mechantcal .19 .54
14. Simple PhysrcaJ Labor .oo .oo
15. Medical-Pattent Care . .
16. Medical-Equipment . .
Or-tented
17. Medical Prccedures . .
18. Simple Nontechnical .02 .I3
Procedures
19. Communicative-Oral .20 .55
20. Communicatrve-Written .*9 .a3
21. General Tasks Or .13 .e
Proceaures
22. Reasoning/Planning/ .05 .2?
Analyzing
23. Scienttfic Math Reasoning .08 .31
Or Calculatrons
24. Speaal Talents .05 ‘9
25. Supervisory .27 24
26. Training .05 .20
Note:
Omega Squared Is The Intrac!ass Correlation fnter-
Rater Agreement: It Is Equivalent To The Fit 1. Pkk
Is The Estimated Reliability For The Mean Rating From
Five Raters.
‘All Tasks Had Zero Ratings; Inter-Rater Agreement
Statistics Are Meaningless.
Figure 2. Interrater Agreement Data
for the 26 Skill and Knowledge Scales.
Our first modeling activity was to fit the regression model of equation 2. The R* for this
model was .65, which is statistically significantly greater than zero: F(451,2255) = 9.5, p < .OOl.
Next, we fit the regression model of equation 3, which replaced task identification variables
with scores on the skill and knowledge scales. The R* for this model was .52, which is also
statistically significant: F( l&$,2525) = 14.9, p < .OOl. If one views the skill and
120