I__. - International Military Testing Association

follows: (I 2 episodes (2) 4 episodes (3) 3 episodes (4) 2 episodes and (5) 3 episodes. Hypothesis 13 was
significant 1y
supported for each level.
Hypothesis 14. The more difficult the episode, the less time the instructors will take to find the TAE test
solutions when compared to the students (novices). The difficulty level of the episode and the dtfference rn
time between instructors and students to find TAE test solutions is negatively correlated but not significant.
Although no significant difference was found, the more difficult the episode, the less time instructors tended
to take to find the TAE test solutions when compared to the students.
Generally, the results were as expected; that is, the more difficult, the more time; at different levels of
difficulty, better performers took less time. An unexpected result was the lack of significant difference between
students and instructors. The difference was, however, strongly in the direction expected.
The consistently significant relationship In this area clearly calls for further investigation and improvement,
particularly in behavioral and cognitive task analyses.
Iime (Table A6). Hypothesis 15. Subjects with a higher level of troubleshooting proficiency will take
si nificantly less total time to find TAE e isode solutions than less proficient subjects. The correlation b&ween
TfE score and total time to find epis OCPe fault is significant at the .OOl level.
Hypothesis 16. Subjects with higher levels of troubleshooting proficiency will take a significantly longer
time than less proficient subjects before making the first test point. The correlation between TAE score and
time to first test point is significant at the .05 level.
Results suggest that analysis of behavior and cognitive protocols could result in a dramatic change In the
way the training community presents troubleshooting training. Here again, behavioral protocol analysis could
provide useful information on training approaches.
Comolex Test Eauioment (Table A-7). Hypothesis 17. Subjects with a higher level of troubleshooting
proficiency will make significantly more tests using an oscilloscope than less proficient subjects. The
correlation between TAE score and the number of oscilloscope tests is not significant.
Given the nature of the hardware system and the resulting TAE delivery system, subjects did not a pear
to have sufficient opportunity to use complex test equipment in the TAE episodes. Therefore, the lac R of a
statistically significant result may have no practical meaning.
m (Table A-8). H pothesis 18. The higher the student’s TAE class rank, the higher the student will
be ranked in terms of trou‘6 leshooting proficiency by instructors or work center supervisors. Hypothesis 18
was supported for two classes at the .OOl level. The correlation between TAE class ranking and in.structorMork
center supervisor ranking was not significant for the other classes. Although not significant, two classes had
an inverse relationshi .
Hypothesis 19. Ph e higher the student’s TAE class rank (final score), the higher will be the student’s
ranking in the class. Hypothesis 19 was supported for one class at the .Ol level of significance. For the other
classes, the correlation between TAE class ranking and ranking in school class was not significant. Although
not significant, two classes indicated a strong positive correlation. Conversely, one class showed a strong
inverse relationship between TAE class ranking and school class ranking.
Hypothesis 20. The higher the instructor ranking of the student in terms of troubleshooting proficiency,
the higher will be the student’s ranking in the class (final score). Hypothesis 20 was supported for three
classes, one class at .OOl level and two at .05 level. Although not significant, one class showed a strong
positive correlation between instructor student ranking and class student ranking. One class showed a weaker
positive correlation and two classes indicated an inverse relationship.
There were no consistent results in rankings across instructors, TAE performance, or school Performance.
In several classes, inverse relationships were shown. Only one class had a consistent significant relationship
across hypotheses.
The results of this area most clearfy attest to the need for an objective evaluation tool of the skill of
troubleshooting. It shows that supervisors, and school results do not have the ability to evaluate personnel
in this skill.
FUTURE EFFORTS
In addition to the recommendations made for each area of investigation, we also have the following general
recommendations for future efforts in this area.
1. Further investigate TAE validity and reliability. Design and development of the TAE approach and
delivery system stron ly support face validity of TAE. Subject matter ex erts were involved in all phases of
the project. They Betermined the factors of evaluation, weights oP the factors, evaluation scheme,
troubleshooting episodes to be used, developed the episodes and participated in the test and evaluation.
Since T&E results are somewhat ambiguous, areas dealing with validity and reliability should be investigated
further.
2. Analyze data to further develop discriminatory/predictive capability. Results of performance Of
subjects on TAE episodes should be subjected to behavioral protocol analyses to develop a model of
troubleshooting and further analyses of approaches used by good vs. bad troubleshooters and ultimately
cognitive Protocol analyses to determine selection, training and evaluation requirements.
376