I__. - International Military Testing Association

The goal in the TAE testing is to find and replace the LRU. Subjects begin TAE testing by reviewing a series
of menus of symptoms, Panels, and diagnostic information; next they select equipment to be tested and
conduct tests or replace a LRU.
ch Hy@heseg, The 20 hypotheses for the TAE Test and Evaluation were organized into seven
categories: experience, electronics knowledge, electronics performance proficiency, difficulty level, time,
complex test equipment, and ranking. The hypotheses in each category, and method of testing each, are
described in the following sections.
METHOD
Test Administration Procedu & Testing was conducted by NPRDC personnel in a classroom at the
Advanced Electronics School: Department (AESD), Service Schools Command, San Diego, California.
Testing was on the Zenith 248 microcomputer. Technical documentation for the hardware system was.in the
classroom. Subjects were assigned randomized test sequences to protect from test order effects. Srxteen
episodes were administered to each subject and each episode required about an hour to comPlete, but
subjects had no specific time limit. Subjects completed all episodes in two to three days. The admrnrstrator
was present in the classroom during testing. Subjects listened to an introduction to the TAE study and the
technical documentation available; read and signed a Privacy Act release statement; and completed a
computerized Learn Program, 2 practice and 14 test troubleshooting episodes. After testing, subjects
received test performance feedback and completed a critique.
Subjects. Subjects for the TAE test and evaluation were students in the “system” phase of the maintenance
course and the system qualified instructors. All subjects were required to have school training on the
subsystems.
Data Data were collected for 53 students and 13 instructors in two data bases, using a standard
statistical Package for analysis. The first contained demographic data: the second, performance data. Data
were collected for seven classes of students between April and September 1989. Demographic data for each
student included: SSN, time in service, Armed Services Vocational Aptitude Battery (ASVAB) scores, school
subsystem scores, school corn rehensive score, school final score, class ranking, TAE ranking, and instructor
ranking. Demographic and TAI! performance data for instructors were collected during September 1989. The
demographic data for instructors included SSN, rate/rating, time in service & paygrade, time system qualified,
time working on the system in the fleet and as a system instructor. The TAE program data for both students
and instructors consisted of scores for 16 episodes encompassing 673 variables. Table A-l describes the
variables for each episode (Episode 1 is presented).
Data files were refined and evaluated. Data for five students were dropped due to missing data, and for
two instructors due to lack of system qualification. Thus, the data of 59 subjects were used for this study, 48
students and 11 instructors. The resultant data base were used to create files for testing the study hypotheses.
The master file was used to create files with variables specifically required to test each hypothesis. The
methods for testing the hypotheses are described in the following subsections.
RESULTS and DISCUSSION
Results of the data analyses are presented in Appendix A, and the specific areas investigated are discussed
in the following:
Demoaraohic Data. For the 48 students, the average time in service was 2.23 years. For the 11 instructors,
9 had a rate of electronics technician first class (ETl) and 2, of ET2; the average paygrade was 5.82. The
average time in service for instructors was 10.41 years and average time in paygrade was 3.64 years.
Instructors were system qualified for an average of 4.67 years and had worked on the system hardware in the
fleet an average of 2.94 years, In addition, they averaged 16.18 months as instructors.
.
EI@WIWZ (Table A-2). Hypothesis 1. Instructors (experts) will score significantly higher on the TAE test
than students (novices). A one-way analysis of variance (ANOVA) was performed to test hypothesis 1. The
F ratio value is not significant.
Hypothesis 2. Sub’ects with a longer time in the electronics rate (i.e., Time in Service - TIS) will score
significantly higher on tlle TAE test than subjects with less time in that rate.
Generally, the relationship between experrence and TAE performance was not statistically significant. This
apparent anomaly may be explained by the fact that instructors of the course are not required to be system
qualified. Students must prove their system qualification to graduate.
The lack of a significant relationship between experience and troubleshooting performance causes one to
uestion if the experience measures were appropriate, if an appropriate set of subjects was tested, if the TAE
1 elivery and evaluation systems are valid, or if there is actually no difference due experience. Given the face
validity of TAE and the high level of expectation by subject matter experts of the relationship between
experience and Performance, further testing is needed to resolve this issue.