2003 IMTA Proceedings - International Military Testing Association

performance value to lowest through the use of PRTFAC, a TASSET output. Those tasks with
the highest co-performance values are most representative of the TM (Phalen, Staley, and
Mitchell, 1987). In addition, the PRTFAC allows an analyst to better characterize and name
each TM, discern finer distinctions between tasks, and create new TMs if necessary. These
adjustments can be manually input into GRPMOD (another TASSET output), re-analyzed and
readjusted until a final GRPMOD TM listing (TM extract) is produced.
Previous research by Snooks and Luster (2003) indicated that labeling task modules
added no measurable benefit; therefore, none of the task modules were named for this study.
Instead, unlabeled TMs were provided for each team that showed tasks grouped by coperformance
only (TM extracts).
As customary, SKT development teams were provided with SKT extracts during the first
week of each project. The SMEs began using the SKT extract data in the first week of each 5week
major revision project by using the data for outline development in accordance with TE
process standards, and then continued using the SKT extracts throughout item development to
validate test items. The TM extracts were presented to the teams between week 2 and week 4 of
the projects, after the teams were familiar with the SKT Extract. Each team member was asked
to review the TM extract and then complete a 10-item, 7-point Likert scale survey (see
Attachment 1). OMS TPs assigned to these projects were also asked to review the TM extracts
and complete a survey, identical to the TP survey, except for the heading (see Attachment 2). In
order to avoid tainting the responses by the enthusiasm of the researchers, and to ensure standard
survey administration, a cover letter (see Attachment 3) was developed that included a brief
explanation of the two survey products and directions for completing the survey. In other words,
the survey was designed for self-administration.
Results
In simple mean comparison tests: the perceived “importance” of the SKT extract was
compared with the perceived “importance” of the TM extract; the perceived “accuracy” of the
SKT extract was compared with the perceived “accuracy” of the TM extract; the perceived “ease
of understanding” the SKT extract was compared with the perceived ‘ease of understanding” the
TM extract; the perceived “ease of using” the SKT extract was compared with the perceived
“ease of using” the TM extract; and the “desire to use” the SKT extract was compared with the
“desire to use” the TM extract. In addition, both groups (SMEs and TPs) were encouraged to
provide comments.
No statistically significant differences were found, however some comparisons can be
made. All respondents (N = 53) appeared to be slightly more positive about the SKT extract
(Mean = 5.1811) than the TM extract (Mean = 4.8604) on all five examined attributes, with the
“desire to use” attribute rated the highest of all (Mean = 5.8679) for the SKT extract and the
“ease of use” attribute rated the lowest (Mean – 4.5660) for the SKT extract. The highest rating
for the TM extract, from all respondents, was for the “importance” attribute (Mean = 5.1698),
while the attribute with the lowest rating from all respondents was “ease of use” (Mean =
4.4339). It is worth noting that the “ease of use” attribute for both products has the lowest
ratings, indicating that it may be the data itself that is difficult to use.
TPs (N = 13, Mean = 5.5000) appeared to be slightly more positive towards both
products on all five examined attributes than the SMEs (N = 40, Mean = 4.4688), with the
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003
33