2003 IMTA Proceedings - International Military Testing Association

Usability Testing II
The second phase of testing occurred over a three and a half month period in the spring of
2003 at RTC Great Lakes. A group of approximately 4,500 participants completed the refined
JOIN (1.0e) instrument. The group was 82% male, 65% white, 20% black, and 15% other.
From a usability perspective, 93.2% of all respondents rated JOIN “good” or “very good.”
Regarding the PC descriptors, 90.4% of respondents felt that the pictures did a “good” or “very
good” job of conveying the information presented in the descriptors, and 80.5% stated that the
items did a “good” or “very good” job of conveying Navy relevant job information to new
recruits. In terms of psychometric quality, the average PC scale α was 0.87. Descriptive
statistics indicate that participants have provided differential responses across and within work
activity scales. The average testing time decreased (from the original version) from an average
of 24 minutes to 13 minutes. The average time spent per item ranges from 8 to 10 seconds
(except for special operations items – 21 seconds). Special programs and Aviation are preferred
communities, with working outside and in a team as the work environment and style of choice.
As in the initial pilot test, the most desirable work activity has been to operate weapons.
Criterion Related Validity Testing
Currently the data collected in the most recent round of testing is also being used to
establish criterion-related validity of the JOIN instrument. As those who completed the
instrument lack prior experience or knowledge of the Navy or Navy ratings, they are an ideal
group to use for establishing predictive validity of the tool. Criterion measures (e.g. A-school
success) will be collected as participants progress through technical training, and those data
become available. Participants’ social-security-numbers (SSN) were collected to link interest
measures to longitudinal data, including the multiple survey 1st Watch source data. Additional
measures will also include attrition prior to End of Active Obligated Service (EAOS), measures
of satisfaction (on the job and in the Navy), propensity to leave the Navy, or desire to re-enlist.
Additionally, JOIN results will be linked with performance criteria.
JOIN MODEL ENHANCEMENT
In addition to the establishment of criterion-related validity, current efforts are focused on
establishing and enhancing the construct validity of the SME model upon which the JOIN
framework exists. As mentioned previously, the tool was developed using the judgments of
enlisted community managers. In addition to decomposing rating descriptions to core elements
and matching photographs with these elements, this group also established the initial scoring
weights, which were limited in the first iteration to unit weights. At present, NPRST researchers
are conducting focus group efforts with Navy detailers, classifiers, and A-school instructors for
the purpose of deriving SME rater determined numerical weights that establish an empirical link
between JOIN components and all existing Navy ratings that are available to first term sailors.
These weights will be utilized in the enhancement of the scoring algorithm that provides an
individual preference score for each Navy rating. A rank ordering (based on preference scores)
of all Navy ratings is provided for each potential recruit.
483
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003