2003 IMTA Proceedings - International Military Testing Association

collection and analysis. Also included was the development of novel approaches for organizing
and representing data.
Following the objective-based training model, training objectives to be measured during the
exercises were identified and developed. In keeping with objective and mission-based
development, the focus remained on objective-based training oriented up to the ship level for the
USN and utilized NMETL-derived measures. Building upon this effort, the RAN objectives
were then aligned with the USN approach for performance measurement. Training objectives
were developed for both procedural and process measurement. Procedural measurement
involves the assessment of individual and team completion of training objectives usually within a
prescribed sequence. Process measurement examines team-level skills necessary for effective
performance (e.g., effective communication).
The development of this new approach for collecting performance data also involved front-end
data organization and analysis software, hand-held software for data collection, and methods for
transferring data between each tool. The front-end data management tool, stored on a PC,
housed the entire population of event scenarios, training objectives, and associated performance
measures. Here, a data collection is be built by linking desired training objectives with event
scenarios. This data collection plan then transfers to the hand-held device for implementation.
Once data has been collected, it is then transferred back into the data management software for
analysis. Output includes summary data regarding performance standards met for the training
objectives (please see Giebenrath, Burns, Hockensmith, Hession, Brewer, and McDonald, 2003
for a detailed description of the methods and tools developed and implemented during the
CReaMS Phase II effort).
Putting it all Together: CReaMS Phase III
In Phase II of the CReaMS Phase effort, a new and novel approach to the development and
implementation of objective-based performance was successfully employed in a coalition
training environment. The use of task sequences provided an effective method for organizing,
storing, and transferring procedurally oriented performance data and while actual data collection
tools differed across sites (laptop computers with Excel-based tools were used in Australia while
the HHD was used for data collection with the USN participants), there was consensus among all
CReaMS participants that the methodology worked well both in terms being “user-friendly” in
data collection and in terms of providing performance data at the right level of detail for exercise
participants.
Building on lessons learned and the successes of Phases I and II, in CReaMS Phase III, a specific
goal was set to conduct a distributed, participative, and facilitated debrief. Once again the
primary participants in the “Virtual Coalition Readiness” (VCR) exercise were the USN
(including the USS HOWARD (DDG-83), and Tactical Training Group Pacific) and the RAN
(including RAN Ship Trainers at HMAS WATSON with watch standers from the ADELAIDE
and the ARUNTA). The VCR exercise was designed to incorporate all of the elements of
CReaMS Phase I and Phase II events with an additional emphasis on the integration of
procedural and process measurement and feedback. While a lengthy description of the CReaMS
Phase III effort is beyond the scope of the current paper, a discussion of key constructs and how
119
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003