2003 IMTA Proceedings - International Military Testing Association

As a result, we are currently developing a performance measurement authoring tool (PMAT) that
will be delivered to the Navy. Based on the positive feedback obtained from workshop
participants, the tool will include the same 7-step framework that was presented in the
workshops. In addition, the authoring tool will include a great deal of guidance (i.e., definitions,
background information, tips) and probing questions in order for instructors of varying levels of
experience to benefit from the tool. Specifically, the tool will be programmed to incorporate a
wizard (i.e., tutor) that will guide the user through the 7-step framework just as the team of the
facilitators did in the afternoon portion of the workshops. Studies are currently underway to test
the decision rules that will be programmed into the authoring tool. Once completed, a prototype
of the tool will be developed and a series of usability tests will be conducted. Finally, the
effectiveness of PMAT will be tested by asking two groups of instructors to develop measures of
human performance. One group of instructors will develop these measures on their own. The
second group will develop their measures with the assistance of PMAT. Participants will then
use the measures that they have developed to assess performance during a scenario-based
training exercise. System effectiveness will be demonstrated by comparing the reliability and
validity of ratings developed with and without the PMAT system. Additionally, the amount of
time that it takes both groups to develop their measures will also be examined.
REFERENCES
Fowlkes, J. E., Lane, N. E., Salas, E., Franz, T., & Oser, R. (1994). Improving the
measurement of team performance: The targets methodology. Military Psychology, 6, 47-61.
Goodman, P. S., & Garber, S. (1988). Absenteeism and accidents in a dangerous
environment: Empirical analysis of underground coal mines. Journal of Applied Psychology,
73, 81-86.
Hollenbeck, J. R., Ilgen, D. R., Tuttle, D. B., & Sego, D. J. (1995). Team performance
on monitoring tasks: An examination of decision errors in contexts requiring sustained attention.
Journal of Applied Psychology, 80(6), 685-696.
Joslyn, S., & Hunt, E. (1998). Evaluating individual differences in response to timepressure
situations. Journal of Experimental Psychology: Applied, 4(1), 16-43.
Marks, M. A., Zaccaro, S. J., & Mathieu, J. E. (2000). Performance implications of
leader briefings and team-interaction training for team adaptation to novel environments.
Journal of Applied Psychology, 85(6), 971-986.
Salas, E., Fowlkes, J. E., Stout, R. J., Milanovich, D. M., & Prince, C. (1999). Does
CRM improve teamwork skills in the cockpit? Two evaluation studies. Human Factors, 41(2),
326-343.
Smith-Jentsch, K. A., Johnston, J. H., & Payne, S. C. (1998). Measuring team-related
expertise in complex environments. In J.A. Cannon-Bowers & E. Salas (Eds.), Making decisions
under stress: Implications for individual and team training (pp. 61-87). Washington, DC:
American Psychological Association.
Stout, R. J., Cannon-Bowers, J. A., Salas, E., & Milanovich, D. M. (1999). Planning,
shared understanding, and coordinated performance: An empirical link is established. Human
Factors, 41(1), 61-71.
693
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003