2003 IMTA Proceedings - International Military Testing Association

526
• Individual Pace & Intensity
The Army-wide FX scales will incorporate descriptions of the anticipated future
conditions listed above. Raters will read the description and rate how effectively they think the
Soldier would perform under each condition. They will also provide a rating of how confident
they are in those ratings. The cluster-specific scales will be based on scenarios much like those
generated for the SJT-X in the NCO21 project (Knapp et al., 2002). These scenarios are more
detailed and are specific to the job cluster. Again, raters will indicate how well they think the
Soldiers would perform in that scenario. We anticipate developing 5-6 scenarios for the Close
Combat cluster, where the anticipated future conditions are much the same for the three MOS,
and 10-11 scenarios for the Surveillance, Intelligence, and Communication cluster, where there
is less overlap among the MOS. Some of these scenarios may be only applicable to one or two
MOS in the cluster. This process will not be as good as addressing independent themes or
constructs, but it will do well at sampling the relevant content for the MOS.
We will collect separate ratings for the Army-wide and cluster-specific FX scales and
conduct statistical analyses to determine whether there is dimensionality in the ratings. However,
because (a) each scenario is likely to involve multiple “dimensions” of performance and (b) a
single dimension of performance is likely to be relevant for more than one scenario, we believe
that dimensionality is very unlikely here. Therefore, for both Army-wide and cluster FX ratings,
we plan to aggregate the ratings into an overall Army-wide rating and an overall cluster rating,
respectively.
Rater Training/Process
Effective rater training is key to getting raters to use the rating scales as intended. We
have considerable experience in developing rater training that has focused on evaluation errors
(e.g., first impression, stereotyping) and response tendency errors (e.g., halo effect, central
tendency). This experience has shown that reducing or eliminating rating error is quite difficult.
Our goal with Select21 rater training is to more clearly focus raters on reading and using
the scales accurately. For all raters, training will emphasize the importance of making accurate
ratings and thinking about a Soldier’s relative strengths and weaknesses. To this end, we will
stress the importance of accurate performance measures to the overall success of the project. In
past work we have found that stressing the fact that the ratings are “for research purposes only”
helps to overcome problems, such as leniency, that are common in operational performance
ratings. We will focus on the importance of reading the anchors, thinking about a Soldier’s
relative strengths and weaknesses, and applying that insight to the ratings. The ranking exercise
described previously should help with this focus, as will the format of the rating scales. We will
address response tendency and evaluation errors. In addition, while raters are working, we will
have facilitators move about the room to keep an eye out for raters who seem to be falling prey
to these visible errors.
We expect that we will collect ratings from most of the raters in a face-to-face setting.
We also expect that a fairly large proportion of raters will not be available during the data
collection period. Identifying raters and collecting their ratings has been a challenge in past
efforts such as this, and we expect that we will encounter the same situation in this project. It is
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003