2003 IMTA Proceedings - International Military Testing Association

performance domains at the same level of specificity (e.g., Sackett et al., 1988). Each of these
two items was rated on a five point Likert scale, ranging from 1 to 5, where higher scores reflect
higher efficiency or quality of team actions. As these two scores were highly correlated (r = .72),
we decided to average them to form a composite team performance score. Given that the ICC (1)
is .35 (p < .05) and ICC(2) = .85, we averaged the scores across raters to form an overall typical
team performance rating for each team.
Team Performance in Maximum Contexts. These ratings were collected during a one-day
assessment center conducted at the end of the team training to evaluate the combat proficiency of
the team. One external assessor was randomly assigned to evaluate the performance of the team
over a series of six military tasks (e.g., the team may be tasked by HQ to evacuate a casualty
from one place to another); these tasks comprehensively summarized the types of tasks
performed as part of the team training. The assessor used a 5 point Likert scale to evaluate the
efficiency and the quality of team actions on each of the tasks. As there was only one assessor
per team, inter-rater reliability was not available; however, the inter-task reliability for the
efficiency measure was .90 while the inter-task reliability for the quality of team actions was .87.
As with the ratings collected under the typical performance context, both of these measures were
highly correlated (r = .67, p < .01) and so a composite team performance measure was created.
Content Equivalence of the Performance Ratings. To ensure the content and specificity
of the performance measures was sufficiently similar across the two performance contexts, we
asked 10 subject matter experts (SMEs) with extensive experience in training and evaluating this
type of combat team to respond to a 10 item survey. The survey sought their opinions about the
overlap of the performance domains being assessed by the supervisors and the assessors under
the two performance conditions. As shown in Table 1, the responses from these SMEs
demonstrate the performance domains assessed by the supervisors and the assessors were highly
similar in terms of content. The mean response from these SMEs on the 10-item survey is 5.2 on
a six-point scale, indicating these raters believed the overlap between the two performance
measures was present to at least “a great extent.” This information, coupled with the fact that
raters were instructed to only consider team behaviors associated with the content of the training,
and that the same performance dimensions and response scales were used for both conditions,
suggests that the performance ratings assessed the same performance domain with the same
degree of specificity. Thus, the ratings obtained under the two contexts differ in terms of the
performance demands placed on the team (typical or maximum), and perhaps also the knowledge
raters had about the teams and leaders (a point we return to in the Limitations section).
Results
Power Analysis and Data
In contrast to research at the individual level of analysis, the difficulty of collecting data
from large samples of intact teams usually results in smaller sample sizes. For instance, Liden,
Wayne, Judge, Sparrowe, Kraimer, and Franz (1999) analyzed 41 workgroups while Marks,
Sabella, Burke, and Zaccaro (2002) analyzed 45 teams. Indeed, Cohen and Bailey (1997) report
the average number of teams in project team research (the type of teams most similar to those in
the present study) averages only 45 per study. Such is the case in this study, making a careful
consideration of power, p-values, and effect size important.
A power analysis shows that given a sample size of 39 teams, there is only a 59% chance
of detecting moderate effects at p < .05 (one-tailed) (Cohen, 1988). However, with a one-tailed
test and p < .10, power becomes 73%. Hence, we considered values with p < .10 (one-tailed) to
be statistically significant instead of p < .05 (one-tailed) because of the low statistical power
639
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003