2003 IMTA Proceedings - International Military Testing Association

276
we surveyed all of the employees. Whenever the SME panels created a new work role or merged
existing work roles, they were required to identify a suitable sample of employees to complete
the survey. Each survey gathered respondent background information and ratings of the core
competencies. For each competency, a dichotomous (yes/no) format item asked whether the
competency is used. When the respondent indicated “yes,” the survey asked three additional
questions: the competency’s importance to performing the work, the extent to which it is needed
upon hire, and the amount of time that the respondent spends using it. The importance, hire, and
time items used 5-point Likert scales, with anchors ranging from “Strongly Disagree” (1) to
“Strongly Agree” (5). Because the survey was electronically-deployed, validation rules
prevented out-of-range values.
Next, we weighted the response data at two steps. The first weighting was made at the
work role level. The purpose of this first weighting was to adjust the PAB ratings to contribute
50 percent of the final weighted response. For each work role, we identified the overall
importance for each competency by calculating the weighted mean and standard deviation of
ratings for need at hire, frequency, and importance across all the incumbent raters for that work
role and its PAB (with the PAB ratings weighted as previously discussed). After calculating the
weighted overall importance ratings, we sorted the competencies within competency type
(knowledge, skill, cross-occupational skill, and tool) in decreasing order of importance (mean)
and increasing consensus about importance (standard deviation). The second weighting, which
was made at the occupation level, was designed to provide greater weight to competencies that
are identified in a large number of roles. Therefore, we weighted each competency by the
number of work roles in which it appeared. For example, if a competency appeared in 5 work
roles, it was weighted by 5; if a competency appeared in only 1 work role, it was not weighted.
This weighting procedure was done to ensure that the most “critical” competencies were in fact
used by a large percentage of the NIMA workforce.
We then used Chronbach’s alpha to calculate the overall degree of inter-rater reliability
(or consensus) within each work role. Cronbach’s alpha represents the degree to which raters
provided consistent evaluations of the importance of each competency. Cronbach’s alpha is
based on both the number of raters, as well as the similarity of the ratings included in calculating
it. Consequently, low values may also be due either to small sample size alone, low consensus
alone, or both low consensus and small sample size. Values for Cronbach’s alpha range from 0.0
(indicating no consistency) to 1.0 (indicating perfect agreement). Values under 0.50 may suggest
that the raters disagreed about the importance of the competencies they rated, and possibly that
the work role may require additional review. However, low values can also result when the work
role had only a small number of reviewers. Conversely, high values indicate agreement among
the reviewers about the importance of the competencies they rated, suggesting that the work role
more accurately describes the work.
Finally, we used the Jaccard similarity coefficient to indicate the degree to which pairs of
work roles had the same competencies identified as being essential for performing work they
describe. Specifically, we considered values for the Jaccard coefficient greater than 0.80 as
suggesting that further review should be considered. As before, statistical results were combined
with expert judgments to determine whether or not individual pairs of work roles should be
combined. Before the results data could be integrated into PeopleSoft, they needed to be
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003