Source Code Analysis Laboratory (SCALe) for Energy ... - CERT

• individual consistency (for example, how consistent is the individual in rendering the same
judgment across time for the same or virtually the same flagged nonconformity; often referred
to as repeatability)
• group accuracy (for example, what percentage of the time does a specific group of SCALe
analysts render the correct judgment)
• group consistency (for example, what percentage of the time does a specific group of SCALe
analysts render the same judgment for a given flagged nonconformity across time; often referred
to as reproducibility)
Any modern statistical package can easily determine the attribute agreement measures, which are
interpreted as follows [Landis 1977]:
• Less than 0 (no agreement)
• 0–0.20 (slight agreement)
• 0.21–0.40 (fair agreement)
• 0.41–0.60 (moderate agreement)
• 0.61–0.80 (substantial agreement)
• 0.81–1 (almost perfect agreement)
A need may arise to assess both accuracy and consistency of SCALe analysts’ judgments with
measures that extend beyond the binary situation (correct or incorrect) to situations in which a
judgment is a gradual measure of closeness to the right answer. In this case, analysts should use
an alternative attribute agreement measure and interpret the output quite similarly to the Kappa
coefficient, with results possible on the dimensions listed above. As such, Kendall coefficients
serve well for judgments on an ordinal scale, in which incorrect answers are closer or farther away
from the correct answer. A hypothetical example of a judgment on an ordinal scale would be if a
SCALe analyst were asked to render judgment of the severity of a flagged nonconformity, say on
a 10-point scale. If the true severity is, for example, 8, and two SCALe analysts provided answers
of 1 and 7, respectively, then a severity judgment of 7 would have a much higher Kendall coefficient
than the severity judgment of 1.
In conclusion, attribute agreement analysis may be conducted via small exercises with SCALe
analysts rendering judgments on a reasonably-sized list of different types of flagged nonconformities
mapped to the set of rules within the scope of a given code conformance test.
2.7.2.2 Formative Evaluation Assessment Using Attribute Agreement Analysis
SCALe analysts participate in a formative evaluation assessment as part of their training and certification.
Certification of a candidate as a SCALe analyst requires attribute agreement scores of
80 percent or higher. In addition, acceptable thresholds for accuracy may be imposed separately
for each rule.
The formative evaluation assessment implements a simple attribute agreement analysis as follows.
First, the SCALe assessment administrator identifies a preliminary set of 20 to 35 different
flagged nonconformities (from the diagnostic output of a software system or code base) for the
evaluation assessment. The administrator ensures that the preliminary set includes a variety of
CMU/SEI-2010-TR-021 | 22