SEKE 2012 Proceedings - Knowledge Systems Institute

Data
Set
TABLE II. DATA SETS
Artifact Name Description # of
Inspectors
1 st
inspection
Faults
2 nd
Inspection
Faults
1 Starkville Theatre System Management of ticket sales and seat assignments for the theatre 8 30 25 55
2 Management of Apartment Managing apartment and town property, assignment of tenants, rent 8 41 64 105
and Town properties
collection, and locating property by potential renters
3 Conference Management Helping the conference chair to manage paper submission,
6 52 42 94
4 Conference Management notification to authors, and other related responsibilities
6 64 54 118
Total
Faults
B. Data Set
The data was drawn from earlier inspection studies
conducted at Mississippi State University (MSU). The original
goal of these studies was to investigate how the use of error
information impacted requirements inspections [20]. Only the
information relevant to our research analysis is presented here:
1) Software Artifacts and Software Inspectors: Inspection
data from 4 artifacts used in this study were developed by
senior-level students enrolled in the Software Engineering
Design Course at MSU during two different years. The course
required student teams to interact with real customers to elicit
and document requirements that they would later implement.
So, even though the developers were students, the artifacts are
realistic for a small project. The subjects were divided into 4
teams (with 8, 8, 6, and 6 students respectively) that developed
the requirement documents for their respective systems as
shown in Table II.
2) Software Inspection Process: Each requirement
document was inspected twice by the same subjects who
created it. During the first inspection, the subjects received
training on the use of a fault checklist. Then, each inspector
individually inspected the requirements using the fault
checklist and logged any faults identified. After the first
inspection, the participants were trained on how to abstract
errors from faults, how to classify the errors, and how to use
the errors to re-inspect the requirements document. Then, each
inspector re-inspected the requirements using the errors to find
the additional faults. The artifacts were not modified or
corrected between inspections (i.e., the same document was
re-inspected). Therefore, we collected inspection data from the
first inspection, the second inspection, and the total for each
artifact. Note that the last three columns in Table II show
faults found during the first inspection, during the second
inspection, and total for all the 4 artifacts. Also, we collected
the time spent (in hours) by each subject during the first and
second inspection cycle and the total time for each artifact.
C. Evaluation Procedure
The following costs and savings for each artifact were
computed after the first and second inspection cycle:
Average cost to detect a fault in inspection (c r ): Adding all
the faults found by the inspectors, the average number of
faults found by an inspector is calculated. From the
available values of time taken by each inspector and the
average number of faults found by an inspector, c r is
calculated as: c r = C r / D r
o
o
The “C r - Inspection cost”, is calculated by adding the
time spent by all the inspectors during the inspection.
The “D r ”, is the total number of unique faults found
by all the inspectors during an inspection cycle.
Virtual Testing Cost (C vt ): is calculated as the product of
the average cost to defect a fault in testing (i.e., c t ) and the
total number of faults present in the product (i.e., D total ).
o The “c t - Average cost to detect a fault in testing”, is
calculated as 6 times of average cost to detect a fault
during the inspection (c r ).
o The “D total - Total fault count”, is the total number of
faults present in the document.
Cost saved from inspection (∆C t ): The testing cost saved
from inspection is the product of the number of unique
faults found during the inspection (D r ) and the average
cost to find a fault during the testing (c t ). i.e., ∆C t =Dr *c t .
The difference in the testing cost saved by the inspection
and the cost spent on the inspection provides the reduction of
the total costs. M k value is then obtained as follows:
M k = (∆C t - C r ) / C vt
This procedure was executed to calculate: (1) M k using the
actual fault count and (2) M k using the CR estimates after each
inspection for all the 4 artifacts. After the first inspection, the
actual and estimated M k values were obtained as follows:
1) M k based on the actual fault count: Because we do not
know the actual number of faults, the total number of
exclusive faults found after both inspections is assumed to be
the actual fault count for the purposes of this study. Therefore,
for each artifact, the faults found during the first inspection
(column 5 of Table II) by all the inspectors and the actual fault
count (column 7 of Table II) were used to calculate the actual
M k value of the first inspection cycle.
2) M k based on the CR estimates: For each artifact, the
faults found during the first inspection by all the inspectors
(column 5) are inserted into a matrix (as described in Section
II.C). The matrix is then fed to the automated tool (CARE-2
[8]) to produce the estimates from all the CR estimators. Using
the estimated fault count and the number of faults found
during the first inspection, the M k values were calculated.
Because the artifacts were unchanged between inspections,
to calculate the cost-effectiveness after second inspection, we
use the total faults found from both inspections (Column 7).
Since we are calculating the testing cost saved by performing
the inspections, it did not make sense to use only the data from
second inspection. Furthermore, using only the data from
48