SEKE 2012 Proceedings - Knowledge Systems Institute

using the CR estimates against the actual fault count. We also
analyzed the ability of the CR method to make a cost-effective
decision post-inspection. The rest of the paper is structured as
follows: Section II describes the inspection cost model and
metrics, the basic principles of CR models in software
inspections. Section III describes the literature review used for
the evaluation study. Section IV describes the design of the
evaluation study. Section V reports the results. Section VI
discusses the threats to validity. Section VII summarizes the
results. Section VIII contains the conclusions and future work
II. BACKGROUND
This section provides information regarding the inspection
costs and savings, different cost-metrics, the basic principles of
the use of CR models in inspections, and a summary of the
empirical studies related to cost-effectiveness of inspections.
A. Inspection Cost Model
The traditional software inspection cost model [13] consists
of the following components. The process of calculating these
components after the inspection is discussed as follows:
D r - number of unique faults found during the inspection;
is determined by comparing the faults found by all the
inspectors.
D - total number of faults present in the product; is not
total
available during the development. In this research, the
overlap in the faults found by multiple inspectors during
the inspection is used to estimate the total fault count using
the CR estimators.
C r – cost spent on an inspection; is measure of the time
taken (in hours) to review the software artifact. In this
research, we only consider the cost invested during the
“individual review/preparation” stage of the inspection
process and is calculated by adding the time taken by each
inspector during the inspection.
C t – cost spent to detect remaining faults in testing; is the
cost required to detect the faults remaining post inspection.
If we consider c t as the average cost to detect a fault in the
testing stage, then C t can be measured as the product of
total number of faults remaining post-inspection (D total –
D r ) and the average cost to detect a fault during testing (c t ).
o c t , – Average cost to detect a fault in testing, is not
available after the inspection. Only the average cost to
detect a fault in inspection is available. Therefore, a
cost ratio of 1:6 (i.e., time spent to find a fault during
the testing is six times the time spent to find a fault
during the inspection) was used to calculate the
average cost to detect a fault in testing. This cost ratio
is derived from literature and described in Section III.
∆ C t – testing cost saved by the inspection. By spending
cost C r during the inspection, cost ∆C t is being saved
during the testing. It is calculated as the product of number
of faults found during an inspection (D r ) and the average
cost to detect a fault in testing (c t ). That is, ∆C t = Dr * c t
C vt – Virtual testing cost, (i.e., testing cost expended if no
inspections are performed) is the sum of the testing cost
required to detect the faults remaining post-inspection (C t )
and the testing cost saved by the inspection (∆C t ). That is,
C vt = (C t + ∆C t ).
B. Software Inspection Cost-Metrics
Meyer [15] and Fagan [11] have used a subset of the
components listed in II.A to propose metrics for evaluating the
effectiveness (number of faults) and efficiency (faults per hour)
of inspections. Their research has neglected the cost spent and
cost saved by the inspections. In this paper, only the inspection
metrics that considered cost factors are discussed.
Collofello’s Metric (M c ): Collofello et al., [9] defined a
cost-effectiveness metric as the ratio of the cost saved by
inspections (∆C t ) to the cost consumed by inspections (C r ).
Although M c considers the cost factors, it does not take into
account the total cost to detect all the faults in the software
work product by inspections and testing. As such, we cannot
compare the M c values across different projects as shown using
an example: Suppose two projects involve inspections and
testing and that in both projects, if inspections had not been
performed, the cost of testing would be 1000 units. The first
project consumes 10 units for their inspections and saves 100
units. Thus, the total cost for fault detection is 910 units. In the
second project, inspections cost 60 units and save 600 units.
Thus, the total cost for fault detection is 460 units, which is far
smaller than the cost in the first project of 910 units. However,
the value of M c in both projects is 10, which doesn’t recognize
the benefit of inspections in the second project. The Kusumoto
metric [13] overcame this problem as discussed below.
Kusumoto Metric (M k ): Kusumoto et al., [13] defined the
cost effectiveness of an inspection in terms of the reduction of
cost to detect and remove all faults from the software product.
M k is calculated as a ratio of the reduction of the total costs to
detect and remove all faults from the software product using
inspections to the virtual testing cost (i.e., testing cost if no
inspection is executed). The M k normalizes the savings by the
potential fault cost. Hence, it can be compared across different
projects. M k is intuitive and appropriate for this research as it
can be interpreted in terms of fault rework savings due to
inspections. Formally stated, the testing cost is reduced (by ∆C t
- C r ) by performing inspections as compared to the testing cost
(Ct + ΔCt) if no inspection is executed. Therefore, the M k can
be derived as: M k = (∆C t - C r ) / (C t + ∆C t ) (1)
C. Using Capture Recapture (CR) in Software Inspections
As mentioned in Section II.A, this research uses the CR
method to estimate the total fault count, which is then used to
estimate the faults remaining post-inspection. Using these fault
estimates, the M k value is then computed using (1).
The use of CR method in biology makes certain
assumptions that do not always hold for software inspections.
The assumptions made by CR in biology include: 1) closed
population (i.e. no animal can enter or leave), 2) equal capture
probability (i.e. all animals have an equal chance of being
captured), and 3) marks are not lost (i.e. an animal that has
been captured can be identified) [8, 22]. When using the CR in
inspections, the closed population assumption is met (i.e., all
inspectors review the same artifact independently and it is not
modified) and the assumption that marks are not lost is met (i.e.
46