SEKE 2012 Proceedings - Knowledge Systems Institute

TABLE I. CAPTURE-RECAPTURE MODELS AND ESTIMATORS [8, 10, 16, 18, 19, 22]
Model Variation Source Estimators Belonging to Each CR Model
M o
All inspectors have the same ability, and all defects are equally likely of being Unconditional Likelihood (M o-UMLE); Conditional
detected.
Likelihood (M o-CMLE); Estimating Equations (M o-EE)
M t Inspectors differ in their abilities, but all defects are equally likely of being found. M o-UMLE; M o-CMLE; M o-EE
M h Inspectors have the same ability, but defects differ in their probability of being found. Jackknife (M h-JK); Sample Coverage (M h-SC); M h-EE
M th Inspectors differ in their ability, and defects differ in their probability of being found. M h-SC; M h-EE
it can be determined if two people report the same fault).
However, because some faults are easier to find than others and
because inspectors have different fault detection abilities, the
equal capture probability assumption is not met [18].
To accommodate these assumptions, 4 different CR models
are built around the 2 sources of variation: Inspector Capability
and Fault Detection Probability. Table I shows the four CR
models along with their source(s) of variation. Each CR model
in Table I has a set of estimators, which use different statistical
approaches to produce the estimates. The estimators for each
CR model are also shown in Table I. The mathematical details
of CR estimators are beyond the scope of this paper [18, 22].
The input data used by all the CR estimators is organized as a
matrix with rows that represent faults and columns that
represent inspectors. A matrix entry is 1 if the fault is found by
the inspector and 0 otherwise.
CR was introduced to software inspections by Eick, et al.
by applying it to real fault data from AT&T. A major result
from this study was the recommendation that an artifact should
be re-inspected if more than 20% of the total faults remain
undetected [5, 10]. Following this study, various empirical
studies in SE have evaluated the use of CR models to
accurately estimate the total fault count using artifacts with a
known number of seeded defects [18]. In addition, our prior
research evaluated the effect of inspection team size on the
quality of the CR estimates [19]. A common finding across all
the studies is that the estimators underestimate the actual fault
count with small number of inspectors, but improve with more
faults and inspectors. While there is evidence on the ability of
the CR estimators to accurately estimate the total fault count
[16], the CR research has neglected the cost spent and cost
saved by an inspection. This research extends our prior work
by evaluating the cost-effectiveness of software inspections
using the CR estimates on real artifacts. These results will
provide guidance on whether the CR estimators can be used to
evaluate the cost-effectiveness of an inspection process in real
software project where the fault count is unknown beforehand.
III. LITERATURE ON INSPECTION VS. TESTING COST RATIO
As mentioned earlier (section II.A), calculating the cost
saved by the inspection requires a count of faults remaining
post-inspection, and the average cost to detect a fault in the
testing stage. While the CR method can be used to estimate the
remaining faults, the average cost to detect a fault in the testing
stage is not available at the end of inspection cycle. In this
research, the average cost (time spent in hours) to find a fault
during the testing, is calculated as a factor of the average cost
(time spent in hours) to find a fault during the inspection. This
section presents a summary of findings from different software
organizations that reported the data on the cost (staff-hours)
spent to find a fault during the inspections versus testing. This
cost ratio is then used to calculate the cost-savings, the virtual
testing cost, and the M k value of the inspections using (1).
Major results regarding the cost spent (in staff hours) to
find a fault during inspections versus the cost spent to find a
fault during testing show a cost ratio of 1:6 [2, 4, 12, 17, 22].
These values are based on actual reported data. Also, Briand
[6], based on the published data has provided probability
distribution parameters for the average effort using different
fault detection techniques according to which the most likely
value for design inspections are 1.58 hours per fault and, for
testing are 6 hours per fault. These values were derived from
various studies on the cost of finding faults in design, code
reviews and testing. Different studies in the literature give
different estimates because of the differences in study settings,
software processes, severity of the faults, review techniques
and other factors. In order to find the most appropriate value,
we computed the median of the reported cost ratio values
resulting from precise data collection. We did not consider
approximations, estimates, or data whose origins were unclear.
As a result, the median cost ratio is 1:5.93. Therefore, for this
research, the inspection to testing cost ratio of 1:6 was used to
calculate the cost-effectiveness of the inspection process.
IV. STUDY DESIGN
Prior software engineering research has validated the fault
detection effectiveness of inspections at the early stages of
development [1,11]. However, there is a lack on empirical
research on the benefits of inspections in terms of the extent to
which the testing costs can be reduced by performing the
inspections of early software artifacts. Furthermore, while
inspections are effective; they cannot provide insights into the
remaining faults which are required to calculate the fault
rework savings. On that end, the CR method has been
evaluated to provide a reliable estimate of the faults remaining
post-inspection using artifacts with seeded faults [16].
However, there is a lack of research on the CR estimator’s
ability to provide a reliable estimate of the remaining faults
when the actual fault count of the software artifact in not
known beforehand. Therefore, this paper evaluates the ability
of the CR estimator’s to accurately predict the costeffectiveness
of the inspection process using the fault data from
4 real software artifacts that contained naturally occurring
faults. In addition, each artifact was inspected twice, which
allowed the analysis of the CR estimator’s ability to accurately
estimate the cost-effectiveness after each inspection cycle.
A. Research Goal
The main goal of this study is to evaluate the ability of CR
estimators to provide an accurate cost-effectiveness value of an
inspection process by comparing the M k values based on the
CR estimates against the actual M k value after each inspection.
47