SEKE 2012 Proceedings - Knowledge Systems Institute

Evaluating the Cost-Effectiveness of Inspecting the
Requirement Documents: An Empirical Study
Narendar Mandala, Gursimran S. Walia
Department of Computer Science
North Dakota State University
Fargo, ND
{narendar.mandala, gursimran.walia}@ndsu.edu
Abstract—Inspections and testing are two widely recommended
techniques for improving software quality. While inspections are
effective at finding and fixing the faults early in the lifecycle,
there is a lack of empirical evidence regarding the extent to which
the testing costs are reduced by performing the inspections of
early software documents. This research applied the Kusumoto
cost-metric that analyzed the costs and benefits of inspections
versus testing using the fault data from four real requirement
documents that were inspected twice. Another aspect of our
research evaluated the use of Capture Recapture (CR) method to
estimate the faults remaining post inspection to decide the need
for a re-inspection. Our results provide a detailed analysis of the
ability of the CR method to accurately estimate the cost savings
(within +/- 20% of the actual value) after each inspection cycle.
Keywords-cost-effectiveness; inspections; requirements.
I. INTRODUCTION
Successful software organizations strive to deliver high
quality products on time and within budget. To manage
software quality, researchers and practitioners have devoted
considerable effort to help developers find and fix faults at the
early stages of the lifecycle (i.e., in requirements and design
documents) and reduce its impact in the later stages [1-3]. In
software engineering research, software inspections have been
empirically validated to improve quality by helping developers
find and fix faults early and avoid costly rework [1-3, 6, 11].
Similar to inspections, testing is also widely recommended
technique for improving software quality. While both are
effective fault detection techniques, testing cannot be
conducted until software has been implemented, whereas
inspections can be applied immediately after software
documents have been created. Empirical evidence suggests that
a majority of the development effort is spent during the testing
stage of project [6, 11, 12]. Furthermore, it is estimated that 40-
50% of the testing effort is spent on fixing problems that
should have been fixed during the early stages of the
development [15, 17, 22]. However, there is little empirical
evidence regarding the rework cost-savings that can be
achieved by performing the inspections of early work products.
Therefore, an empirical evaluation of the costs and benefits
of inspections (against the testing cost that would be spent if no
inspections are performed) can help manage the software
quality. To evaluate the costs and benefits of inspections and
testing, this research reports the results from the application of
Kusumoto cost-metric [13] that analyzed the testing costs that
were reduced by performing inspections of software
requirement documents immediately after its development.
Furthermore, the kusumoto metric calculation requires
information regarding the total fault count of the document and
the faults remaining post-inspection. However, inspections only
provide the information about the faults that are found during
an inspection. During software development, project managers
need reliable estimates of the remaining faults after an
inspection to help decide whether to invest in a re-inspection or
to pass the document to the next phase. To that end, our prior
research has shown that, among the different approaches that
are available for estimating the total number of faults in the
artifact (e.g., defect density, subjective assessment, historical
data, capture-recapture, curve-fitting), capture-recapture (CR)
method is the most appropriate and objective approach [19].
Capture-recapture (CR) is a statistical method originally
developed by biologists to estimate the size of wildlife
populations. To use CR, a biologist captures a fixed number of
animals, marks them, and releases them back into the
population. Then another trapping occasion occurs. If an
animal that was ‘marked’ during the first trapping is caught
again, it is said to have been recaptured. The process of
trapping and marking can be repeated multiple times. The size
of the population is then estimated using: 1) the total number of
unique animals captured across all trappings, and 2) the number
of animals that were re-captured [7-8, 22]. Using the same
principle, the CR method can be used during the inspection
process to estimate the number of faults in an artifact. During
an inspection, each inspector finds (or captures) some faults. If
the same fault is found by more than one inspector it has been
re-captured [5, 10, 16, 19]. The total number of faults is
estimated using the same process as in wildlife research, except
that the animals are replaced by faults and the trappings are
replaced by inspectors. The inspection team can use the
estimate of the total fault count along with the number of faults
already detected to estimate the number of faults remaining in
the artifact post inspection.
This paper reports a comprehensive evaluation of Kusumoto
metric in conjunction with the CR method using inspection
data from real artifacts that contain natural faults made during
the development. The artifacts were inspected two times, and
we compared the cost-effectiveness after each inspection cycle
45