SEKE 2012 Proceedings - Knowledge Systems Institute

Therefore, the calculation of RT S under Partial-CATESR can
be given by the following process.
RT S ←
if TS HC ≠ ∅ then RT S HC
else if TS MC ≠ ∅ then RT S MC
else RT S LC
(3)
Full-CATESR is a naive strategy by simply combining three
sub test suites into one test suite. The construction process of
RT S can be defined as follows:
RT S ← RT S HC ∪ RT S MC ∪ RT S LC (4)
IV. EMPIRICAL STUDIES
To evaluate the effectiveness of our CATESR approach, we
perform empirical studies to answer the following research
questions:
RQ1: To what extent can CATESR approach decrease the
size of reduced test suite compared to HGS approach?
RQ2: Can the FDA be kept, when CATESR approach
generates a relatively smaller reduced test suite, comparing
to HGS approach?
RQ3: Can the FDA be weakened due to the partial coverage
of test requirements?
A. Subjects and Experiment Setup
1) Experiment Subjects: We adopt seven small C programs
in Siemens suite and one large-scale C program named space
in our empirical study. The Siemens suite is originally contributed
by Ostrand et al. for a study of the fault detection
abilities of control-flow and data-flow coverage criteria [12],
and has been partially modified by researchers for further
studies. space is a program for interpreting statements written
in some specific array definition language (ADL). Each subject
contains a single correct version and a set of versions with a
single fault.
TABLE II
EXPERIMENT SUBJECTS
Subject # Test Cases # Versons LOC Description
printtok 4130 7 402 Lexical Analyzer
printtok2 4115 10 483 Lexical Analyzer
replace 5542 29 516 Pattern Replacement
schedule 2650 9 299 Priority Scheduler
schedule2 2710 9 297 Priority Scheduler
tcas 1608 40 138 Altitude Separation
totinfo 1052 23 346 Information Measure
space 13585 35 6218 ADL Interpreter
The characteristics of these subjects are summarized in
Table II. Each subject contains a large test pool with at least
1052 test cases, and 13585 at most. Each subject contains
multiple single-faulty versions, with the count between 7 and
40. For subjects in Siemens suite, the number of lines of code
(LOC) ranges from 138 to 516, which is relatively small comparing
to practical programs. Therefore, we introduce space,
which contains 6218 LOC, to verify the scalability of our
CATESR approach. These subjects are available from Subject
Infrastructure Repository (SIR) at University of Nebraska-
Lincoln 2 [4].
2) Experiment Setup: Since there exists randomness in
our approach, we independently perform the experiment 1000
times on each faulty version for each subject. During each
iteration, we firstly generate a test suite by randomly choosing
test cases from the test pool, then we adopt both Partial-
CATESR and Full-CATESR in test suite reduction. To show
the effectiveness of our approach, we also implement HGS
algorithm [7] as a baseline. When randomly generating a test
suite, we firstly determine the size n of test suite. The value
of n is determined by LOC of the subject timing a random
number ranging from 0 to 0.5. Then we randomly choose n
test cases from the test pool and construct TS. Finally, when
test cases in TS cannot cover all feasible requirements covered
by the test pool, we will add additional test cases by randomly
choosing test cases from the remainder of the test pool. If no
test case in TS can detect the fault in this faulty version, we
will discard TS and regenerate the test suite. This experiment
design is motivated by Rothermel et al. [19].
B. Results and Analysis
In this subsection, we analyze the data gathered from our
experiments to answer RQ1, RQ2, and RQ3.
1) Experiment Metrics: In our empirical study, we mainly
focus on two metrics: time cost and FDA.
In real testing scenarios, time cost includes test environment
setup, test case execution, and test result examination. Since
test suite reduction can significantly reduce the time cost of
regression testing, here we only consider the size of test suite
as a metric to measure the time cost. Consequently, we can
adopt the average extent of reduction to original test suite in
percentage over 1000 times independent experiments, given
by the following formula:
TS Reduced = |TS| avg −|RT S| avg
∗ 100 (5)
|TS| avg
To measure the effectiveness of our approach, we also need
to check FDA after test suite reduction. Since each faulty
version contains only one fault, we define an indicator function
g f as follows:
g f (TS)=
{
1 if fault f can be detected by TS,
0 other wise.
Consequently, we can take the average value g f (TS) of
indicator over 1000 experiments as the metric of fault detection
ability, denoted by FDA.
2) Test Suite Size Analysis: To answer RQ1, we analyze the
data, and fetch the size of reduced test suites. The reduction
extent is measured by formula 5 and visualized in Figure 2.
For each subject over all available faulty versions, we mark
three key values to show the reduction extent TS Reduced for
HGS, Partial-CATESR, and Full-CATESR,respectively in the
same vertical drop line.
2 http://sir.unl.edu/php/index.php
(6)
221