SEKE 2012 Proceedings - Knowledge Systems Institute

Figure 3. The PF and PTS results for the subject
program
Figure 4. The AP F D and AP F D R results for
the subject program
4.2 Results
First, we see the effectiveness of test case selection. Figure
3 shows the results of the PF and PTS values for the
subject program during its evolution. From the results of
PF, we see that our approach could select the test cases
covering most of the faults in the program. The percentage
of the identified faults can reach 80% above in most cases.
In addition, Figure 3 also shows the percentage of test cases
that were selected for each version of the subject. It illustrates
that the percentage of selected test cases varies widely
from about 30% to 80%, which is similar to the results from
other studies evaluating test case selection on procedural
programs [13]. The results do not show any obvious difference
that is peculiar to object-oriented paradigm. Hence,
from the results, we see that our approach can select a small
set of test cases which can identify most of the faults.
Second, we see the effectiveness of the test case prioritization.
To evaluate this, we manually interrupted the newly
test suite T ′ , and randomly gave another ordering test
suite T ′′
based on T ′ . Then we computed the AP F D value
for T ′′
(AP F D R). Figure 4 shows the AP F D and
AP F D R values using our prioritization approach and the
random prioritization approach, respectively. The data illustrates
that our regression testing prioritization approach
has higher AP F D values than that of random prioritization
approach in all cases. Moreover, in most cases, the AP F D
values of our approach are 10% (or above) higher than that
of random prioritization approach. This shows that our prioritization
approach is more effective compared to the random
prioritization for its better ability of early fault detection.
4.3 Threats to Validity
In this section, we discuss some threats to the validity of
our empirical study. The main external threat is the repre-
sentativeness of our subjects and mutation faults. The subject
program is of small size. Thus we cannot guarantee
that the results can be generalized to other more complex
or arbitrary programs. However, it is a real-world program
widely used in our lab [14]. A second concern is the threat
to construct validity. The goal of test case prioritization is
to maximize some predefined criteria by executing the test
cases in a certain order. Here, our focus is to maximize the
rate of fault detection and we used AP F D to measure it.
However, AP F D is not the only possible measure for fault
detection rate. Some other measures may obtain different
results. Finally, we consider the threat to internal validity.
In our experiment, we utilized mutation faults to study
the test case prioritization. Some other methods to generate
the faults may be better to evaluate our regression testing
approach. However, it is widely used by the academic community
in evaluating test case prioritization [4, 5].
5 Related Work
Anumber of approaches have been studied to facilitate
the regression testing process. Test case selection and prioritization
are two effective techniques to conduct regression
testing.
Test case selection attempts to reduce the size of original
test suite, and focuses on identification of the modified
parts of the program. To date, many regression test
selection techniques have been proposed. Rothermel et al.
proposed a safe and efficient regression test selection technique
based on comparison of differences on the control
flow graph [13]. In addition, regression testing based on
slicing techniques has attracted much attention for a long
time. And Binkley summarized the application of slicing
techniques to support regression testing [1]. Orso et al. also
proposed a similar regression testing approach to ours
in this paper, which used the impact results from the CIA
process to support regression testing [12]. They leveraged
456