SEKE 2012 Proceedings - Knowledge Systems Institute

are implemented on Weka 3.6.1 by using its default setting
since this experiment is designed to answer the three research
questions.
The second independent variable TrainingSet refers to
the number of instances (i.e., executions) in a training set.
As the number of available training instances may be associated
with the scale of a program, we use the ratio between
the number of training instances and the number of executable
statements to represent TrainingSet, which is set to
be 20%, 40%, 50%, 60%, 80%, and 100% in our empirical
study.
The third independent variable Attribute refers to which
type of execution data has been collected during the execution
of the program and used to classify the execution
data. In our empirical study, the values of variable
Attribute can be any of the following: statement count
(abbreviated as #statements, which is the number of times
each statement has been executed), statement coverage (abbreviated
as ?statement, which is whether each statement
has been executed), method count (abbreviated as #method,
which is the number of times each method has been executed),
method coverage (abbreviated as ?method, which is
whether each method has been executed), and branch coverage
(abbreviated as ?branch, which is whether each branch
has been executed). To record the execution information
on branches when running the program, we used “GCOV”
command of GCC, whose collected branches are predicates
within a branch condition, not the whole branch condition.
Moreover, the predicates whose value is true or false are
taken as different branches.
The dependent variables in our empirical study are the
results of execution-data classification measured by ROC
analysis [4] since it has advantages overs other measures
like precision-recall and accuracy. Specifically, our study
uses the “Area under a ROC curve (usually abbreviated as
AUC)” to measure the performance of an execution-data
classification approach. The AUC is always between 0 and
1. The bigger the AUC is, the higher performance the corresponding
classification approach has. Moreover, a realistic
classifier is always no less than 0.5 since the AUC for the
random guessing curve is 0.5.
3.2. Process
First, we ran each subject with its test cases by recording
the five types of execution data as well as the outcomeofan
execution (i.e., whether it is passing or failing).
Second, we took all the test cases with their execution
information (i.e., execution data and outcome) as instances
and split all the instances into a training set and a testing
set. Although all the test cases are labeled with either passing
or failing, our experiment randomly selected some test
cases into a training set, and took the rest test cases as a
testing set. The training set is the set of test cases whose
execution data and outcome are taken as input to build a
classifier, whereas the testing set is the set of test cases that
are used to verify whether the classified outcome based on
the classifier is correct. Since all the test cases are known to
be passing or failing in our empirical study before building
the classifier, we know whether the classification is correct
or not by comparing the classification result with its actual
outcome.
Moreover, for each faulty program, our empirical study
randomly selected n ∗ 20%, n ∗ 40%, n ∗ 50%, n ∗ 60%,
n ∗ 80%, orn test cases from its whole test collection as
a training set and took the rest test cases as a testing set,
where n is the number of executable statements for each
subject shown by Table 1. To reduce the influence of random
selection, we repeated each test-selection process 100
times.
Third, we applied each of the three machine-learning algorithms
to each training set and recorded the classified outcome
of test cases in its corresponding testing set. As the
outcome of each test case is known, we know whether the
classification is correct. We recorded the number of passing
test cases that have been correctly classified to be passing,
the number of passing test cases that have been classified to
be failing, the number of failing test cases that have been
correctly classified to be failing, and the number of failing
test cases that have been classified to be passing. Then
for each faulty program with a machine-learning algorithm,
we calculated its corresponding AUC. As each subject has
several faulty versions, we calculated their average AUC as
the result of the corresponding subject. Our experiment is
performed on an Intel E7300 Dual-Core Processor 2.66GHz
with 2G memory.
3.3. Threats to Validity
The construct threat of our empirical study lies in the
measures, although ROC analysis has been widely used in
evaluating classifiers in machine learning because it has advantages
over the accuracy, error rate, precision, recall and
can decouple classifier performance from class skew and
error costs [4]. The main external threat comes from the
subjects and the seeded faults. Although these subjects including
the faults have been widely used in the literature of
software testing and analysis, we will perform more experiments
on larger programs with real faults.
3.4. Results and Analysis
In our empirical study we exclude the faulty programs
whose corresponding test collection has less than 5% failing
test cases because its percentage of correct classification
would be larger than 95% if the test cases are always classified
to be “passing”. After excluding such biased data,
285