SEKE 2012 Proceedings - Knowledge Systems Institute

of execution-data classification. Third, their study investigated
execution data only in the form of counts for methods,
statements and so on, but did not investigate execution data
in the form of coverage, which is widely used in practice.
In this paper, we report an empirical study that evaluates
the impact of these three factors on the effectiveness
of execution-data classification. The aim of our empirical
study is to provide evidence on the determination of
the three important factors in practice. For the first factor,
we investigate three popular machine-learning algorithms:
the Random Forest algorithm (RF), the Sequential Minimal
Optimization algorithm(SMO), and the Naive Bayes algorithm(NB).
For the second factor, we investigate training
sets of different sizes. For the third factor, we investigate
five types of execution data: the statement count, the statement
coverage, the method count, the method coverage, and
the branch coverage. The main contributions of this paper
are as follows. First, to our knowledge, our study is
the first empirical study on the impact of the three factors
in execution-data classification based on machine learning.
Second, our study provides guidelines for the application
of execution-data classification based on machine learning
regarding the choice of machine-learning algorithms, the
number of training instances, and the type of execution data.
2. Execution-Data Classification Based on Machine
Learning
Haran and colleagues [5] proposed to collect execution
data by instrumenting instances of a released software system
and then classify passing executions and failing executions
based on these collected execution data using the Random
Forest algorithm, which is an implementation of treebased
classification. This technique can be viewed as an
application of a classification algorithm in machine learning
for execution-data classification. In the following, we generalize
Haran and colleagues’ technique to execution-data
classification based on machine learning.
For the ease of presentation, let us consider the execution
data based on the statement count. For a program consisting
of n statements (denoted as ST={st 1 ,st 2 , ..., st n }), the
execution data of a test case (denoted as t) is represented as
a series of numbers, each of which is the number of times a
statement is executed when using t to test the program. All
the test cases (whose execution data are already obtained)
are divided into two sets: the training set and the testing
set 1 . Each test case in the training set is also associated
with a label that indicates whether the test case is a passing
one or a failing one. For test cases in the testing set, it is
unknown whether each such test case is a passing one or a
failing one. The problem of execution-data classification is
1 The training set and the testing set are two terminologies in machine
learning. Note that the testing set is not related to software testing.
Table 1. Subjects
Program Ver. Test TrainingSet LOC(Exe) Met. Bra.
print tokens 7 4,072 41∼203 565(203) 18 35
print tokens2 10 4,057 41∼203 510(203) 18 70
replace 32 5,542 58∼289 563(289) 21 61
schedule 9 2,627 33∼162 412(162) 18 25
schedule2 10 2,683 29∼144 307(144) 16 31
tcas 41 1,592 14∼67 173(67) 9 16
tot info 23 1,026 14∼136 406(136) 7 36
Space 38 13,585 1,244∼6,218 9,564(6218) 136 530
to use the training set to predict whether each test case in
the testing set is a passing one or a failing one.
3. Experimental Study
In this experiment, we plan to evaluate the impact of
three factors in execution-data classification so as to answer
the following three research questions. RQ1— which
machine-learning algorithm produces the best results in
execution-data classification? RQ2— what is the minimal
number of training instances that are needed to produce
good enough results in execution-data classification?
RQ3— which type of execution-data produces the best results
in execution-data classification?
Table 1 presents the details of the eight C subjects used in
this experiment, whose source code and test collections are
available from Software-artifact Infrastructure Repository
(http://sir.unl.edu/portal/index.php) [3]. Specifically, the table
gives the name of each subject, number of faulty versions,
number of test cases in the test collections, the range
of the numbers of training instances used to construct a classification
model in our empirical study, the number of lines
of code, and the number of branch statements. Moreover,
the “Exe” (abbreviation of executable statements) within
parentheses represents the number of executable statements,
which presents the real size of a subject. As the faulty programs
in Table 1 are all single-fault programs, we totally
constructed 80 multiple-fault programs for the eight subjects
by extracting the faults from the single-fault programs
of each subject and then seeding more than one faults into
each subject.
3.1. Independent and Dependent Variables
The independent variables come from the three research
questions and are defined as follows.
The first independent variable Algorithm refers to the
machine-learning algorithm that is used to build a classification
model. Many algorithms have been proposed in
the literature of machine learning. Here we investigate the
following three representative algorithms: Random Forest
(RF) [2], Naive Bayes (NB) [8], and Sequential Minimal
Optimization (SMO) [12], because these algorithms have
been found to be effective in the literature of machine learning
and have been widely used. Moreover, these algorithms
284