SEKE 2012 Proceedings - Knowledge Systems Institute

Figure 1. Boxplots of average AUC results for
various machine-learning algorithms
Table 2. Paired Samples T-Test on Machine-
Learning Algorithms (α=0.05)
Single-Fault Programs t-value Sig. Result
SMO - RF -18.767 0.000 Reject
SMO - NB -14.576 0.000 Reject
RF - NB 3.970 0.000 Reject
Multiple-Fault Programs t-value Sig. Result
SMO - RF -30.602 0.000 Reject
SMO - NB -16.362 0.000 Reject
RF - NB 6.950 0.000 Reject
our empirical study has only the results of six single-fault
programs except for print tokens and schedule2, and seven
multiple-fault programs except for Space. Our empirical
study does not acquire the results of execution-data classification
for Space based on its statement coverage or statement
count because the maximal memory of Java Virtual
Machine is not large enough to classify executions which
have a large number of attributes (i.e., the statement coverage
or the statement count).
(1)RQ1: Machine-Learning Algorithms
Figure 1 shows the distribution of the average AUC of
the execution-data classification approach with the same
machine-learning algorithm for each single-fault subject.
The horizontal axis represents the three machine-learning
algorithms, whereas the vertical axis represents the average
AUC. According to Figure 1, the classification approach
with SMO is usually much less effective than the
approach with RF. The classification approach with RF produces
close AUC results to the approach with NB. Moreover,
sometimes (e.g., for replace) the execution-data classification
approach with RF is better than the execution-data
classification approach with NB, whereas sometimes (e.g.,
for schedule) the latter approach is better than the former.
To further compare the three machine-learning algorithms,
we performed the paired samples t-test on the average
AUC results by comparing each pair of results of the
execution-data classification approaches with the same subject,
the same percentage of training instances, and the same
type of execution data, but different machine-learning algorithms.
The results are shown by Table 2. Here the t-test
is performed separately on results of single-fault programs
and multiple-fault programs because the machine-learning
based execution-data classification approach is intuitively
more effective to classify an execution for single-fault programs
than for multiple-fault programs considering the impact
of multiple faults.
According to this table, the hypothesis that neither RF
nor NB is superior to the other in execution-data classification
is rejected (for both single-fault and multiple-fault
programs). Moreover, execution-data classification using
RF wins the classification approach using NB since its calculated
t-value is positive. That is, for single-fault and
multiple-fault programs, execution-data classification with
RF is significantly better than the approach with NB. Similarly,
execution-data classification with either RF or NB is
significantly better than the approach with SMO.
(2)RQ2: Training Set
According to the average AUC, with the increase of the
number of training instances (i.e., variable TrainingSet increases
from 20% to 100%), the average AUC of each subject
usually increased except for a few data.
However, it is not practical to construct a classifier based
on a large number of training instances (i.e., execution
data with known outcome) in execution-data classification.
Thus, we are more interested in the experimental results
of execution-data classification using a small number of
training instances. According to the experimental results,
as we increase the number of training instances, the average
AUC results increase slightly. That is, the executiondata
classification approach fed with 20% training instances
has close AUC results to the approach with 100% training
instances, which produces reliable classification results.
Consequently, for any subject whose number of executable
statements is n, execution-data classification has been evaluated
to be reliable even if the number of training instances
is n ∗ 20%.
(3)RQ3: Type of Execution Data
To show the influence of the types of execution data on
AUC results of execution-data classification, we draw some
boxplots for single-fault programs by statically analyzing
average AUC results of same type of execution data. For
each program, the five boxplots for various types of execution
data have observable difference. For instance, the
AUC results of execution-data classification with the branch
coverage usually distribute in a smaller range than the approach
with any of the other types of execution data. That
is, execution-data classification with the branch coverage
is more stable than the approach with the other type of
execution-data although some other types of execution data
286