SEKE 2012 Proceedings - Knowledge Systems Institute

1) Classifiers: In this study, software defect prediction models are
built with three well-known classification algorithms: naïve Bayes
(NB), multilayer perceptron (MLP), and logistic regression (LR). All
three learners themselves do not have a built-in feature selection capability
and are commonly used in the software engineering and other
data mining applications. All classifiers were implemented in the
WEKA tool with default parameter settings [6] except MLP. Based
on preliminary research, the parameters of MLP were set as follows.
The ‘hiddenLayers’ parameter was set to 3 to define a network with
one hidden layer containing three nodes. The ‘validationSetSize’
parameter was set to 10 to cause the classifier to leave 10% of the
training data aside to be used as a validation set to determine when
to stop the iterative training process.
2) Performance Metric: Because using traditional performance
measures such as classification accuracy can give misleading results
on imbalanced data, we use a performance metric that considers
the ability of a classifier to differentiate between the two classes:
the area under the ROC (Receiver Operating Characteristic) curve
(AUC). A perfect classifier provides an AUC that equals 1. It has
been shown that AUC has lower variance and more reliability than
other performance metrics (such as precision, recall, F-measure) [15].
Note that the metric used to measure the performance of the classifiers
is completely independent from the metric in the TBFS algorithm.
AUC is used both to select the most predictive subset of features in
TBFS and to evaluate the classification models constructed using this
set of features.
3) Experimental Results: During the experiments, ten runs of fivefold
cross-validation were performed. First, we ranked the attributes
using the eighteen rankers separately. Once the attributes are ranked,
the top six attributes are selected to yield the final training data. After
feature selection, we applied the classifier to the training datasets
with the selected features, and then we used AUC to evaluate the
performance of the classification model. In total, 18 rankers × 4
datasets × 10 runs × 5 folds = 3600 combinations of feature ranking
techniques were employed, and correspondingly 3600 × 3 classifiers
= 10800 classification models were built.
All the results are reported in Table VI through Table VIII. Note
that each value presented in the tables is the average over the ten
runs of five-fold cross-validation outcomes. From these tables, we
can observe that although a given ranker might perform best in
combination with one learner, this may not be true when other
learners are used to evaluate models. For example, S2N performed
best on average in terms of AUC when the NB and LR classifiers
are used. However, this is not true when the MLP classifier is used;
in that case, PO performed best. The results also demonstrate that
although no particular ranker dominates the others, we can conclude
that S2N, PO, and PRC are most often the best techniques, while
PR, OR, and GR are rarely optimal. Our recent study [5] shows
that the reduced feature subsets can have better or similar prediction
performance compared to the complete set of attributes (original data
set).
We also conducted a two-way ANalysis Of VAriance (ANOVA)
F test ([16]) to statistically examine the various effects on the
performances of the classification models. The two-way ANOVA test
in this study includes two factors: the first represents 18 rankers, and
the second represents the three learners. In this ANOVA test, the
results from all four datasets were taken into account together. In
this study, we also performed the multiple comparison tests using
Tukey’s honestly significant difference (HSD) criterion [16]. All
tests of statistical significance utilize a significance level α of 5%.
Both ANOVA and multiple comparison tests were implemented in
TABLE VI
CLASSIFICATION PERFORMANCE, NB
SP1 SP2 SP3 SP4 Average
CS 0.7846 0.8108 0.8184 0.7696 0.7958
GR 0.7346 0.7613 0.7808 0.7519 0.7571
IG 0.7831 0.8081 0.8118 0.7794 0.7956
RF 0.7879 0.8053 0.8305 0.7731 0.7992
RFW 0.7882 0.8081 0.8190 0.7735 0.7972
SU 0.7865 0.7729 0.7882 0.7592 0.7767
FM 0.7822 0.8074 0.8176 0.7731 0.7951
OR 0.7405 0.8060 0.7181 0.7558 0.7551
PO 0.7891 0.8071 0.8141 0.8023 0.8031
PR 0.7345 0.7963 0.7179 0.7605 0.7523
GI 0.7341 0.7982 0.7678 0.6997 0.7500
MI 0.7739 0.8010 0.8119 0.7788 0.7914
KS 0.7722 0.7750 0.8125 0.7588 0.7796
DV 0.7820 0.8099 0.8163 0.7874 0.7989
GM 0.7716 0.7740 0.8165 0.7586 0.7802
AUC 0.7685 0.8072 0.7947 0.7683 0.7847
PRC 0.7885 0.8131 0.8120 0.7953 0.8022
S2N 0.7995 0.8142 0.8067 0.8129 0.8083
TABLE VII
CLASSIFICATION PERFORMANCE, MLP
SP1 SP2 SP3 SP4 Average
CS 0.7943 0.8117 0.8126 0.7914 0.8025
GR 0.7475 0.7545 0.7688 0.7464 0.7543
IG 0.7926 0.8099 0.8209 0.8103 0.8084
RF 0.7948 0.8119 0.8191 0.7619 0.7969
RFW 0.7955 0.8139 0.8303 0.7598 0.7999
SU 0.7875 0.7847 0.7843 0.7504 0.7767
FM 0.7917 0.8127 0.8163 0.7924 0.8033
OR 0.7665 0.8058 0.7244 0.7550 0.7630
PO 0.7955 0.8133 0.8261 0.8010 0.8090
PR 0.7666 0.7970 0.7299 0.7576 0.7628
GI 0.7660 0.7963 0.7813 0.7495 0.7733
MI 0.7855 0.7945 0.8249 0.7865 0.7978
KS 0.7836 0.7749 0.8201 0.7607 0.7848
DV 0.7923 0.8095 0.8180 0.7959 0.8039
GM 0.7812 0.7779 0.8150 0.7701 0.7861
AUC 0.7754 0.8099 0.8260 0.7769 0.7970
PRC 0.7977 0.8151 0.8105 0.7908 0.8035
S2N 0.8047 0.8039 0.8108 0.7972 0.8042
TABLE VIII
CLASSIFICATION PERFORMANCE, LR
Ranker SP1 SP2 SP3 SP4 Average
CS 0.8021 0.8229 0.8354 0.8153 0.8189
GR 0.7688 0.7935 0.7805 0.7816 0.7811
IG 0.8014 0.8176 0.8361 0.8216 0.8192
RF 0.8103 0.8221 0.8354 0.8118 0.8199
RFW 0.8091 0.8233 0.8387 0.8142 0.8213
SU 0.7993 0.7909 0.8040 0.7802 0.7936
FM 0.8023 0.8235 0.8298 0.8088 0.8161
OR 0.7816 0.8158 0.7422 0.7768 0.7791
PO 0.8041 0.8255 0.8334 0.8202 0.8208
PR 0.7784 0.8053 0.7467 0.7808 0.7778
GI 0.7787 0.8062 0.7918 0.7780 0.7887
MI 0.7934 0.7983 0.8354 0.7942 0.8053
KS 0.7902 0.7788 0.8338 0.7705 0.7933
DV 0.8020 0.8186 0.8323 0.8164 0.8173
GM 0.7886 0.7755 0.8327 0.7799 0.7942
AUC 0.7845 0.8154 0.8318 0.7882 0.8049
PRC 0.8041 0.8265 0.8282 0.8123 0.8178
S2N 0.8176 0.8279 0.8336 0.8231 0.8256
98