SEKE 2012 Proceedings - Knowledge Systems Institute

Stability of Filter-Based Feature Selection Methods
for Imbalanced Software Measurement Data
Kehan Gao
Eastern Connecticut State University
Willimantic, Connecticut 06226
gaok@easternct.edu
Taghi M. Khoshgoftaar
Florida Atlantic University
Boca Raton, Florida 33431
khoshgof@fau.edu
Amri Napolitano
Florida Atlantic University
Boca Raton, Florida 33431
amrifau@gmail.com
Abstract—Feature selection (FS) is necessary for software quality
modeling, especially when a large number of software metrics are
available in data repositories. Selecting a subset of features (software
metrics) that best describe the class attribute (module’s quality) can bring
many benefits such as reducing the training time of learners, improving
the comprehensibility of the resulting classifier models, and facilitating
software metrics collection, organization, and management. Another
challenge of software measurement data is the presence of skewed or
imbalanced distributions between the two types of modules (e.g., many
more not-fault-prone modules than fault-prone modules found in those
datasets). In this paper, we use data sampling to deal with this problem.
Previous research usually evaluates FS techniques by comparing the
performance of classifiers before and after the training data is modified.
This study assesses FS techniques from a different perspective: stability.
Stability is important because FS techniques that reliably produce the
same features are more trustworthy. We consider six filter-based feature
selection methods and six data sampling approaches. We also vary the
number of features selected in the feature subsets. We want to examine
the effect of data sampling approaches on the stability of FS when using
the sampled data. The experiments were performed on nine datasets
from a real-world software project. The results demonstrate that different
FS techniques may have quite different stability behaviors. In addition,
other factors, such as the sampling technique used and the number of
attributes retained in the feature subset, may also greatly influence the
stability results.
Index Terms—software defect prediction, software metrics, feature
selection, data sampling, stability
I. INTRODUCTION
Software defect prediction is a process of building a classifier by
using software metrics and fault data collected during the previous
software project and then applying this classifier to predict the quality
of new program modules (e.g., classify the program modules as either
fault-prone (fp) or not-fault-prone (nfp)) [1]. The benefit of such
prediction is that the project resources can be strategically allocated
to the program modules according to the prediction. For instance,
intensive inspection and testing can first be applied to the potentially
problematic modules, thereby improving the quality of the product.
Two problems that often come with the software measurement data
are high-dimensionality and class imbalance. High-dimensionality
refers to the situation where the number of available software metrics
is too large to easily work with. Several problems may arise due to
high-dimensionality, including longer learning time of a classification
algorithm and a decline in prediction performance of a classification
model. Class imbalance occurs when instances of one class in a
dataset appear more frequently than instances of the other class. This
phenomenon is more prevalent in high-assurance and mission-critical
software systems, where the type of nfp modules is always dominant
between the two types (fp and nfp) ofmodules in a given dataset.
The primary weakness of such imbalanced data is that a traditional
classification algorithm tends to classify fp modules as nfp, resulting
in more customer-discovered faults that have serious consequences
and high repair cost.
Feature selection and data sampling are often employed to deal
with these problems. Feature selection (FS) is a process of choosing
a subset of input variables by eliminating features with little or no
predictive information. Although FS techniques have been studied
in a variety of domains [2], [3] for many years, research working
on improving software defect prediction through metric (feature)
selection just started recently [4], [5]. Data sampling isacommon
technique to alter the relative proportion of the different types of
modules, therefore achieving a more balanced dataset. Note that in
this study, the training dataset is sampled to change the relative
proportion of the nfp and fp modules before FS is performed.
To evaluate a FS technique, most previous research focuses on
comparing the performance of classification models before and after
a specific FS technique is performed. In this paper, we use a different
way to assess FS techniques – stability. Stability of a FS technique
usually refers to the sensitivity of the technique to variations in
the training set. Practitioners may prefer a FS algorithm that can
produce consistent results despite such variations. For example, if a
FS technique produces the same orsimilar results when using the
entire training dataset or only half of it, a practitioner may save the
computation time and use this FS technique on the smaller training
dataset to get the same reliable results.
In this study, we are more interested in investigating the stability
of FS techniques with respect to various data sampling approaches.
The strategy we adopted is that the ranking of features from each
sampled dataset is compared to the ranking from the original dataset
which it came from. Those FS techniques that are able to produce
consistent outputs with respect to the different perturbations (due to
sampling) in the input data are considered stable (robust), or say they
are insensitive to that particular data sampling technique. Since the
purpose of data sampling here is to alter class proportions rather than
changing the size of the training dataset, the consistent outputs for
feature rankings imply that the data sampling technique has less or no
effect on the FS technique. To our knowledge, limited research has
been done on studying the impact of data sampling on the stability
of feature selection.
The case study of this paper is performed on nine datasets from a
real-world software project. We examine six filter-based FS methods,
including five threshold-based techniques (mutual information (MI),
Kolmogorov-Smirnov statistic (KS), geometric mean (GM), area
under the ROC curve (AUC), and area under the precision-recall
curve (PRC)), and the signal-to-noise ratio (S2N) approach. We employ
three data sampling techniques (random undersampling (RUS),
random oversampling (ROS), and synthetic minority oversampling
(SMO)), each combined with two post-sampling class ratios. Besides,
we vary from 2to10thenumber of features retained in the feature
74