SEKE 2012 Proceedings - Knowledge Systems Institute

An Empirical Study of Software Metric Selection Techniques
for Defect Prediction
Huanjing Wang, Taghi M. Khoshgoftaar, Randall Wald, and Amri Napolitano
{huanjing.wang@wku.edu, khoshgof@fau.edu, rwald1@fau.edu, amrifau@gmail.com}
Abstract—In software engineering, a common classification problem
is determining the quality of a software component, module, or release.
To aid in this task, software metrics are collected at various states of
a software development cycle, and these metrics can be used to build
a defect prediction model. However, not all metrics are relevant to
defect prediction. One solution to finding the relevant metrics is the data
preprocessing step known as feature selection. We present an empirical
study in which we evaluate the similarity of eighteen different feature
selection techniques and how the feature subsets chosen by each of these
techniques perform in defect prediction. We look at similarity in addition
to classification because many applications seek a diverse set of rankers,
and similarity can be used to find which rankers are too close together
to provide diversity. The classification models are trained using three
commonly-used classifiers. The case study is based on software metrics
and defect data collected from multiple releases of a large real-world
software system. The results show that the features fall into a number
of identifiable clusters in terms ofsimilarity. In addition, the similarity
clusters were somewhat predictive of the clusters based on classification
ranking: rankers within a similarity cluster had similar classification
performance, and thus ended up in the same or adjacent classification
clusters. The reverse was not true, with some classification clusters
containing multiple unrelated similarity clusters. Overall, we found that
the signal-to-noise and ReliefF-W rankers selected good features while
being dissimilar from one another, suggesting they are appropriate for
choosing diverse but high-performance rankers.
I. INTRODUCTION
In the practice of software quality assurance, software metrics
are often collected and associated with modules, which have their
number of pre- and post-release defects recorded. A software defect
prediction model is often built to ensure the quality of future software
products or releases. However, not all software metrics are relevant
for predicting the fault proneness of software modules. Software
metrics selection (or feature selection) prior to training a defect
prediction model can help separate relevant software metrics from
irrelevant or redundant ones.
In this paper, we focus on feature selection of software metrics
for defect prediction. During the past decade, numerous studies have
examined feature selection with respect to classification performance,
but very few studies focus on the similarity of feature selection
techniques. The purpose of studying this similarity is to make it easier
to select a set of diverse rankers, ensuring that none of those chosen
are so similar to the others as to provide no additional diversity
in the collection of rankers. In this study, we perform similarity
analysis on eighteen different feature selection techniques, eleven of
which were recently developed and implemented by our research
group. We evaluate the similarity of two filters on a dataset by
measuring the consistency between the two feature subsets chosen.
We also evaluate the effectiveness of defect predictors that estimate
the quality of program modules, e.g., fault-prone (fp) or not-faultprone
(nfp). Three different classifiers (learners) are used to build our
prediction models. The empirical validation of the similarity measure
and model performance was implemented through a case study of
four consecutive releases of a very large telecommunications software
system (denoted as LLTS). To our knowledge this is the first study
to examine both similarity and classification performance of feature
rankers in the software engineering domain.
The experimental results show that using both the similarity and
classification performance of the rankers gave different types of
clusters. Similarity produced a number of smaller clusters, with four
or five rankers being about the largest sizes found (and two and one
being common sizes as well). Classification gave a smaller number
of clusters, with the largest cluster (also the best-performing cluster)
having nine members. In both cases, the clusters often contained
feature rankers which would not immediately seem to have much in
common; although this makes sense for classification performance,
it is an intriguing result for the similarity (indicating that the rankers
may be more related than one might expect). We also found that
the similarity groups were generally predictive of the classification
groupings: members within a single similarity group are in the same
or adjacent classification groups. Finally, we noted that signal-tonoise
and ReliefF-W performed very well in terms of classification
while choosing two or fewer features in common.
The rest of the paper is organized as follows. We review relevant
literature on feature selection techniques in Section II. Section III
provides detailed information about the 18 feature selection techniques.
Section IV describes the datasets used in the study, presents
similarity results and analysis, and shows model performance results
and analysis. Finally, in Section V, the conclusion is summarized and
suggestions for future work are indicated.
II. RELATED WORK
The main goal of feature selection is to select a subset of
features that minimizes the prediction errors of classifiers. Feature
selection can be broadly classified as feature ranking and feature
subset selection. Feature ranking sorts the attributes according to
their individual predictive power, while feature subset selection finds
subsets of attributes that collectively have good predictive power.
Feature selection can also be categorized as filters and wrappers.
Filters are algorithms in which a feature subset is selected without
involving any learning algorithm. Wrappers are algorithms that use
feedback from a learning algorithm to determine which feature(s) to
include in building a classification model.
A number of papers have studied the use of feature selection
techniques as a data preprocesing step. Guyon and Elisseeff [1]
outline key approaches used for attribute selection, including feature
construction, feature ranking, multivariate feature selection, efficient
search methods, and feature validity assessment methods. A study by
Liu and Yu [2] provides a comprehensive survey of feature selection
algorithms and presents an integrated approach to intelligent feature
selection. Jeffery et al. [3] compare the similarity between gene lists
produced by 10 different feature selection methods. They conclude
that sample size clearly affects the ranked gene lists produced by
different feature selection methods.
94