SEKE 2012 Proceedings - Knowledge Systems Institute

subsets. The empirical results demonstrate that S2N, AUC, and PRC
show higher stability performance than the other filters. Moreover, the
filters with the ROS sampling technique have higer stability behavior
than they do with the other sampling methods. Finally, the stability
of the FS techniques increases as the number of attributes retained
in the feature subset increases.
The remainder of the paper is organized as follows: Section
II discusses related work. Section III outlines the methods and
techniques used in this paper. Section IV describes nine datasets
used in the case study. Section V presents the case study including
design, results, and analysis. Finally, we summarize our conclusions
and provide suggestions for future work in Section VI.
II. RELATED WORK
Feature selection (FS), also known as attribute selection or variable
selection, is a process of selecting some subset of the features which
are useful in building a classifier. FS techniques can be divided into
wrapper and filter categories [3]. Wrappers use a search algorithm to
search through the space of possible features, evaluate each subset
through a learning algorithm, and determine which ones are finally
selected in building a classifier. Filters use a simpler statistical
measure to evaluate each subset or individual feature rather than using
a learning algorithm. Feature selection may also be categorized as
ranking or subset selection [3]. Feature ranking scores the attributes
based on their individual predictive power, while subset selection
selects subset of attributes that collectively have good prediction
capability. In this study, the FS techniques used belong to the filterbased
feature ranking category.
Class imbalance, which appears in various domains, is another
significant problem in data mining. One effective method for alleviating
the adverse effect of skewed class distribution is sampling [6],
[7]. While considerable work has been done for feature selection
and data sampling separately, research on investigating both together
started recently. Chen et al. [8] have studied data row pruning (data
sampling) and data column pruning (feature selection) in the context
of software cost/effort estimation. However, the data sampling in their
study was not specifically for the class imbalance problem, and also
the classification models were not for binary problems.
To evaluate FS techniques, most existing research works on comparing
the classification behaviors of models built with the selected
features to those built with the complete set of features. Instead
of using classification performance, the present work assesses FS
techniques using stability. The stability of a FS algorithm is normally
defined as the degree of consensus between the output of that FS
method as it pertains to randomly-selected subsets of the same input
data. Lustgarten et al. [9] presented an adjusted stability measure
that computes robustness of a FS method with respect to random FS.
Saeys et al. [10] assessed the robustness of FS techniques using the
Spearman rank correlation coefficient and Jaccard index. Abeel et
al. [11] presented a general framework for stability analysis of the
FS techniques. They showed that stability could be improved through
ensemble FS. Alelyani et al. [12] jointly considered both sample sets’
similarity and feature list similarity in stability assessment for FS
algorithms.
III. METHODOLOGY
A. Filter-based feature ranking techniques
The procedure of filter-based feature ranking is to score each
feature (attribute) according to a particular method (metric), allowing
the selection of the best set of features. In this study, we use five
threshold-based feature selection techniques and the signal-to-noise
ratio method.
1) Threshold-based feature selection (TBFS) methods: The TBFS
techniques were proposed by our research team and implemented
within WEKA [2]. The procedure is shown in Algorithm 1. Each
independent attribute works individually with the class attribute, and
that two-attribute dataset is evaluated using different performance
metrics. More specifically, the TBFS procedure includes two steps:
(1) normalizing the attribute values so that they fall between 0 and
1; and (2) treating those values as the posterior probabilities from
which to calculate classifier performance metrics.
Analogous to the procedure for calculating rates in a classification
setting with a posterior probability, the true positive (TPR), true
negative (TNR), false positive (FPR), and false negative (FNR)
rates can be calculated at each threshold t ∈ [0, 1] relative to the
normalized attribute ˆX j . Precision PRE(t) is defined as the fraction
of the predicted-positive examples which are actually positive.
The feature rankers utilize five metrics: Mutual Information (MI),
Kolmogorov-Smirnov Statistic (KS), Geometric Mean (GM), Area
Under the ROC Curve (AUC), and Area Under the Precision-Recall
Curve (PRC). The value is computed in both directions: first treating
instances above the threshold (t) as positive and below as negative,
then treating instances above the threshold as negative and below as
positive. The better result is used. Five metrics are calculated for
each attribute individually, and attributes with higher values for MI,
KS, GM, AUC, and PRC are determined to better predict the class
attribute. In this manner, the attributes can be ranked from most to
least predictive based on each of the five metrics. For more detailed
information about these five metrics, please reference the work of
Dittman et al. [2].
2) Signal-to-Noise Ratio (S2N) Technique: S2N represents how
well a feature separates two classes. The equation for signal-to-noise
is:
S2N =(μ P − μ N )/(σ P + σ N ) (1)
where μ P and μ N are the mean values of that particular attribute
in all of the instances which belong to a specific class, which is
either P or N (the positive and negative classes). σ P and σ N are
the standard deviations of that particular attribute as it relates to the
class. The larger the S2N ratio, the more relevant a feature is to the
dataset [13].
B. Data Sampling Techniques
We here present three sampling techniques, which stand for the
major paradigms in data sampling: random and intelligent under and
oversampling.
1 Random Sampling Techniques
The two most common data sampling techniques are random
oversampling (ROS) and random undersampling (RUS). Random
oversampling duplicates instances (selected randomly) of
the minority class. Random undersampling randomly discards
instances from the majority class.
2 Synthetic Minority Oversampling Technique
Chawla et al. proposed an intelligent oversampling method
called Synthetic Minority Oversampling Technique (SMOTE)
[6]. SMOTE (denoted SMO in this work) adds new, artificial
minority examples by extrapolating between preexisting minority
instances rather than simply duplicating original examples.
The newly created instances cause the minority regions of the
feature-space to be fuller and more general.
75