6th European Conference - Academic Conferences

Suzan Arslanturk et al.
Knowledge Analysis) is used. Weka is an open source Java based data mining tool that performs pre
processing, classification, clustering and many other machine learning tools by using a graphical user
interface. First the dataset is uploaded to the Weka Explorer mode and the data is discretized. Then all
the attribute selection algorithms are performed. Sensitivity and specificity are used to evaluate the
performances in a statistical manner. Sensitivity i.e. true positive rate; measures the proportion of actual
positives which are correctly identified. While specificity (1 – false positive probability) measures the
proportion of negatives which are correctly identified. The algorithms with higher sensitivity and higher
specificity are the ones that performs significantly better than the others.
5. Results and discussions
Our experiment applied five different feature selection algorithms to the simulation dataset and the results
are compared when noise, missing values and multicollinearity are added to the data. The first set of
experiments was designed to evaluate the following feature selection methods to see which one handles
additive noise better: J48, Relief, information gain, consistency based feature selection and correlation
based feature selection.
In order to refer to a feature selection algorithm as robust, it has to both has a high sensitivity and a high
specificity. Fig. 1 shows that J48 algorithm performs well in terms of sensitivity but there is a huge
decrease in the specificity curve (Fig. 2) which makes the algorithm less desirable than the others.
Without any noise Information gain and ReliefF algorithms both perform well. Consistency based feature
evaluation has a low specificity and J48 decision tree classification algorithm has a low sensitivity. The
results do not change in Cfs, Information Gain and Relief algorithms when the noise level is 2%, 5%,
10%, 15%, ,respectively. In spite, in J48 decision tree the specificity decreases and the sensitivity
increases rapidly when the noise level increase. The sensitivity of consistency based feature selection
was 11% higher than the average sensitivity of other methods. However, its specificity was 37% lower
than that of the average. It is important to note that data method that maximizes both the sensitivity and
specificity is of interest. The best method, when both sensitivity and specificity are considered, was
information gain. This method outperformed the average performance of the other methods by 1% and
20.5% when we considered its sensitivity and specificity, respectively. Consistency based subset
evaluation and J48 algorithms can not handle noise. The next set of experiments was designed to
evaluate the feature selection methods to see which one handles missing values better. Fig. 3 and Fig. 4
show the results of different missing value levels. In
this case, when considering sensitivity and specificity, Relief and information gain were proved to perform
better compared to the other methods of our study by 7.2% and 12.4%, respectively. Despite, the
Consistency based feature selection and J48 algorithms can not handle missing values effectively. Our
studies also showed that when multicollinearity was embedded into the dataset without any noise and
missing values, the correlation based feature selection method outperformed other methods. In summary,
Relief and information gain were the best in all three situations (noise, missing value, multicollinearity)
when both sensitivity and specificity were considered.
Figure 1: Noise-sensitivity
21