SEKE 2012 Proceedings - Knowledge Systems Institute

Fig. 3.
Stability comparisons over three groups of Eclipse datasets
datasets show the same/similar patterns. It is clearly seen that the FS
methods had highest stability performance on the Eclipse3 datasets,
then on Eclipse2, and finally on Eclipse1. This is especially true when
using the RUS35, RUS50, SMO35, and SMO50 sampling approaches.
VI. CONCLUSION
This paper presents a strategy that uses feature selection (FS) and
data sampling together to cope with the high-dimensionality and
class imbalance problems in the context of software defect prediction.
Instead of assessing FS techniques by measuring classification
performance after the training dataset is modified, this study focuses
on another important property of FS – stability, more specifically, the
sensitivity of a FS method when used with a data sampling technique.
More stable FS techniques will reliably give the same features even
after sampling has been used, so practitioners can be more confident
in those features.
We examined six filter-based feature ranking techniques, five
of which are threshold-based feature selection methods (MI, KS,
GR, AUC, and PRC), and the remaining one is the signal-to-noise
ratio (S2N) method. The three sampling techniques adopted are
random undersampling (RUS), random oversampling (ROS), and
synthetic minority oversampling (SMO), each combined with two
post-sampling class ratios (35:65 and 50:50). The experiments were
performed on three groups of datasets from a real-world software
project.
The results demonstrate that 1) S2N, AUC, and PRC had higher
stability performance than other rankers; 2) ROS35 and ROS50
produced higher stability values than other sampling approaches; (3)
post-sampling class ratio between fp and nfp of 35:65 showed higher
stability than the ratio of 50:50 for the RUS and SMO sampling
techniques; (4) the stability performance generally increased along
the increment of the number of the attributes retained in the feature
subset, especially when the dataset is relatively skewed; and (5) the
less imbalanced original datasets (prior to sampling) were more likely
to obtain higher stability performance when data sampling techniques
(such as RUS or SMO) was performed on them.
Future work will include additional case studies with software
measurement datasets of other software systems. In addition, different
data sampling and FS techniques will be considered in future
research.
REFERENCES
[1] S. Lessmann, B. Baesens, C. Mues, and S. Pietsch, “Benchmarking
classification models for software defect prediction: A proposed framework
and novel findings,” IEEE Transactions on Software Engineering,
vol. 34, no. 4, pp. 485–496, July-August 2008.
[2] D. J. Dittman, T. M. Khoshgoftaar, R. Wald, and J. V. Hulse, Handbook
of Data Intensive Computing. Springer, 2011, ch. Feature Selection
Algorithms for Mining High Dimensional DNA Microarray Data, pp.
685–710.
[3] H. Liu, H. Motoda, R. Setiono, and Z. Zhao, “Feature selection: An
ever evolving frontier in data mining,” in Proceedings of the Fourth International
Workshop on Feature Selection in Data Mining, Hyderabad,
India, 2010, pp. 4–13.
[4] K. Gao, T. M. Khoshgoftaar, H. Wang, and N. Seliya, “Choosing software
metrics for defect prediction: an investigation on feature selection
techniques,” Softw., Pract. Exper., vol. 41, no. 5, pp. 579–606, 2011.
[5] S. Shivaji, J. E. W. Jr., R. Akella, and S. Kim, “Reducing feature to
improve bug prediction,” in ASE ’09 Proceedings of the 2009 IEEE/ACM
International Conference on Automated Software Engineering, Auckland,
New Zealand, 2009, pp. 600–604.
[6] N. V. Chawla, K. W. Bowyer, L. O. Hall, and P. W. Kegelmeyer,
“SMOTE: Synthetic minority over-sampling technique,” Journal of Artificial
Intelligence Research, vol. 16, pp. 321–357, 2002.
[7] C. Seiffert, T. M. Khoshgoftaar, J. Van Hulse, and A. Napolitano,
“Rusboost: A hybrid approach to alleviate class imbalance,” IEEE
Transactions on Systems, Man & Cybernetics: Part A: Systems and
Humans, vol. 40, no. 1, January 2010.
[8] Z. Chen, T. Menzies, D. Port, and B. Boehm, “Finding the right data
for software cost modeling,” IEEE Software, no. 22, pp. 38–46, 2005.
[9] J. L. Lustgarten, V. Gopalakrishnan, and S. Visweswaran, “Measuring
stability of feature selection in biomedical datasets,” in AMIA Annu Symp
Proc. 2009, 2009, pp. 406–410.
[10] Y. Saeys, T. Abeel, and Y. Peer, “Robust feature selection using ensemble
feature selection techniques,” in ECML PKDD ’08: Proceedings of the
European conference on Machine Learning and Knowledge Discovery
in Databases - Part II. Berlin, Heidelberg: Springer-Verlag, 2008, pp.
313–325.
[11] T. Abeel, T. Helleputte, Y. Van de Peer, P. Dupont, and Y. Saeys,
“Robust biomarker identification for cancer diagnosis with ensemble
feature selection methods,” Bioinformatics, vol. 26, no. 3, pp. 392–398,
February 2010.
[12] S. Alelyani, Z. Zhao, and H. Liu, “A dilemma in assessing stability of
feature selection algorithms,” in Proceedings of the 13th International
Conference on High Performance Computing and Communications
(HPCC), Banff, Canada, 2011, pp. 701–707.
[13] X. Chen and M. Wasikowski, “Fast: a roc-based feature selection metric
for small samples and imbalanced data classification problems,” in KDD
’08: Proc. 14th ACM SIGKDD Int’l Conf. Knowldege Discovery and
Data Mining. New York, NY: ACM, 2008, pp. 124–132.
[14] L. I. Kuncheva, “A stability index for feature selection,” in Proceedings
of the 25th conference on Proceedings of the 25th IASTED International
Multi-Conference: artificial intelligence and applications. Anaheim,
CA, USA: ACTA Press, 2007, pp. 390–395.
[15] K. Dunne, P. Cunningham, and F. Azuaje, “Solutions to Instability Problems
with Sequential Wrapper-Based Approaches To Feature Selection,”
Department of Computer Science, Trinity College, Dublin, Ireland, Tech.
Rep. TCD-CD-2002-28, 2002.
[16] P. Křížek, J. Kittler, and V. Hlaváč, “Improving stability of feature
selection methods,” in Proceedings of the 12th international conference
on Computer analysis of images and patterns, ser. CAIP’07. Berlin,
Heidelberg: Springer-Verlag, 2007, pp. 929–936.
[17] G. Boetticher, T. Menzies, and T. Ostrand. (2007) Promise
repository of empirical software engineering data. [Online]. Available:
http://promisedata.org/
[18] T. Zimmermann, R. Premraj, and A. Zeller, “Predicting defects for
eclipse,” in Proceedings of the 29th International Conference on Software
Engineering Workshops.
Society, 2007, p. 76.
Washington, DC, USA: IEEE Computer
79