SEKE 2012 Proceedings - Knowledge Systems Institute

anches is more applicable than the approaches with either
statements or methods.
4. Related Work
As the testing and analysis activities on released software
systems are different from those in-house, researchers have
proposed some techniques to facilitate testing and analysis
of released software systems. Pavlopoulou and Young [11]
proposed a residual test coverage monitoring tool to selectively
instrument a Java program under test so that the performance
overhead is acceptable. As a software system has
distributed in sites of many remote users, a monitoring task
can be divided into several subtasks, and each subtask is
assigned to different instances of the software system and
can be implemented by minimal instrumentation. This approach,
called the Gamma system, isimplemented by Orso
and colleagues [1, 10]. The preceding research aims to
propose different instrumentation strategies in monitoring
a released software system. However, our work uses execution
data, which is collected by these instrumentation techniques,
to distinguish failing executions from passing ones.
The execution data collected from the field can be analyzed
and used for the purpose of software maintenance and
evolution. Liblit and colleagues [6, 7] proposed to gather
executions by using Bernoulli process and then isolate bugs
based on the sampling executions. Orsa and colleagues [9]
investigated the use of execution-data from the field to support
and improve impact analysis and regression testing inhouse.
Although both of these research and ours use the
execution data, they have different aims. These work aims
at fault localization, impact analysis or testing, whereas our
work aims at classifying executions.
5. Conclusion
In this paper, we performed an empirical study on
execution-data classification by varying the three factors:
machine learning algorithms, number of training instances,
and type of execution data. Among the three machinelearning
algorithms, the Random Forest algorithm makes
the classification approach produce significantly better results
than the Naive Bayes algorithm. Either of these algorithms
makes the classification approach produce significantly
better results than the Sequential Minimal Optimization
algorithm. As we increased the number of training
instances, the classification model usually produces better
classification results. However, execution-data classification
approach still correctly classifies most execution data
when fed with a small number of training instances (i.e.,
1/5 of the number of executable statements contained by a
program). Moreover, when the type of execution data is
branch coverage, the corresponding classification approach
produces better results than the approach with the other
types of execution data.
Acknowledgements
This work is supported by the National Basic Research
Program of China under Grant No. 2009CB320703, Science
Fund for Creative Research Groups of China under
Grant No. 60821003, the National Natural Science Foundation
of China under Grant No. 91118004, and the National
Natural Science Foundation of China under Grant No.
60803012.
References
[1] J. Bowring, A. Orso, and M. J. Harrold. Monitoring deployed
software using software tomograph. In Proc. ACM
SIGPLAN-SIGSOFT Workshop on Program Analysis for
Software Tools and Engineering, pages 2–8, 2002.
[2] L. Breiman. Random forests. Machine Learning, 45(1):5–
32, Oct. 2001.
[3] H. Do, S. Elbaum, and G. Rothermel. Supporting controlled
experimentation with testing techniques: An infrastructure
and its potential impact. Empirical Software Engineering,
10(4):405–435, 2005.
[4] T. Fawcett. An introduction to roc analysis. Pattern Recognition
Letters, 27:861–874, 2006.
[5] M. Haran, A. Karr, A. Orso, A. Porter, and A. Sanil. Applying
classification techniques to remotely-collected program
execution data. In Proc. 13th ACM SIGSOFT International
Symposium on Foundations of Software Engineering, pages
146–155, 2005.
[6] B. Liblit, A. Aiken, A. X. Zheng, and M. I. Jordan. Bug
isolation via remote program sampling. In Proc. ACM SIG-
PLAN 2003 Conference on Programming Languages Design
and Implementation, pages 141–154, 2003.
[7] B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan.
Scalable statistical bug isolation. In Proc. 2005 ACM SIG-
PLAN Conference on Programming Language Design and
Implementation, pages 15–26, June 2005.
[8] A. McCallum and K. Nigam. Acomparison of event models
for naive bayes text classification. In Proc. AAAI-98
Workshop on Learning for Text Categorization, pages 41–
48, 1998.
[9] A. Orso, T. Apiwattanaapong, and M. J. Harrold. Leveraging
field data for impact analysis and regression testing. In
Proc. 10th ACM SIGSOFT Symposium on the Foundations
of Software Engineering, pages 128–137, 2003.
[10] A. Orso, D. Liang, M. J. Harrold, and R. Lipton. Gamma
system: Continuous evolution of software after deployment.
In Proc. the International Symposium on Software Testing
and Analysis, pages 65–69, July 2002.
[11] C. Pavlopoulou and M. Young. Residual test coverage monitoring.
In Proc. 21st International Conference on Software
Engineering, pages 277–284, May 1999.
[12] J. C. Platt. Fast training of support vector machines using sequential
minimal optimization. Advances in kernel methods:
support vector learning, pages 185 – 208, 1999.
288