SEKE 2012 Proceedings - Knowledge Systems Institute

CATESR: Change-aware Test Suite Reduction
Based on Partial Coverage of Test Requirements
Lijiu Zhang † , Xiang Chen ‡ , Qing Gu †∗ , Haigang Zhao † , Xiaoyan Shi † , Daoxu Chen †
† State Key Laboratory for Novel Software Technology
Nanjing University, Nanjing, China
Email: jssyzlj@gmail.com
‡ School of Computer Science and Technology
Nantong University, Nantong, China
Email: xchencs@ntu.edu.cn
Abstract—Test suite reduction is an effective technique to
save the cost of regression test. This technique is performed by
identifying and eliminating redundant test cases from regression
test suite. However, fault detection ability of original test suite
may also be seriously weakened due to excessive reduction in
test suite. In this paper, we propose a new change-aware test
suite reduction approach CATESR, based on the conjecture that
test cases which are more likely to cover the changes after code
modification can gain a higher probability to reveal potential
faults in the modified code. To assess the effectiveness of our
approach, we implement CASTER and conduct a set of empirical
studies on eight real C programs. The results show that our
approach can not only greatly decrease the size of reduced test
suite but also keep, sometimes even improve, the fault detection
ability compared to HGS approach.
Index Terms—Regression Testing, Test Suite Reduction, Test
Requirement Classification, Change-aware
I. INTRODUCTION
During software development and maintenance, developers
often modify codes and then trigger software evolution.
Code modification may be caused by new user requirements,
faults detected, code refactoring, or performance improvement.
Regression testing is an important method to guarantee the
quality of software during its evolution. Statistical data shows
that the cost of regression testing accounts for over 1/3 of
total development cost [15]. Main research issues in regression
testing include test suite reduction, test case selection, test case
prioritization, and test suite augmentation [8].
In this paper, we mainly concentrate on test suite reduction
issue. Some of previous research mainly focus on proposing
novel approaches to reduce test suites according to a single test
coverage criterion [2] [3] [7] [16] [21]. However, Rothermel et
al. observed that test suite reduction can significantly decrease
the size of test suites but at the cost of seriously losing
original fault detection ability (FDA) [19]. To solve this issue,
researchers have proposed some approaches based on multiple
test coverage criteria. For example, Jeffrey and Gupta proposed
a selective redundancy concept [13]. Black et al. performed
test suite reduction using bi-criteria binary ILP model [1].
⋆ Correspondence author. Email: guq@nju.edu.cn
Then Hsu and Orso extended this work [1] and developed
a general tool MINTS [11].
Differing from previous research, we propose a changeaware
test suite reduction approach. According to the previous
approaches analysis, we find these approaches seldom analyzed
the relationship between code changes and test requirements.
In our research, we want to focus on test cases that
are more likely to cover the changes after code modification
in hope of detecting potential faults. Based on this conjecture,
we propose CATESR approach to perform test suite reduction
base on partial coverage of test requirements. In particular,
given a test suite TS and a set of test requirements R, we
firstly label R into three categories after code change analysis.
Secondly, we classify TS into three sub test suites based on the
requirement label result. Finally, we design a strategy named
Partial-CATESR, which only performs test suite reduction
over the sub test suite with the highest ability to cover the
code changes. However, this strategy maybe lose partial test
requirement coverage in most cases. Therefore, to verify the
effectiveness of Partial-CATESR, we also design a strategy
named Full-CATESR, which performs test suite reduction over
all three sub test suites respectively and combine these test
suites into a single test suite.
In our empirical study, we choose 7 real C programs in
Siemens suite and 1 large-scale C program named space
as our experiment subjects. In experiment setup, we choose
HGS approach as our test suite reduction approach and make
comparisons among HGS, HGS-based Partial-CATESR, and
HGS-based Full-CATESR. After data analysis, we find: (1)
Partial-CATESR can not only greatly decrease the size of
reduced test suite, but also keep, sometimes even improve,
the FDA of reduced test suite compared to HGS approach.
(2) The fault detection ability of reduced test suite are not
weakened due to the partial coverage of test requirements.
The main contributions of this paper include:
• we propose a change-aware test suite reduction approach
CATESR based on partial coverage of test requirements.
• we conduct an empirical study to verify the effectiveness
of our approach.
217