SEKE 2012 Proceedings - Knowledge Systems Institute

Siemens set contains seven C programs, and all of them can be
downloaded from the SIR repository [17]. Each of the
programs has a correct version, a number of faulty versions
seeded with a single fault, and a corresponding test suite. We
compare the output results of the correct versions with that of
the corresponding seeded versions to determine the failures.
Failures determined in this manner are called output-based
failures. According to the execution profile, if a faulty element
is executed during a test case, but no output-based failure is
detected, we categorize the test case as coincidentally correct.
Our study only takes into account 115 seeded versions, and
excludes the other versions because they contain code-missing
errors or the faulty statements are not executable. Figure 1
summarizes the result. It illustrates the exhibited level of
coincidental correctness is significant. The horizontal axis
represents the percentages of coincidentally correct tests (each
bar corresponds to a range of size 10%). The vertical axis
represents the percentage of seeded versions that exhibited a
given range. As can be seen, coincidental correctness is
common in software testing.
2.2 Safety Reducing Effect
Denmat et al. [15] pointed out the limitation of CBFL and
argued that the effectiveness of this technique largely depended
on the hypothesis that executing the faulty statements leads
most of the time to a failure.
In the following, we use Ochiai as an example to show that
coincidental correctness is a potential safety-reducing factor.
As shown in L. Naish et al., [16], the suspiciousness metric of
Ochiai is defined as:
M(e) =
( a
ef
aef
anf
)( a
ef
a
e = faulty program element
a ef = number of failed runs that execute e
a nf = number of failed runs that do not execute e
a ep = number of passed runs that execute e
Assume that there are k tests which execute e but do not
raise a failure. Two strategies can be applied on these tests to
improve the accuracy of the CBFL technique. The first strategy
% Faulty Versions
30
25
20
15
10
5
0
10 20 30 40 50 60 70 80 90 100
% Coincidental Correctnes(|Tcc|/|T|)
Figure 1. Frequency of Coincidental Correctness
ep
)
is to remove these tests from the test suite, that is, to subtract k
from a ep . Consequently, the suspiciousness metric will be:
M'(e) =
( a
ef
aef
anf
)( a
ef
a
ep
k)
It is easy to know that M(e) ≤ M’(e). To verify:
M’(e) ≥ 0, M(e) ≥ 0 and M’(e)/M(e) ≥ 1 => M(e) ≤
M’(e).
The second strategy is to relabel those tests from “passed”
to “failed”, i.e., to subtract k from a ep and add it to a ef , the
suspiciousness metric will be:
M'' (e) =
( a
ef
aef
k
anf
k)(
a
ef
a
It is easy to know that M(e) ≤ M’’(e). To verify:
M’’ 2 (e) -M 2 (e) ≥ 0 => M(e) ≤ M’’(e). It can be seen
that ignoring coincidentally correct test cases will leads to an
underestimating of the suspiciousness of the faulty element.
3 METHODOLOGY
3.1 General Process
Some symbols we use throughout the rest of the paper are
explained as follows:
T: the test suite used for a given program.
Tp: the set of passed test cases.
Tf:the set of failed test cases.
Tcc: the set of coincidentally correct test cases.
Ticc: the set of identified coincidentally correct tests.
Given a test suite T, which is comprised of Tp and Tf, the
goal is to identify Tcc from Tp. The result is Ticc, and each
element of Ticc is a potential candidate of the members of Tcc.
In this paper, we propose a clustering-based strategy to
obtain Ticc. The goal of cluster analysis is to partition objects
into clusters such that objects with similar attributes are placed
in the same cluster, while objects with dissimilar attributes are
placed in different clusters [7]. So, execution profiles are used
as the features fed to a clustering algorithm. Specifically, test
cases which execute the faulty elements and have similar
execution profiles with the failed test cases are likely to be
clustered together. Therefore, if a cluster consists of both failed
test cases and passed test cases, the passed test cases within this
cluster are very likely to be coincidentally correct. Note that
our approach is based on the single-fault assumption. Multifault
programs are not within the discussion of this paper, but
will be explored in the near future.
For a developer to find the fault with the help of automatic
fault-localization techniques, he/she can use the following
procedure to take advantage of our strategy to improve the
effectiveness of the diagnosis: First, a set of test cases is
executed on the given program. As a result, each test case is
labeled "passed" or "failed" according to its output result.
Execution profiles which reveal the coverage information are
ep
)
268