SEKE 2012 Proceedings - Knowledge Systems Institute

adequate to another criterion. This measure is important to
evaluate empirically the inclusion relation among criteria.
During the generation of test cases was observed the
effectiveness of each criteria to reveal the existent faults.
Some programs of the benchmarks Inspect, Helgrind and
Rungta have injected fault. If a test case belonging to an
adequate test set is able to reveal fault, this criterion is considered
effective to reveal this kind of fault.
4 Experimental Study: Results
The analysis and interpretation of the results are made
based on principles of the descriptive statistics and hypothesis
testing. Using the analysis of variance (ANOVA) [9], it
was verified whether it is possible to reject the null hypothesis
based on collected data set and statistical tests. The
descriptive analysis is useful to describe and to show graphically
interesting aspects of the study. There are different
perspectives to evaluate the cost of a testing criterion. In
this study we choose the size of the adequate test set and
the number of required elements. Table 1 presents information
of cost for some programs of the experiment, showing
the size of the adequate test set and the number of feasible
required elements for each testing criterion. The cost
data obtained for all programs of the experiment can suggest
an order to apply these criteria, considering initially
the criteria with minor cost: All-w-nodes (ANW), All-pnodes
(ANP), All-p-uses (APU), All-c-uses (ACU), Allnodes
(AN), All-comm-c-uses (ACCU), All-comm-p-uses
(ACPU), All-sync-uses (ASU), All-s-edges (AES) and Alledges
(AE).
Criteria MMult Lazy01 Jacobi Stateful06 Effectiveness
All-nodes 12/340 2/26 4/260 1/26 75%
All-p-nodes 12/54 1/11 2/31 1/7 50%
All-w-nodes 12/49 1/9 3/27 1/6 50%
All-edges 52/425 5/19 11/149 2/13 100%
All-s-edges 52/328 5/15 11/75 2/8 100%
All-c-uses 12/247 1/3 2/67 1/8 50%
All-p-uses 12/209 - 2/64 1/6 66%
All-sync-uses 52/270 4/9 - 2/6 75%
All-comm-c-uses 24/320 2/2 6/100 1/1 75%
All-comm-p-uses - 4/6 7/61 - 100%
effectiveness =
number of faults found
∗ 100
number of faults injected
In this analysis was considered the programs with injected
faults, totaling 23 programs. The results are presented
in Table 1. The criteria ACPU, AE and AES are the
most effective to reveal the faults, however there are some
kind of faults that are revealed for few testing criteria and,
in some cases, testing criteria with less effectiveness are
responsible to identify specific kind of faults. The results
of the ANOVA analysis suggest that is possible reject the
null hypothesis (NH2) and accept the alternative hypothesis
(AH2), because the p-value obtained is 0.007, less than
0.05. These results indicate that there is a considerable difference
of effectiveness among the testing criteria.
The equation below is used to calculate of strength of a
criterion C 1 :
strength C1 =
Number of elements covered by T C2
T otal of required elements − infeasible elements
Data analysis related to the strength was performed using
the statistical method of cluster analysis in order to identify
if there is an inclusion relation among the criteria. Dendrogram
charts are used to illustrate the results about inclusion
relation among criteria. If a criterion C1 includes another
criteria C2 by applying the adequate test set T C1 , these criteria
are at same level in the graph. Analysing the Dendogram
1 we can conclude that the ACCU criterion includes
the ANW criterion because both are at the same level in
the graph. Based on this result the null hypothesis (NH3)
can be rejected and the alternative hypothesis (AH3) is accepted,
indicating that the criteria can be complementary.
Table 1. Costs and effectiveness for some
programs in the experiment.
The test set size was used to evaluate the cost and to perform
the hypothesis testing. Based on the ANOVA analysis,
the null hypothesis (NH1) is rejected because the p-value
obtained for a pair of different criteria was 0.0009, less than
the significance level of 0.05. This result suggests that there
exist differences among the costs of these testing criteria. In
this case, the alternative hypothesis (AH1) is accepted.
The effectiveness is calculated by the following equation:
Figure 1. Dendrogram to All-comm-c-uses criterion.
The cluster analysis for the remaining criteria indicates
478