SEKE 2012 Proceedings - Knowledge Systems Institute

Table 1. The precision and recall results of
the empirical study
CS θ ɛ P IIS R IIS ) P FIS R FIS P TCG R TCG
Transaction: V 0− >V1
2 0.72 0.55 0.51 0.83 0.47 0.93
0.2 3 0.72 0.56 0.51 0.84 0.47 0.93
4 0.69 0.61 0.52 0.84 0.47 0.93
5 0.66 0.63 0.52 0.84 0.47 0.93
2 0.75 0.51 0.59 0.79 0.47 0.93
0.3 3 0.71 0.57 0.57 0.79 0.47 0.93
4 0.71 0.57 0.59 0.77 0.47 0.93
5 0.67 0.64 0.55 0.82 0.47 0.93
20 2 0.79 0.51 0.72 0.65 0.47 0.93
0.4 3 0.70 0.59 0.63 0.72 0.47 0.93
4 0.71 0.59 0.65 0.69 0.47 0.94
5 0.66 0.66 0.58 0.76 0.47 0.93
2 0.75 0.49 0.67 0.56 0.47 0.93
0.5 3 0.75 0.55 0.68 0.62 0.47 0.93
4 0.71 0.60 0.65 0.68 0.47 0.93
5 0.65 0.65 0.58 0.73 0.47 0.93
Transaction: V 1− >V2
2 0.42 0.59 0.31 0.75 0.25 0.85
0.2 3 0.39 0.65 0.29 0.79 0.25 0.85
4 0.40 0.64 0.29 0.79 0.25 0.85
5 0.37 0.73 0.31 0.81 0.25 0.85
2 0.45 0.58 0.35 0.69 0.25 0.84
0.3 3 0.38 0.59 0.31 0.72 0.25 0.84
4 0.37 0.59 0.31 0.73 0.25 0.85
5 0.36 0.66 0.30 0.75 0.25 0.84
13 2 0.47 0.56 0.41 0.63 0.25 0.84
0.4 3 0.39 0.65 0.33 0.72 0.25 0.84
4 0.38 0.64 0.33 0.73 0.25 0.86
5 0.36 0.64 0.33 0.72 0.25 0.84
2 0.42 0.61 0.40 0.61 0.25 0.85
0.5 3 0.39 0.62 0.34 0.64 0.25 0.84
4 0.38 0.63 0.33 0.65 0.25 0.85
5 0.37 0.66 0.33 0.69 0.25 0.84
Transaction: V 2− >V3
2 0.85 0.67 0.25 0.93 0.17 0.93
0.2 3 0.80 0.72 0.25 0.94 0.17 0.94
4 0.81 0.77 0.25 0.93 0.17 0.93
5 0.52 0.84 0.25 0.94 0.17 0.94
2 0.82 0.64 0.57 0.91 0.17 0.94
0.3 3 0.84 0.71 0.58 0.90 0.17 0.93
4 0.83 0.78 0.56 0.92 0.17 0.94
5 0.48 0.84 0.37 0.92 0.17 0.93
12 2 0.85 0.66 0.78 0.83 0.17 0.93
0.4 3 0.83 0.72 0.76 0.84 0.17 0.94
4 0.81 0.76 0.73 0.84 0.17 0.93
5 0.47 0.82 0.44 0.88 0.17 0.93
2 0.87 0.66 0.84 0.74 0.17 0.93
0.5 3 0.79 0.72 0.76 0.79 0.17 0.93
4 0.80 0.77 0.77 0.83 0.17 0.93
5 0.51 0.83 0.50 0.88 0.17 0.93
the transaction V 1− >V2 are bad but they are very good
for transaction V 2− >V3. As we trace the changes to the
transactions of these two version evolution, we know that,
for V 1− >V2 transaction, the changes made to the system
are loosely scattered in different parts, in other words,
these changes are implemented individually with very few
relationships among them. But for V 2− > V3 transaction,
the changes are implemented in only some local parts
of the system and some of them are implemented together.
This phenomenon shows that interference among the
changes affects the effectiveness of the CIA. Hence, from
the discussion above, we see that selection of different CS,
θ and ɛ have impact on the CIA technique. The tendency of
their impacts are summarized in Figure 2. From this figure,
we know that when we know more information about the
changes, our CIA is better. And different values of these
two thresholds can control the precision and recall of the
CIA technique.
Then, we see the effectiveness of our CIA technique.
Our CIA computes two impact sets (IIS and FIS). Table
1 shows that the precision and recall of the IIS and
Figure 2. Impact of CS, θ and ɛ on the CIA
technique
FIS are different, to say in detail, in most of the time, the
precision of the IIS is better than that of the FIS while
the recall of the IIS is worse than that of FIS. The results
conform to our expectation and are useful in practice. In
practice, we can first use the IIS to more precisely check
the real change ripples. After this, we use the FIS to check
other change ripples to ensure the integrity of the change.
In addition, Table 1 also presents the precision and recall of
the traditional call graph based CIA technique (TCG)in
the last two columns. As it shows, the precision of our technique
is much higher than that of the TCG. But the recall
is a little worse than the TCG. This indicates that our CIA
technique effectively removes some false-positives from the
TCGtechnique at the cost of a little recall.
From the data and analysis above, we obtain some preliminary
conclusions: (1) the precision and recall of the impact
sets are affected by the ɛ and θ; (2) our CIA technique
is particularly useful to tackle multiple changes which have
some relationships (interference) among them. (3) our CIA
computes the IIS with higher precision and the FIS with
higher recall. Compared with traditional call graph based
CIA technique, our technique can effectively remove some
false-positives, but at the cost of a little recall.
3.3 Threats to Validity
Like any empirical validation, ours has its limitations.
Firstly, we have considered the application of our CIA approach
to only one Java program. Therefore, to allow for
generalization of our results, large-scale experimental evaluation
on different projects in different object oriented languages
is necessary to address such threats to external validity.
However, the program used here is real and non-trivial
programs, moreover, the subject program is selected from
open projects and widely used for empirical validation [11].
In addition, the change sets are obtained by randomly selecting
some differences between consecutive versions. S-
ince there exists random variation, we may not obtain the
same results if we repeat our case study. However, we performed
a statistical testing of our CIA technique with a variation
of the percentage of the number of changed methods
in the change set ranging from 10% to 50%. Using this s-
11