SEKE 2012 Proceedings - Knowledge Systems Institute

Figure 1. An example to illustrate the process of CIA
3.1 Setup
We use the subject NanoXML 1 as our research object.
NanoXML is a small XML parser for Java. We extracted
four consecutive versions of NanoXML from its CV S
repository for evaluation. There are about 26 classes, 245
methods, and 7631 lines of code for NanoXML. During the
evolution of these versions, there is a basic version (V 0)
from which to start.
The focus of our evaluation is on the accuracy of the CIA
technique. The accuracy of the CIA is in terms of precision
(P ) and recall (R). Precision is an inverse measure of
false-positives while recall is an inverse measure of falsenegatives.
They are widely used in an information retrieval
scenario [2], defined as follows:
P =
|Estimated Impact Set|
R =
Here, Actual Impact Set is the set of methods which are
really changed during versions evolution. This is obtained
by comparing the changed methods between consecutive
program versions. Estimated Impact Set is the set of methods
estimated based on the change set using the CIA technique.
As we can not obtain the information that which
changed methods are pre-actively changed to cope with the
change request and which changed methods are reactively
changed to cope with the impacts induced by these changes,
a statistical testing of our CIA technique with a variation of
the percentage of the number of changed methods ranging
from 10% to 50% of the total number of change methods is
performed for each program transaction. In the study, the
average number of the methods in the change set for each
program transaction (i.e., V i − >V i+1 ) is shown in the first
column in Table 1.
3.2 Results
|Actual Impact Set ∩ Estimated Impact Set|
|Actual Impact Set ∩ Estimated Impact Set|
|Actual Impact Set|
During the process of the evaluation, we should first find
appropriate values for the threshold ɛ and θ. For space limi-
1 http://nanoxml.sourceforge.net/orig
tation, we only provide some results with high quality here.
More details of the preliminary evaluation results are available
online 2 . Based on the empirical results, the precision
and recall of both IIS and FIS are reasonable and better
with the range of ɛ =2, 3, 4, 5, θ =0.2, 0.3, 0.4, 0.5. So the
experimental results only show the precision and recall of
these threshold ranges, as shown in Table 1. In addition, for
comparison, we also present the precision and recall of the
impact set computed based on traditional call graph based
CIA technique (TCG) [3]. In the following, we discuss the
data collected from the case study.
First, we see whether different threshold values have impact
on the accuracy of our CIA. In Table 1, the third column
shows different ɛ values. The threshold ɛ indicates the
cohesion of the relationship among the methods and is used
to control the precision of the initial impact set. From this
table, we see that in most cases, with the increase of the ɛ
values, the recall of both the IIS and FISis also increased,
but their precision values are decreased. From these results,
we see that the FIS relies on the IIS, i.e., if the IIS
is better, the FIS is better. Thus, the ɛ threshold has impact
on both the IIS and FIS generated by our CIA technique.
Then, we see the impact of the θ threshold on our
CIA technique. This threshold relies on the quality of the
IIS, and is used to indicate the possibility of the methods
potentially affected by the IIS. As Table 1 shows, with
the increase of the θ values, the precision of the FIS is also
increased, but the recall is decreased. For example, for
transaction V 0− >V1, in case of ɛ =3, when θ =0.2, the
precision and recall of FIS is 0.51 and 0.84, respectively.
When θ =0.3, its precision is increased to 0.57, but its recall
is decreased to 0.79. With the increase of θ values, the
same phenomenon occurs. So the effectiveness of the CIA
technique is also affected by the θ threshold. In addition,
we see an interesting phenomenon in Table 1, i.e., for different
transactions of the versions evolution, the precision
and recall of the impact sets with the same ɛ and θ threshold
value are different. For example, the precision and recall for
2 http://ise.seu.edu.cn/people/XiaobingSun/seke2012.xls
10