SEKE 2012 Proceedings - Knowledge Systems Institute

TABLE IV
TOOL’S CLASSIFICATION ACCORDING WITH TESTING LEVELS PLUS
REGRESSION TESTING
Unit Testing
SSTT [13],[25],[27],
[14],[15],[31],
[16],[32],[17],
[33],[35],[19],
[36]
SPLTT [40],[41],[43],
[44]
Integr.
Testing
[13],[26],
[31],[33]
System
Testing
[23],[28],
[29],[30],
[21],[22]
[39],[37],
[45]
Accept.
Testing
[23],[24]
[18],[20]
[42] [38]
Regr.
Testing
[27],[17],
[34],[20]
same idea for SPLTT. Every year, since 2004, at least one tool
for Unit Testing Level was published.
There is a clear evolution of tools at the Unit Testing Level.
We identified that JTest [14], Jwalk [16], and JUnitMX [35]
evolved from the xUnit [41]. A possible reason for this fact
was the release of new versions of the Java programming
language, which allowed the development of more complex
functionalities. Nevertheless, there is no visible evolution in
other testing levels of single system and SPL testing tools
including regression testing.
V. MAIN FINDINGS
The analysis of the results enables us to identify what have
been emphasized in past research and also to identify gaps
and possibilities for future research.
Based on the analyzed tools, it was possible to identify that
the tools are usually developed to support a specific testing
level, under the justification that there are no tools supporting
all functionalities of a testing process. None of the selected
tools supports the overall SPL life cycle. For example, the
prototype tool developed by [39] was created to assist the
ScenTED technique, and does not support a general testing
process. Moreover, the majority of the testing tools were
implemented to solve a specific problem.
The amount of SPL testing tools in academy (78%) is
higher than the number of single system testing tools (58%). In
industry this percentage is inverted: 25% of the SSTT and 11%
of SPLTT. The percentage is equivalent when the tools are
applied in both industry and academy (17% for single system
and 11% for SPL). In accordance with Table III, the amount
of projects applied in industrial context lacks investments.
During the research, we identified the possibility to use
single system testing tools such as JUnitMX [35], and Code-
Genie [31] to support the SPL testing process, however, it
will be necessary to change the methodology of using in
order to suit the tool for the SPL context. Another possibility
should be implement specific functionalities from SPL such as
variability management of the assets, but this will be possible
only for open-source applications. Tools such as AsmL [24],
and SimC [15] were developed to solve specific problems of a
specific environment, thus, the effort to adapt these tools can
be impracticable.
Despite the number of SSTT be twice higher than the
number of SPLTT, when the tools are organized by testing
levels, the percentage of single system and SPL testing tools
are equivalent. The only exception was the Integration Testing
Level since there was no SPLTT identified for this level.
The amount of SSTT used in industry added with tools used
in both academy and industry correspond to 41% of the total.
When we focus on SPLTT, this number decreases to 22%.
Thus, we believe that SPLTT do not achieve maturity enough
to be applied in industrial projects so far. Moreover, the effort
required to implement a SPL testing process into organizations
hampers the development of SPLTT.
Companies are interested in developing tools that could be
used by a large number of consumers. For this reason, they
look for attacking problems common to most customers. These
tools are implemented to test general functionalities, performance,
security, etc. Meanwhile, academy aims to produce
tools that solve complex problems, which can explain the large
number of Solution Proposal.
Figure 5 shows that 2007 was the year with a higher
number of publications describing SSTT (8 papers): 3 in the
International Conference on Software Engineering (ICSE’07),
2 in the International Conference on Automated Software
Engineering (ASE’07), and 3 in other conferences. These
two conferences together published the largest number of
papers describing SSTT over the years, 6 papers each one.
Coincidentally, 2007 was also the year with more tools focused
on System Testing Level. From all the tools of System Testing
Level analyzed, only 3 were applied in industry: 2 SSTT and
1 SPLTT.
The lack of SPLTT since 2009 could be explained by the
absence of the Software Product Lines Testing Workshop
(SPLiT) which is a great responsible by publications of this
kind of tools. The SPLiT is part of the Software Product Lines
Conference (SPLC) which is the main event of the SPL area.
Finally, we identified a tendency directed to Automatic Generation
Tools (50% of the analyzed tools) and according to [46]
“The dream would be a powerful integrated test environment
which by itself, as a piece of software is completed and
deployed, generating the most suitable test cases, executing
them and finally issuing a test report”.
However, tools for automatic test case, test input, and test
data generation, work independently. For this reason, the need
for a framework to integrate these tools still exists [3].
VI. THREATS TO VALIDITY
The main threats to validity identified in the review are
described next:
• Tools selection. A possible threat in such review is to
exclude some relevant tool. In order to reduce this possibility,
the tools selection was based on the identification
of the key research portals in computer science and a wide
range of web search engines, besides the main journals
and conferences. The defined criteria intended to select
tools supporting some functionalities of the SPL Test
Process and not just supporting specifics requirements.
• Data Extraction. In order to ensure the validity, multiple
sources of data were analyzed, i.e. papers, prototypes,
632