SEKE 2012 Proceedings - Knowledge Systems Institute

Validation
Research
SSTT [13],[14],[15],
[16],[17],[18],
[19],[20]
TABLE II
STUDIES BY RESEARCH TOPICS
Evaluation Solution Proposal
Research
[21],[22] [23],[24],[25],[26],[27],[28],
[29],[30],[31],[32],[33],
[34],[35],[36]
SPLTT [37],[38] [39] [40],[41],[42],[43],[44],[45]
identify different facets within the selected studies. More
details about the type of research as defined by [4] can be
seen in: http://wp.me/p157qN-3i
E. Data Extraction and Synthesis
All the 33 studies were fully read and submitted to a
predefined form to accurately record the information obtained
by the researchers from the primary studies. The form for data
extraction provides some standard information, such as:
• Tool’s name;
• Date of data extraction;
• Title, authors, journal, publication details (if available);
• Prototype information (if available);
• Website (if available); and,
• A list of each conclusion and statement found for each
question.
Based on the research results, inclusion and exclusion criteria,
a set of tools were selected. A brief description presented
ordered by the publication year of SSTT tools can be seen
in: http://wp.me/p157qN-2Z and SPLTT tools can be seen in:
http://wp.me/p157qN-32
IV. OUTCOMES
In this section, each research question is answered by
analyzing different point of views, highlighting the evidences
gathered from the data extraction process. All of these results
populated the classification scheme, which evolved while
doing the data extraction.
Initially, we analyzed the studies distribution regarding to
the research topics. The classification scheme, detailed in Table
II, allowed us to infer that many of the studies are Validation
Research (31%), Evaluation Research (9%) and Solution Proposal
(60%). On the other hand, no Philosophical, Opinion,
and Experience Papers were reported. For this reason, Table
II did not show them.
A. Tools Development and Usage
From the selected tools, most of them were developed in the
academic environment (14 for single systems and 7 for SPL),
while 7 were developed exclusively in industry (6 for single
system and 1 for SPL). The remaining 5 tools were developed
in both environments, academic and industrial (4 for single
systems and 1 for SPL), as detailed in Table III.
[13], [17], and [19], present case studies executed in an
academy context. [21], [22], and [39], describe case studies
in industry. Finally, [14], [15], [16], [18], [20], [37], and [38],
report experiments in order to evaluate the tools. The other
studies did not describe empirical evaluation.
TABLE III
WHERE THE TOOLS WERE DEVELOPED AND USED
Academy Industry Both
SSTT [23],[25],[26],[27],[15],[14],[28],
[29],[30],[22],[31],[16],[36],[20]
[24],[21],[32],
[34],[35],[19]
SPLTT [41],[42],[43],[37],[44],[38],[45] [40] [39]
[13],[17],
[33],[18]
B. Software Product Lines Adaptability
According to the results (see Table II), only 33% of the
single system testing tools can be used to test SPL. The
other ones (Solution Proposal) were implemented to specific
programming languages, techniques and approaches which
cannot be suitable to the SPL context.
Websob [29], Korat [30], CodeGenie [31], JWalk [16],
Smart [32], REST [34] and JUnitMX [35] are tools able to
be utilized in the SPL testing process.
Testing tools with specific purpose such as: test management,
bug reports, security test, can be used in Software
Product Lines testing if the methodology of using it is adapted
to suit the SPL necessities. Commonalities and variabilities
should be considered also.
C. Testing Levels
Many of the analyzed tools have similar functionalities.
Moreover, the extracted functionalities have analogous goals,
so it was possible to group them. This classification matched
the description presented in [6], which defines the following
groups:
• Unit Testing - Tools that test the smallest unit of software
implementation. This unit can be basically a class, or even
a module, a function, or a software component [10].
• Integration Testing - Tools that test the integration
between modules or within the reference in domainlevel
when the architecture calls for specific domain
components to be integrated in multiple systems [10].
• System Testing - Tools that ensure that the final product
matches the required features [37].
• Acceptance Testing - Tools that will be used by customers
during the validation of applications [10].
• Regression Testing - Even though regression testing is
not a test level [6], some tools were developed to work
with it. For this reason, we considered regression testing
as part of the classification.
This classification can be applied not only for single system
testing tools but also for SPL testing tools. The main difference
is that SPL divide Testing according to two activities [10]:
core asset (grouping Unit and integration testing) and product
development (grouping system and acceptance testing). Table
IV details the classification of the tools according to the testing
level plus regression testing.
D. Testing Tools Evolution
In order to identify the evolution of the tools, we constructed
a timeline for SSTT showed in Figure 5. Figure 6 shows the
631