SEKE 2012 Proceedings - Knowledge Systems Institute

A Mapping Study on
Software Product Lines Testing Tools
Crescencio Rodrigues Lima Neto 1,3 , Paulo Anselmo Mota Silveira Neto 3
Eduardo Santana de Almeida 2,3 , Silvio Romero de Lemos Meira 1,3
1 Center for Informatics - Federal University of Pernambuco (CIn/UFPE)
2 Computer Science Department - Federal University of Bahia (DCC/UFBA)
3 Reuse in Software Engineering Labs (RiSE)
{crln, srlm}@cin.ufpe.br, pamsn@rise.com.br, esa@dcc.ufba.br
Abstract—The benefits of using a software testing tool in order
to achieve significant reductions in cost and time to market,
and, at the same time, increasing the quality has encouraged
the adoption of testing tools both in single systems and product
lines. In this context, this study focuses on the following goals:
analyze how the available tools are supporting the Software
Product Lines (SPL) Testing Process, investigate the state-of-theart
on single system and SPL testing tools, synthesize available
evidence, and identify gaps among the tools, available in the
literature. A mapping study was undertaken to analyze important
aspects that should be considered when adopting testing tools. A
set of four research questions were defined in which 33 studies,
dated from 1999 to 2011, were evaluated. From the total of 33
studies considered, 24 of them described single system testing
tools and the other 9 described SPL testing tools. However, there
is insufficient information about publications describing tools
used in the industry. There is no tool suitable to all testing levels
of a SPL, researchers need to consider the feasibility of adapting
existing tools or constructing new tools.
Keywords—Software Testing; Software Product Lines; Software
Reuse; Testing Tools; Systematic Literature Reviews
I. INTRODUCTION
In order to achieve the ability to build individualized products,
to meet individual customer needs, companies need high
investments, which consequently leads to higher prices for
final products. In this context, the increasing adoption of Software
Product Lines (SPL) practices in industry has decreased
implementation costs, reduced time-to-market and improved
quality of products. Although the benefits are promising, SPL
advantages demand mature software engineering practices,
planned and managed reuse [1].
Testing is an essential part of software development to
address product quality, since it examines the core assets, the
product-specific software, the interaction between them, and
finally the completed products [2].
Testing tools are important to increase the quality of testing
and at the same time reduce the effort to perform them.
However, existing testing tools (see Section IV) are mostly
unable to support SPL Testing [3].
This study has been structured combining ideas from [4]
with good practices defined in the guidelines proposed by
[5], such as the protocol definition. Therefore, we applied a
process for a mapping study (MS), including best practices for
conducting systematic reviews (SR), making the best use of
both methods.
The objective of this study is to investigate how do the
available tools support the Software Product Lines Testing
process? This study aims to map out existing testing tools,
to synthesize evidence to suggest important implications for
practice, as well as to identify research trends, open issues, and
areas for improvement. We derived four research questions:
• RQ1 - In which context the proposed tools were proposed
and evaluated? It aims at identifying where the tools were
developed (industry or academia) and how they have been
validated. Through these descriptions, it is possible to
map the current adoption of the tools.
• RQ2 - Is it possible to use the single system testing
tools to test software product lines? It aims to analyze
the possibility of using the single system testing tools to
test SPL instead of creating tools from scratch.
• RQ3 - Which testing levels are supported by existing
tools? It aims to classify the tools according to the testing
levels in order to identify what the tools can offer. It
can provide information regarding to further research in
a specific area or testing level.
• RQ4 - How are testing tools evolving? It aims to analyze
how the existing testing tools are evolving, regarding to
the different testing levels.
Furthermore, we believe that this paper can provide insights
for new research in the software testing tools area, serving as
a baseline, by analyzing the available tools and presenting the
existing gaps.
The remainder of this paper is structured as follows: Section
II presents the related work. Section III describes the research
method used. Section IV presents the analysis of the results.
Section V presents a discussion on the results and the main
findings of the study. Section VI presents the threats to validity,
and finally, Section VII discusses the main conclusions.
II. RELATED WORK
Several studies [6], [3], [7], [8], emphasize the need for
testing tools to support the Testing Process, however, few ones
628