Lecture Notes in Computer Science 3472

11 Evaluating Coverage Based Testing 295
relatedtomodels:forexample,acoveragecriterion may be to cover all the states
of a model.
This chapter provides an overview of different research activities dealing with
the coverage based testing techniques and their evaluation. In Section 11.2, we
discuss how coverage criteria can help to select meaningful test suites. In Section
11.3, we present common coverage criteria. In Section 11.4 we concentrate
on providing quantitative elements to evaluate the ability to detect faults of coverage
based testing techniques. Finally, in Section 11.5, we summarize the main
results and present open problems.
11.2 Coverage Criteria
To address the question Which test cases should be extracted from a possibly
infinite set of test cases ? it is possible to use advanced specifications describing
which test cases to choose. Such specifications are usually called test case
specifications (cf. Figure 11.1), and are, strictly speaking, selection criteria.
The main reasons for an infinitely large test suite are that a specification
can represent an infinite number of traces (e.g., caused by variables ranging over
infinite domains) and that a specification can contain infinite traces (e.g., caused
by loops). For the first reason the problem is to select traces for the second reason
the problem is to chop traces. Basically, the specification (ideally) describes all
possible test cases while the test case specification describes which of these test
cases are likely to find errors and, consequently, increase the confidence in the
correctness of the implementation. Most of the criteria used for selecting test
cases can be coarsely classified according to the following three aspects:
(1) The test case specification is a description of a structural criterion which
should be covered.
(2) The test case specification is a description of functional aspects, also called
scenarios, which should be covered.
(3) The test case specification contains stochastic information about different
aspects of the implementation which is be used to concentrate on particular
details.
The third variant is also used to restrict a test case specification if this
represents an infinite model (for example, if path coverage is required which is
usually infeasible to achieve as explained in the following section).
11.2.1 Structural Criteria
Basically, coverage criteria are used to measure the quality or, more precisely,
the adequacy of test suites: A test suite is adequate according to a criterion if a
designated percentage of this coverage criterion is reached. Depending on this,
testing is continued or stopped. A structural criterion is an assertion about the
structure of the specification. For example, in model based specifications states