Lecture Notes in Computer Science 3472

14 Tools for Test Case Generation 435
These theoretical results coincide with the results obtained with the benchmarking
experiment discussed below.
Regarding the other theoretical aspects we have tried to give as much information
as possible in the tool descriptions. Not all facts (especially complexity
issues) are known for every tool and some aspects are still actual research topics.
Examples of the latter are compositionality, complex data and real time issues.
14.3.2 Benchmarking
The benchmarking approach takes the view that, as the proof of the pudding is
in the eating, the comparison (testing) of the test tool is in seeing how successful
they are at finding errors. To make comparison easier, a controlled experiment
can be set up. In such an experiment, a specification (formal or informal) is
provided, together with a number of implementations. Some of the implementations
are correct, others contain errors. Each of the tools is then used to try to
identify the erroneous implementations. Ideally, the persons doing the testing do
not know which implementations are erroneous, nor do they know details about
the errors themselves. Also, the experience that they have with the tools should
be comparable (ideally, they should all be expert users, to give each tool the
best chance in succeeding).
In the literature we have found a few references to benchmarking or similar
experiments.
Other disciplines, for example model checking, have collected over time a
common body of cases or examples, out of which most tool authors pick their
examples when they publish results of their new or updated tools, such that
their results can be compared to those of others.
In (model-based) testing this is much less the case, in our experience. Often
papers about model-based testing tools do refer to case studies done with the
tools, but usually the case studies are one-time specific ones. Moreover, many
of the experiments done for those cases cannot be considered controlled in the
sense that one knows in advance which SUTs are erroneous. This does make those
experiments more realistic – which is no coincidence since often the experiments
are done in collaboration with industry – but at the same time it makes it hard
to compare the results, at least with respect to error-detecting power of the tools.
Of course, there are exceptions, where controlled model-based testing experiments
are conducted and the results are published. In some cases those experiments
are linked with a particular application domain. For example, Lutess has
participated in a Feature Interaction contest [dBZ99].
Also independent benchmarking experiments have been set up, like the “Conference
Protocol Benchmarking Experiment” [BFdV + 99, HFT00, dBRS + 00] that
we will discuss in more detail below. The implementations that are tested in such
an experiment are usually much simpler than those that one has to deal with in
day-to-day real-life testing – if only to limit the resources (e.g. time) needed to
conduct or participate in the experiment. There is not much one can do about
that.