Lecture Notes in Computer Science 3472

15 Case Studies 459
abstract entities (states, operations etc.) of the model and concrete instructions
for the SUT’s test platform. By doing so, the tc-translator can easily be adjusted
for different test platforms. Often one operation in the model is a macro for the
tc-translator and is substituted by many instructions at concrete level (communication
abstraction). Furthermore the table may specify how to determine the
precise timing of operations in a concrete test case if the test model abstracts
from timing issues.
Other approaches pre-process the abstract test case before they translate
them. For example, Dushina et al. [DBG01] remove all variables of the abstract
test sequence not corresponding to inputs of the SUT from the generated test
sequence. Furthermore they extend the test sequence with reset instructions
before it translated in micro code.
15.8 Test Execution and Test Evaluation
In general, a simulation environment or a test framework (we call it test platform)
of the SUT executes and evaluates the concrete test cases. A test case is evaluated
to the verdicts pass or fail. Pass means that SUT conforms with the abstract
test model restricted to the behavior specified by the executed test case. Fail
means the reverse, i.e., an error was detected during test case execution. Clarke
et al. [CJRZ01] introduce in addition to pass and fail a third verdict called
inconclusive. Inconclusive means that test execution reached a point where no
error has occurred but the test purpose cannot be satisfied any more.
Test evaluation is based on a abstraction/concretization relation between the
traces of the abstract test model and the concrete system runs of the SUT that
has to be determined. None of the reviewed papers defines this relation explicitly.
Only Belinfante et al. [BFdV + 99] state that it is important to have it but do
neither describe it nor declare it. Nevertheless this relation is indicated implicitly
in the literature by the description of the applied test evaluation method. We
found the following approaches:
• The approach of Farchi et al. [FHP02] translates the complete abstract test
case consisting of inputs and expected outputs or observable states of the
SUT. The abstract inputs are translated to concrete stimuli of the SUT.
The abstract outputs and states are translated to executable verification
statements (verdicts) which decide if the output of the SUT conforms with
the test model or not. Thus the evaluation of the test is done at the concrete
level within the concrete test case.
• The approaches of Philipps et al. [PPS + 03] and Dushina et al. [DBG01]
translate only the inputs of the abstract test case to concrete level. These
concrete stimuli are injected in the SUT by the test platform. The test
platform monitors the outputs of the SUT, abstracts them to the abstract
level of the test model and compares them with the expected outputs encoded
in the abstract test case. Hence the test evaluation in done at the abstract
level by comparing the abstracted SUT outputs with the expected output of
the test model.