here - Stefan-Marr.de

8.2. Methodology
Generalizability beyond Interpreters The low performance of the RoarVM
precludes any generalization beyond interpreters. Since its implementation
is comparatively inefficient, it can hide performance effects that are caused
by the OMOP implementation. In any highly optimized VM implementation,
newly introduced overhead will have a higher impact than in an implementation
that is already slow to begin with.
Consequently, it is not possible to generalize the results to highly optimized
VMs that use JIT compilers. This is a serious limitation, but addressing it is
outside the scope of this dissertation and part of future work (cf. Sec. 9.5.2).
However, the results are transferable to common bytecode-based interpreters,
e. g., Lua 5.2, Python 3.3, Ruby 1.9, because they have similar execution characteristics.
8.2. Methodology
This section gives an overview of the methodology used for the execution of
the experiments as well as for the reporting in this dissertation.
8.2.1. Precautions for Reliable Results
The benchmarking methodology is derived from the advice of Georges et al.
[2007] for a practical statistically rigorous performance evaluation. Consequently,
it takes the following precautions for setting up the experiments:
• Measure steady-state performance: bytecode-interpreter VMs do not
have a warmup phase, however the CogVM (cf. Sec. 4.3) with its baseline
JIT compiler needs proper warmup to reach its steady-state. Currently,
it is sufficient for the CogVM to execute the benchmark twice before
the effectively measured run to ensure that the code is compiled and
ready. Since the CogVM does not support adaptive compilation, steady
state is reached and performance will remain unchanged for subsequent
execution.
• Minimize nondeterminism incurred by concurrently executing threads:
Since the performance evaluation focuses on the general performance
impact (cf. Sec. 8.1.3) and since the OMOP itself does not affect the characteristics
of parallel execution, 15 this evaluation focuses on sequential
15 As shown with the case studies (cf. Sec. 6.2), the OMOP does not change execution semantics,
but enables language implementers to enforce desired semantics more directly. It does
207