here - Stefan-Marr.de

8.3. Baseline Assessment
marking framework 17 . The benchmarks are executed based on a configuration
that consists of the lists of benchmarks and their parameters, as well as the
VMs and the necessary command-line arguments. ReBench 18 uses this configuration
to execute the benchmarks accordingly. The configuration is given in
Appendix B.2.
For the CogVM, an official 32-bit binary is used (cf. Sec. 8.1.3). The RoarVM
is equally compiled as a 32-bit binary. The compiler used is either Clang
3.0 or GNU GCC 4.2 with full compiler optimization enabled, i. e., using the
compiler switch -O3.
8.2.2. Presentation
Beanplots Most of the following diagrams use beanplots [Kampstra, 2008].
Beanplots are similar to violin and box plots. They show the distribution of
measurements, and thus, enable an visual comparison of benchmark results,
including an assessment of the significance of the observed differences (cf.
Fig. 8.1). Note that the beanplots are directly based on the measured data
points, i. e., they depict the distribution of the actual measurements and are
not synthesized from aggregated values. Asymmetric beanplots such as in
Fig. 8.3 are a variant that facilitates direct comparison of two data sets by
depicting both data sets for each criterion side-by-side. The first example in
the following section discusses the semantics of these diagrams in more detail.
Geometric Mean Note that all averages reported in this dissertation are
based on the geometric mean. For most experiments normalized measurements
are reported, and thus, the geometric mean needs to be used because
the arithmetic mean would be meaningless (cf. Fleming and Wallace [1986]).
8.3. Baseline Assessment
This section discusses baseline performance of the VMs used for the experiments.
The goal of this discussion is to provide an intuition of the performance
characteristics of the individual VMs before discussing the impact of
the modifications for the OMOP. This evaluation uses the kernel benchmarks
described in Appendix B.1.2. As discussed in Sec. 8.2, every benchmark is executed
100 times for each of the VMs. All benchmarks have been executed
with identical parameters.
17 http://www.squeaksource.com/SMark.html
18 http://github.com/smarr/ReBench
209