here - Stefan-Marr.de

8.3. Baseline Assessment
the OMOP is likely to show up more significantly in the RoarVM (opt) with
its improved performance.
Since the Clang 3.0 compiler is used for development and experiments, its
performance is compared to OS X’s standard compiler GCC 4.2. Both compilers
were instructed to apply full optimizations using the -O3 compiler switch
(cf. Sec. 8.2).
The results are depicted in Fig. 8.3. This beanplot uses asymmetric beans to
allow an intuitive comparison between the results for Clang and GCC. The
results are normalized with respect to the mean runtime of the RoarVM compiled
with Clang 3.0. Thus, the black line indicating the mean is directly on
the 1.0 line, i. e., the baseline. The resulting distributions for GCC are depicted
in gray and indicate that the Clang compiler produces a slightly better result.
The RoarVM compiled with GCC is on average 3% slower and the results vary
over the different benchmarks between 1% and 5%. Since this difference is insignificant
compared to the overall performance differences between RoarVM
and CogVM, the evaluation proceeds with the Clang 3.0 compiler.
Another important aspect depicted in Fig. 8.3 22 is the relatively wide range
of measurement errors. The measurements show outliers up to 10%, even
though, the setup eliminates a range of common causes for nondeterminism
(cf. Sec. 8.2), others such as optimizations in the operating system, and caching
remain and need to be considered as potential measurement bias. For the
given results however, the distribution of measurements is visible in the graph
and indicates that the measured results are statistically significant.
Conclusion The measurements of the baseline performance of the RoarVM
and the CogVM indicate that the CogVM is about 11.0x faster on the kernel
benchmarks than the RoarVM (opt) (min 7.1x, max 14.4x). Consequently, the
CogVM is used as execution platform for the AST-OMOP, so as to provide it
with the best possible execution performance available.
The RoarVM (opt) is about 23% faster on the kernel benchmarks than the
RoarVM (std) (min 6%, max 43%). This evaluation therefore relies on these
optimizations to have an optimal baseline performance in order to assess the
overhead that the changes for the OMOP introduce.
22 The beanplot library of R enforced a linear scale for this graph.
213