here - Stefan-Marr.de

8. Evaluation: Performance
General Performance Ratios The beanplot in Fig. 8.1 is a visualization of
the distribution of the kernel benchmark results for three RoarVM variants,
as well as the results for the CogVM.
Note that Fig. 8.1 uses an unconventional approach to represent overall performance
of the VMs. The presented data have not been aggregated to produce
the plot. Instead, it is an overview of all measurements, which Fig. 8.2
presents in more detail. The goal of this plot is to combine all measurements
and present them so as to compare performance effects of different VMs
within a single representation.
The experiment uses eight kernel benchmarks for the measurements and
the bean plot for the RoarVM (opt) shows these eight benchmarks as eight
distinct accumulations of measurements. Each of the benchmarks has a distinct
average around which the measurements are concentrated. This becomes
visible in this representation as pearls on a necklace. For each of the VMs, the
plot is generated from 800 distinct measurements, which are depicted as distributions.
Comparing the RoarVM (opt) and the RoarVM (std) shows that there are two
benchmarks that are affected less by the differences in VMs than the other six
benchmarks, i. e., the two pearls at the bottom move barely visible, while the
other six pearls move up and also differ in their distance between each other.
This representation visualizes the impact on the execution of the corresponding
benchmarks, but only provides a rough overview without detail.
The only aggregation in this representation is the black average line, which
averages over all measurements for a VM, and provides an additional intuition
of the overall performance compared to each other.
Since the overall performance of the CogVM is a magnitude better, results
are closer together and the individual benchmarks become indistinguishable
for the scale chosen for the plot. A plot with a scale from 0 to 500 would
reveal a similar pattern as is visible for the RoarVM variants.
The main insight to be derived from this plot is that the CogVM is significantly
faster than any of the RoarVM variants. Furthermore, the differences
between the RoarVM variants have a different impact on the benchmarks.
Consider all benchmarks separately as depicted in Fig. 8.2, the CogVM is on
average 11.0x 19 faster than the RoarVM (opt). 20 Depending on the benchmark,
the difference is between 7.1x and 14.4x. These performance variations for the
different benchmarks are a result of their specific characteristics and their
19 The reported averages in this chapter consistently refer to the geometric mean.
20 A speedup or slowdown reported as nx refers to the ratio VM 1 /VM baseline . In this case it is:
CogVM/RoarVM (opt).
210