here - Stefan-Marr.de

8.4. Ad hoc vs. OMOP Performance
Table 8.1.: Experiments for Evaluation of Ad Hoc vs. OMOP-based Performance
Ad Hoc vs OMOP-based
AmbientTalkST CogVM vs CogVM with AST-OMOP
RoarVM (opt) vs RoarVM+OMOP (opt)
LRSTM CogVM vs CogVM with AST-OMOP
RoarVM (opt) vs RoarVM+OMOP (opt)
VM-based implementation (RoarVM+OMOP) can be compared. Ad hoc implementations
are executed on top of the RoarVM (opt) without OMOP support
to reflect the most plausible use case. Since VM-support for the OMOP
introduces an inherent performance overhead (cf. Sec. 8.5.2), it would put the
ad hoc implementations at a disadvantage if they were to execute on top of
RoarVM+OMOP (opt).
The proposed evaluation only measures minimal variation and uses a simple
bar chart with error bars indicating the 0.95% confidence interval. Since
measurement errors are insignificant, the error bars are barely visible and the
resulting Fig. 8.4 is more readable than a beanplot would have been. Fig. 8.4
only shows microbenchmarks. The runtime has been normalized to the mean
of the ad hoc implementation’s runtime measured for a specific benchmark.
The y-axis shows this runtime ratio with a logarithmic scale, and thus, the
ideal result is at the 1.0 line or below, which would indicate a speedup. All
values above 1.0 indicate a slowdown, i. e., the benchmark running on top
of the OMOP-based implementation took more time to complete than the
benchmark on top of the ad hoc implementation.
Results for Microbenchmarks The first conclusion from Fig. 8.4 is that most
LRSTM benchmarks on RoarVM+OMOP show on par or better performance
(gray bars). Only Array Access (STM) (28%), Class Var Binding (STM) (14%),
and InstVar Access (STM) (18%) show slowdowns. This slowdown can be explained
by the general overhead of about 17% for OMOP support in the VM
(cf. Sec. 8.5.2).
The AmbientTalkST benchmarks on the other hand experience slowdowns
throughout. These benchmarks indicate that the differences in how message
sends and primitives are handled in the ad hoc and the RoarVM+OMOP
implementation, have a significant impact on performance.
The results for the AST-OMOP implementation show more significant slowdowns.
The main reason for slowdown is that the implementation has to reify
215