here - Stefan-Marr.de

8. Evaluation: Performance
benchmarks, i. e., benchmarks that are executed on a single processor
core. This removes a strong source of noise in the results and improves
the quality of our measurements.
• Minimize nondeterminism incurred by Intel’s TurboBoost: Modern processors
often support dynamic frequency changes for processor cores to
manage power consumption and heat dissipation. To avoid an influence
of such techniques on measurements, they need to be disabled. 16
• Minimize nondeterminism incurred by garbage collection: To avoid measurement
noise coming from different GC strategies in the different
VMs, the heap size are adjusted to a large enough value of 1024MB,
and thus, GC operations are avoided.
• Problem sizes: The problem size of all benchmarks is adjusted to yield
runtimes in the order of hundreds of milliseconds, ideally between
300ms to 1000ms. In some experiments, the runtimes go well beyond
this range, because of the different performance characteristics of the
VMs used. However, fixing the lower bound to an appropriate value
avoids influence of potentially imprecise timer resolutions provided by
the different VMs.
Execution With these precautions in place, systematic bias in the experiments
is reduced. However, there remain other nondeterministic influences.
For instance, operating system features such as virtual memory and scheduling
of other processes cannot completely be disabled. Adjusting process priority
helps partially, but other techniques such as caching in the processors
remain. The remaining influences are handled by executing each benchmark
100 times to obtain statistically significant results.
Hardware And Software Setup The machine used for these experiments is
a MacPro (version 4,1) with two quad-core Intel Xeon E5520 processors at
2.26 GHz with 8 GB of memory. Note that since the benchmarks are purely
sequential, the number of cores is not relevant. The operating system is OS
X 10.6.7 (10K549). The benchmarks are implemented using the SMark benchnot
provide mechanisms that interfere with parallel execution: it facilitates concurrent programming
in contrast to parallel programming (cf. Sec. 2.3.2).
16 On Mac OS X this is possible by using a kernel module:
https://github.com/nanoant/DisableTurboBoost.kext
208