Performance Modeling and Benchmarking of Event-Based ... - DVS

5.1. SPECJMS2007 - A STANDARD BENCHMARK 97
For vertical and horizontal scaling, many of these values are either fixed or are automatically
calculated based upon the scaling factor.
With this knowledge, the controller instantiates the software components of the SUT. It
begins this by starting an RMI server and connecting to a satellite process on each node machine
identified as part of this test to give it specific instructions. In all places (throughout the
benchmark) where lists are given, work is distributed equally using a round-robin algorithm.
Period 2: Starting Agents The satellite is a simple part of the framework that knows how
to build the correct environment to start the required JVM processes. It takes the controller’s
configuration and starts the agents relevant to that node. There is an architectural lower bound
of one Agent-JVM per class of location (SP, SM, DC, HQ), meaning a minumum of four AgentJVMs
in the SUT. Each agent connects back to the controller to signal their readiness.
Period 3. Starting Event Handlers The controller signals all agents to initialise their
event handler threads and connect to the JMS resources they will be using (this includes both
incoming and outgoing Destinations). Each event handler is implemented as a Java thread.
Period 4: Warmup Period Load-generating threads (drivers) ramp up their throughput
from zero to their configured rate over the Warmup Period. This helps ensure the SUT is not
swamped by an initial rush when many of its constituent elements may not yet be fully prepared.
The agents are the only parts of the SUT which perform JMS operations (i.e. talk directly
to the JMS middleware).
Period 5: Measurement Period The measurement period is also known as the steadystate
period. All agents are running at their configured workload and no changes are made. The
controller will periodically (thirty seconds by default) check that there are no errors and may
also collect periodic realtime performance statistics.
Period 6: Drain Period In order to make sure all produced messages have an opportunity
to be consumed, the controller signals agents to pause their load-generating threads (drivers).
This period is not expected to be long in duration as a noticeable backlog in messages would
invalidate audit requirements on throughput anyway.
Period 7: Stopping Event Handlers Agents will terminate all event handlers but remain
present themselves so that the controller can collect final statistics.
Period 8: Post-processing Results Having collected statistics from all parties, the controller
begins post-process including auditing and preparation of the final results.
Reporter Framework
The benchmark includes a reporter framework that prepares detailed reports about the run. For
this purpose, the reporter framework enriches collected measurement data with other information,
e.g., throughput predictions and configuration settings.
The reporter framework has two major components:
1. Final Reporter
The controller takes formal measurements at three points during the run (see Figure 5.17).
The first two, the beginning and end of the measurement period, are used to audit the
messaging throughput. The final measurement, at the end of the drain period, is used to
audit the final message counts.