Performance Modeling and Benchmarking of Event-Based ... - DVS

5.1. SPECJMS2007 - A STANDARD BENCHMARK 99
Audit Test Scope Description
Input rate is within +-5%
of configured value
Interaction For each Interaction, the observed input rate is calculated
as count/time for the measurement period.
This must be within 5% of the value prescribed by
Total message count is
within +-5% of configured
value
Input rate distribution deviations
do not exceed
20%
90th percentile of Delivery
Times under 5000ms
All messages sent were received
Interaction
Interaction
Message
Destination
Message
Destination
the topology.
Using a model of the scenario, the benchmark knows
how many messages should be processed as part of
each Interaction. The observed number of messages
sent and received by all parties must be within 5% of
this value.
The percentage of pacing misses (in driver threads)
the benchmark will allow. A miss is when the timing
code found itself behind where it thought it should
be.
Messages are timestamped when sent and received.
The consequent delivery time is recorded as a histogram
(for the measurement period only) in each
event handler and the 90th percentile of this histogram
must be on or under 5000ms.
Fails if the final results (taken after the Drain Period)
show that not all sent messages were received. For
publish-subscribe topics this means received by all
subscribers.
Table 5.8: Audit Tests
Auditor
To validate the correctness of a benchmark run, SPECjms2007 comes with two auditor components:
• Pre-Auditor (executed in Phase I): The pre-auditor checks the availability of JMS
message destinations (queues / topcis) and whether messages exist before starting a benchmark
run which would invalidate a run (but might otherwise only be detected when the
test is completed). If the pre-audit fails for any reason the benchmark is not started.
• Auditor (executed in Phase III): The auditor component is responsible for making
sure that the run has been executed properly and that the results are valid. The Auditor
is automatically launched at the end of the run to validate the results. It performs an
explicit audit of the results against the specified requirements marking each audit test as
P ASS or F AIL in the summary reports. A list of all audit tests in provided in Table 5.8.
Minimizing the Impact of Non-MOM-Related Components
As discussed in Section 5.1.1, the SPECjms2007 workload should be focused on measuring
the performance and scalability of a MOM server’s software and hardware components and
minimize the impact of other components that are not directly related to the MOM services.
While developing SPECjms2007 two concerns had to be addressed in order to achieve this goal.
The first one is how to avoid using a database and the second one is how to minimize the
message processing overhead on the client. Given the fact that MOM servers, in their role as
mediators in interactions, are typically less loaded than database servers or application servers,