Performance Modeling and Benchmarking of Event-Based ... - DVS
Performance Modeling and Benchmarking of Event-Based ... - DVS
Performance Modeling and Benchmarking of Event-Based ... - DVS
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
78 CHAPTER 5. BENCHMARKING OF EVENT-BASED SYSTEMS<br />
validate them, but can also be exploited to study the effect <strong>of</strong> different platform configuration<br />
parameters on the overall system performance. However, if a benchmark is to be useful <strong>and</strong><br />
reliable, it must fulfill the following fundamental requirements [123]:<br />
• It must be based on a workload representative <strong>of</strong> real-world applications.<br />
• It must exercise all critical services provided by platforms.<br />
• It must not be tuned/optimized for a specific product.<br />
• It must generate reproducible results.<br />
• It must not have any inherent scalability limitations<br />
The major goal <strong>of</strong> the SPECjms2007 benchmark is to provide a st<strong>and</strong>ard workload <strong>and</strong> metrics<br />
for measuring <strong>and</strong> evaluating the performance <strong>and</strong> scalability <strong>of</strong> JMS-based MOM platforms 1 .<br />
In addition, the benchmark should provide a flexible framework for JMS performance analysis.<br />
To achieve this goal, the workload must be designed to meet a number <strong>of</strong> requirements that can<br />
be grouped according the following five categories:<br />
1. Representativeness<br />
2. Comprehensiveness<br />
3. Focus<br />
4. Configurability<br />
5. Scalability<br />
We now briefly discuss each <strong>of</strong> these requirements.<br />
Representativeness<br />
No matter how well a benchmark is designed, it would be <strong>of</strong> little value if the workload it is<br />
based on does not reflect the way platform services are exercised in real-life systems. Therefore,<br />
the most important requirement for the SPECjms2007 benchmark is that it is based on a<br />
representative workload scenario including a representative set <strong>of</strong> interactions, message types,<br />
message sizes <strong>and</strong> message delivery modes. The communication style <strong>and</strong> the types <strong>of</strong> messages<br />
sent <strong>and</strong> received by the different parties in the scenario should represent a typical transaction<br />
mix. The goal is to allow users to relate the observed behavior to their own applications <strong>and</strong><br />
environments.<br />
Comprehensiveness<br />
Another important requirement is that the workload is comprehensive in that it exercises all<br />
platform features typically used in the major classes <strong>of</strong> JMS applications. Both the point-topoint<br />
<strong>and</strong> publish/subscribe messaging domains should be covered. The features <strong>and</strong> services<br />
stressed should be weighted according to their usage in real-life systems. There is no need to<br />
cover features <strong>of</strong> MOM platforms that are used very rarely in practice.<br />
The following dimensions have to be considered when defining the workload transaction mix:<br />
• Transactional vs. non-transactional messages.<br />
• Persistent vs. non-persistent messages.<br />
1 For an introduction to MOM <strong>and</strong> JMS see Section 2.2.4