Performance Modeling and Benchmarking of Event-Based ... - DVS

Chapter 7
Conclusions and Outlook
With the growing popularity of EBS and their gradual adoption in mission critical areas, the
need for novel techniques for benchmarking and performance modeling of EBS is increasing.
Since their reliability is crucial for the whole IT infrastructure, a certain QoS level has to be
ensured. The motivation for this thesis was to support the development and maintenance of
EBS that meet certain QoS requirements. Given that EBS differ from traditional software in
fundamental aspects such as their underlying communications paradigm, specific solutions and
concepts are needed. System architects and deployers need tools and methodologies, which
allow us to evaluate and forecast system performance and behavior in certain situations to identify
potential performance problems and bottlenecks. Benchmarks and performance modeling
techniques are usually the means of choice to answer these questions. However, no general performance
modeling methodologies focusing on EBS have been published yet. Furthermore, there
was a lack of test harnesses and benchmarks using representative workloads for EBS. Consequently,
we focused on the development of a performance modeling methodology of EBS as well
as on approaches to benchmark them. We summarize now our main contributions and proposed
approaches.
To comprehend our contributions, an understanding of the fundamental ideas of EBS is essential.
Therefore, we discussed our understanding of events in detail and introduced definitions
for different kinds of events and related concepts. Generally spoken, EBS are software systems
in which an observed event triggers a reaction. We evaluated the variety of underlying technologies
with a focus on DEBS and MOMs and provided a survey of existing DEBS and MOM
products and standards. In our review of existing work, we identified a lack of benchmarks and
performance modeling approaches for EBS. To support a structural evaluation of benchmarks,
we introduced five categories of requirements: (i) Representativeness: the benchmark has to be
based on a representative workload. (ii) Comprehensiveness: exercise all platform features typically
used in applications. (iii)Focus: place the emphasis on the technology server and minimize
the impact of other services, e.g., databases. (iv) Configurability: provide a configurable tool for
performance analysis. (v) Scalability: provide ways to scale the workload in a flexible manner.
None of the existing benchmarks met all our requirements. Therefore, we saw a strong need
for independent and standardized benchmarks for EBS fulfilling the requirements. We launched
a project inside the SPEC with the goal to develop the first industry standard benchmark for
EBS. As underlying technology plattform we chose JMS. This was motivated by the fact that
MOMs are widely used in industry and the quasi-standard for MOMs is JMS. Under the lead
of TU Darmstadt a team with members from organizations such as IBM, Sun and BEA was
formed. Our efforts resulted in the SPECjms2007 standard benchmark. Its main contributions
were twofold:
Standard Workload: Based on the feedback of our industrial partners, we specified a comprehensive
workload with different scaling options that fullfils all our requirements. It contains
135