Performance Modeling and Benchmarking of Event-Based ... - DVS

More documents

Recommendations

Info

78 CHAPTER 5. BENCHMARKING OF EVENT-BASED SYSTEMS validate them, but can also be exploited to study the effect of different platform configuration parameters on the overall system performance. However, if a benchmark is to be useful and reliable, it must fulfill the following fundamental requirements [123]: • It must be based on a workload representative of real-world applications. • It must exercise all critical services provided by platforms. • It must not be tuned/optimized for a specific product. • It must generate reproducible results. • It must not have any inherent scalability limitations The major goal of the SPECjms2007 benchmark is to provide a standard workload and metrics for measuring and evaluating the performance and scalability of JMS-based MOM platforms 1 . In addition, the benchmark should provide a flexible framework for JMS performance analysis. To achieve this goal, the workload must be designed to meet a number of requirements that can be grouped according the following five categories: 1. Representativeness 2. Comprehensiveness 3. Focus 4. Configurability 5. Scalability We now briefly discuss each of these requirements. Representativeness No matter how well a benchmark is designed, it would be of little value if the workload it is based on does not reflect the way platform services are exercised in real-life systems. Therefore, the most important requirement for the SPECjms2007 benchmark is that it is based on a representative workload scenario including a representative set of interactions, message types, message sizes and message delivery modes. The communication style and the types of messages sent and received by the different parties in the scenario should represent a typical transaction mix. The goal is to allow users to relate the observed behavior to their own applications and environments. Comprehensiveness Another important requirement is that the workload is comprehensive in that it exercises all platform features typically used in the major classes of JMS applications. Both the point-topoint and publish/subscribe messaging domains should be covered. The features and services stressed should be weighted according to their usage in real-life systems. There is no need to cover features of MOM platforms that are used very rarely in practice. The following dimensions have to be considered when defining the workload transaction mix: • Transactional vs. non-transactional messages. • Persistent vs. non-persistent messages. 1 For an introduction to MOM and JMS see Section 2.2.4
5.1. SPECJMS2007 - A STANDARD BENCHMARK 79 • Usage of different message types, e.g. TextMessages, ObjectMessages, StreamMessages or MapMessages. • Usage of messages of different sizes (small, medium, large). • Publish/subscribe vs. point-to-point messages (queues vs. topics). • One-to-one vs. one-to-many vs. many-to-many interactions. • Durable vs. non-durable subscriptions. • Ratio of message producers over message consumers. In the case of SPECjms2007 the definition of the subset of JMS functionality and weighting of the different functions was done with respect to the observations in practice of different vendors. Focus The workload should be focused on measuring the performance and scalability of the JMS server’s software and hardware components. It should minimize the impact of other components and services that are typically used in the chosen application scenario. For example, if a database would be used to store business data and manage the application state, it could easily become the limiting factor of the benchmark, as experience with previous benchmarks shows [126]. This is especially true in the case of SPECjms2007, since JMS servers, in their role as mediators in interactions, are typically less loaded than database or application servers. Another potential concern is the client side of the benchmark where messages are sent and received. The impact of client-side operations, such as XML parsing, on the overall performance of the benchmark should be minimized. Configurability As mentioned earlier, in addition to providing standard workload and metrics for JMS performance, SPECjms2007 aims to provide a flexible performance analysis framework which allows users to configure and customize the workload according to their requirements. Producing and publishing standard results for marketing purposes will be just one usage scenario for SPECjms2007. Many users will be interested in using the benchmark to tune and optimize their platforms or to analyze the performance of certain specific MOM features. Others could use the benchmark for research purposes in academic environments where, for example, one might be interested in evaluating the performance and scalability of novel methods and techniques for building high-performance MOM servers. All these usage scenarios require that the benchmark framework allows the user to precisely configure the workload and transaction mix to be generated. Providing this configurability is a challenge because it requires that interactions are designed and implemented in such a way that one could run them in different combinations depending on the desired transaction mix. The ability to switch interactions off implies that interactions should be decoupled from one another. On the other hand, it should be ensured that the benchmark, when run in its standard mode, behaves as if the interactions were interrelated according to their dependencies in the real-life application scenario. Scalability The SPECjms2007 application scenario must provide a way to scale the workload along the following two dimensions: 1. Horizontal scaling 2. Vertical scaling
Page 1:
Performance Modeling and Benchmarki
Page 4 and 5:
priorities. Finally, we validated o
Page 6 and 7:
4.3.4 Tool Extension . . . . . . .
Page 8 and 9:
5.7 Proportions of the Interactions
Page 11 and 12:
List of Acronyms Acronym λ t,k j A
Page 13 and 14:
Acronym SPEC-OSG SPE SPN SQL ST SUT
Page 15 and 16:
Chapter 1 Introduction 1.1 Motivati
Page 17 and 18:
PC Server 310 PC Server 310 PC Serv
Page 19 and 20:
1.3. APPROACH AND CONTRIBUTIONS OF
Page 21 and 22:
1.4. THESIS ORGANIZATION 7 • Stan
Page 23 and 24:
Chapter 2 Background In this chapte
Page 25 and 26:
2.1. EVENT-BASED SYSTEMS 11 is gene
Page 27 and 28:
2.2. TECHNOLOGY PLATFORMS OF EVENT-
Page 29 and 30:
Page 31 and 32:
Page 33 and 34:
Page 35 and 36:
2.3. INTRODUCTION TO QUEUEING PETRI
Page 37 and 38:
2.3. INTRODUCTION TO QUEUEING PETRI
Page 39 and 40:
Chapter 3 Related Work In this chap
Page 41 and 42: 3.2. BENCHMARKING OF EVENT-BASED SY
Page 43 and 44: 3.2. BENCHMARKING OF EVENT-BASED SY
Page 45 and 46: 3.4. CONCLUDING REMARKS 31 hierarch
Page 47 and 48: Chapter 4 Performance Engineering o
Page 49 and 50: 4.1. MODELING METHODOLOGY FOR EBS 3
Page 57 and 58: 4.2. PERFORMANCE MODELING PATTERN 4
Page 87 and 88: 4.3. EXTENSIONS OF QPNS 73 ,-).$%&'
Page 89 and 90: 4.4. CONCLUDING REMARKS 75 Without
Page 91: Chapter 5 Benchmarking of Event-Bas
Page 95 and 96: 5.1. SPECJMS2007 - A STANDARD BENCH
Page 115 and 116: 5.2. CASE STUDY I: SPECJMS2007 101
Page 125 and 126: 5.3. JMS2009-PS - A PUBLISH /SUBSCR
Page 127 and 128: 5.3. JMS2009-PS - A PUBLISH /SUBSCR
Page 129 and 130: 5.4. CASE STUDY II: JMS2009-PS 115
Page 131 and 132: 5.4. CASE STUDY II: JMS2009-PS 117
Page 133 and 134: Chapter 6 Performance Modeling of E
Page 135 and 136: 6.2. MODELING SPECJMS2007 121 Scena
Page 137 and 138: 6.2. MODELING SPECJMS2007 123 BASE*
Page 139 and 140: 6.2. MODELING SPECJMS2007 125 1‘O
Page 141 and 142: 6.2. MODELING SPECJMS2007 127 Sc. 1
Page 143 and 144:
6.2. MODELING SPECJMS2007 129 Perce
Page 145 and 146:
6.2. MODELING SPECJMS2007 131 Serve
Page 147 and 148:
6.3. CONCLUDING REMARKS 133 the mod
Page 149 and 150:
Chapter 7 Conclusions and Outlook W
Page 151 and 152:
7.1. ONGOING AND FUTURE WORK 137 JA
Page 153 and 154:
7.1. ONGOING AND FUTURE WORK 139 th
Page 155 and 156:
Bibliography [1] J. Abbott, K. B. M
Page 157 and 158:
BIBLIOGRAPHY 143 [29] BiCEP Researc
Page 159 and 160:
BIBLIOGRAPHY 145 [61] G. Cugola and
Page 161 and 162:
BIBLIOGRAPHY 147 [92] R. Henjes, M.
Page 163 and 164:
BIBLIOGRAPHY 149 [128] S. Kounev an
Page 165 and 166:
BIBLIOGRAPHY 151 [162] Object Compu
Page 167 and 168:
BIBLIOGRAPHY 153 [197] K. Sachs, S.
Page 169 and 170:
BIBLIOGRAPHY 155 [228] P. Tran, P.
Page 171:
Erklärung Hiermit erkläre ich, di
show all

Performance Modeling and Benchmarking of Event-Based ... - DVS

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?