Performance Modeling and Benchmarking of Event-Based ... - DVS

More documents

Recommendations

Info

136 CHAPTER 7. CONCLUSIONS AND OUTLOOK several configuration parameters for evaluating selected performance and scalability aspects of MOM. Benchmarking Framework: The benchmark was implemented using a newly developed complex framework. The framework offers many additional features, is highly configurable and easy to extend. Examples of its features are a comprehensive reporter and the availability of statistics at run time. We introduced a methodology for the performance evaluation of MOMs using the SPECjms2007 standard benchmark. We showed how the workload can be customized to exercise and evaluate selected aspects of MOM performance to reflect a given target customer workload. Additionally, we demonstrated our methodology in a case study of a leading JMS platform, the WebLogic server, and conducted in-depth performance analyses of the platform for a number of different workload and configuration scenarios. The SPECjms2007 business scenario was specified independently from the underlying technology. Therefore, its usage is not limited to a specific type of EBS. We illustrated how the standardized workload can be applied to other EBS using the example of jms2009-PS, a benchmark for publish/subscribe-based communication. This benchmark provides a flexible framework for performance analysis with a strong focus on research. Both benchmarks, SPECjms2007 and jms2009-PS, are used in several projects by industry and academia and, since they exercise MOM in a realistic way, they are also used as reference applications. To the best of our knowledge, no work introducing a general methodology for modeling EBS has been published yet. As a consequence, we investigated whether and how traditional performance modeling approaches are suitable to model the specifica of EBS. We introduced a formal definition of EBS and their performance aspects, which allows us, among other things, to describe workload properties and routing behavior in a structured way. Resulting from our analysis of existing modeling techniques, we proposed a novel approach to characterize the workload and to model the performance aspects of EBS. We used operational analysis techniques to describe the system traffic and derived an approximation for the mean event delivery latency. We then showed how more detailed performance models based on QPNs could be built and used to provide more accurate performance prediction. We chose queueing Petri nets as performance modeling technique because of their modeling power and expressiveness. Our approach allows evaluating and predicting the performance of an EBS and provides detailed system models. It can be used for an in-depth performance analysis and to identify potential bottlenecks. A further contribution was a novel terminology for performance modeling patterns that we used to introduce eleven patterns targeting common aspects of event-based applications. These performance modeling patterns were the first ones published, which a) target EBS applications and b) use QPNs. To additionally improve the modeling power of QPNs, we suggested several extensions of the standard QPNs, which allow building models in a more flexible and general way and address several limitations of QPNs. By introducing a level of abstraction, it is possible to distinguish between logical and physical layers in our models. This enables to flexibly map logical to physical resources and thus makes it easy to customize the model to a specific deployment. The different layers allow us to reuse one logical model in several physical models or to map several logical models to one physical model. Further, we addressed two limiting aspects of standard QPNs: constant cardinalities and the lack of transition priorities. By introducing non-constant cardinalities of transitions we increased the modeling flexibility and minimized the number of transition modes. The missing support of transition priorities in standard QPNs limits the control of the transition firing order. We addressed this restriction by incorporating priorities for transitions into QPNs and discussed several ways to implement them. Our extensions were integrated in the QPME / SimQPN software tools or are planned for the upcoming release. Finally, we validated the approach in two case studies. We applied our methodology to model EBS and predicted their performance and system behavior under load successfully. As part of the
7.1. ONGOING AND FUTURE WORK 137 JA(27%-+09+,#5"76#+ D0%(>0$)E+F11276(>0$)+($*++ G$*#52-7$8+H#6:$02087#)+ @A$>.#+B0$7%057$8+C+ 45#*76>0$+ !"#$%&'()#*+,-)%#.)+ /0.12#3+!"#$%+4506#))7$8+ 0$+++ '#$6:.(5;7$8+09+!',++ '#$6:.(5;7$8+ B#%:0*0208-+ 4#5905.($6#++ !$87$##57$8+ '#$6:.(5;+ !$87$##57$8+ D#I+H5#$*)+ !"#$%&'!$$%()"(!$$%&'*' Figure 7.1: Open Research Issues first case study we extended SIENA, a well-known DEBS, with a runtime measurement framework. We evaluated a system with 15 brokers, 8 publishers and 16 subscribers and predicted the runtime behavior including delivery latency for a basic workload with a single event type. In the second case study, we developed a comprehensive model of the complete SPECjms2007 workload including the persistent layer, point-to-point and publish/subscribe communication. We applied several of our performance modeling patterns. Furthermore, the workload was modeled using the proposed QPN extensions. We evaluated its accuracy in a commercial middleware environment. To validate our modeling technique we investigated deployments of the benchmark in representative environments comparing the model predictions against measurements on the real systems. A number of different scenarios with varying workload intensity (up to 30,000 messages / 4,500 transaction per second) and interaction mixes were taken into account. By means of the proposed models we were able to predict the performance accurately. To the best of our knowledge, no models of realistic systems of the size and complexity of the one considered in this thesis exist in the literature. The results of both case studies demonstrated the effectiveness and practicality of the proposed modeling and prediction methodology in the context of a real-world scenario. The advantage of the proposed approach is that it is both practical and general, and it can be readily applied for performance evaluation of DEBS and MOM. The technique can be exploited as a powerful tool for performance prediction and capacity planning during the software engineering lifecycle of message-oriented event-driven systems. 7.1 Ongoing and Future Work Based on our experience and the findings of this work, we identified three interesting areas for future research: • Benchmark engineering • Workload characterization and benchmarking of event-based systems
Page 1:
Performance Modeling and Benchmarki
Page 4 and 5:
priorities. Finally, we validated o
Page 6 and 7:
4.3.4 Tool Extension . . . . . . .
Page 8 and 9:
5.7 Proportions of the Interactions
Page 11 and 12:
List of Acronyms Acronym λ t,k j A
Page 13 and 14:
Acronym SPEC-OSG SPE SPN SQL ST SUT
Page 15 and 16:
Chapter 1 Introduction 1.1 Motivati
Page 17 and 18:
PC Server 310 PC Server 310 PC Serv
Page 19 and 20:
1.3. APPROACH AND CONTRIBUTIONS OF
Page 21 and 22:
1.4. THESIS ORGANIZATION 7 • Stan
Page 23 and 24:
Chapter 2 Background In this chapte
Page 25 and 26:
2.1. EVENT-BASED SYSTEMS 11 is gene
Page 27 and 28:
2.2. TECHNOLOGY PLATFORMS OF EVENT-
Page 29 and 30:
Page 31 and 32:
Page 33 and 34:
Page 35 and 36:
2.3. INTRODUCTION TO QUEUEING PETRI
Page 37 and 38:
2.3. INTRODUCTION TO QUEUEING PETRI
Page 39 and 40:
Chapter 3 Related Work In this chap
Page 41 and 42:
3.2. BENCHMARKING OF EVENT-BASED SY
Page 43 and 44:
3.2. BENCHMARKING OF EVENT-BASED SY
Page 45 and 46:
3.4. CONCLUDING REMARKS 31 hierarch
Page 47 and 48:
Chapter 4 Performance Engineering o
Page 49 and 50:
4.1. MODELING METHODOLOGY FOR EBS 3
Page 51 and 52:
Page 53 and 54:
Page 55 and 56:
Page 57 and 58:
4.2. PERFORMANCE MODELING PATTERN 4
Page 59 and 60:
Page 61 and 62:
Page 63 and 64:
Page 65 and 66:
Page 67 and 68:
Page 69 and 70:
Page 71 and 72:
Page 73 and 74:
Page 75 and 76:
Page 77 and 78:
Page 79 and 80:
Page 81 and 82:
Page 83 and 84:
Page 85 and 86:
Page 87 and 88:
4.3. EXTENSIONS OF QPNS 73 ,-).$%&'
Page 89 and 90:
4.4. CONCLUDING REMARKS 75 Without
Page 91 and 92:
Chapter 5 Benchmarking of Event-Bas
Page 93 and 94:
5.1. SPECJMS2007 - A STANDARD BENCH
Page 95 and 96:
Page 97 and 98:
Page 99 and 100: 5.1. SPECJMS2007 - A STANDARD BENCH
Page 115 and 116: 5.2. CASE STUDY I: SPECJMS2007 101
Page 125 and 126: 5.3. JMS2009-PS - A PUBLISH /SUBSCR
Page 127 and 128: 5.3. JMS2009-PS - A PUBLISH /SUBSCR
Page 129 and 130: 5.4. CASE STUDY II: JMS2009-PS 115
Page 131 and 132: 5.4. CASE STUDY II: JMS2009-PS 117
Page 133 and 134: Chapter 6 Performance Modeling of E
Page 135 and 136: 6.2. MODELING SPECJMS2007 121 Scena
Page 137 and 138: 6.2. MODELING SPECJMS2007 123 BASE*
Page 139 and 140: 6.2. MODELING SPECJMS2007 125 1‘O
Page 141 and 142: 6.2. MODELING SPECJMS2007 127 Sc. 1
Page 143 and 144: 6.2. MODELING SPECJMS2007 129 Perce
Page 145 and 146: 6.2. MODELING SPECJMS2007 131 Serve
Page 147 and 148: 6.3. CONCLUDING REMARKS 133 the mod
Page 149: Chapter 7 Conclusions and Outlook W
Page 153 and 154: 7.1. ONGOING AND FUTURE WORK 139 th
Page 155 and 156: Bibliography [1] J. Abbott, K. B. M
Page 157 and 158: BIBLIOGRAPHY 143 [29] BiCEP Researc
Page 159 and 160: BIBLIOGRAPHY 145 [61] G. Cugola and
Page 161 and 162: BIBLIOGRAPHY 147 [92] R. Henjes, M.
Page 163 and 164: BIBLIOGRAPHY 149 [128] S. Kounev an
Page 165 and 166: BIBLIOGRAPHY 151 [162] Object Compu
Page 167 and 168: BIBLIOGRAPHY 153 [197] K. Sachs, S.
Page 169 and 170: BIBLIOGRAPHY 155 [228] P. Tran, P.
Page 171: Erklärung Hiermit erkläre ich, di
show all

Performance Modeling and Benchmarking of Event-Based ... - DVS

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?