Performance Modeling and Benchmarking of Event-Based ... - DVS

More documents

Recommendations

Info

6 CHAPTER 1. INTRODUCTION Server, conducting an in-depth performance analysis of the platform under a number of different workload and configuration scenarios and illustrated how the workload may be customized to exercise and evaluate selected aspects of MOM performance. The SPECjms2007 workload scenario was designed in a general fashion and can be easily applied to specific types of EBS. For this purpose, we explained how the workload can be implemented for a pure publish/subscribe environment considering jms2009-PS—a benchmark for publish/subscribe-based communication—as an example. jms2009-PS provides a flexible framework for performance analysis with a strong focus on research and implements the workload using topic-based pub/sub. It allows the user to define complex scenarios in an easy and flexible way. In a case study we demonstrated how the benchmark can be used as test harness to compare different topic deployments and analyzed the influence of message filtering. Both benchmarks are actively used by industry and academia. Since they exercise MOMs in a realistic way, they are used as benchmarks, as test harnesses, and as reference applications. Resulting from our efforts on benchmarking and performance analysis, we have established and maintained several successful collaborations with middleware vendors and academic institutions: e.g., we cooperated with JBoss and the Apache Foundation to publish official SPECjms2007 results (reviewed by OSG Java Subcommittee of SPEC 1 ) and Karlsruhe Institute of Technology [215]. The results of our work focusing on benchmark development and SPECjms2007 (including workload characterization and framework) were published in [203, 202, 198, 130]. Our methodology for performance evaluation using standard benchmarks was introduced in [201]. The jms2009-PS benchmark is within the focus of [199, 197]. An overview of our activities in the area of MOM benchmarking is provided in [200], and in [12] we demonstrated an approach to benchmark AMQP-based middleware using SPECjms2007 and jms2009-PS. 1.3.3 Summary The contributions of this thesis are in the areas of benchmarking and performance engineering of event-based systems. The main contributions can be outlined as follows: 1. Conceptually: (a) Analysis and Classification of Benchmark Requirements An analysis of the requirements that a benchmark and its workload must meet in order to be meaningful. We deduce a classification for these requirements. (b) Performance Evaluation Methodology Based on Benchmarks A new methodology to analyze the performance of messaging middleware using standard benchmarks. (c) Performance Modeling Methodology for EBS A novel approach to model EBS, using queueing Petri nets for predicting system and performance behavior. (d) Queueing Petri Net (QPN) Extensions An extension of the QPN formalism in several ways, e.g., by simplifying the abstractions for modeling logical software entities, such as, message destinations. 2. Practically: (a) A Novel Representative Workload and Benchmark for EBS The first standard benchmark for MOM—SPECjms2007 : 1 Standard Performance Evaluation Corporation (SPEC) is a non-profit organization whose mission is to establish, maintain, and endorse standardized benchmarks to evaluate performance for the newest generation of computing systems. With more than 80 members and associates from industry and academia, SPEC is one of the world’s largest and most successful performance standardization organizations.
1.4. THESIS ORGANIZATION 7 • Standardized by the Standard Performance Evaluation Corporation (SPEC). • A comprehensive and representative workload for message-oriented EBS. • Additionally, a flexible benchmark framework for in-depth analysis. (b) Performance Evaluation of a State-of-the-Art MOM A case study of a leading commercial MOM conducting an in-depth performance analysis of the platform under a number of different workload and configuration scenarios using the SPECjms2007 standard benchmark. (c) Performance Modeling Case Studies of EBS Case Study I: A simple application deployed on a representative DEBS platform. Case Study II: A complex and realistic application deployed on a representative MOM. 1.3.4 Related Activities and Publications Event-Based Applications and Enabling Technologies In [101], we introduced the basic notions of event processing to create a common understanding, presented the enabling technologies that are used for the implementation of event-based systems, surveyed a wide range of applications identifying their main features, and discussed open research issues. ECA Rule Engines We demonstrated in [83] an implementation of an ECA rule engine of an embedded system. Such ECA rule engines provide a flexible environment for supporting the management, reconfiguration, and execution of business rules. However, modeling the performance of a rule engine is challenging because of its reactive nature. In [84], we presented an analytical performance model for ECA rule engines. We discussed the difficulties in building a performance model of an ECA rule engine and introduced a novel methodology for the performance evaluation of ECA rule engines. We introduced the concept of event paths and showed how ECA rules can be mapped to queueing networks. QoS of Event-Based Systems In [11], we provided a general overview of QoS in event-based systems. We introduced an architecture that supports different types of QoS in an EBS and discussed the QoS in the context of MOM and complex event processing (CEP). Statistical Inference and Performance Models Statistical inference is the process of drawing conclusions by applying statistics to observations or hypotheses based on quantitative data. The goal is to determine the relationship between input and output parameters. In [88], we proposed an approach to the statistical inference of performance models based on observation data. In our case study, we used multivariate adaptive regression splines (MARS) and genetic optimization to estimate the influence of message sizes and arrival rates on the system performance of a MOM. We considered message delivery time, throughput, and resource utilization as part of our analysis. 1.4 Thesis Organization This thesis is organized as follows. In Chapter 2, we provide an overview of the state-of-theart and related work in the areas of event-based systems (EBS), performance modeling, and
Page 1: Performance Modeling and Benchmarki
Page 4 and 5: priorities. Finally, we validated o
Page 6 and 7: 4.3.4 Tool Extension . . . . . . .
Page 8 and 9: 5.7 Proportions of the Interactions
Page 11 and 12: List of Acronyms Acronym λ t,k j A
Page 13 and 14: Acronym SPEC-OSG SPE SPN SQL ST SUT
Page 15 and 16: Chapter 1 Introduction 1.1 Motivati
Page 17 and 18: PC Server 310 PC Server 310 PC Serv
Page 19: 1.3. APPROACH AND CONTRIBUTIONS OF
Page 23 and 24: Chapter 2 Background In this chapte
Page 25 and 26: 2.1. EVENT-BASED SYSTEMS 11 is gene
Page 27 and 28: 2.2. TECHNOLOGY PLATFORMS OF EVENT-
Page 35 and 36: 2.3. INTRODUCTION TO QUEUEING PETRI
Page 37 and 38: 2.3. INTRODUCTION TO QUEUEING PETRI
Page 39 and 40: Chapter 3 Related Work In this chap
Page 41 and 42: 3.2. BENCHMARKING OF EVENT-BASED SY
Page 43 and 44: 3.2. BENCHMARKING OF EVENT-BASED SY
Page 45 and 46: 3.4. CONCLUDING REMARKS 31 hierarch
Page 47 and 48: Chapter 4 Performance Engineering o
Page 49 and 50: 4.1. MODELING METHODOLOGY FOR EBS 3
Page 57 and 58: 4.2. PERFORMANCE MODELING PATTERN 4
Page 71 and 72:
4.2. PERFORMANCE MODELING PATTERN 5
Page 73 and 74:
Page 75 and 76:
Page 77 and 78:
Page 79 and 80:
Page 81 and 82:
Page 83 and 84:
Page 85 and 86:
Page 87 and 88:
4.3. EXTENSIONS OF QPNS 73 ,-).$%&'
Page 89 and 90:
4.4. CONCLUDING REMARKS 75 Without
Page 91 and 92:
Chapter 5 Benchmarking of Event-Bas
Page 93 and 94:
5.1. SPECJMS2007 - A STANDARD BENCH
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
Page 101 and 102:
Page 103 and 104:
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
5.2. CASE STUDY I: SPECJMS2007 101
Page 117 and 118:
Page 119 and 120:
Page 121 and 122:
Page 123 and 124:
Page 125 and 126:
5.3. JMS2009-PS - A PUBLISH /SUBSCR
Page 127 and 128:
5.3. JMS2009-PS - A PUBLISH /SUBSCR
Page 129 and 130:
5.4. CASE STUDY II: JMS2009-PS 115
Page 131 and 132:
5.4. CASE STUDY II: JMS2009-PS 117
Page 133 and 134:
Chapter 6 Performance Modeling of E
Page 135 and 136:
6.2. MODELING SPECJMS2007 121 Scena
Page 137 and 138:
6.2. MODELING SPECJMS2007 123 BASE*
Page 139 and 140:
6.2. MODELING SPECJMS2007 125 1‘O
Page 141 and 142:
6.2. MODELING SPECJMS2007 127 Sc. 1
Page 143 and 144:
6.2. MODELING SPECJMS2007 129 Perce
Page 145 and 146:
6.2. MODELING SPECJMS2007 131 Serve
Page 147 and 148:
6.3. CONCLUDING REMARKS 133 the mod
Page 149 and 150:
Chapter 7 Conclusions and Outlook W
Page 151 and 152:
7.1. ONGOING AND FUTURE WORK 137 JA
Page 153 and 154:
7.1. ONGOING AND FUTURE WORK 139 th
Page 155 and 156:
Bibliography [1] J. Abbott, K. B. M
Page 157 and 158:
BIBLIOGRAPHY 143 [29] BiCEP Researc
Page 159 and 160:
BIBLIOGRAPHY 145 [61] G. Cugola and
Page 161 and 162:
BIBLIOGRAPHY 147 [92] R. Henjes, M.
Page 163 and 164:
BIBLIOGRAPHY 149 [128] S. Kounev an
Page 165 and 166:
BIBLIOGRAPHY 151 [162] Object Compu
Page 167 and 168:
BIBLIOGRAPHY 153 [197] K. Sachs, S.
Page 169 and 170:
BIBLIOGRAPHY 155 [228] P. Tran, P.
Page 171:
Erklärung Hiermit erkläre ich, di
show all

Performance Modeling and Benchmarking of Event-Based ... - DVS

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?