Cost-Based Optimization of Integration Flows - Datenbanken ...

More documents

Recommendations

Info

7 Conclusions Existing approaches before this thesis either used the rule-based optimize-once model, where the integration flow is only optimized once during the initial deployment, or the optimize-always model, where optimization is triggered for each plan instance. In contrast, we presented the first entire cost-based optimizer for imperative integration flows using the cost-based optimization models of periodical and on-demand re-optimization. The major advantage of these new optimization models is robustness in terms of (1) high optimization opportunities that allows (2) the adaptation to changing workload characteristics with (3) low risk of optimization overheads. Beside the investigation of additional optimization techniques, we see four major research challenges regarding future work of the cost-based optimization of integration flows: • Mid-Instance Optimization: Our asynchronous re-optimization model is based on the assumption of many plan instances with rather small amounts of data per instance. In order to make it also suitable for (1) long running plan instances and (2) adhoc integration flows (as required for situational BI and mashup integration flows), an extension to synchronous mid-instance re-optimization would be necessary. The challenge is to define a hybrid model for both use cases of integration flows. • Multi-Objective Optimization: We focused on the optimization objectives of execution time and throughput. However, facing new requirements, this might be extended by multiple objectives including for example, monetary measures, execution and latency time, throughput, energy consumption, resiliency or transactional guarantees. • Optimization of Multiple Deployed Plans: The cost-based optimization of integration flows—with few exceptions—aims to optimize a single deployed plan. However, typically, multiple different integration flows are deployed and concurrently executed by the integration platform. The major question is if we could exploit the knowledge about workload characteristics of all plans and their inter-influences for more efficient scheduling of plan instances or influence-aware plan rewriting. • Optimization of Distributed Integration Flows: With regard to load balancing and the emerging trend towards virtualization, distributed integration flows might be used as well. The resulting research challenge is to optimize the entire distributed integration flow, rather than just the subplans of local server nodes. One aspect of this might be the extension of cost-based vectorization to the distributed case. Thus, we can conclude that there are many directions for future work in this new field of the cost-based optimization of integration flows. Similarly to cost-based query optimization, this might be a starting point for the continuous development of new execution and optimization techniques: ”In my view, the query optimizer was the first attempt at what we call autonomic computing or self-managing, self-timing technology. Query optimizers have been 25 years in development, with enhancements of the cost-based query model and the optimization that goes with it, and a richer and richer variety of execution techniques that the optimizer chooses from. We just have to keep working on this. It’s a never-ending quest for an increasingly better model and repertoire of optimization and execution techniques. So the more the model can predict what’s really happening in the data and how the data is really organized, the closer and closer we will come [to the ideal system].” — Patricia G. Selinger, IBM Research – Almaden, 2003 [Win03] 200
Bibliography [AAB + 05] Daniel J. Abadi, Yanif Ahmad, Magdalena Balazinska, Ugur Çetintemel, Mitch Cherniack, Jeong-Hyon Hwang, Wolfgang Lindner, Anurag Maskey, Alex Rasin, Esther Ryvkina, Nesime Tatbul, Ying Xing, and Stanley B. Zdonik. The Design of the Borealis Stream Processing Engine. In CIDR, pages 277–289, 2005. [ABD + 10] M. Abhirama, Sourjya Bhaumik, Atreyee Dey, Harsh Shrimal, and Jayant R. Haritsa. On the Stability of Plan Costs and the Costs of Plan Stability. Proceedings of the VLDB Endowment, 3(1):1137–1148, 2010. [AC99] [ACG + 08] Ashraf Aboulnaga and Surajit Chaudhuri. Self-tuning Histograms: Building Histograms Without Looking at Data. In SIGMOD Conference, pages 181–192, 1999. Himanshu Agrawal, Girish Chafle, Sunil Goyal, Sumit Mittal, and Sougata Mukherjea. An Enhanced Extract-Transform-Load System for Migrating Data in Telecom Billing. In ICDE, pages 1277–1286, 2008. [AH00] Ron Avnur and Joseph M. Hellerstein. Eddies: Continuously Adaptive Query Processing. In SIGMOD Conference, pages 261–272, 2000. [AHL + 04] Ashraf Aboulnaga, Peter J. Haas, Sam Lightstone, Guy M. Lohman, Volker Markl, Ivan Popivanov, and Vijayshankar Raman. Automated Statistics Collection in DB2 UDB. In VLDB, pages 1146–1157, 2004. [AN08] Alexander Albrecht and Felix Naumann. Managing ETL Processes. In NTII, pages 12–15, 2008. [AN09] [ANY04] [AT08] [BABO + 09] [BB05] Alexander Albrecht and Felix Naumann. METL: Managing and Integrating ETL Processes. In VLDB Ph.D. Workshop, pages 1–6, 2009. Sanjay Agrawal, Vivek R. Narasayya, and Beverly Yang. Integrating Vertical and Horizontal Partitioning Into Automated Physical Database Design. In SIGMOD Conference, pages 359–370, 2004. David Aumüller and Andreas Thor. Mashup-Werkzeuge zur Ad-hoc Datenintegration im Web. Datenbankspektrum, 26:4–10, 2008. Manish Bhide, Manoj Agarwal, Amir Bar-Or, Sriram Padmanabhan, Srinivas Mittapalli, and Girish Venkatachaliah. XPEDIA: XML ProcEssing for Data IntegrAtion. Proceedings of the VLDB Endowment, 2(2):1330–1341, 2009. Shivnath Babu and Pedro Bizarro. Adaptive Query Processing in the Looking Glass. In CIDR, pages 238–249, 2005. 201
Page 1:
Cost-Based Optimization of Integrat
Page 4 and 5:
of traditional data management syst
Page 7 and 8:
Contents 1 Introduction 1 2 Prelimi
Page 9:
Contents 6.5 Experimental Evaluatio
Page 12 and 13:
1 Introduction for integration flow
Page 14 and 15:
1 Introduction violated. We present
Page 16 and 17:
2 Preliminaries and Existing Techni
Page 18 and 19:
Page 20 and 21:
Page 22 and 23:
Page 24 and 25:
Page 26 and 27:
Page 28 and 29:
Page 30 and 31:
Page 32 and 33:
Page 34 and 35:
Page 36 and 37:
Page 38 and 39:
Page 40 and 41:
Page 42 and 43:
Page 44 and 45:
3 Fundamentals of Optimizing Integr
Page 46 and 47:
Page 48 and 49:
Page 50 and 51:
Page 52 and 53:
Page 54 and 55:
Page 56 and 57:
Page 58 and 59:
Page 60 and 61:
Page 62 and 63:
Page 64 and 65:
Page 66 and 67:
Page 68 and 69:
Page 70 and 71:
Page 72 and 73:
Page 74 and 75:
Page 76 and 77:
Page 78 and 79:
Page 80 and 81:
Page 82 and 83:
Page 84 and 85:
Page 86 and 87:
Page 88 and 89:
Page 90 and 91:
Page 92 and 93:
Page 94 and 95:
Page 96 and 97:
Page 98 and 99:
4 Vectorizing Integration Flows •
Page 100 and 101:
4 Vectorizing Integration Flows exe
Page 102 and 103:
4 Vectorizing Integration Flows Ove
Page 104 and 105:
4 Vectorizing Integration Flows Alg
Page 106 and 107:
4 Vectorizing Integration Flows two
Page 108 and 109:
4 Vectorizing Integration Flows Inv
Page 110 and 111:
4 Vectorizing Integration Flows (a)
Page 112 and 113:
4 Vectorizing Integration Flows o 2
Page 114 and 115:
4 Vectorizing Integration Flows In
Page 116 and 117:
4 Vectorizing Integration Flows 2.
Page 118 and 119:
4 Vectorizing Integration Flows The
Page 120 and 121:
4 Vectorizing Integration Flows ord
Page 122 and 123:
4 Vectorizing Integration Flows 4.3
Page 124 and 125:
4 Vectorizing Integration Flows P
Page 126 and 127:
4 Vectorizing Integration Flows P
Page 128 and 129:
4 Vectorizing Integration Flows t1:
Page 130 and 131:
4 Vectorizing Integration Flows We
Page 132 and 133:
Page 134 and 135:
Page 136 and 137:
Page 138 and 139:
4 Vectorizing Integration Flows 4.7
Page 140 and 141:
5 Multi-Flow Optimization cannot be
Page 142 and 143:
5 Multi-Flow Optimization the query
Page 144 and 145:
5 Multi-Flow Optimization The incom
Page 146 and 147:
5 Multi-Flow Optimization example p
Page 148 and 149:
5 Multi-Flow Optimization not allow
Page 150 and 151:
5 Multi-Flow Optimization partition
Page 152 and 153:
5 Multi-Flow Optimization mention i
Page 154 and 155:
5 Multi-Flow Optimization • Case
Page 156 and 157:
5 Multi-Flow Optimization k ′ . F
Page 158 and 159:
5 Multi-Flow Optimization partition
Page 160 and 161: 5 Multi-Flow Optimization the waiti
Page 162 and 163: 5 Multi-Flow Optimization Execution
Page 164 and 165: 5 Multi-Flow Optimization Thus, for
Page 166 and 167: 5 Multi-Flow Optimization Thus, T L
Page 168 and 169: 5 Multi-Flow Optimization • P 5 :
Page 170 and 171: 5 Multi-Flow Optimization decreasin
Page 172 and 173: 5 Multi-Flow Optimization (a) Fixed
Page 174 and 175: 5 Multi-Flow Optimization reached,
Page 176 and 177: 5 Multi-Flow Optimization (2) plan
Page 178 and 179: 6 On-Demand Re-Optimization categor
Page 180 and 181: 6 On-Demand Re-Optimization present
Page 182 and 183: 6 On-Demand Re-Optimization stratum
Page 184 and 185: 6 On-Demand Re-Optimization o 3 o 4
Page 186 and 187: 6 On-Demand Re-Optimization For on-
Page 188 and 189: 6 On-Demand Re-Optimization 6.3.1 O
Page 190 and 191: 6 On-Demand Re-Optimization such th
Page 192 and 193: 6 On-Demand Re-Optimization Join En
Page 194 and 195: 6 On-Demand Re-Optimization f γ((
Page 196 and 197: 6 On-Demand Re-Optimization The res
Page 198 and 199: 6 On-Demand Re-Optimization project
Page 200 and 201: 6 On-Demand Re-Optimization (a) Sel
Page 202 and 203: 6 On-Demand Re-Optimization (a) Loa
Page 204 and 205: 6 On-Demand Re-Optimization ical re
Page 206 and 207: 6 On-Demand Re-Optimization evaluat
Page 208 and 209: 6 On-Demand Re-Optimization 6.6 Sum
Page 212 and 213: Bibliography [BBD05a] Shivnath Babu
Page 214 and 215: Bibliography [BHP + 09b] [BHP + 11]
Page 216 and 217: Bibliography [CM95] Sophie Cluet an
Page 218 and 219: Bibliography [GZ08] [HA03] [Haa07]
Page 220 and 221: Bibliography [IKNG09] [INSS92] [Ioa
Page 222 and 223: Bibliography [LX09] [LZ05] Rubao Le
Page 224 and 225: Bibliography [OMG07] OMG. XML Metad
Page 226 and 227: Bibliography [Sto02] Michael Stoneb
Page 228 and 229: Bibliography [ZRH04] Yali Zhu, Elke
Page 230 and 231: List of Figures 3.27 Workload Adapt
Page 233: List of Tables 2.1 Interaction-Orie
Page 237: Selbstständigkeitserklärung Hierm
show all

Cost-Based Optimization of Integration Flows - Datenbanken ...

Create successful ePaper yourself

Delete template?

Save as template?