Cost-Based Optimization of Integration Flows - Datenbanken ...

More documents

Recommendations

Info

3 Fundamentals of Optimizing Integration Flows data properties of external systems, we use the described lightweight concept for explicitly taking into account conditional probabilities and correlation at the same time using a incremental reordering approach that is tailor-made for integration flows. 3.4 Optimization Techniques In this section, we discuss specific optimization techniques that are used within the described core optimization algorithm. For selected techniques, we present the core idea, the optimality conditions, possible execution time reduction, the rewriting algorithm and its time complexity, as well as possible side effects to other techniques. Cost-Based Techniques Reordering of Switch-Paths (WD1) Merging of Switch-Paths (WD2) Execution Pushdown to External Systems (WD3) Early Selection Application (WD4) Early Projection Application (WD5) Early GroupBy Application (WD6) Materialization Point Insertion (WD7) Orderby Insertion / Removal (WD8) Join-Type Selection (WD9) Join Enumeration (WD10) Setoperation-Type Selection (WD11) Splitting / Merging of Operators (WD12) Precomputation of Values (WD13) Early Translation Application (WD14) (WC1) Rescheduling Start of Parallel Flows (WC2) Rewriting Sequences to Parallel Flows (WC3) Rewriting Iterations to Parallel Flows (WC4) Merging Parallel Flows (WM1) Message Indexing (WM2) Recycling Locally Created Intermediates (WM3) Recycling Externally Loaded Intermediates (Vect) Cost-Based Vectorization see Chapter 4 see Chaper 5 Multi-Flow Optimization (MFO) Data Flow Double Variable Assignment Removal (RD1) Unnecessary Variable Assignment Removal (RD2) Unnecessary Variable Declaration Removal (RD3) Two Sibling Translation Operation Merging (RD4) Two Sibling Validation Merging (RD5) Unnecessary Switch-Path Elimination (RD6) Algebraic Simplification (RD7) (HLB) Heterogeneous Load Balancing (RC1) Redundant Control Flow Rewriting (RC2) Unreachable Subgraph Elimination (RC3) Local Subprocess Inline Compilation (RC4) Static Node Compilation Control Flow Rule-Based Techniques Figure 3.11: Cost-Based Optimization Techniques Figure 3.11 distinguishes the used cost-based optimization techniques into control-floworiented and data-flow-oriented optimization techniques and emphasizes (with bold font) the techniques that we will describe in detail. All optimization techniques presented in this section follow the optimization objective of minimizing the average execution time of a plan. In addition to these techniques, we will discuss in very detail the cost-based vectorization in Chapter 4 and the multi-flow optimization in Chapter 5, which both follow the optimization objective of maximizing the message throughput. Apart from these techniques, we refer the interested reader to our details on message indexing [BHW + 07, BHLW08d] and rule-based optimization techniques [BHW + 07] that are omitted in this thesis. The latter includes, for example, relational algebraic simplifications as described by Dadam [Dad96]. 3.4.1 Control-Flow-Oriented Techniques Control-flow-oriented optimization techniques address the interaction- and control-floworiented operators of our flow meta model. These techniques try to exploit the specific characteristics of operators like alternatives (Switch operator), loops (Iteration operator), and parallel subflows (Fork operator, constrained by Invoke operators). 60
3.4 Optimization Techniques One of the core concepts is to leverage parallelism of operator execution in order to minimize the execution time. Due to typically low CPU utilization reasoned by (1) singlethreaded, instance-based plan execution, (2) significant waiting times for external systems, and (3) IO-bottlenecks for message persistence, these decisions are made with cost-based optimality conditions rather than statically during the initial deployment. Rescheduling the Start of Parallel Flows The technique WC1: Rescheduling the Start of Parallel Flows rewrites existing Fork operators. The execution time of the Fork operator o is determined by its most time-consuming subflow r i with ⎛ ⎞ W (o) = max |r| i=1 ∑m i ⎝ W (o i,j ) + i · W (Start Thread) ⎠ (3.16) j=1 because the individual subflows are started in sequence and temporally joined (synchronized) at the end of this operator. The core principle of this technique is to reduce waiting time by rewriting the concurrent subflows such that the subflows are started in descending order of their execution time. In other words, the start sequence of parallel subflows is optimal if the condition Ŵ (r i) ≥ Ŵ (r i+1 ) holds, where Ŵ (r i) = ∑ m i j=1 Ŵ (o i,j). The execution time can be reduced by (|r|− 1)·W (Start T hread), where W (Start T hread) denotes the constant costs for creation and start of a thread. The rewriting algorithm essentially consists of two steps. First, for each subflow, we recursively sum up the costs of all individual operators. Second, we check if the optimality condition holds and order the subflows according to the estimated costs if required. The time complexity of this algorithm is given by O(|r| · log|r|). Example 3.9 (Rescheduling Parallel Flows). Assume our example plan P 7 and the monitored execution times shown in Figure 3.12(a). Furthermore, assume the cost for starting of a parallel subflow (thread) to be W (Start T hread) = 3 ms. The estimated costs of the Fork operator are then given by Ŵ (o 2) = 2 · 3 ms + 230 ms = 236 ms. Rescheduling the parallel flows yields the alternative plan shown in Figure 3.12(b). Using this plan, the costs are reduced to Ŵ (o 2) = 3 ms + 230 ms = 233 ms. Fork (o2) Fork (o2) 5ms 10ms 7ms 4ms 10ms 4ms 5ms 7ms Assign (o3) Assign (o6) Assign (o9) Assign (o11) Assign (o6) Assign (o11) Assign (o3) Assign (o9) 130ms 150ms 90ms 120ms 150ms 120ms 130ms 90ms Invoke (o4) Invoke (o7) Invoke (o10) Invoke (o12) Invoke (o7) Invoke (o12) Invoke (o4) Invoke (o10) 70ms Translation (o5) 205ms 70ms Translation (o8) 230ms 97ms 85ms Translation (o13) 209ms 70ms Translation (o8) 85ms Translation (o13) 70ms Translation (o5) 230ms 209ms 205ms 97ms (a) Plan P 7 (b) Optimized Plan P ′ 7 Figure 3.12: Example Rescheduling Parallel Flows Clearly, the benefit of this optimization technique is limited but the potential increases with increasing number of subflows. This technique should be used after WC2 and WC3 61
Page 1:
Cost-Based Optimization of Integrat
Page 4 and 5:
of traditional data management syst
Page 7 and 8:
Contents 1 Introduction 1 2 Prelimi
Page 9:
Contents 6.5 Experimental Evaluatio
Page 12 and 13:
1 Introduction for integration flow
Page 14 and 15:
1 Introduction violated. We present
Page 16 and 17:
2 Preliminaries and Existing Techni
Page 18 and 19:
2 Preliminaries and Existing Techni
Page 20 and 21: 2 Preliminaries and Existing Techni
Page 44 and 45: 3 Fundamentals of Optimizing Integr
Page 98 and 99: 4 Vectorizing Integration Flows •
Page 100 and 101: 4 Vectorizing Integration Flows exe
Page 102 and 103: 4 Vectorizing Integration Flows Ove
Page 104 and 105: 4 Vectorizing Integration Flows Alg
Page 106 and 107: 4 Vectorizing Integration Flows two
Page 108 and 109: 4 Vectorizing Integration Flows Inv
Page 110 and 111: 4 Vectorizing Integration Flows (a)
Page 112 and 113: 4 Vectorizing Integration Flows o 2
Page 114 and 115: 4 Vectorizing Integration Flows In
Page 116 and 117: 4 Vectorizing Integration Flows 2.
Page 118 and 119: 4 Vectorizing Integration Flows The
Page 120 and 121:
4 Vectorizing Integration Flows ord
Page 122 and 123:
4 Vectorizing Integration Flows 4.3
Page 124 and 125:
4 Vectorizing Integration Flows P
Page 126 and 127:
4 Vectorizing Integration Flows P
Page 128 and 129:
4 Vectorizing Integration Flows t1:
Page 130 and 131:
4 Vectorizing Integration Flows We
Page 132 and 133:
4 Vectorizing Integration Flows (a)
Page 134 and 135:
Page 136 and 137:
Page 138 and 139:
4 Vectorizing Integration Flows 4.7
Page 140 and 141:
5 Multi-Flow Optimization cannot be
Page 142 and 143:
5 Multi-Flow Optimization the query
Page 144 and 145:
5 Multi-Flow Optimization The incom
Page 146 and 147:
5 Multi-Flow Optimization example p
Page 148 and 149:
5 Multi-Flow Optimization not allow
Page 150 and 151:
5 Multi-Flow Optimization partition
Page 152 and 153:
5 Multi-Flow Optimization mention i
Page 154 and 155:
5 Multi-Flow Optimization • Case
Page 156 and 157:
5 Multi-Flow Optimization k ′ . F
Page 158 and 159:
5 Multi-Flow Optimization partition
Page 160 and 161:
5 Multi-Flow Optimization the waiti
Page 162 and 163:
5 Multi-Flow Optimization Execution
Page 164 and 165:
5 Multi-Flow Optimization Thus, for
Page 166 and 167:
5 Multi-Flow Optimization Thus, T L
Page 168 and 169:
5 Multi-Flow Optimization • P 5 :
Page 170 and 171:
5 Multi-Flow Optimization decreasin
Page 172 and 173:
5 Multi-Flow Optimization (a) Fixed
Page 174 and 175:
5 Multi-Flow Optimization reached,
Page 176 and 177:
5 Multi-Flow Optimization (2) plan
Page 178 and 179:
6 On-Demand Re-Optimization categor
Page 180 and 181:
6 On-Demand Re-Optimization present
Page 182 and 183:
6 On-Demand Re-Optimization stratum
Page 184 and 185:
6 On-Demand Re-Optimization o 3 o 4
Page 186 and 187:
6 On-Demand Re-Optimization For on-
Page 188 and 189:
6 On-Demand Re-Optimization 6.3.1 O
Page 190 and 191:
6 On-Demand Re-Optimization such th
Page 192 and 193:
6 On-Demand Re-Optimization Join En
Page 194 and 195:
6 On-Demand Re-Optimization f γ((
Page 196 and 197:
6 On-Demand Re-Optimization The res
Page 198 and 199:
6 On-Demand Re-Optimization project
Page 200 and 201:
6 On-Demand Re-Optimization (a) Sel
Page 202 and 203:
6 On-Demand Re-Optimization (a) Loa
Page 204 and 205:
6 On-Demand Re-Optimization ical re
Page 206 and 207:
6 On-Demand Re-Optimization evaluat
Page 208 and 209:
6 On-Demand Re-Optimization 6.6 Sum
Page 210 and 211:
7 Conclusions Existing approaches b
Page 212 and 213:
Bibliography [BBD05a] Shivnath Babu
Page 214 and 215:
Bibliography [BHP + 09b] [BHP + 11]
Page 216 and 217:
Bibliography [CM95] Sophie Cluet an
Page 218 and 219:
Bibliography [GZ08] [HA03] [Haa07]
Page 220 and 221:
Bibliography [IKNG09] [INSS92] [Ioa
Page 222 and 223:
Bibliography [LX09] [LZ05] Rubao Le
Page 224 and 225:
Bibliography [OMG07] OMG. XML Metad
Page 226 and 227:
Bibliography [Sto02] Michael Stoneb
Page 228 and 229:
Bibliography [ZRH04] Yali Zhu, Elke
Page 230 and 231:
List of Figures 3.27 Workload Adapt
Page 233:
List of Tables 2.1 Interaction-Orie
Page 237:
Selbstständigkeitserklärung Hierm
show all

Cost-Based Optimization of Integration Flows - Datenbanken ...

Create successful ePaper yourself

Delete template?

Save as template?