Cost-Based Optimization of Integration Flows - Datenbanken ...

More documents

Recommendations

Info

3 Fundamentals of Optimizing Integration Flows number of parallel subflows. The rewriting of iterations to parallel flows is beneficial if ⎛ ⎞ |r| ∑m i ∑m i max ⎝ Ŵ (o i,j ) + i · W (Start Thread) ⎠ < r · W (o i,j ) i=1 j=1 j=1 (3.18) with Ŵ (o |r| i,j) = min(|r|, k) · (Ŵ (o i,j) − wait(o i,j )) + wait(o i,j ). Due to foreach semantics, r must be estimated in terms of the frequency of iterations. Rewriting is realized by the following algorithm. We check if there are no dependencies between iterations and if we are allowed to change the temporal order. In case of independence, we compile all operators of the iteration body to r subflows plus one additional subflow. Each subflow references a specific data partition of the inbound data set, while the last subflow is used for all partitions that exceed the estimated number of iterations. In conclusion, this optimization technique offers—similar to the rewriting of sequences— high optimization opportunities. In contrast to sequences, the rewriting relies heavily on the estimated number of iterations. The possibility of arbitrary input data sets might result in unused subflows (overestimation) or in executing multiple partitions with the last subflow (underestimation). One can further enhance this by using dynamic runtime scheduling of parallel subflows (e.g., guided self-scheduling [PK87], or factoring [HSF91]). In addition, this technique can be combined with rewriting sequences (the operators of one iteration) to parallel flows and this technique should be applied before the techniques WC1 (rescheduling parallel flows) and WC4 (merging parallel flows). Merging Parallel Flows Recall the costs of a Fork operator that are determined by the most time-consuming subflow. The idea of the technique WC4: Merging Parallel Flows is that if the costs of the subflow with maximum costs subsume the costs of two or more other subflows, the subsumed subflows can be rewritten to one subflow in order to reduce the costs by W (T hread) that denotes the costs for thread creation, starting, and monitoring (in contrast to the previously used W (Start T hread) that denotes the time required for thread creation only). Therefore, all Fork operators with more than two subflows have to be considered. We achieve an execution time reduction because less threads are required, while the most time-consuming subflow is unchanged. In general, this problem of a given maximum constraint per partition and the optimization objective of minimizing the number of partitions, is reducible to the offline bin packing problem that is known to be an NP-hard problem. Therefore, we use an extension of the heuristic first fit algorithm [Joh74] that works as follows. First, we determine the subflow with maximum costs. Second, for each old subflow, we check if it can be merged with an existing new subflow. If this is possible, we merge the subflow with the first fit subflow by temporally concatenating the sequences of operators; otherwise, we create a new subflow. In the worst case, for each old subflow, we check each new subflow. Thus, the time complexity is given by O(m 2 ). In the following, we use an example to illustrate this heuristic algorithm. Example 3.11 (Merging Parallel Flows). Recall our example plan P 7 from Example 3.9. Further, assume changed execution times as shown in Figure 3.14(a). First, we determine the fourth subflow (349 ms) as upper bound for our extended first fit algorithm. Second, 64
3.4 Optimization Techniques Fork (o2) 5ms 10ms 4ms Assign (o3) Assign (o6) Assign (o11) Fork (o2) 130ms Invoke (o4) 150ms Invoke (o7) 220ms Invoke (o12) 5ms Assign (o3) 130ms Invoke (o4) 10ms Assign (o6) 150ms Invoke (o7) 7ms Assign (o9) 90ms Invoke (o10) 4ms Assign (o11) 220ms Invoke (o12) 70ms Translation (o5) 7ms Assign (o9) 70ms Translation (o8) 125ms Translation (o13) 70ms Translation (o5) 205ms 70ms Translation (o8) 230ms 97ms 125ms Translation (o13) 349ms 90ms Invoke (o10) 302ms 230ms 349ms (a) Plan P 7 (b) Optimized Plan P ′ 7 Figure 3.14: Example Merging Parallel Flows we iterate over all subflows in order to minimize the total number of subflows. There, for the first old subflow, we create a new subflow. Then, we check if the second old subflow fits into the first. Due to 205 ms + 230 ms > 349 ms, we create a second new subflow. Subsequently, we repeat this with the third old subflow and observe that we can merge it with the first new subflow. Finally, the fourth old subflow, obviously, cannot fit into an existing subflow and hence, we create a third new subflow. Figure 3.14(b) illustrates the result of this algorithm. As a result, this optimization technique reduces the number of required threads without sacrificing the estimated execution time of a Fork operator. Due to less synchronization and thread monitoring overhead, this results in execution time reductions. Note that this technique should be applied after WC2 and WC3 (rewriting sequences and iterations to parallel flows), but before WC1 (rescheduling parallel flows). Apart from these purely control-flow-oriented optimization techniques, there are several optimization techniques that can be classified as hybrid techniques because they combine data-flow- and control-flow-oriented aspects in order to achieve lower execution time. We will discuss them within the context of data-flow oriented optimization techniques. 3.4.2 Data-Flow-Oriented Techniques The data-flow-oriented optimization techniques address the interaction-oriented operators and the data-flow-oriented operators of our flow meta model. This includes various techniques from traditional query processing, hybrid techniques combining data-flow and control-flow-oriented aspects, and techniques that are tailor-made for integration flows. Reordering and Merging of Switch Paths Programming language compilers and modern processors use only path probabilities for static and dynamic branch prediction in order to support speculative path computation. In contrast, expression evaluation within integration flows is typically more expensive due to the expression evaluation on messages (e.g., XPath expressions on XML messages). The techniques WD1: Reordering of Switch Paths and WD2: Merging of Switch Paths are applied to the Switch operator depending on the workload characteristics. For this 65
Page 1:
Cost-Based Optimization of Integrat
Page 4 and 5:
of traditional data management syst
Page 7 and 8:
Contents 1 Introduction 1 2 Prelimi
Page 9:
Contents 6.5 Experimental Evaluatio
Page 12 and 13:
1 Introduction for integration flow
Page 14 and 15:
1 Introduction violated. We present
Page 16 and 17:
2 Preliminaries and Existing Techni
Page 18 and 19:
Page 20 and 21:
Page 22 and 23:
Page 24 and 25: 2 Preliminaries and Existing Techni
Page 44 and 45: 3 Fundamentals of Optimizing Integr
Page 98 and 99: 4 Vectorizing Integration Flows •
Page 100 and 101: 4 Vectorizing Integration Flows exe
Page 102 and 103: 4 Vectorizing Integration Flows Ove
Page 104 and 105: 4 Vectorizing Integration Flows Alg
Page 106 and 107: 4 Vectorizing Integration Flows two
Page 108 and 109: 4 Vectorizing Integration Flows Inv
Page 110 and 111: 4 Vectorizing Integration Flows (a)
Page 112 and 113: 4 Vectorizing Integration Flows o 2
Page 114 and 115: 4 Vectorizing Integration Flows In
Page 116 and 117: 4 Vectorizing Integration Flows 2.
Page 118 and 119: 4 Vectorizing Integration Flows The
Page 120 and 121: 4 Vectorizing Integration Flows ord
Page 122 and 123: 4 Vectorizing Integration Flows 4.3
Page 124 and 125:
4 Vectorizing Integration Flows P
Page 126 and 127:
4 Vectorizing Integration Flows P
Page 128 and 129:
4 Vectorizing Integration Flows t1:
Page 130 and 131:
4 Vectorizing Integration Flows We
Page 132 and 133:
4 Vectorizing Integration Flows (a)
Page 134 and 135:
Page 136 and 137:
Page 138 and 139:
4 Vectorizing Integration Flows 4.7
Page 140 and 141:
5 Multi-Flow Optimization cannot be
Page 142 and 143:
5 Multi-Flow Optimization the query
Page 144 and 145:
5 Multi-Flow Optimization The incom
Page 146 and 147:
5 Multi-Flow Optimization example p
Page 148 and 149:
5 Multi-Flow Optimization not allow
Page 150 and 151:
5 Multi-Flow Optimization partition
Page 152 and 153:
5 Multi-Flow Optimization mention i
Page 154 and 155:
5 Multi-Flow Optimization • Case
Page 156 and 157:
5 Multi-Flow Optimization k ′ . F
Page 158 and 159:
5 Multi-Flow Optimization partition
Page 160 and 161:
5 Multi-Flow Optimization the waiti
Page 162 and 163:
5 Multi-Flow Optimization Execution
Page 164 and 165:
5 Multi-Flow Optimization Thus, for
Page 166 and 167:
5 Multi-Flow Optimization Thus, T L
Page 168 and 169:
5 Multi-Flow Optimization • P 5 :
Page 170 and 171:
5 Multi-Flow Optimization decreasin
Page 172 and 173:
5 Multi-Flow Optimization (a) Fixed
Page 174 and 175:
5 Multi-Flow Optimization reached,
Page 176 and 177:
5 Multi-Flow Optimization (2) plan
Page 178 and 179:
6 On-Demand Re-Optimization categor
Page 180 and 181:
6 On-Demand Re-Optimization present
Page 182 and 183:
6 On-Demand Re-Optimization stratum
Page 184 and 185:
6 On-Demand Re-Optimization o 3 o 4
Page 186 and 187:
6 On-Demand Re-Optimization For on-
Page 188 and 189:
6 On-Demand Re-Optimization 6.3.1 O
Page 190 and 191:
6 On-Demand Re-Optimization such th
Page 192 and 193:
6 On-Demand Re-Optimization Join En
Page 194 and 195:
6 On-Demand Re-Optimization f γ((
Page 196 and 197:
6 On-Demand Re-Optimization The res
Page 198 and 199:
6 On-Demand Re-Optimization project
Page 200 and 201:
6 On-Demand Re-Optimization (a) Sel
Page 202 and 203:
6 On-Demand Re-Optimization (a) Loa
Page 204 and 205:
6 On-Demand Re-Optimization ical re
Page 206 and 207:
6 On-Demand Re-Optimization evaluat
Page 208 and 209:
6 On-Demand Re-Optimization 6.6 Sum
Page 210 and 211:
7 Conclusions Existing approaches b
Page 212 and 213:
Bibliography [BBD05a] Shivnath Babu
Page 214 and 215:
Bibliography [BHP + 09b] [BHP + 11]
Page 216 and 217:
Bibliography [CM95] Sophie Cluet an
Page 218 and 219:
Bibliography [GZ08] [HA03] [Haa07]
Page 220 and 221:
Bibliography [IKNG09] [INSS92] [Ioa
Page 222 and 223:
Bibliography [LX09] [LZ05] Rubao Le
Page 224 and 225:
Bibliography [OMG07] OMG. XML Metad
Page 226 and 227:
Bibliography [Sto02] Michael Stoneb
Page 228 and 229:
Bibliography [ZRH04] Yali Zhu, Elke
Page 230 and 231:
List of Figures 3.27 Workload Adapt
Page 233:
List of Tables 2.1 Interaction-Orie
Page 237:
Selbstständigkeitserklärung Hierm
show all

Cost-Based Optimization of Integration Flows - Datenbanken ...

Create successful ePaper yourself

Delete template?

Save as template?