Cost-Based Optimization of Integration Flows - Datenbanken ...

More documents

Recommendations

Info

4 Vectorizing Integration Flows two additional hash maps as secondary index structures for source operators and target operators over the set of data dependencies D. Then, for any operator o i we only need to iterate over the lists of its own source and target dependencies. This leads to a quadratic worst-case time complexity for the algorithm part of graph creation. However, for the general case, where we need to iterate over all dependencies, we have cubic complexity. Furthermore, for low numbers of data dependencies, the overhead of the A-PV is moderate and therefore, we use the more general form. Rewriting Context-specific Operators In addition to the core rewriting algorithm, we now discuss rewriting rules for the contextspecific operators Switch, Iteration, and Savepoint as well as specific situations, where multiple writes to external systems must be synchronized (Invoke) in order to guarantee semantic correctness when applying plan vectorization. Rewriting Switch operators. When rewriting Switch operators, we must be aware of their ordered if-elseif-else semantics. Here, message sequences are routed along different switch-paths, which will eventually be merged. Assume a message sequence of m 1 and m 2 , where m 1 is routed to path A, while m 2 is routed to path B. If W (A) ≥ W (B) + W (Switch B ), m 2 arrives earlier at the merging point than m 1 does. Hence, a message outrun has taken place. In order to overcome this problem, we could use timestamp comparison at the merging point. Therefore, we introduced the XOR operator that is inserted just before the single switch paths are merged. It reads from all queues, compares the timestamps of read messages and forwards the oldest. Due to the possibility of message starvation (we are not allowed to forward a message until we read a younger message from all other switch paths) in combination with possibly nested Switch operators, we use a so-called synchronization queue that represents the temporal order of messages and thus, by comparing the message source IDs with the read synchronization IDs, we overcome the problem of starvation because we can output messages according to this sequence of IDs. The dedicated synchronization queue is required due to arbitrarily nested Switch operators, where the assumption of a cohesive sequence of source IDs does not hold. Example 4.3 (Rewriting Switch Operators). Assume the dependency graph DG(P 1 ) of Switch (o2) [in: msg1] Receive (o1) [service: s3, out: msg1] δ msg1 D Translation (o3) [in: msg1, out: msg2] Translation (o5) [in: msg1, out: msg2] δ msg1 D @type='MATMAS04' Switch (o2) [in: msg1] @type='MATMAS05' δ msg1 D Assign (o4) [in: msg2, out: msg3] Assign (o6) [in: msg2, out: msg3] δ msg2 D Translation (o3) [in: msg1, out: msg2] Assign (o4) [in: msg2, out: msg3] Translation (o5) [in: msg1, out: msg2] Assign (o6) [in: msg2, out: msg3] δ msg2 D XOR (ox) [in: msg3, out: msg3] δ msg3 D Invoke (o7) [service s1, in: msg3] δ msg3 D Invoke (o7) [service s1, in: msg3] (a) Dependency Graph DG(P 1) of Subplan P 1 (b) Vectorized Subplan P ′ 1 Figure 4.6: Rewriting Switch Operators 96
4.2 Plan Vectorization a subplan of our example plan P 1 shown in Figure 4.6(a). The vectorized plan P 1 ′ is the output of the A-PV and it is shown in Figure 4.6(b). There, the Switch-specific rewriting technique has been applied, where we have created two pipeline branches (one for each switch-path) and changed the data flow such that messages are passed through the Switch operator. In order to avoid message outrun, we inserted the XOR operator and the synchronization queue. Note that the full plan of P 1 required additional Copy and And operators for the last Invoke operator because it depends directly on the output of the Translation operator. This serialization concept will be discussed separately. Rewriting Iteration operators. When rewriting Iteration operators, the main problem is also the message outrun. We must ensure that all iteration loops (for a message) have been processed before the next message enters. Basically, a foreach Iteration is rewritten to a sequence of (1) one Split operator, (2) operators of the Iteration body and (3) one Setoperation (UNION ALL) operator. Using this strategy, inherently leads to the highest degree of parallelism, while it requires only moderate additional costs for splitting and merging. In contrast to this, iterations with while semantics are not vectorized (one single execution bucket) because we cannot guarantee semantic correctness. Rewriting Savepoint operators. Within the instance-based model, we can use messagespecific and context-specific savepoints. Due to the missing global context, we need to reduce the savepoint semantics to the storage of messages, where context information needs to be stored via specific messages. However, in order to ensure the semantic correctness, we require two different rewriting methodologies. The message-specific savepoint is simply vectorized in a standard manner. In contrast to this, the context-specific savepoint, that stores all current messages at a certain plan position, must be rewritten in a more complex way. Here, we need to insert one savepoint into each parallel data flow branch with respect to the operator position in the instance-based case. Rewriting Invoke operators. In order to realize the serialization of external behavior (precondition for transparency), we must ensure that explicitly modeled sequences of writing interactions (Invoke operators) are serialized (see Rule 3 of Definition 3.1). Hence, we use the And operator for synchronization purposes. If (1) two Invoke operators have a temporal dependency within P , (2) they perform a writing interaction to the same external system, and (3) they are included in different pipelines in P ′ , we insert an And operator right before the second Invoke operator as well as a synchronization queue between the first Invoke operator and the And operator. The And operator reads from the synchronization queue and from the original queue and synchronizes the external behavior by deferring all messages until its source message ID is available in the synchronization queue. We use an example to illustrate this concept. Example 4.4 (Serialization of External Behavior). Assume a dependency graph DG(P 8 ) of a subplan of our example plan P 8 (see Figure 4.7(a)) to be part of a data-driven integration flow. If we vectorize this subplan to P 8 ′ (see Figure 4.7(b)) with two pipeline branches, we need to ensure the serialized external behavior. We inserted an And operator, where the first Invoke sends synchronizing source message IDs to this operator using the introduced synchronization queue. Only in the case that the Assign as well as the first Invoke have been processed successfully, the payload message of the right pipeline branch is forwarded to the second Invoke. In summary, when rewriting instance-based plans to vectorized plans, we guarantee semantic correctness for context-specific operators as well. 97
Page 1:
Cost-Based Optimization of Integrat
Page 4 and 5:
of traditional data management syst
Page 7 and 8:
Contents 1 Introduction 1 2 Prelimi
Page 9:
Contents 6.5 Experimental Evaluatio
Page 12 and 13:
1 Introduction for integration flow
Page 14 and 15:
1 Introduction violated. We present
Page 16 and 17:
2 Preliminaries and Existing Techni
Page 18 and 19:
Page 20 and 21:
Page 22 and 23:
Page 24 and 25:
Page 26 and 27:
Page 28 and 29:
Page 30 and 31:
Page 32 and 33:
Page 34 and 35:
Page 36 and 37:
Page 38 and 39:
Page 40 and 41:
Page 42 and 43:
Page 44 and 45:
3 Fundamentals of Optimizing Integr
Page 46 and 47:
Page 48 and 49:
Page 50 and 51:
Page 52 and 53:
Page 54 and 55:
Page 56 and 57: 3 Fundamentals of Optimizing Integr
Page 98 and 99: 4 Vectorizing Integration Flows •
Page 100 and 101: 4 Vectorizing Integration Flows exe
Page 102 and 103: 4 Vectorizing Integration Flows Ove
Page 104 and 105: 4 Vectorizing Integration Flows Alg
Page 108 and 109: 4 Vectorizing Integration Flows Inv
Page 110 and 111: 4 Vectorizing Integration Flows (a)
Page 112 and 113: 4 Vectorizing Integration Flows o 2
Page 114 and 115: 4 Vectorizing Integration Flows In
Page 116 and 117: 4 Vectorizing Integration Flows 2.
Page 118 and 119: 4 Vectorizing Integration Flows The
Page 120 and 121: 4 Vectorizing Integration Flows ord
Page 122 and 123: 4 Vectorizing Integration Flows 4.3
Page 124 and 125: 4 Vectorizing Integration Flows P
Page 126 and 127: 4 Vectorizing Integration Flows P
Page 128 and 129: 4 Vectorizing Integration Flows t1:
Page 130 and 131: 4 Vectorizing Integration Flows We
Page 138 and 139: 4 Vectorizing Integration Flows 4.7
Page 140 and 141: 5 Multi-Flow Optimization cannot be
Page 142 and 143: 5 Multi-Flow Optimization the query
Page 144 and 145: 5 Multi-Flow Optimization The incom
Page 146 and 147: 5 Multi-Flow Optimization example p
Page 148 and 149: 5 Multi-Flow Optimization not allow
Page 150 and 151: 5 Multi-Flow Optimization partition
Page 152 and 153: 5 Multi-Flow Optimization mention i
Page 154 and 155: 5 Multi-Flow Optimization • Case
Page 156 and 157:
5 Multi-Flow Optimization k ′ . F
Page 158 and 159:
5 Multi-Flow Optimization partition
Page 160 and 161:
5 Multi-Flow Optimization the waiti
Page 162 and 163:
5 Multi-Flow Optimization Execution
Page 164 and 165:
5 Multi-Flow Optimization Thus, for
Page 166 and 167:
5 Multi-Flow Optimization Thus, T L
Page 168 and 169:
5 Multi-Flow Optimization • P 5 :
Page 170 and 171:
5 Multi-Flow Optimization decreasin
Page 172 and 173:
5 Multi-Flow Optimization (a) Fixed
Page 174 and 175:
5 Multi-Flow Optimization reached,
Page 176 and 177:
5 Multi-Flow Optimization (2) plan
Page 178 and 179:
6 On-Demand Re-Optimization categor
Page 180 and 181:
6 On-Demand Re-Optimization present
Page 182 and 183:
6 On-Demand Re-Optimization stratum
Page 184 and 185:
6 On-Demand Re-Optimization o 3 o 4
Page 186 and 187:
6 On-Demand Re-Optimization For on-
Page 188 and 189:
6 On-Demand Re-Optimization 6.3.1 O
Page 190 and 191:
6 On-Demand Re-Optimization such th
Page 192 and 193:
6 On-Demand Re-Optimization Join En
Page 194 and 195:
6 On-Demand Re-Optimization f γ((
Page 196 and 197:
6 On-Demand Re-Optimization The res
Page 198 and 199:
6 On-Demand Re-Optimization project
Page 200 and 201:
6 On-Demand Re-Optimization (a) Sel
Page 202 and 203:
6 On-Demand Re-Optimization (a) Loa
Page 204 and 205:
6 On-Demand Re-Optimization ical re
Page 206 and 207:
6 On-Demand Re-Optimization evaluat
Page 208 and 209:
6 On-Demand Re-Optimization 6.6 Sum
Page 210 and 211:
7 Conclusions Existing approaches b
Page 212 and 213:
Bibliography [BBD05a] Shivnath Babu
Page 214 and 215:
Bibliography [BHP + 09b] [BHP + 11]
Page 216 and 217:
Bibliography [CM95] Sophie Cluet an
Page 218 and 219:
Bibliography [GZ08] [HA03] [Haa07]
Page 220 and 221:
Bibliography [IKNG09] [INSS92] [Ioa
Page 222 and 223:
Bibliography [LX09] [LZ05] Rubao Le
Page 224 and 225:
Bibliography [OMG07] OMG. XML Metad
Page 226 and 227:
Bibliography [Sto02] Michael Stoneb
Page 228 and 229:
Bibliography [ZRH04] Yali Zhu, Elke
Page 230 and 231:
List of Figures 3.27 Workload Adapt
Page 233:
List of Tables 2.1 Interaction-Orie
Page 237:
Selbstständigkeitserklärung Hierm
show all

Cost-Based Optimization of Integration Flows - Datenbanken ...

Create successful ePaper yourself

Delete template?

Save as template?