Cost-Based Optimization of Integration Flows - Datenbanken ...

4 Vectorizing Integration Flows
t1: Pcur
o1 o2 o3
o4 o5
W(o1)=3 W(o2)=2 W(o3)=3 W(o4)=5 W(o5)=3
t2: Pcur
1 2
3 1
o1 o2 o3
o4 o5
W(o1)=3 W(o2)=4 W(o3)=3 W(o4)=5 W(o5)=4
Pnew
o1 o2 o3
o4 o5
W(o1)=3 W(o2)=4 W(o3)=3 W(o4)=5 W(o5)=4
(a) Deployed Plan P cur
(b) New Plan P new
Figure 4.18: Example Periodical Re-Optimization
Hence, we created the new plan shown in Figure 4.18(b). First, we determine the costs
for flushing the current pipeline, assuming the new statistics with
W flush (P ′′
cur) = 3 · W (b 2 ) + 5 ms + 4 ms = 30 ms.
Then, we compute the benefit of changing the plan by
W change = (n + 5 − 1) · 5 ms − (n + 4 − 1) · (4 ms + 3 ms) = −2 ms · n − 1 ms.
Subsequently, we use the monitored message rate R = 10.7 msg /s and the optimization period
∆t = 10 s as estimation for the number of processed messages n = 10.7 msg /s · 10 s = 107
during the next period and compare the costs with the benefit, by assuming full system
utilization, as follows:
(W flush + W change = 30 ms + (−2 ms · 107 − 1 ms)) ≤ 0.
Finally, we decide to exchange plans because in the next period ∆t, we will yield an improvement
of 185 ms, including the overhead for rewriting.
If the evaluation of the rewriting benefit resulted in the decision to exchange plans, we
need to dynamically rewrite the existing plan during runtime. In the following, we explain
this step in more detail.
Dynamic Plan Rewriting
The major problem when rewriting a vectorized plan during runtime is posed by loaded
queues. One approach would be explicit state migration and state re-computation [ZRH04].
However, re-computation might is impossible for integration flows due to interactions with
external systems that have to be executed exactly once. Therefore, plan rewriting is realized
by stopping execution buckets and flushing of intermediate queues.
For example, in order to merge two execution buckets b i and b i+1 with a queue q i+1 in
between, we need to stop the execution bucket b i , while bucket b i+1 is still working. Over
time, we flush q i+1 and wait until it contains zero messages. We then merge the execution
buckets to b i , which contains an instance-based subplan with all operators of the merged
subplans, and simply remove q i+1 . This concept can be used for bucket merging and
splitting, respectively and we never loose a message during dynamic plan rewriting.
Putting it all together, we introduced the general concept of vectorization as a controlflow-oriented
optimization technique that aims to improve the message throughput. Furthermore,
we generalized this concept to the cost-based plan vectorization and explained
how to take multiple deployed plans into account as well. Finally, we described how this
technique is embedded into our general cost-based optimization framework. Although, this
118