Cost-Based Optimization of Integration Flows - Datenbanken ...

4.5 Periodical Re-Optimization
changed the current plan, there is a need for dynamic rewriting of vectorized plans due
to the required state migration of loaded message queues and intra-operator states. Both
challenges are addressed in the following.
Evaluating Re-Optimization Potential
If the current logical plan has been changed during optimization, we need to evaluate
the benefit of rewriting the physical plan during runtime. This is required because a
vectorized plan exhibits a state in the sense of all messages that are currently within the
standing process (operators and loaded queues) and thus, we cannot simply generate a new
physical plan. Hence, there is a trade-off between the overhead of exchanging the plan and
the benefit yielded by the newly computed best plan. In general, the same is true for the
general optimization framework as well. However, in contrast to the efficient inter-instance
plan change, this trade-off has much higher importance when rewriting vectorized plans
due to the need for state migration or flushing of pipelines.
The intuition behind our evaluation approach is to compare the costs of flushing the
pipelines of the current plan, with the estimated benefit we gain by using P new ′′ instead of
P cur ′′ for the next period ∆t. We restrict the cost comparison to ∆t because at the next
evaluation timestamp, we might revert the plan change due to changed execution statistics
and hence, we cannot estimate the benefit for a period longer than ∆t. Although we might
miss optimization opportunities in case of a constant workload, we use this approach in
order to ensure robustness of optimizer choices in terms of plan stability.
In detail, the costs of flushing the pipelines are affected by the number of messages in
the queues q i and the execution time of the most time-consuming operator. We determine
the queue cardinalities and compute the costs by
W flush (P ′′
cur) = W (b x ) ·
|b|
l x∑ ∑
bi
|q i | + W (b i ) with W (b x ) = max
k ∑
W (o l ). (4.20)
i=1
i=x
These costs are given by the number of messages in front of the most time-consuming execution
bucket multiplied by the costs of this bucket W (b x ) plus the costs for the remaining
buckets after b x . Those are the approximated costs for flushing the whole pipeline. Note
that incremental rewriting (merging and splitting) is possible as well.
For computing the benefit of dynamic rewriting, we use the message rate R to compute
the number of processed messages by n = R · ∆t. Then, the benefit of exchanging plans
is given by
W change (P ′′ ) = (n + |b| P ′′
new − 1) · W P ′′ new (b x1) − (n + |b| P ′′
cur − 1) · W P ′′ cur (b x2), (4.21)
where W change < 0 holds by definition because the optimizer will only return a new plan
P new if W (P new ) < W (P cur ). Finally, we would change plans if
j=1
l=1
W flush + W change ≤ 0. (4.22)
We illustrate this evaluation approach with the following example.
Example 4.12 (Evaluating Rewriting Benefit). Assume the current plan shown in Figure
4.18(a). The figure also shows the statistics that were present when creating this plan
(t 1 ) as well as the current state (t 2 ) in the form of the numbers of messages in queues.
During the period ∆t = 10 s, average execution times changed (W (o 2 ) = 4 and W (o 5 ) = 4).
117