Cost-Based Optimization of Integration Flows - Datenbanken ...

6 On-Demand Re-Optimization
categories that reasoned the use of periodical re-optimization for integration flows. First,
integration flows are deployed once and executed many times, with rather small amounts
of data per instance. Hence, there is no need for mid-instance (inter- or intra-operator) reoptimization.
Second, in contrast to continuous-query-based systems, many independent
instances of an integration flow are executed over time. Thus, there is no need for state migration
during plan rewriting. Further advantages are (1) the asynchronous optimization
independent of any instance execution, (2) the fact that all subsequent instances (until
the next plan change) rather than only the current query benefit from re-optimization,
and (3) the efficient inter-instance plan change without state migration. However, this
optimization model exhibits also major drawbacks, which we reveal in the following.
Periodical
Re-Optimization
Execution Time
per
Instance of Plan
P, P’, P’’
On-Demand
Re-Optimization
(4) high-influence parameter
optimization interval
re-optimization steps
∆t
P P’ P’’
(1) many unnecessary
re-optimization steps
initial
plan P
workload change
modified
plan P’
(2) missed optimization
opportunity (3) overhead of maintaining
unnecessary statistics
workload change
Time
Figure 6.1: Drawbacks of Periodical Re-Optimization
P’’
Figure 6.1 shows the execution time of plan instances that have been executed over time
in a scenario with two workload shifts. Re-optimization is triggered periodically using a period
∆t, where we only find a new plan if a workload shift occurred meanwhile. We observe
the potential problems of (1) many unnecessary re-optimization steps, where each step is a
full re-optimization and (2) adaptation delays, where we miss optimization opportunities.
Furthermore, we might (3) maintain statistics that are not used by the optimizer and (4)
the chosen optimization interval has high influence on the execution time. Depending on
the optimization interval, periodical re-optimization can even degrade to the unoptimized
execution. To tackle these problems, we propose the on-demand re-optimization that directly
reacts to workload shifts if a new plan is certain to be found. This implies only
necessary re-optimization steps and no missed optimization opportunities.
Example 6.1 (Periodical Plan Optimization). Recall our example plan P 5 that consists of
m = 9 operators, which is illustrated in Figure 6.2. It receives messages from the system
s 3 , executes three Selection operators (according to different attributes). Subsequently,
a Switch operator routes the incoming messages with content-based predicates to schema
mapping Translation operators. Finally, the result is loaded into the system s 6 . For each
received message, conceptually, an independent instance of this plan is initiated. In order
to enable cost-based optimization, statistics are monitored for each operator. We assume
that re-optimization is periodically triggered with period ∆t, as shown in Figure 6.1. During
this re-optimization, all gathered statistics are aggregated and used as cost estimates.
However, in this particular example, there are only few rewriting possibilities: In detail, the
sequence of Selection operators can be reordered according to their selectivities (optimality
conditions oc 1 -oc 3 ; e.g., oc 1 : sel(o 2 ) ≤ sel(o 3 ) with sel = |ds out1 |/|ds in1 |), and the paths
of the Switch operator can be reordered according to their cost-weighted path probabilities
(oc 4 ). Each single re-optimization is a full optimization, where our transformation-based
168