Cost-Based Optimization of Integration Flows - Datenbanken ...

6 On-Demand Re-Optimization
For on-demand re-optimization, we only maintain statistics that are required to evaluate
the optimality conditions. All other statistics are declined by the PlanOptTree such
that they are not stored and aggregated. When atomic statistics are updated, we also
maintain the aggregate, update the hierarchy of complex statistics measures, and evaluate
optimality conditions that are reachable children of this statistic node. Arbitrary workload
aggregation methods, as described in Subsection 3.3.2, can be used for aggregation
of atomic statistics. Due to this incremental maintenance and immediate condition evaluation,
the use of Exponential Moving Average (EMA) is most suitable because (1) it is
incrementally maintained and (2) no negative statistics maintenance (sliding window) is
necessary due to the exponentially decaying weights.
The naïve application of triggering re-optimization based on the incrementally monitored
statistics could lead to the problem of frequently changing plans.
Example 6.5 (Problem of Frequent Plan Changes). Assume the optimality condition of
sel(σ A ) ≤ sel(σ B ). There are two problems that can cause frequent plan changes (instability).
First, due to unknown statistics, we are only able to monitor conditional selectivities
sel(σ A ) and sel(σ B |σ A ). If A and B are correlated, the optimality condition might be
violated even after re-optimization. Second, if the selectivities are constant but alternate
around equality with sel(σ A ) ≈ sel(σ B ), we would also change the plan back and forth. In
both cases, we would get frequent re-optimization steps that are not amortized.
We explicitly address this problem of missing robustness (instability) when triggering
re-optimization with the following strategies:
• Correlation Tables: As described in Subsection 3.3.4, we explicitly compute conditional
selectivities using a lightweight correlation table. Essentially, we maintain
selectivities over multiple versions of a plan, where we store and maintain a row
of atomic and conditional selectivities for each pair of operators with direct data
dependency within the current plan. Unless we see the second operator ordering,
we assume statistical independence. However, based on the maintenance of conditional
selectivities, we do not make a wrong decision based on correlation twice. For
on-demand re-optimization, the use of this correlation table is even more important.
The integration into the PlanOptTree is realized by a specific complex statistic node
(CSNode) Conditional Selectivity that maintains and reads the correlation table.
• Minimal Existence Time: We use the time period ∆t from periodical re-optimization
as minimal existence time of a plan. This means that no optimality conditions are
evaluated during ∆t after the last re-optimization. During this interval, we only collect
statistics but we do not aggregate and evaluate them. As a result, the adaptation
sensibility is reduced in order to avoid that re-optimization is triggered multiple times
in case that (1) we have not finished another asynchronous re-optimization step or
(2) the workload has changed abruptly and caused the violation of multiple optimality
conditions with short delay. However, ∆t determines only the minimum period of
re-optimization and hence, can be set independently of the workload characteristics
(low-influence parameter). After that, we continuously check optimality conditions
and adapt faster to workload changes than periodical re-optimization does.
• Lazy Condition Violation: When evaluating an optimality condition, we might not
have seen all atomic statistics of a plan instance. Similar to known control strategies,
re-optimization is lazily triggered if the condition is violated ∑ m ′
1 s′ times, which is
176