Cost-Based Optimization of Integration Flows - Datenbanken ...

6 On-Demand Re-Optimization
6.6 Summary and Discussion
The periodical re-optimization exhibits the drawbacks of (1) full monitoring of all operator
statistics regardless if they are used by the optimizer or not, (2) many unnecessary
periodical re-optimization steps, where for each step a full optimization is executed, (3)
missed optimization opportunities due to potentially slow workload adaptation, and (4)
the optimization period ∆t as a high-influence parameter, whose configuration requires
context knowledge of the integration flows as well as knowledge of the frequency of workload
changes. These drawbacks are mainly reasoned by the strict separation between
optimizer, statistics monitoring and plan execution.
In this chapter, we presented the concept of on-demand re-optimization to overcome
these drawbacks. We introduced the PlanOptTree that models optimality of the current
plan by exploiting context knowledge from the optimizer. With this approach, statistics
are monitored according to optimality conditions and directed re-optimization is triggered
if and only if such a condition is violated. In conclusion, this approach always reduces the
total execution time because (1) if the workload does not change, we avoid unnecessary reoptimization
steps, and (2) if there are workload changes, we do not miss any optimization
opportunities due to direct re-optimization, while the overhead for evaluating optimality
conditions is negligible. In addition, it allows for predictable performance without the
need for elaborate parameterization. In conclusion, on-demand re-optimization has the
same advantages but overcomes the disadvantages of periodical re-optimization.
However, on-demand re-optimization has also some limitations. Most importantly, the
on-demand re-optimization is more sensitive with regard to workload changes than periodical
re-optimization is. On the one side this is advantageous because we directly adapt to
these changes and therefore, reduce the execution time. On the other side, more care with
regard to robustness (stability) is required. For example, correlation-awareness is required
because otherwise, we might result in frequent plan changes, which hurt performance. For
this reason, we introduce the concepts of correlation tables, minimal existence time, and
lazy condition violation, which ensures robustness of on-demand re-optimization.
In conclusion of our experimental evaluation, the on-demand re-optimization, in comparison
to periodical re-optimization, achieves additional cumulative execution time improvements,
while it requires much less re-optimization steps and therefore, significantly
reduces the total optimization overhead. Furthermore, there are plenty issues for future
work. This includes, for example, the investigation of (1) the extension of inter-instance
re-optimization to intra-instance re-optimization (mid-query re-optimization) in order to
support ad-hoc integration flows or long running plan instances, (2) specific approaches
for directed re-optimization with regard to complex optimization techniques (e.g., join
enumeration, eager group-by), and (3) the combination of progressive parametric query
optimization with on-demand re-optimization in order to reuse generated physical plans.
Although the on-demand re-optimization approach is tailor-made for integration flows that
are deployed once and executed many times, it is also applicable in other areas. Examples
for these areas are continuous queries in DSMS, re-occurring queries in DBMS, and
incremental maintenance of data mining results.
198