Cost-Based Optimization of Integration Flows - Datenbanken ...

6 On-Demand Re-Optimization
The overall cost-based optimization framework used so far relies on incremental statistic
maintenance and periodical re-optimization to meet the high performance demands of integration
flows and to overcome the problems of unknown statistics and changing workload
characteristics. The potential problems of (1) many unnecessary re-optimization steps and
(2) missed optimization opportunities due to adaptation delays are caused by the strict
separation of optimization, execution and statistic monitoring. In order to overcome these
major drawbacks of periodical re-optimization, in this chapter, we introduce the novel
concept of on-demand re-optimization.
Our aim is to reduce the overhead for statistics monitoring and re-optimization and
at the same time to adapt to changing workload characteristics as fast as possible. We
achieve this by extending the optimizer interface of our overall cost-based re-optimization
framework in the sense of modeling optimality of a plan by its optimality conditions
and triggering re-optimization only if workload changes violate these conditions. First,
we define the Plan Optimality Tree (PlanOptTree) and describe how to create such a
PlanOptTree for a given plan to model optimality of this plan by its optimality conditions
rather than considering the complete search space. We explain how to use it for
statistic maintenance and how this triggers re-optimization if optimality conditions are
violated. Second, we exploit these violated conditions for search space reductions during
re-optimization. In detail, we explain the directed re-optimization and the update of
PlanOptTrees after successful re-optimization. Finally, we describe how common optimization
techniques are extended in order to enable on-demand re-optimization and we
present experimental evaluation results, which compare the periodical re-optimization with
this novel on-demand re-optimization approach. According to the experimental evaluation,
we achieve improvements concerning re-optimization overhead as well as adaptation
sensibility and thus reduce the total execution time.
6.1 Motivation and Problem Description
Efficiency of integration flows is ensured by cost-based optimization in order to (1) exploit
the full optimization potential and (2) to adapt to changing workload characteristics
[MSHR02, IHW04, BMM + 04, DIR07, LSM + 07, CC08, NRB09] such as varying cardinalities,
selectivities and execution times (for example, reasoned by unpredictable workloads
of external systems or temporal variations of network properties). With regard to the low
risk of re-optimization overhead and good optimization opportunities, the state-of-the-art
cost-based optimization models of integration flows are (1) the periodical re-optimization
(see Chapter 3) or (2) the optimize-always optimization model [SVS05, SMWM06]. Within
the optimize-always model optimization is triggered for each plan instance, which fails in
the case of many plan instances with rather small amounts of data per instance because
the optimization time can be even higher than the execution time of a single plan instance.
As mentioned in Subsection 2.2.4, there are fundamental differences to other system
167