Cost-Based Optimization of Integration Flows - Datenbanken ...

6 On-Demand Re-Optimization
projection, early translation, early group-by, join enumeration), (2) techniques that insert
or remove specific operators (e.g., execution pushdown, order-by insertion, rewriting
sequences/iteration to parallel flows), and (3) techniques that choose between different
physical operator implementations (e.g., join type selection, setoperation type selection).
Fourth, existing cost models (e.g., cost formulas and aspects such as correlation) can
be reused. Only the optimizer interface and the single optimization techniques require
some changes with regard to (1) expressing optimality with partial PlanOptTrees and (2)
allowing for directed re-optimization of subplans. We explained these modifications for
selected optimization techniques. In conclusion, the concept of on-demand re-optimization
is generally applicable for existing optimizer architectures as well.
6.5 Experimental Evaluation
In this section, we present selected experimental evaluation results comparing the ondemand
re-optimization with the periodical re-optimization that has been evaluated already
in Chapter 3. Therefore, we concentrate on the comparison of the different optimization
models rather than comparing the on-demand re-optimization approach with the
unoptimized execution, or regarding the benefit of individual optimization techniques. In
general, the experiments show that:
• Most importantly, the cumulative costs of plan execution are reduced due to the fast
adaptation to changing workload characteristics that prevents missed optimization
opportunities. In addition to the fast adaptation, the total execution time does not
depend on the chosen optimization interval anymore.
• The re-optimization costs are typically reduced because re-optimization is only triggered
if required. Re-optimization might be triggered more frequently than for periodical
re-optimization in case of continuously changing workload characteristics.
For this reason of ensuring robustness, the use of our lightweight correlation table
has higher importance as for the periodical re-optimization.
• For complex integration flows, directed re-optimization significantly improves the optimization
time. This is especially true for optimization techniques with super-linear
time complexity, where the benefit of reducing the number of evaluated combinations
has huge impact on the overall optimization time.
• The overhead of evaluating optimality conditions for each monitored atomic statistic
is fairly low due to minimal statistic maintenance. Furthermore, this low overhead
is negligible compared to the cumulative execution time improvements.
In detail, the description of our experimental results is structured as follows. First, we
present the end-to-end comparison of on-demand re-optimization with periodical and no
optimization using simple and complex integration flows. We also show the scalability with
varying parameters. Second, we discuss the benefit of directed re-optimization and the
overhead of maintaining optimality conditions. Third, we explain the effects of correlation
and how we achieve robustness of on-demand re-optimization.
Experimental Setting
We implemented the on-demand re-optimization, within our WFPE (workflow process engine).
This includes the PlanOptTree data structure and related algorithms, as well as
188