Cost-Based Optimization of Integration Flows - Datenbanken ...

3.6 Summary and Discussion
(a) Selectivity Variations
(b) Execution Time
Figure 3.29: Comparison Scenario of Periodical Re-Optimization with Correlation
maintaining conditional probabilities over multiple versions of a plan. Due to the involvement
of three correlated operators, incremental optimization required deleting records
from this correlation table. As a result, there are also some plan switches to suboptimal
plans (e.g., after workload shift *2, we observe three wrong plan switches). However, over
time, the conditional selectivity estimates converge to the real selectivities, which reasons
a 10% improvement with regard to the cumulative execution time. In conclusion, the use
of our correlation table ensures robustness in the presense of correlated data or conditional
probabilities, while the overhead is negligible (in this comparison scenario, we maintained
three entries in this correlation table).
The details of our exhaustive evaluation have shown that significant performance improvements
can be achieved by periodical re-optimization in the sense of minimizing the
average execution time of a plan, while only moderate overhead is imposed by statistics
monitoring and periodical re-optimization. Even in the case, where no optimization
techniques could be applied, no significant performance penalty was measured. Most importantly,
the optimized plans show a better scalability than unoptimized plans. Thus,
typically, the relative performance improvements increase with an increasing input data
size or an increasing number of operators. Finally, with the right choice of parameters
the self-adjusting cost model in combination with correlation awareness enables a fast but
still robust adaptation to changing workload characteristics.
3.6 Summary and Discussion
To summarize, in this chapter, we introduced the cost-based optimization of imperative
integration flows to overcome the major problem of inefficiently performing integration
flows in the presence of changing workload characteristics. The incremental maintenance
of execution statistics addresses missing knowledge about data properties when integrating
heterogeneous and highly distributed systems and applications. In the area of integration
flows, cost-based re-optimization has been considered for the first time. Our solution comprises
the dependency analysis and details on the monitoring of workload and execution
statistics as well as the definition of the double-metric cost model. Based on these foundations,
we discussed the NP-hard Periodic Plan Optimization Problem (P-PPO), including
85