Cost-Based Optimization of Integration Flows - Datenbanken ...

of traditional data management systems and the included optimization techniques focus
on execution time minimization only, we additionally introduce two novel cost-based optimization
techniques that are tailor-made for integration flows, which both follow the
optimization objective of throughput maximization. First, we explain the concept of
cost-based vectorization of integration flows in order to optimally leverage pipeline parallelism
of plan operators and thus, increase the message throughput. Second, we discuss
the concept of multi-flow optimization via horizontal message queue partitioning that increases
throughput by executing operations on message partitions instead of on individual
messages and thus, it reduces work of the integration platform such as the costs for querying
external systems. Finally, the major drawbacks of periodical re-optimization are (1)
many unnecessary re-optimization steps, where we find a new plan, only if workload characteristics
have changed, and (2) adaptation delays after a workload change, where we
use a suboptimal plan until re-optimization and miss optimization opportunities. Therefore,
we refine the re-optimization approach from periodical re-optimization to on-demand
re-optimization, where only necessary statistics are maintained and re-optimization is immediately
triggered only if a new plan is certain to be found.
The positive consequences of the cost-based optimization of integration flows are, in general,
(1) the continuous adaptation to dynamically changing workload characteristics and
(2) performance improvements in the sense of minimizing execution times and maximizing
message throughput by exploiting the full optimization potential of rewriting decisions.
In particular, the parameterless on-demand re-optimization achieves a fast but robust
adaptation to changing workload characteristics with minimal overhead for incremental
statistics maintenance and directed re-optimization. Finally, this cost-based optimization
framework of integration flows can be used for investigating additional integration-flowspecific
optimization techniques. Those optimizations are strongly needed in order to meet
the continuously increasing performance requirements on integration platforms.
iv