Cost-Based Optimization of Integration Flows - Datenbanken ...

3 Fundamentals of Optimizing Integration Flows
our transformation-based optimization algorithm, approaches for search space reduction
and adjusting the sensibility of workload adaptation as well as a lightweight concept for
handling conditional probabilities and correlation. Further, we explained selected optimization
techniques that are specific to integration flows because they exploit both the
data flow and the control flow in a combined manner. Our evaluation shows significant
performance improvements with moderate overhead for periodical re-optimization.
In conclusion, our cost-based optimization approach can be integrated seamlessly into
the major products in the area of integration platforms. Based on the observation of
many independent instances of integration flows, this approach of periodical cost-based
re-optimization is tailor-made for integration flows. In detail, the advantages of periodical
re-optimization are (1) the asynchronous optimization independently of executing certain
instances, (2) the fact that all subsequent instances rather than only the current query
benefit from re-optimization, and (3) the inter-instance plan change without the need
of state migration. This general optimization framework can be used as foundation for
further rewriting techniques and optimization approaches.
Apart from these re-optimization advantages, the optimization framework presented so
far has still several shortcomings. First, only the optimization objective of minimizing
the average plan execution time was considered. This is not always a suitable optimization
objective because in high load scenarios often the major optimization objective is
throughput maximization, while moderate latency times are acceptable. Therefore, in the
following, we will present two integration-flow-specific optimization techniques that have
the potential to significantly increase the message throughput. In detail, we present the
cost-based vectorization (a control-flow-oriented optimization technique) in Chapter 4 and
the multi-flow optimization (a data-flow-oriented optimization technique) in Chapter 5.
Second, also the periodical re-optimization algorithm itself has several drawbacks. This
includes the generic gathering of statistics for all operators that causes the maintenance of
statistics that might not be used by the optimizer. While for the evaluated workload aggregation
methods, this overhead was negligible, there might be performance issues when
using more complex forecast models. In addition, there is the problem of periodically
triggered re-optimization, where a new plan is only found if workload characteristics have
changed. Otherwise, we trigger many unnecessary invocations of the optimizer that evaluates
the complete search space. Depending on the used optimization techniques this can
have notable performance implications. However, if a workload change occurs, it takes a
while until re-optimization is triggered. During this adaptation delay, we thus use a suboptimal
plan and miss optimization opportunities. Finally, the parameter ∆t (optimization
period) has high influence on optimization and execution times and hence, parameterization
requires awareness of changing workloads. These four drawbacks are addressed with
the concept of on-demand re-optimization that we will present in Chapter 6. However, the
periodical re-optimization already provides a reasonable optimization framework including
many fundamental concepts and thus, is used as the conceptual basis of this thesis.
86