Cost-Based Optimization of Integration Flows - Datenbanken ...

3 Fundamentals of Optimizing Integration Flows
Problem 3.2 (Changing Workload Characteristics). In the context of integration flows
that integrate distributed systems and applications, the workload characteristics—in the
form of statistics such as selectivities, cardinalities, and execution times—can change significantly
over time [IHW04, NRB09, DIR07, CC08, LSM + 07, BMM + 04, MSHR02]. This
can be caused by unpredictable workloads of the external systems (e.g., number of update
transactions, amount of data to be integrated, waiting times for external systems) or temporal
variations of infrastructural properties (e.g., network traffic or bandwidth). These
influences lead to changing workload characteristics of the integration platform in the sense
of different numbers of plan instances, different cardinalities and selectivities as well as
different execution times when accessing external systems.
As a result of Problem 3.2, rule-based optimized plans, where no execution statistics
are used for optimization, fall short and may perform inefficiently over time. For the
same reason, also manually optimized plans, where a fixed (hand-crafted) plan is specified
by an administrator, cannot be applied [Win03]. Cost-based optimization, using data
properties and execution statistics, has the potential to overcome this deficit because
optimal execution plans are generated with regard to the current statistics. By keeping
these statistics up-to-date, adaptation to changing workload characteristics is possible.
However, for integration flows, there is the additional problem of missing statistics:
Problem 3.3 (Missing Knowledge about Statistics of External Systems). One of the
main problems of integration flow optimization is the lack of knowledge about data characteristics
(e.g., cardinalities, selectivities, ordering) and execution statistics (execution
times, bandwidth) of the different data sources [IHW04]. Due to the integration of autonomous
(loosely-coupled) source systems, the integration platform usually has no access
to statistical information of the external systems—if they exist at all.
However, execution statistics are required for cost-based optimization. Due to the missing
statistics, the central integration platform has to incrementally maintain the workload
characteristics and execution statistics by itself. The combination of Problem 3.2 and
Problem 3.3 leads to the need for a cost-based optimization approach that incrementally
maintains execution statistics and re-optimizes given plans if necessary. Unfortunately, existing
cost-based approaches [SMWM06, SVS05] follow an optimize-always model, where
optimization is synchronously trigged for each plan instance before it is executed. This
optimize-always model falls short in the presence of many short-running plan instances,
where the optimization time might be even higher than the execution time of an instance.
In addition, these existing approaches do not consider an overall cost-based optimization
framework but only investigate selected cost-based optimization techniques in isolation.
In consequence, we introduce the general concept of cost-based optimization of imperative
integration flows. As a starting point, we follow the optimization objective of
minimizing the average execution time of a plan, which implicitly increases the message
throughput as well. The core concept is (1) to incrementally monitor workload characteristics
and execution statistics, and (2) to periodically re-optimize given plans using a set of
cost-based optimization techniques. As a result, this approach enables cost-based rewriting
decisions and it achieves a suitable adaptation to changing workload characteristics,
while it requires less optimization overhead than the optimize-always model.
The cost-based optimization of integration flows differs from cost-based optimization for
(1) programming languages or (2) data management systems for several reasons. First,
although optimizers of programming language compilers optimize imperative programs,
34