Cost-Based Optimization of Integration Flows - Datenbanken ...

Abstract
Integration flows are increasingly used to specify and execute data-intensive integration
tasks between several heterogeneous systems and applications. There are many different
application areas such as (near) real-time ETL (Extraction Transformation Loading) and
data synchronization between operational systems. For the reasons of (1) an increasing
amount of data, (2) typically highly distributed IT infrastructures, and (3) high requirements
for data consistency and up-to-dateness, many instances of integration flows—with
rather small amounts of data per instance—are executed over time by the central integration
platform. Due to this high load as well as blocking synchronous source systems or
client applications, the performance of the central integration platform is crucial for an
IT infrastructure. As a result, there is a need for optimizing integration flows. Existing
approaches for the optimization of integration flows tackle this problem with rule-based
optimization in the form of algebraic simplifications or static rewriting decisions during
deployment. Unfortunately, rule-based optimization exhibits two major drawbacks. First,
we cannot exploit the full optimization potential because the decision on rewriting alternatives
often depends on dynamically changing costs with regard to execution statistics
such as cardinalities, selectivities and execution times. Second, there is no re-optimization
over time and hence, the adaptation to changing workload characteristics is impossible.
In conclusion, there is a need for adaptive cost-based optimization of integration flows.
This problem of cost-based optimization of integration flows is not as straight-forward as
it may appear at a first glance. The differences to optimization in traditional data management
systems are manifold. First, integration flows are reactive in the sense that they process
remote, partially non-accessible data that is received in the form of message streams.
Thus, proactive optimization such as dedicated physical design is impossible. Second,
there is also the problem of missing knowledge about data properties of external systems
because, in the context of loosely coupled applications, statistics are non-accessible or do
not exist at all. Third, in contrast to traditional declarative queries, integration flows are
described as imperative flow specifications including both data-flow-oriented and controlflow-oriented
operators. This requires awareness with regard to semantic correctness when
rewriting such flows. Additionally, further integration-flow-specific transactional properties
such as the serial order of messages, the cache coherency problem when interacting
with external systems, and the compensation-based rollback must be taken into account
when optimizing such integration flows. In conclusion, the cost-based optimization of
integration flows is a hard but highly relevant problem in today’s IT infrastructures.
In this thesis, we introduce the concept of cost-based optimization of integration flows
that relies on incremental statistics maintenance and inter-instance plan re-optimization.
As a foundation, we propose the concept of periodical re-optimization and present how
to integrate such a cost-based optimizer into the system architecture of an integration
platform. This includes integration-flow-specific (1) prerequisites such as the dependency
analysis and a cost model for interaction-, control-flow- and data-flow-oriented operators
as well as (2) specific statistic maintenance strategies, optimization algorithms and
optimization techniques. While this architecture was inspired by cost-based optimizers
iii