Cost-Based Optimization of Integration Flows - Datenbanken ...

1 Introduction
for integration flows exist [Sch01]. First, the horizontal integration describes the integration
of systems within one level. A typical example is the integration of operational systems
by EAI servers (adapter-based integration of arbitrary systems and applications) or MOM
systems (efficient message transport via messaging standards), where every update within
an operational system can initiate data synchronization with other operational systems
and hence, data is exchanged by transferring many small messages [HW04]. Second, the
vertical integration describes the integration across the levels of the information pyramid.
The most typical example is the integration of data from the operational systems into
the dispositive and strategical systems (data warehouses) by ETL tools. In this context,
there is a trend towards operational BI (Business Intelligence) that requires immediate
synchronization between the operational source systems and the data warehouse in order
to achieve high up-to-dateness for analytical query results [DCSW09, O’C08, WK10].
This requirement is typically addressed with a near-real-time approach [DCSW09], where
the frequency of periodical delta load is simply increased, or with data-driven ETL flows,
where data changes of the operational source systems are directly propagated to the data
warehouse infrastructure as so-called trickle-feeds [DCSW09, SWCD09].
As a result of both horizontal and vertical integration, many independent instances of
integration flows are executed over time. In addition to this high load of flow instances,
the performance of source systems depends on the execution time of synchronous datadriven
integration flows, where the source systems are blocked during execution. For
these reasons, there are high performance demands on integration platforms in terms of
minimizing execution and latency times. Furthermore, from an availability perspective in
terms of the average response times, the performance of synchronous integration flows has
also direct monetary influences. For example, Amazon states that just 0.1s increase in
average response times will cost them 1% in sales [Bro09]. Similarly, Google recognized
that just 0.5s increase in latency time caused the traffic to drop by a fifth [Lin06]. In
consequence, optimization approaches are required.
Existing optimization approaches of integration flows are mainly rule-based in the sense
that a flow is only optimized once during the initial deployment. Thus, only static rewriting
decisions can be made. Further, the optimization of integration flows is a hard problem
with regard to the characteristics of imperative flow specifications with interaction-,
control-flow- and data-flow-oriented operators as well as specific transactional properties
such the need for preserving the serial order of incoming messages. The advantage is low
optimization overhead because optimization is only executed once. However, rule-based
optimization has two major drawbacks. First, many optimization opportunities cannot
be exploited because rewriting decisions can often only be made dynamically based on
costs with regard to execution statistics such as operator execution times, selectivities
and cardinalities. Second, it is impossible to adapt to changing workload characteristics,
which commonly vary significantly over time [IHW04, NRB09, DIR07, CC08, LSM + 07,
BMM + 04, MSHR02]. This would require rewriting an integration flow in a cost-based
manner according to the load of flow instances and specific execution statistics.
Contributions
In order to address the high performance demands on integration platforms and to overcome
the drawbacks of rule-based optimization, we introduce the concept of cost-based
optimization of integration flows with a primary focus on typically used imperative in-
2