Cost-Based Optimization of Integration Flows - Datenbanken ...

7 Conclusions
I can’t change the direction of the wind,
but I can adjust my sails to always reach my destination.
— Jimmy Dean
From the reactive perspective of an integration platform, we cannot change the workload
characteristics (the direction of the wind) in the sense of incoming messages that initiate
plan instances of integration flows. However, we can incrementally maintain execution
statistics and use these for the cost-based optimization of integration flows (the adjustment
of sails) in order to improve their execution time, latency and thus, the message throughput
(in order to reach the destination). This allows the workload-based adjustment of the
current plan and thus, for continuous adaptation to changing workload characteristics.
Based on emerging requirements of complex integration tasks that (1) stretch beyond
simple read-only applications, (2) involve many types of heterogeneous systems and applications,
and (3) require fairly complex procedural aspects, typically, imperative integration
flows are used for specification and execution of these tasks. In this context, we observe
that many independent instances of such integration flows with rather small amounts of
data per instance are executed over time in order to achieve (1) high consistency between
data of operational systems or (2) high up-to-dateness of analytical query results in data
warehouse infrastructures. In addition to this high load of flow instances, the performance
of source systems depends on the execution time and availability of synchronous
data-driven integration flows. For these reasons, there are high performance demands on
integration platforms that execute imperative integration flows.
To tackle this problem of high performance requirements on the execution of integration
flows, we introduced the cost-based optimization of these imperative integration flows. In
detail, we described the fundamentals of cost-based optimization including novel techniques
such as the first entirely defined cost model for imperative integration flows, a
transformation-based rewriting algorithm with several search space reduction approaches,
the asynchronous periodical re-optimization as well as techniques for workload adaptation
and handling of correlated data. Essentially, this cost-based optimizer exploits the major
integration flow specific characteristics of being deployed once and executed many times
with rather small amounts of data per instance. In addition, we introduced several controlflow
and data-flow-oriented optimization techniques, where we adapted on the one side
techniques from data management systems, and programming language compilers as well
as on the other side, we defined techniques tailor-made for integration flows. Additionally,
we described in detail two novel optimization techniques for throughput optimization
of integration flows, namely the cost-based vectorization and the multi-flow optimization.
Finally, we introduced the novel concept of on-demand re-optimization of integration flows
that overcomes the major drawbacks of periodical re-optimization, while still exploiting
the major characteristics of integration flows. Our experiments showed that significant execution
time and throughput improvements are possible, while only moderate additional
optimization overhead is required.
199