Cost-Based Optimization of Integration Flows - Datenbanken ...

5 Multi-Flow Optimization
Similar to the vectorization of integration flows, in this chapter, we introduce the multiflow
optimization [BHL10, BHL11] as a data-flow-oriented optimization technique that is
tailor-made for integration flows. This technique tackles the problem of expensive external
system access as well as it exploits the optimization potential that equivalent work (e.g.,
same queries to external systems) is done multiple times. The core idea is to horizontally
partition inbound message queues and to execute plan instances for message batches rather
than for individual messages. Therefore, this technique is applicable for asynchronous
data-driven integration flows, where message queues are used at the inbound side of the
integration platform. As a result, the message throughput is increased by reducing the
amount of work (external system access and local processing steps) done by the integration
platform. We call this technique multi-flow optimization because sequences of messages
that would initiate multiple plan instances are processed together.
In order to enable multi-flow optimization, in Section 5.1, we introduce the batch creation
via horizontal message queue partitioning. Essentially, two major challenges arise
in the context of multi-flow optimization. In Section 5.2, we discuss the challenge of plan
execution on batches of messages. Furthermore, in Section 5.3, we describe how this optimization
technique is embedded within the periodical re-optimization framework and
we address the challenge of computing the optimal waiting time with regard to message
throughput maximization. In addition, we provide formal analysis results such as optimality
and latency guarantees in Section 5.4. Finally, the experimental evaluation, which
is presented in Section 5.5, shows that significant performance improvements in the sense
of an increased message throughput are achieved by multi-flow optimization.
5.1 Motivation and Problem Description
In the context of integration platforms, especially in scenarios with huge numbers of plan
instances, the major optimization objective is throughput maximization [LZL07] rather
than the execution time minimization of single plan instances. The goal is (1) to maximize
the number of messages processed per time period, or synonymously in our context, (2) to
minimize the total execution time of a sequence of plan instances. Here, depending on the
application area, moderate latency times of single messages, in the orders of seconds to
minutes, are acceptable [UGA + 09]. When addressing this general optimization objective,
the following concrete problems have to be considered:
Problem 5.1 (Expensive External System Access). External system access can be really
time-consuming caused by network latency (minimal roundtrip time), external query processing,
network traffic, and message transformations from external formats into internal
structures. Depending on the involved external systems and on the present infrastructure,
the fraction of these influences with regard to the required total access time may vary significantly.
However, in particular when accessing custom applications and services, data
129