Cost-Based Optimization of Integration Flows - Datenbanken ...

5 Multi-Flow Optimization
the waiting time at the lower border of the defined T L function interval (line 7). Finally,
we compute the total latency time, the resulting number of messages per batch k ′ and
check the validity condition in order to react on overload situations (line 8-11). Due to
the linear classification of operator costs and the analytical determination of ∆tw, this
algorithm exhibits a linear complexity of O(m) according to the number of operators m.
Despite the described simplification of waiting time computation, where we compute
the special case ∆tw = W (P ′ , k ′ ), it would not be sufficient to simply collect messages for
the execution time of the current message partition. Most importantly, this is reasoned
by horizontal (value-based) partitioning that leads to internal out-of-order execution. For
high message rates in combination with many distinct partitions, the average message
latency as well as the effective system output rate would degrade due to message synchronization
at the outbound side. By computing the waiting time or maximal partition size,
respectively, we flush oldest messages out of the system such that better average message
latency times and output rates are achieved and we also reduce the effort for outbound
message synchronization. Finally, we can also compute the global minimum for the general
case of arbitrary cost models.
In this subsection, we have described how to compute the optimal waiting time with
regard to minimizing the total latency time under a maximum latency constraint. This
maximizes the message throughput with regard to a single deployed plan. With regard
to multiple deployed plans this approach is applicable as well. However, changing the optimization
objective to minimizing the total execution time under the maximum latency
constraint can lead to an even higher throughput because this minimizes potentially overlapping
execution. However, the computation approach is realized similarly as for the case
of a single deployed plan despite the difference that we always try to determine the upper
border (lc) of the defined T L function interval rather than the lower border. Finally, the
waiting time computation is a central aspect of the multi-flow optimization technique and
of its integration into our general cost-based optimization framework.
5.3.4 Optimization Algorithm
Putting it all together, we now describe how the multi-flow optimization technique is
integrated into our general cost-based optimization framework. This includes changes of
the deployment process as well as the cost-based re-optimization.
The deployment process is modified such that it now additionally includes the automatic
derivation of partitioning attributes, as described in Subsection 5.2. Apart from
that, most aspects of the multi-flow optimization technique are integrated into the feedback
loop of our cost-based optimization framework. The major issues are the derivation
of partitioning schemes, the rewriting of plans and the computation of the optimal waiting
time. First, according to the monitored selectivities of partitioning attributes that have
been derived during the initial deployment, we derive the optimal partitioning scheme in
case of multiple partitioning attributes. Second, if there are at least two attributes and
if we have found a new partitioning scheme during re-optimization, we rewrite the plan
according to this partitioning scheme in order to enable the execution of operations on
horizontally partitioned message batches. For a single partitioning attribute, rewriting is
not required because all operators can work transparently on a single partition level as
described in Subsection 5.2.3. This rewriting includes also the requirement of dynamic
state migration in the sense of transforming an existing partition tree that indexes collected
messages from one partitioning scheme into another scheme. In order to ensure the
150