Cost-Based Optimization of Integration Flows - Datenbanken ...

5.6 Summary and Discussion
total execution time), while not reducing the total latency time. However, for a data
size of d = 1 as an example, we achieved significant overall relative improvements of 82%
(P 1 ), 72% (P 2 ), 74% (P 5 ), and 55% (P 7 ). Second, we observe that typically, the highest
optimization benefits are achieved by vectorization and multi-flow optimization, where
these plans (P 1 , P 2 , and P 5 ) do not have a high CPU utilization in the unoptimized case.
Hence, the optimization benefits are partially overlapping. In contrast, for plans such as
P 7 , where the local processing steps dominate the execution time (high CPU utilization),
the standard cost-based optimiztaion techniques have higher influence. In this case, the
different optimization benefits are not overlapping and hence the joint application achieves
significant improvements. Furthermore, we see different scalability of the different optimization
techniques with incresing data size according to the used plan. The application
of all optimization techniques balances these effects such that finally, we observe good
scalability with increasing data size for all plans with almost constant improvement. We
can conclude that, especially, with regard to the scalability and the maximum benefit, it
is advantageous to use all optimization techniques in combination.
Finally, we can state that MFO achieves significant throughput improvement by accepting
moderate additional latency time for single messages. Furthermore, the serialized
external behavior can be guaranteed as well. Anyway, how much we benefit from MFO
depends on the used plans and on the concrete workload. The benefit of MFO is caused by
two main facts. First, even for one-message partitions, there is only a moderate runtime
overhead (Figures 5.18(b) and 5.22). Second, only a small number of messages is required
within one partition to yield a significant speedup (Figure 5.18(b)).
5.6 Summary and Discussion
To summarize, in this chapter, we introduced the data-flow-oriented multi-flow optimization
(MFO) technique for throughput maximization of integration flows. Both MFO and
the control-flow-oriented vectorization technique achieve throughput improvements. In
contrast to vectorization that relies on parallelization, MFO reduces executed work by
employing horizontal data partitioning of inbound message queues and executing plans
for batches of messages. First, we discussed the plan execution of message partitions that
includes the definition of the partition tree as a queue data structure for message partitions
as well as the automatic derivation of partitioning attributes, the derivation of partitioning
schemes, and the rewriting of plans. Second, we explained the required cost model extensions,
the computation of the optimal waiting time with regard to message throughput
improvement, and the integration into our overall cost-based optimization framework.
In conclusion of our formal analysis and experimental evaluation, the multi-flow optimization
technique achieves significant throughput improvement by accepting moderate
additional latency time for single messages. Furthermore, we guarantee constraints of
maximum latency for single messages and serialized external behavior. Thus, MFO is
applicable for arbitrary asynchronous, data-driven integration flows in many different application
areas. Finally, it is important to note that MFO and vectorization can already
be applied together in order to achieve the highest throughput due to the integration of
both techniques within the surrounding cost-based optimization framework.
Further, MFO opens several opportunities for further optimizations. Future work might
consider, for example, (1) the execution of partitions independent of their temporal order,
165