Cost-Based Optimization of Integration Flows - Datenbanken ...

5 Multi-Flow Optimization
reached, where the message rate is too high. For example, consider a message rate of
R = 20 msg /s, this break-even point occurs at ≈ 422,000 distinct partitions in the queue.
Side Effects of Optimization Techniques
Putting it all together, we conducted an additional experiment in order to evaluate the
influences between MFO, vectorization, and the other cost-based optimization techniques.
(a) Plan P 1 (b) Plan P 2
(c) Plan P 5 (d) Plan P 7
Figure 5.23: Use Case Scalability Comparison with Varying Data Size d
We reused our scalability experiment with increasing data size. In contrast to the already
presented scalability results, we now use the different cost-based optimization techniques
from Chapter 3-5 in combination with each other. In detail, we executed 20,000 plan
instances for the plans P 1 , P 2 , P 5 , and P 7 and compared the optimized plans with their
unoptimized versions varying the input data size d ∈ {1, 4, 7} (in 100 kB). In contrast to
all other experiments of this chapter, we measured the elapsed scenario time (the total
latency time of the message sequence) rather than the total execution time because for
vectorized execution, the execution times of single plan instances cannot be aggregated
due to overlapping message execution (pipeline semantics). Furthermore, we varied the
input data size of these plans (the size of the received message) but did not change the
size of externally loaded data. We fixed a batch size of k ′ = 10, an optimization interval
of ∆t = 5 min, a sliding window size of ∆w = 5 min and EMA as the workload aggregation
method. The results (total elapsed time) are shown in Figure 5.23. Essentially, we
observe two major effects. First, the application of all optimization techniques consistently
shows the highest performance compared to single optimization techniques. Only
plan P 2 performed slightly worse when using full optimization compared to vectorization
only, because the operators that benefit from partitioning are not part of the most timeconsuming
bucket and thus, MFO introduces additional latency (although it reduces the
164