Cost-Based Optimization of Integration Flows - Datenbanken ...

5 Multi-Flow Optimization
• P 5 : The plan P 5 shows the lowest benefit. There, the Switch operator o 5 is executed
only once for a batch because the switch expression attribute /resultsets/
resultset/row/A1 Orderdate is used as partitioning attribute. Additional benefit
comes from the Assign and Invoke operators o 8 and o 9 . However, we achieved only
an improvement of 25% because the Selection operators in front of the operators
that benefit from partitioning consume most of the time and significantly reduces
the cardinality of intermediate results.
• P 7 : In contrast to the other plans, plan P 7 does not contain any partitioning attribute
candidate. Therefore, a system partitioning with sel = 1.0 is used (this
case is similar to the time-base batch creation strategy but without the drawback
of distinct messages in a batch). Many operators benefit from partitioning. First,
the queries to external systems are prepared only once (Assign operators o 3 , o 6 , o 9 ,
and o 11 ). Second, also the external queries and the subsequent schema mapping is
only executed once (Invoke operators o 4 , o 7 , o 10 , and o 12 as well as Translation
operators o 5 , o 8 , o 13 ). Third, additional benefit is achieved by the final Assign and
Invoke operators o 18 and o 19 . In total, this led to an improvement of 30%.
Figure 5.15(b) illustrates the optimization overhead imposed by the cost-based multiflow
optimization. This includes the derivation of partitioning attributes, the creation of
a partitioning scheme, the plan rewriting, as well as the continuous waiting time computation.
Essentially, we observe that the overhead is moderate, where the differences are
mainly reasoned by the different numbers of operators.
Finally, based on the observed results, we can conclude that the multi-flow optimization
technique can be used by default (if small additional latency for single messages is acceptable)
because the throughput improvements clearly amortize the optimization overhead.
This is true for arbitrary asynchronous, data-driven integration flows because each flow
has at least one combination of writing Assign and Invoke operators.
Scalability
We now investigate the scalability of plan execution, which includes (1) the scalability
with increasing input data sizes and (2) the scalability with increasing batch sizes.
First, we used our example plans in order to investigate the scalability of optimization
benefits with increasing input data size. We reused the scalability experiment with increasing
data size from Section 3.5. In contrast to the already presented scalability results, we
now disabled all optimization techniques except multi-flow optimization. In detail, we executed
20,000 plan instances for the plans P 1 , P 2 , P 5 , and P 7 and compared the optimized
plans with their unoptimized counterparts varying the input data size d ∈ {1, 2, 3, 4, 5, 6, 7}
(in 100 kB). Again, we varied the input data size of these plans (the size of the received
message) only but did not change the size of externally loaded data. Further, we fixed
a batch size of k ′ = 10, an optimization interval of ∆t = 5 min, a sliding window size
of ∆w = 5 min and EMA as the workload aggregation method. The results are shown in
Figure 5.16. In general, the plans scale with increasing data size but with a decreasing
relative improvement. With regard to the different plans, we observe different scalability
behavior. The plan P 1 scales best with increasing data size and shows almost constant
relative improvement because this plan mainly benefits from reduced costs for writing
interactions that linearly depend on the data size. Further, also plan P 2 shows good scalability
with increasing data size. However, the relative improvement is decreasing because
158