Cost-Based Optimization of Integration Flows - Datenbanken ...

3 Fundamentals of Optimizing Integration Flows
number of parallel subflows. The rewriting of iterations to parallel flows is beneficial if
⎛
⎞
|r| ∑m i
∑m i
max ⎝ Ŵ (o i,j ) + i · W (Start Thread) ⎠ < r · W (o i,j )
i=1
j=1
j=1
(3.18)
with Ŵ (o |r|
i,j) =
min(|r|, k) · (Ŵ (o i,j) − wait(o i,j )) + wait(o i,j ).
Due to foreach semantics, r must be estimated in terms of the frequency of iterations.
Rewriting is realized by the following algorithm. We check if there are no dependencies
between iterations and if we are allowed to change the temporal order. In case of independence,
we compile all operators of the iteration body to r subflows plus one additional
subflow. Each subflow references a specific data partition of the inbound data set, while
the last subflow is used for all partitions that exceed the estimated number of iterations.
In conclusion, this optimization technique offers—similar to the rewriting of sequences—
high optimization opportunities. In contrast to sequences, the rewriting relies heavily on
the estimated number of iterations. The possibility of arbitrary input data sets might
result in unused subflows (overestimation) or in executing multiple partitions with the
last subflow (underestimation). One can further enhance this by using dynamic runtime
scheduling of parallel subflows (e.g., guided self-scheduling [PK87], or factoring [HSF91]).
In addition, this technique can be combined with rewriting sequences (the operators of
one iteration) to parallel flows and this technique should be applied before the techniques
WC1 (rescheduling parallel flows) and WC4 (merging parallel flows).
Merging Parallel Flows
Recall the costs of a Fork operator that are determined by the most time-consuming
subflow. The idea of the technique WC4: Merging Parallel Flows is that if the costs
of the subflow with maximum costs subsume the costs of two or more other subflows,
the subsumed subflows can be rewritten to one subflow in order to reduce the costs by
W (T hread) that denotes the costs for thread creation, starting, and monitoring (in contrast
to the previously used W (Start T hread) that denotes the time required for thread
creation only). Therefore, all Fork operators with more than two subflows have to be
considered. We achieve an execution time reduction because less threads are required,
while the most time-consuming subflow is unchanged.
In general, this problem of a given maximum constraint per partition and the optimization
objective of minimizing the number of partitions, is reducible to the offline bin
packing problem that is known to be an NP-hard problem. Therefore, we use an extension
of the heuristic first fit algorithm [Joh74] that works as follows. First, we determine the
subflow with maximum costs. Second, for each old subflow, we check if it can be merged
with an existing new subflow. If this is possible, we merge the subflow with the first fit
subflow by temporally concatenating the sequences of operators; otherwise, we create a
new subflow. In the worst case, for each old subflow, we check each new subflow. Thus,
the time complexity is given by O(m 2 ). In the following, we use an example to illustrate
this heuristic algorithm.
Example 3.11 (Merging Parallel Flows). Recall our example plan P 7 from Example 3.9.
Further, assume changed execution times as shown in Figure 3.14(a). First, we determine
the fourth subflow (349 ms) as upper bound for our extended first fit algorithm. Second,
64