Cost-Based Optimization of Integration Flows - Datenbanken ...

3 Fundamentals of Optimizing Integration Flows
(parallelizing sequences and iterations) because otherwise we might miss optimization opportunities.
Furthermore, this technique should be used also after WC4 (merging parallel
flows) because otherwise, we might perform unnecessary optimization efforts.
Rewriting Sequences to Parallel Flows
The technique WC2: Rewriting Sequences to Parallel Flows is used to optimize sequences
of operators. Such sequences can be found as implicit children of each Plan, Switch path,
Fork subflow, and Iteration. The costs of a sequence of operators o is given by the sum
of their execution times with W (o) = ∑ m
i=1 W (o i).
The core concept is to rewrite a sequence of operators to parallel subflows of a Fork
operator by analyzing the dependencies between the single operators. Recall that the
execution time of the Fork operator is determined by the subflow with highest cost. For
analyzing the optimality of such a rewriting, we take into account the number of logical
processors (hardware threads) k as well as the CPU utilization of the involved operators.
There, the CPU utilization of an operator with regard to a single logical processor is
computed by (W (o i ) − wait(o i ))/W (o i ), where the waiting time can be monitored (e.g.,
waiting time for external systems); otherwise we assume wait(o i ) = W (o i ) · 0.05 as a
heuristic. Such a rewriting of a sequence to |r| parallel subflows is advantageous if
⎛
⎞
|r|
max
i=1
∑m i
⎝ Ŵ (o i,j ) + i · W (Start Thread) ⎠ <
j=1
with Ŵ (o i,j) =
m∑
Ŵ (o i )
i=1
|r|
min(|r|, k) · (W (o i,j) − wait(o i,j )) + wait(o i,j ),
(3.17)
which means that the estimated most time-consuming parallel subflow must have lower
costs than the plan sequence of operators. There, the costs of operators within parallel
subflows are estimated by the waiting time plus the execution time that depends on the
number of logical processors k and the number of parallel subflows |r|. Intuitively, this
represents the increased execution time if parallel subflows share hardware resources. This
is a worst-case consideration because for an exact model, the temporal overlap of waiting
times and execution times would be required as well.
Rewriting sequences to parallel flows is realized with the following algorithm. First, we
split the given sequence into disjoint subsequences according to Rule 3 of Definition 3.1
(preserve temporal order of writing interactions to the same external system). Second, for
each of these subsequences, we create a new Fork operator and partition the individual
operators, where we iterate over the operators and determine if they depend on other
operators of the same subsequence. If so, we add this operator to the existing subflow,
where the operator referred by the dependency exists; otherwise, we create a new subflow
and add the operator as a child. If an operator depends on multiple operators from
different subflows, this operator splits the subsequence into two subsequences. Third,
if a Fork operator contains only one subflow, we rewrite it back to a simple sequence.
Fourth, and finally, we check the optimality condition for each Fork operator. The time
complexity of this algorithm is O(m 2 ) due to the dependency checking for each operator.
In the following, we use an example to illustrate this rewriting concept in more detail.
Example 3.10 (Rewriting Sequences to Parallel Flows). Recall our example plan P 8 that
is essentially a sequence of operators and assume the monitored execution times shown in
62