Cost-Based Optimization of Integration Flows - Datenbanken ...

4 Vectorizing Integration Flows
Theorem 4.3. The cost-based plan vectorization algorithm solves (1) the P-CPV and (2)
the constrained P-CPV with linear time complexity of O(m). There, the cost constraints
hold but the number of execution buckets might not be minimized.
Proof. Assume a plan that comprises a sequence of m operators. First, the maximum of
a value list (line 2) is known to exhibit a linear time complexity of O(m). Second, we see
that the bucket number is at least 1 (all operators assigned to one bucket) and at most
m (each operator assigned to exactly one bucket). Third, in both cases of k = 1, and
k = m, there are at most 2m − 1 possible operator evaluations. If we assume constant
time complexity for all set operations, we can now conclude that the cost-based plan
vectorization algorithm exhibits a linear complexity with O(m) = O(3m − 1). However,
due to the importance of the concrete order of operator evaluations, we might require a
higher number of execution buckets k than optimal. Hence, Theorem 4.3 holds.
We use an example to illustrate this heuristic cost-based plan vectorization algorithm
and the influence of the λ parameter regarding the constrained optimization objective φ c .
Example 4.7 (Heuristic Cost-Based Plan Vectorization). Assume a plan with m = 6
operators shown in Figure 4.13. Each operator o i has assigned execution times W (o i ). The
maximum operator execution time is given by W (o max ) = max m i=1 W (o i) = W (o 3 ) = 5 ms.
given operator
sequence o
o1 o2 o3 o4 o5 o6
W(o1)=1 W(o2)=4 W(o3)=5 W(o4)=2 W(o5)=3 W(o6)=1
λ=0 (max W(bi)=5)
λ=1 (max W(bi)=6)
λ=2 (max W(bi)=7)
b1
b2 b3 b4
o1 o2 o3 o4 o5 o6
b1
b2 b3
o1 o2 o3 o4 o5 o6
b1
b2
b3
o1 o2 o3 o4 o5 o6
k=4
k=3
k=3
Figure 4.13: Bucket Merging with Different λ
The Constrained P-CPV describes the search for the minimal number of execution buckets,
where the cumulative costs of each bucket must not be larger than the determined maximum
plus a user-defined cost increase λ. Hence, we search for those k buckets whose cumulative
costs of each bucket are, at most, equal to five. If we increase λ, we can reduce the number
of buckets by increasing the allowed maximum and hence, the work cycle of the vectorized
plan. This example also shows the heuristic character of the algorithm. For the case of
λ = 2 ms, we find a scheme with (k = 3, W (b 1 ) = 5 ms, W (b 2 ) = 7 ms, W (b 3 ) = 4 ms),
while our exhaustive approach would find the more balanced scheme (k = 3, W (b 1 ) =
5 ms, W (b 2 ) = 5 ms, W (b 3 ) = 6 ms).
In conclusion, this heuristic approach ensures the maximum benefit of pipeline parallelism
and often minimizes the number of execution buckets and hence, reduces the
number of threads as well as the length of the pipeline at the same time. Cammert et
al. introduced a similar optimization objective in terms of a stall-avoiding partitioning
of continuous queries [CHK + 07] in the context of data stream management systems. In
contrast to their approach, our algorithm is tailor-made for integration flows with control
flow semantics and materialized intermediate results within an execution bucket. In
addition, (1) we presented exhaustive and heuristic computation approaches as well as
108