Cost-Based Optimization of Integration Flows - Datenbanken ...

4 Vectorizing Integration Flows
o 2 p 1 o 1 o 3 o 4
p 2 o 2 o 3 o 4
o 1
o 1 o 2 o 3 o 4
p 3
time t
t 0(p 1) t 1(p 1) t 0(p 2)
t 1(p 2) t 0(p 3)
t 1(p 3)
(a) Instance-Based Plan P
p 1
o 1
o 2 o 3 o 4
o 2 o 3 o 4
p 2 o 1
o 1
o 2 o 3 o 4
p 3
t 0(p 1) t 1(p 1)
t 0(p 2) t 0(p 3)
t 1(p 2)
t 1(p 3)
time t
(b) Fully Vectorized Plan P ′
p 1 o 1
o 2 o 3 o 4
p 2 o 1 o 2 o 3 o 4
p 3 o 1 o 2 o 3 o 4
t 0(p 1) t 0(p 2) t 0(p 3) t 1(p 1)
t 1(p 2)
t 1(p 3)
time t
(c) Cost-Based Vectorized Plan P ′′
Figure 4.11: Work Cycle Domination by Operator o 3
of plan P ′ in Figure 4.11(b). In conclusion, we leverage the waiting time during work
cycles of the data flow graph and merge operators into execution buckets if applicable.
Formally, this optimization objective is defined as follows:
⎛ ⎞
φ = min
m
l k | ∀i ∈ [1, k] : ∑ bi
⎝ W (o j ) ⎠ ≤ W (o max ) (4.7)
k=1
The goal is to find the minimal number of execution buckets k under the restriction that the
execution time of each bucket b i (sum of execution times of the l bi operators of this bucket)
does not exceed the execution time of the most time-consuming operator. As a result, we
achieve the highest degree of parallelism with a minimal number of threads. Further
advantages of this concept are reduced latency time for single messages and robustness
in the case of many plan operators but limited thread resources. The special case of
the P-CPV with optimization objective φ, where all operators are independent (no data
dependencies), is reducible to the NP-hard offline bin packing problem [Joh74].
Typically, the optimization objective φ allows to find a scheme that exploits the highest
pipeline parallelism but requires fewer threads than the full vectorization. However, in
special cases such as (1) where all operators exhibit almost the same execution time or
(2) where a plan contains too many operators, the problem of a large number of required
threads still exist. In order to overcome this general problem, we extend the P-CPV by a
parameter to allow for higher robustness. In detail, this extended optimization problem
is defined as follows:
Definition 4.3 (Constrained P-CPV). With regard to the P-CPV, find the minimal number
of k buckets and an assignment of operators o j with j ∈ [1, m] to those execution buckets
j=1
102