Cost-Based Optimization of Integration Flows - Datenbanken ...

4 Vectorizing Integration Flows
2. Schema Evaluation: Evaluate all distribution schemes according to the optimization
objective φ or φ c .
3. Plan Rewriting: Rewrite the plan according to the distribution scheme.
In the following, we briefly describe each of those steps with more technical depth.
1: Scheme Enumeration: In order to enumerate all possible distribution schemes
of a plan P with m operators, we recursively use Algorithm 4.2. As a first step, we
create a MEMO table with m columns. In a second step, for each k ∈ [1, m], we create a
record of length k and invoke the recursive A-EDS. Conceptually, this algorithm varies the
number of operators of bucket 1 (line 5) and recursively invokes itself in order to distribute
the remaining operators across buckets 2 to k. It then varies the number of operators of
bucket 2 and so on. Finally, if the remaining operators should be distributed across the last
bucket, we insert the tuple into the MEMO structure but we could also directly evaluate the
enumerated scheme. As a result, the MEMO structure holds all 2 m−1 candidate distribution
schemes. Note that this approach is used recursively for complex operators and it contains
different loop conditions for the case of sets of operators.
Algorithm 4.2 Enumerate Distribution Schemes (A-EDS)
Require: number of operators m, number of buckets k, record r, position pos
1: if k = 1 then
2: r.pos[1] ← m
3: insert r into MEMO
4: else
5: for i ← 1 to m − k + 1 do // for each operator o i
6: r.pos[pos] ← i
7: A-EDS(m − i, k − 1, r, pos + 1) // recursively enumerate distribution schemes
2: Scheme Evaluation: Having enumerated all candidates, we can now iterate over
the MEMO structure and evaluate those schemes in order to determine the optimal scheme
according to the optimization objectives φ or φ c . Recall the problem definition of costbased
vectorization, i.e., the overall performance of vectorized plans depends on the most
time-consuming operator. Here, the costs of a bucket are defined as the sum of all operators
in that bucket. We then determine the bucket with maximum costs. The overall
optimization objective φ is to minimize the number of buckets under the condition of
lowest possible maximum bucket costs. In general, all 2 m−1 candidate schemes need to
be evaluated. However, we could prune schemes, where (1) we already determined that
a bucket exceeds the maximum execution time and (2) the number of buckets exceeds
the minimum number of buckets seen so far. These pruning techniques can be realized
on-the-fly during scheme enumeration or with skip-list structures as known from other
research areas such as join enumeration [HKL + 08] or time series analysis [GZ08].
3: Plan Rewriting: Finally, we use the optimal scheme in order to rewrite the given
plan P . For that, the A-PV can be reused with minor changes. Here, we do not create an
execution bucket for each operator but we consider the computed k. All operators of one
bucket can be copied as a subplan, while data dependencies across execution buckets are
replaced by queues. The general model of execution buckets (see Subsection 4.2) is reused
as it is with the difference of arbitrary subplans instead of single operators.
Similar to the analysis of Section 4.2, the rewriting algorithm still has a worst-case complexity
of O(m 3 ) for the case of k = m. Furthermore, the evaluation of a single distribution
106