Cost-Based Optimization of Integration Flows - Datenbanken ...

5 Multi-Flow Optimization
not allow a discrete counting of outrun messages due to the possible outrun of partitions.
5.2.2 Deriving Partitioning Schemes
The partitioning scheme in terms of the optimal layout of the partition tree is derived
automatically. This includes (1) deriving candidate partitioning attributes from the given
plan and (2) to find the optimal partitioning scheme for the overall partition tree.
Candidate partition attributes are derived from the single operators o i of plan P . We
realize this by searching for attributes that are involved in predicates, expressions and
dynamic parameter assignments. This is a linear search over all operators with O(m).
Due to different semantics of those attributes, we distinguish between the following three
types of partitioning attributes, which have been introduced in Definition 5.1:
1. Value: This scheme causes a data partitioning by exact value. Thus, for all 1/sel
distinct values of this attribute, a partition is used. An example for this type is an
equality predicate of a query to an external system (Assign and Invoke operator).
2. Value List: Due to disjunctive predicates of external queries or local Switch operators,
we can also use a list of exact values with or-semantics.
3. Range: According to range query predicates or inequalities range partitioning is used.
Examples for this type are expressions of Switch operators or range predicates of
queries to external systems (Assign and Invoke operator).
After having derived the set of candidate partitioning attributes and the type of each
partitioning attribute, we need to select candidates that are advantageous to use. First,
we remove all candidates, where a partitioning attribute refers to externally loaded data
because these attribute values are not present for incoming messages at the inbound side of
the integration platform. Second, we compare the benefit of using a partitioning attribute
(see Section 5.3) with a user-specified cost reduction threshold τ and remove all candidates
that are evaluated as being below this threshold.
Based on the set of partitioning attributes, we create a concrete partitioning scheme for
the partition tree. For h partitioning attributes, there are h! different partitioning schemes.
Due to this factorial complexity, we use a heuristic for finding the optimal scheme. The
intuition is to minimize the number of partitions in the index. Assuming no correlations 14 ,
we order the index attributes according to their selectivities with
min
h∑
|b ∈ ba i | iff sel(ba 1 ) ≥ sel(ba i ) ≥ sel(ba h ). (5.1)
i=1
Hence, finding the best partitioning scheme exhibits a complexity of O(h log h) due to
the requirement of sorting the h partitioning attributes. Another approach would be to
take the additional costs of rewritten plans into account. However, the costs of additional
operators (splitting and merging of partitions) have shown to be negligible.
Example 5.5 (Minimizing the Number of Partitions). Recall Example 5.4 and assume
the partitioning attributes ba 1 (customer, value) and ba 2 (total price, range) as well
14 The MFO approach works also for correlated partitioning attributes, where queue management might become
more expensive due to mis-estimated selectivities. Alternatively, the correlation table introduced
in Subsection 3.3.4 could be used for correlation-aware ordering of partitioning attributes.
138