Cost-Based Optimization of Integration Flows - Datenbanken ...

3.4 Optimization Techniques
determining only the most time-consuming subflow rather than all subflows. A detailed
cost estimation example of a rewritten subgraph (applying this technique) was given with
Example 3.2 (Cost Estimation) in Subsection 3.2.2
Finally, this technique should be applied after the join enumeration WD10 because
it requires the optimal join order as the basis for its rewriting algorithm but it should
be applied before WD9 because the join type selection might change according to the
estimated cardinalities. Interdependencies to join enumeration are deferred to the next
invocation of the optimizer.
Set Operations with Distinctness
For the Setoperation operator, we distinguish different types, among others the UNION
DISTINCT and the UNION ALL. While the UNION ALL can be computed with low costs of
|ds in1 | + |ds in2 |, the costs of UNION DISTINCT are computed by
|ds in1 | + |ds in2 | · |ds out|
. (3.27)
2
We transfer the first data set completely into the result (|ds in1 |) and then, for each tuple
of the second data set ds in2 , we need to check whether or not this tuple is already in the
result. In the average case, this causes costs of |ds out |/2 for each tuple. As a result, UNION
DISTINCT has a quadratic time complexity of O(N 2 ).
The core idea of WD11: Setoperation-Type Selection in combination with the WD8:
Orderby Insertion / Removal is to sort both input data sets by their distinct key in order
to enable the application of an efficient merge algorithm for ensuring distinctness. Hence,
only costs of |ds in1 | + |ds in2 | (similar to a UNION ALL) would be necessary to compute
the UNION DISTINCT. Including the costs for sorting, the result is a time complexity of
O(N log N). Despite the internal order-preserving XML representation, sorting of message
content is applicable in dependence on the source adapter types of these messages.
In the following, we consider the required optimality conditions. There are three alternative
subplans for a union distinct R ∪ S. First, there is the normal union distinct
operator with costs that are given by C(R ∪ S) = |R| + |S| · |R ∪ S|/2 (two plans due to
asymmetric costs), where |R| ≤ |R ∪ S| ≤ |R| + |S| holds. Second, we can sort both input
data sets and apply a merge algorithm (third plan), where the costs are computed by
C (sort(R) ∪ M sort(S)) = |R| + |S| + |R| · log 2 |R| + |S| · log 2 |S|. (3.28)
In conclusion, for arbitrary cardinalities, the optimality conditions are |R| ≥ |S| and
|R| + |S| ·
|R ∪ S|
2
≤ |R| + |S| + |R| · log 2 |R| + |S| · log 2 |S|
|R ∪ S| ≤ 2 + 2 · |R| · log 2|R|
|S|
+ 2 · log 2 |S|.
(3.29)
We see that this decision depends on the union output cardinality and both input cardinalities.
If one input is known to be sorted, the corresponding Orderby operator is omitted
and the optimality conditions are modified accordingly.
Example 3.15 (Setoperation-Type Selection). Assume our example plan P 6 (see Figure
3.18(a)) that includes two Setoperation operators of type UNION DISTINCT. Using
71