Cost-Based Optimization of Integration Flows - Datenbanken ...

3 Fundamentals of Optimizing Integration Flows
and σ B and the following gathered statistics:
σ A : |ds in | = 100, |ds out | = 40 σ B : |ds in | = 40, |ds out | = 8
⇒P (A) = 0.4
P (B) = P (B|A) = 0.2
P (A ∧ B) = 0.08.
//monitored
//computed assuming independence
//monitored
Assuming statistical independence, we set P (B) = P (B|A) and based on the comparison
of P (A) > P (B), we reorder the sequence (σ A , σ B ) to (σ B , σ A ). Using the new plan, we
gather the following statistics:
σ B : |ds in | = 100, |ds out | = 68 σ A : |ds in | = 68, |ds out | = 8
⇒P (B) = 0.68
P (A) = P (A|B) ≈ 0.12
P (A ∧ B) = 0.08.
//monitored
//computed assuming independence
//monitored
Clearly, P (A) and P (B) are strongly correlated (P (¬A∧¬B) = 0). Due to the simplifying
assumption of independence, we would assume that P (B) > P (A). Hence, we would
reorder the operators back to the initial plan. As a result, even in the presence of constant
statistics, we would reorder the plan back and forth and thus, produce inefficient plans.
Our approach of conditional selectivities explicitly takes those conditional probabilities
into account in order to overcome that problem when reordering selective dataflow-oriented
operators (e.g., Selection, Projection with duplicate elimination, Join,
Groupby, Setoperation) or the paths of a Switch operator. Essentially, we maintain
selectivity statistics over multiple versions of a plan, independently of the sliding time
window statistics. Therefore, for each pair of data-flow-oriented operators with direct
data dependency within the current plan, we maintain a row of selectivities:
(o 1 , o 2 , P (o 1 ), P (o 2 ), P (o 1 ∧ o 2 )), (3.14)
where both operators o 1 and o 2 are identified, and we store the selectivity as well as the
conjunct selectivity of both operators. The approach works similar for binary operators
and reordered Switch paths. Due to the binary comparison approach of only two operators
at-a-time, the overhead is fairly low. In the worst case, there are m 2 selectivity tuples,
where m denotes the number of operators. We revisit the example in order to explain that
concept in detail.
Example 3.8 (Monitored Conditional Selectivities). Assume the same setting as in Example
3.7. When reordering σ A and σ B at timestamp T 1 , we create a new statistic tuple
as shown in Table 3.5.
Table 3.5: Conditional Selectivity Table
o 1 o 2 P (o 1 ) P (o 2 ) P (o 1 ∧ o 2 )
T 1 σ A σ B 0.4 0.08
T 2 σ A σ B 0.4 0.68 0.08
We only include probabilities that are known not to be conditional (the first operator and
the combination of both operators). If we evaluate the temporal order of those operators
58