Cost-Based Optimization of Integration Flows - Datenbanken ...

3.3 Periodical Re-Optimization
at T 2 , we enrich the selectivity tuple with the missing information (due to reordering, we
now have independent statistics for both operators). Over time, we use the exponential
moving average (EMA) to adapt those statistics independently of the sliding time window.
Finally, we see that the sequence of (σ A , σ B ) is the best choice because P (A) < P (B).
We might make wrong decisions at the beginning but the probability estimates converge to
the real statistics over time. Hence, we do not make wrong decisions twice as long as the
probability comparison and the data dependency between both operators do not change.
As long as a selectivity tuple contains a missing value for an operator o 2 , we make the
assumption of statistical independence and hence compute P (o 2 ) = P (o 1 ∧ o 2 )/P (o 1 ).
Furthermore, we compute the conditional probabilities for a given operator—based on the
described selectivity tuple—as follows:
P (o 2 |o 1 ) = P (o 1 ∧ o 2 )
P (o 1 )
⎧
P (o 1 ∧ o 2 )
⎪⎨
= P (o 1 ) P (o 2 ) = NULL
P (o 1 |o 2 ) =
P (o 2 |o 1 )
P (o 1 ∧ o 2 )
⎪⎩
otherwise.
P (o 2 )
(3.15)
Thus, we explicitly use the monitored conditional probabilities if available. It is important
to note that we can only maintain the probability of the first operator as well as the joint
probability of both operators. Hence, the additional problem of starvation in reordering
decisions might occur if the real probability of the second operator decreases because
we cannot monitor this effect. In order to tackle this problem of starvation in certain
rewriting decisions, we use an aging strategy, where the probability of the second operator
is slowly decreased over time. This prevents starvation because over time we reorder both
operators due to the decreased probability and can monitor the actual probability of this
second operator. Although this can cause suboptimal plans in case of no workload changes,
it prevents starvation and converges to the real selectivities and probabilities.
For arbitrary chains of operators, the correlation table is used recursively along operators
that are directly connected by data dependencies such that an operator might be included
in multiple entries of the correlation table. For such chains, we allow only reordering
directly connected operators. After a reordering, all entries of the correlation table that
refer to a removed data dependency are removed as well and new entries for new data
dependencies are created.
In conclusion, the adaptive behavior of our condition selectivity approach ensures more
robust and stable optimizer decisions for conditional probabilities and correlated data,
even in the context of changing workload characteristics. Note that there are more sophisticated,
heavyweight approaches (e.g., by using a maximum entropy approach [MHK + 07,
MMK + 05] or by using a measure of clusteredness [HNM03]) for correlation-aware selectivity
estimation in the context of DBMS. We could also maintain adaptable multidimensional
histograms [BCG01, AC99, LWV03, MVW00, KW99]. However, in contrast
to DBMS, where data is static and statistics are required for arbitrary predicates, we optimize
the average plan execution time with known predicates but dynamic data. Hence,
the introduced lightweight correlation table can be used instead of heavyweight multidimensional
histograms, where we would need to maintain exact or approximate frequencies
with regard to arbitrary conjunct predicates [Pol05, TDJ10]. However, other
approaches can be integrated as well. Due to the problem of missing statistics about
59