Cost-Based Optimization of Integration Flows - Datenbanken ...

3.3 Periodical Re-Optimization
The MA causes the slowest adaptation because of the simple average, where all items
are equally weighted, while WMA (linear weights) and EMA (exponential weights) support a
faster adaptation due to the highest influence of the latest items. However, LR achieves the
fastest adaptation because the estimate is extrapolated from the last items. Unfortunately,
LR tends to over- and underestimate on abrupt changes. Further, full re-computation with
all methods can be realized with linear complexity of O(n). A detailed explanation of
the different adaptation parameters with regard to monitored execution statistics is given
within the experimental evaluation. We also experimented with Polynomial Regression
(PR) with up to a degree of four. Due to high over- and underestimation on abrupt changes,
we do not use this aggregation method. Apart from these aggregation methods, one can
use forecast models types [DB07, BJR94] in order to detect reoccurring patterns and thus,
increase the accuracy of estimation. However, doing so is a trade-off between estimation
overhead and benefit achieved by more accurate estimation. Experiments have shown that
the simple exponential moving average (EMA) robustly achieves the highest accuracy with
low estimation overhead such that we use this method as our default workload aggregation
strategy. There, we do not use any automatic parameter estimation. However, the automatic
evaluation and adaptation of this parameter could be seamlessly integrated into the
periodical re-optimization framework.
Recall the parameters optimization interval ∆t and sliding time window size ∆w. If
∆t ≥ ∆w, statistic tuples are only used at one specific optimization timestamp. In this
case, we simply compute those statistics from scratch using the workload aggregation methods.
However, if ∆t < ∆w, this is an inefficient approach because we aggregate portions
of statistic tuples multiple times. In such a case, incremental maintenance of workload
statistics—in the sense of updating the aggregate with new statistics—is required. In general,
incremental statistics maintenance is possible for all of these aggregation methods.
However, note that MA and LR require negative (implicitly removed tuples based on time)
and positive maintenance (new tuples) according to the sliding time window size ∆w (and
thus, atomic statistics must be stored), while EMA and WMA does not require negative maintenance
due to the increasing weights, where the influence of older tuples can be neglected.
In conclusion, we use the EMA as default workload aggregation method.
Finally, the workload aggregation can be optimized. Similar to the basic counting
algorithm [DGIM02], approximate incremental statistics maintenance over the sliding
window—with summarizing data structures—is possible. Further, according to workload
shift detection approaches [HR07, HR08], it might be possible to minimize the optimization
costs by deferring the cost re-estimation until predicted workload shifts (anticipatory
re-optimization).
3.3.4 Handling Correlation and Conditional Probabilities
In the context of missing knowledge about data properties of external systems, the main
problem of cost estimation is correlation and conditional probabilities (more precisely,
relative frequencies). Assume two successive Selection operators σ A and σ B . In fact, we
are only able to monitor P (A) and P (B|A). A naïve approach would be to assume that
all predicates and the resulting selectivities and cardinalities are independent. However,
this can lead to wrong estimates [BMM + 04] that would result in non-optimal plans or
changing a plan back and forth.
Example 3.7 (Monitored Selectivities). Assume two successive Selection operators σ A
57