Cost-Based Optimization of Integration Flows - Datenbanken ...

3 Fundamentals of Optimizing Integration Flows
for typical workloads. Second, we evaluate the parameters of these workload aggregation
methods. Figure 3.28(c) illustrates the influence of the parameter EMA smoothing constant
α. We used a sliding time window size of ∆w = 10,000 s and illustrated the estimated costs
continuously (∆t = 1 s). Clearly, an decreasing parameter α causes slower adaptation and
therefore more robust estimation. However, for typical parameter settings of 0.05 to 0.001
very fast but still robust adaptation can be achieved. Note that for α ∈ {0.2, 0.02, 0.002}
we obtained similar results for the sliding window size of ∆w = 1,000 s from the previous
experiment. However, for α = 0.0002 (with ∆w = 1,000 s) the estimates varied significantly
which was caused by too few statistics in the time window in combination with
a low smoothing factor such that the estimated values were significantly determined by
the initial value (first statistic in the window) because the adaptation took too long. In
order to analyze the influence of the sliding window size ∆w in general, we conducted an
additional experiment. Figure 3.28(d) illustrates the influence of the sliding time window
size, where we fixed Agg = MA and varied ∆w from 10 s to 10,000 s. Clearly, the adaptation
slows down with an increasing ∆w. However, both extremes can lead to wrong (large error)
estimations. The choice of the window size should be made based on the specific plan
because, for example, a long-running plan or a infrequently used plan need a longer time
window than plans with many instances per time period. The EMA method, typically, does
not need sliding window semantics due to the time-decaying character where older items
can be neglected. However, if a sliding window is used, the sliding window size ∆w should
be set according to the plan and the used smoothing constant α such that enough statistics
are available as already discussed. Furthermore, the optimization interval influences
the re-estimation granularity. With ∆t = 1 s, we get a continuous cost function, while an
increasing ∆t causes a slower adaptation because this influences the maximal delay of ∆t
until re-estimation. Obviously, parameter estimators, which minimize the error between
forecast values and real values could be used to determine optimal parameter values for
∆t and ∆w. However, when and how to adjust these parameters is a trade-off between
additional statistic maintenance overhead and cost estimation accuracy that is beyond the
scope of this thesis.
With regard to precise statistic estimation, handling of correlated data and conditional
probabilities are important. Therefore, we conducted an experiment in order to evaluate
our lightweight correlation table approach in detail. We reused our end-to-end comparison
scenario (see Figure 3.20), where we executed 100,000 instances of our example plan P 5
and compared the resulting execution time when using periodical re-optimization with and
without the use of our correlation table. In contrast to the original comparison scenario, we
generated correlated 7 data. Figure 3.29(a) illustrates the conditional selectivities P (o 2 ),
P (o 3 |o 2 ), and P (o 4 |o 2 ∧ o 3 ) of the three Selection operators, where we additionally set
P (o 3 |¬o 2 ) = 1 and P (o 4 |¬o 2 ∨ ¬o 3 ) = 1. As a result, o 3 strongly depends on o 2 as well as
o 4 strongly depends on o 2 and o 3 .
Figure 3.29(b) illustrates the resulting execution time with and without the use of our
correlation table. We observe that without the use of the correlation table, the optimization
technique selection reordering assumes statistical independence and thus, changed the
plan back and forth, even in case of constant workload characteristics. This led to the periodic
use of suboptimal plans, where the optimization interval ∆t = 5 min prevented more
frequent plan changes. In contrast, the use of the correlation table ensured robustness by
7 We did not use the Pearson correlation coefficient and known data generation techniques [Fac10] in order
to enable the exact control of unconditional and conditional selectivities.
84