Cost-Based Optimization of Integration Flows - Datenbanken ...

3.5 Experimental Evaluation
Figure 3.25: Optimization Overhead of Join Enumeration
rithm [Moe09] and (2) our join reordering heuristic with quadratic time complexity that
we have described in Subsection 3.3.2. Before optimization, we randomly generated statistics
for input cardinalities and join selectivities. The experiment was repeated ten times.
Figure 3.25 illustrates the results of this experiment using a log-scaled y-axis. The optimization
time of the full join enumeration increases exponentially, while for the heuristic
re-optimization, the optimization time increases slightly super-linear. However, we observe
acceptable absolute optimization time of exhaustive join enumeration until eight Join operators,
where the clique query is the worst-case for the DPSize algorithm. This justifies
our algorithm selection rule of using the full optimization algorithm until eight joins and
to use the heuristic for larger numbers of joins. As a result, we can guarantee that (1)
the time required by our optimization algorithm will not increase exponentially with the
complexity of plans and (2) the algorithm will find the optimal plan in the presence of
small numbers of Join operators.
Second, we analyzed the overhead of statistics monitoring and statistics aggregation.
If statistics monitoring is enabled, each operator propagates at least three (|ds in |, |ds out |,
and W (o i )) statistics to the Estimator. However, there are operators that propagate
more statistics such as Switch path frequencies, number of iterations and the cardinality
of multiple input and output data sets. Thus, the efficiency of statistics monitoring and
workload aggregation is important in order to achieve moderate re-optimization overheads.
Figure 3.26: Cumulative Statistic Maintenance Overhead
We used the statistic trace from our first comparison scenario (see Figure 3.20), where
we executed n = 100,000 plan instances of P 5 . We used three statistics (execution time,
input and output cardinalities) from seven of nine operators of this plan that results in
a test set of 2,100,000 statistic tuples. All sub experiments were repeated 1,000 times.
Figure 3.26 illustrates the results for different aggregation strategies. Full aggregation
refers to a single estimation using all statistics, which is applicable if the optimization
81