Cost-Based Optimization of Integration Flows - Datenbanken ...

3.3 Periodical Re-Optimization
We can monitor all cardinalities |R|, |S|, and |T | but only the selectivities f R,S and
f (R⋊⋉S),T . To estimate f R,T , we need to derive it with f R,T θ f (R⋊⋉S),T , where θ is a
function representing the correlation. If we assume statistical independence of selectivities,
we can set f R,T = f (R⋊⋉S),T . However, in general, the selectivities are derived from the
present conjunction of join predicates. This cost comparison is applied for each subset
of join operators of a plan that can be fully reordered. Note that this heuristic does not
necessarily requires that the cost model exhibits the ASI property [Moe09]. In addition,
simple sorting of join operands is not applicable due to the use of arbitrary correlation
functions and the need for evaluating if join inputs are connected.
Although this algorithm often produces good results, obviously, it does not guarantee
to find the optimal join order. This is reasoned by (1) the restrictions of considering only
nested loop joins and left-deep-join trees, (2) the selection of the first join operand based
on minimum cardinality, and (3) reordering only directly connected join operands rather
than partial subtrees.
In contrast to our transformation-based join reordering heuristic, recent approaches of
heuristic query optimization [BGLJ10] use merge-based techniques, where ranked subplans
are merged iteratively to an overall plan. While this bottom-up approach is advantageous
for declarative queries, our transformation-based reordering is more advantageous for imperative
integration flows with regard to the characteristic of an initially specified plan.
In general, this heuristic join reordering algorithm exhibits a quadratic time complexity
of O(m 2 ), where m denotes the number of operators with m = n − 1 and n denotes the
number of joined input data sets. This is reasoned as follows. First, we iterate over all n
input data sets in order to determine the minimum cardinality. Second, we iterate over all
input data sets and for each, compare the costs assuming a reordering with its predecessors
(similar to selection sort). Thus, in total, we execute at most
n +
n∑
i=3
(i − 2) = n2 − n
2
+ 1 (3.9)
iterations during this algorithm. Finally, note that—except the awareness of temporal
dependencies (join enumeration restrictions)—this heuristic join reordering algorithm can
be applied in data management systems as well. As a result, the use of this heuristic
join enumeration algorithm (combined with an extended heuristic first fit algorithm for
merging parallel flows that we will describe in Section 3.4) reduces the overall complexity
of the periodic plan optimization problem to polynomial time, where most of our other
optimization techniques exhibit a linear or quadratic time complexity.
3.3.3 Workload Adaptation Sensibility
The core optimization algorithm can be influenced by a number of parameters. We can
leverage these parameters in order to adjust the sensibility of adaptation to changing
workload characteristics. For our core estimation approach, workload statistics of the
current plan P are monitored. If an alternative plan P ′ has been created, we estimate the
missing operator statistics with Ŵ (o′ i ) = C(o′ i )/C(o i) · W (o i ) using our cost model, where
workload statistics of P are aggregated over the sliding time window. In this context, the
following three parameters influence the sensibility of workload adaptation:
Workload Sliding Time Window Size ∆w (time interval used for statistic aggregation):
Monitored statistics of plan instances p i with p i ∈ [T k − ∆w, T k ] are included in the
55