Cost-Based Optimization of Integration Flows - Datenbanken ...

3 Fundamentals of Optimizing Integration Flows
within the critical path. When reordering selective operators (e.g., joins), this can have
tremendous impact for following operators (that are included in the critical path). For
example, we might exclude a join operator from the optimization because it is not within
the critical path, which might lead to a suboptimal join order.
In conclusion, there are three cases where this critical-path approach is advantageous.
First, it can be used if the optimization interval is really short and thus, optimization
time is more important than in other cases. Second, it is beneficial if the parallel subflows
are fully independent of the rest of the plan because then, we do not miss any global
optimization potential. Third, its application is promising if there is a significant difference
in the costs of the single subflows; otherwise, the critical path might change.
Heuristic Join Enumeration
Since the complexity of the A-PMO is dominated by the complexity of join enumeration,
we typically use our tailor-made heuristic optimization algorithm, if the number of join
operators of a plan exceed a certain number. In contrast, for the second problem with
high complexity (merging parallel flows), we apply a heuristic by default because the plan
cost influence of join enumeration is much higher than the number of parallel flows. Thus,
the A-HPMO is essentially equivalent to the A-PMO except that we do not apply the full
join enumeration but the heuristic described here.
Similar concepts are also used in DBMS, where existing approaches typically fall back
to some kind of greedy heuristics or randomized algorithms if a certain optimization time
is exceeded [Neu09]. In contrast, we use—similar to selected DBMS such as Postgres—the
number of joins as an indicator when to use the heuristic because otherwise, intermediate
results of join enumeration cannot be exploited when using the DPSize (bottom-up dynamic
programming) join enumeration algorithm and thus, the elapsed optimization time
would be wasted if we have to fall back to the heuristic.
Before discussing the join enumeration heuristic, we need to define the join enumeration
restrictions that must be taken into account when reordering joins in order to preserve
the semantic correctness. Most importantly, Rule 2 from Definition 3.1 applies. Thus, if
there is a dependency between an interaction-oriented operator and another operator, the
temporal order of them must be equivalent in P and P ′ . In addition to this, the input
data of interaction-oriented operators (data that is sent to external systems) must also be
equivalent in P and P ′ (preventing the external behavior from being changed). This has
influence of applicable join re-orderings. We use an example to illustrate the consequences
of these join enumeration restrictions.
Example 3.6 (Join Enumeration Restrictions). Recall the example plan P 7 and a slightly
different plan P 7 ′ (where we changed the initial order of the Invoke operator o 19) that
are illustrated in Figure 3.9. While for plan P 7 (Figure 3.9(a)) a full reordering is applicable
(in case of a clique query type, where all data sets are directly connected), for P 7
′
(Figure 3.9(b)), we are not allowed to reorder all Join operators of this plan. The plan
P 7 ′ is an example, where we might change the external behavior if we consider full join
enumeration. The reason is, that the external system s 6 requires the result of operators
o 14 , o 15 and o 16 . During full join reordering, we might use operator o 17 earlier in this
chain of Join operators and thus, we would be unable to produce the required result in
case of selective joins (or we need at least a combination of Selection and Projection
operators in order to hide additional data). In conclusion, only partial join reordering
52