Cost-Based Optimization of Integration Flows - Datenbanken ...

5 Multi-Flow Optimization
(2) plan partitioning in the sense of compiling different plans for different partitioning attribute
values 17 , and (3) MFO for multiple plans by an extended waiting time computation
for scheduling overlapping plan executions with regard to the hardware environment.
The main differences of the MFO approach to prior work are twofold: First, the concept
of horizontal message queue partitioning simplifies the naïve (time-based) approach
because it can be applied also for local operators and no rewriting of queries and local
post-processing of query results is required. In addition, horizontal partitioned execution
leads to predictable and higher throughput. Second, the approach of computing the optimal
waiting time ensures the adaptation to current workload characteristics by adjusting
the waiting time according to throughput and latency time. In consequence, the MFO
approach can be applied in other domains as well. For example, it might be used for (1)
scan sharing, where queries are indexed according to predicate values [UGA + 09], or (2)
for transient views over equivalent query predicates [ZLFL07].
Beside the achievable throughput improvements, MFO has also some limitations that
one should be aware of. First, while caching might lead to using outdated data, the
execution of message partitions might cause us to use data that is more current than it
was when the message arrived. Despite our guarantee of ensuring eventual consistency
as sketched in Subsection 5.1, both caching and MFO, cannot ensure monotonic reads
over multiple data objects, which might be a problem if there are hidden dependencies
between data objects within the external system. Second, if the number of distinct values
is too high, we will not benefit from partitioning due to the additional runtime overhead
(partitioned enqueue of messages, serialization at the outbound side) and a fairly low
maximum waiting time due to the worst-case latency time consideration according to the
number of partitions. However, with regard to the experimental evaluation, there are three
facts why we typically benefit from MFO. First, even for one-message partitions, there is
only a moderate runtime overhead. Second, throughput optimization is required if and
only if high message load (peaks) exists. In such cases, it is very likely that messages
with equal attribute values are in the queue. Third, only a small number of messages is
required within one partition to yield a significant speedup for different types of operators.
Finally, we consolidate the results from the Chapters 3-5. The general cost-based optimization
framework for integration flows, defined in Chapter 3, minimizes the average plan
execution time by employing control-flow- and data-flow-oriented optimization techniques
but it neglected the alternative optimization objective of throughput improvement. This
drawback has been addressed with the integration-flow-specific optimization techniques
vectorization (Chapter 4) and multi-flow optimization (Chapter 5). However, the periodical
re-optimization algorithm has still several drawbacks. Most importantly, there are the
problems of (1) many unnecessary re-optimization steps, where we do not find a new plan
if workload characteristics have not changed, and (2) adaptation delays after a workload
change, where we use a suboptimal plan until re-optimization and miss optimization opportunities.
To tackle these additional problems, in the following Chapter 6, we introduce
the concept of on-demand re-optimization.
17 For example, correlated data inherently leads to data partitions, where each partition has specific statistical
characteristics and thus a different optimal plan [Pol05, BBDW05, TDJ10]. MFO in combination
with different plans for different data can address this within our cost-based optimization framework.
In addition, we can apply plan simplifications (e.g., remove Switch operators).
166