Cost-Based Optimization of Integration Flows - Datenbanken ...

5 Multi-Flow Optimization
The incoming messages m i are partitioned according to the partitioning attribute customer
name that was extracted with ba = m i /Customer/Cname at the inbound side. A plan
instance of the rewritten plan P 2 ′ reads the first partition from the queue and executes the
single operators for this partition. Due to the equal values of the partitioning attribute, we
do not need to rewrite the query to the external system s 4 . Every batch contains exactly
one distinct attribute value according to ba. In total, we achieve performance benefits for
the Assign, as well as the Invoke operators. Thus, the throughput is further improved
because the execution time of such a batch does not include any distinct messages anymore.
Note that the incoming order of messages was changed (arrows in Figure 5.5) and therefore
needs to be serialized at the outbound side.
It is important to note that beside external queries (Invoke operator), also local operators
(e.g., Assign and Switch) can directly benefit from horizontal partitioning. Partitioning
attributes are derived from the plan (e.g., query predicates, or switch expressions).
This benefit is caused by operation execution on partitions instead of on individual messages.
A similar underlying concept is also used for pre-aggregation [IHW04] or earlygroup-by
[CS94]. In addition, all operators that work on partitions of equal messages
(e.g., loaded once from an external system) also need to be executed only once.
Our cost-based optimizer realizes the optimization objective of throughput maximization
by monitoring several statistics of the stream of incoming messages and by periodical
re-optimization, where the optimal waiting time ∆tw is computed. In case of low message
rates (no full utilization of the integration platform), the waiting time is decreased in order
to ensure low latency of single messages. As the message rate increases, the waiting time
is increased accordingly to increase the message throughput by processing more messages
per batch, while preserving maximum latency constraints for single messages.
MQO (Multi-Query Optimization) and OOP (Out-of-Order Processing) [LTS + 08] have
already been investigated for other system types. In contrast to existing work, we present
the novel MFO approach that is tailor-made for integration flows and that maximizes
the throughput by employing horizontal message queue partitioning and computing the
optimal waiting time. MFO is also related to caching and the recycling of intermediate
results [IKNG09]. While caching might lead to use of outdated data, the partitioned execution
might cause reading more recent data of different objects. However, we cannot
ensure strong consistency by using asynchronous integration flows (decoupled from clients
with message queues) anyway. Furthermore, with regard to eventual consistency [Vog08],
we guarantee (1) monotonic writes, (2) monotonic reads with regard to individual data
objects 12 , (3) read-your-writes/session consistency, (4) semantic correctness as defined in
Definition 3.1, (5) that the temporal gap of up-to-dateness is at most equal to a given
latency constraint, and (6) that no outdated data is read. In contrast, caching cannot
guarantee read-your-writes/session consistency and that no outdated data is read. In conclusion,
caching is advantageous if data of external sources is static and the amount of data
is rather small, while MFO is beneficial if data of external sources changes dynamically.
Finally, the major research challenges of MFO via horizontal partitioning are (1) to
enable plan execution of horizontally partitioned message batches and (2) to compute
the optimal waiting time ∆tw during periodical re-optimization. We address these two
challenges in Section 5.2 and Section 5.3, respectively.
12 Both caching and MFO cannot ensure monotonic reads over multiple data objects due to different read
times of certain data objects.
134