Cost-Based Optimization of Integration Flows - Datenbanken ...

5.2 Horizontal Queue Partitioning
as the monitored average value selectivities sel(ba 1 ) = 1/3 and sel(ba 2 ) = 1/10. This
results in two possible partitioning schemes with maximum partition numbers of |ba 1 , ba 2 | =
3 + 3 · 10 = 33 and |ba 2 , ba 1 | = 10 + 10 · 3 = 40. Thus, we select (ba 1 , ba 2 ) as the best
partitioning scheme because sel(ba 1 ) ≥ sel(ba 2 ). For a given message subsequence, this
results in three instead of ten plan instances.
Having minimized the total number of partitions, we minimized the overhead of queue
maintenance and more importantly maximized the number of messages per top-level partition,
which reduces the number of required plan instances. The result is the optimal
partitioning scheme with regard to relative execution time and thus message throughput.
5.2.3 Plan Rewriting Algorithm
With regard to executing hierarchical message partitions, only slight changes of physical
operator implementations are necessary. All other changes are made on logical level when
rewriting a plan P to P ′ during the initial deployment or during periodical re-optimization.
For the purpose of plan rewriting, several integration flow meta model extensions are required.
First, the message meta model is extended in such a way that an abstract message
can be either an atomic message or a message partition, where the latter is described by a
partitioning attribute ba i as well as the type and the values of this partitioning attribute.
In addition, the message partition can be a node partition, which references child partitions,
or a leaf partition, which references atomic messages. All operators that benefit from
partitioning (e.g., Invoke, Assign, or Switch) are modified accordingly, while all other
operators transparently split the incoming message partition into all atomic messages, are
executed for each message, and then they repartition the messages after execution. Second,
the flow meta model is extended by two additional operators that are described in
Table 5.1. They represent inverse functionalities as shown in Figure 5.8.
Based on these two additional operators, the logical plan rewriting is realized with the
so-called split and merge approach. From a macroscopic view, a plan receives the toplevel
partition, dequeued from the partition tree. Then, we can execute all operators that
benefit from the top-level partitioning attribute. Just before an operator that benefits
from a lower-level partition attribute, we need to insert a PSplit operator that splits the
top-level partition into the 1/sel(ba 2 ) subpartitions (worst case) as well as an Iteration
operator (foreach) that iterates over these subpartitions. The sequence of operators that
benefit from this granularity are used as iteration body. After this iteration, we insert a
PMerge operator in order to merge the resulting partitions back to the top-level partition if
required (e.g., if a subsequent operator benefit from higher level partitioning attributes).
Table 5.1: Additional Operators for Partitioned Plan Execution
Name Description Input Output Complex
PSplit
PMerge
Reads a message partition, splits this partition
into the next level partitions, and returns
a directly accessible array of abstract
messages.
The inverse operation to a PSplit operator
reads an array of message partitions and
groups this messages into a single partition.
(1,1) (1,*) No
(1,*) (1,1) No
139