Cost-Based Optimization of Integration Flows - Datenbanken ...

5.2 Horizontal Queue Partitioning
5.2 Horizontal Queue Partitioning
In order to enable plan execution of message partitions, several preconditions are required.
First, we describe the tailor-made message queue data structure partition tree. It enables
horizontal partitioning according to multiple attributes because a single plan might include
several predicates, where we can benefit from partitioning. Second, we explain changes of
the deployment process, which include (1) the derivation of partitioning attributes from
a given plan, (2) the derivation of the optimal partitioning scheme in case of multiple
attributes, and (3) the rewriting of plans according to the chosen partitioning scheme.
5.2.1 Maintaining Partition Trees
As the foundation for multi-flow optimization, we introduce the partition tree as a partitioned
message queue data structure. Essentially, this tree is a simplified multi-dimensional
B * -Tree (MDB-Tree) [SO82] with specific extensions, where the messages are horizontally
(value-based) partitioned. Similar to a traditional MDB-Tree, each tree level represents a
different partitioning attribute. The major difference to a traditional MDB-Tree is that
the partitions are sorted according to their timestamps of creation rather than according
to the key values of index attributes. This is reasoned by queuing semantics that imply a
temporal order. Thus, at each tree level, a list of partitions, unsorted with regard to the
attribute values, is stored. Formally, the partition tree is defined as follows:
Definition 5.1 (Partition Tree). The partition tree is an index for multi-dimensional
attributes. It contains h levels, where each level represents a partition attribute ba i ∈
{ba 1 , ba 2 , . . . , ba h }. For each attribute ba i , a list of batches (partitions) b are maintained.
Those partitions are ordered according to their timestamps of creation t c (b i ) with t c (b i−1 ) ≤
t c (b i ) ≤ t c (b i+1 ). Only the last index level ba h contains the queued messages. A partition
attribute has a type with type(ba i ) ∈ {value, value-list, range}.
Such a partition tree is used as our message queue representation. Similar to usual
message queues, it decouples the inbound adapters from the process engine in order to
enable receiving incoming messages even in the case of overload situations. For example,
such situations might be caused by (1) workload peaks, or (2) temporarily unavailable
target systems. Additionally, the partition tree realizes the horizontal queue partitioning.
We use an example in order to explain the structure of this partition tree more clearly.
Example 5.4 (Partition Tree). Assume two partitioning attributes ba 1 (customer, value)
and ba 2 (total price, range) that have been derived from a plan P . Figure 5.6 shows this
Inbound
Adapters
enqueue()
Partitioned Queue
dequeue()
Process
Engine
ba1
(Customer)
ba2
(Totalprice )
partition b5 [“CustB“] partition b2 [“CustC“] partition b1 [“CustA“]
tc(b5)
partition b5.2
[“
partition b5.1
[“[10,200)“]
msg5 [„CustB“]
msg9 [„CustB“]
>
tc(b2)
partition b1.3
[“[10,200)“]
msg4 [„CustA“]
msg8 [„CustA“]
msg10 [„CustA“]
tc(b1)
partition b1.2
[“>200“]
msg3 [„CustA“]
msg7 [„CustA“]
partition b1.1
[“
>
>
node message partition
leaf message partition
atomic message
135