Cost-Based Optimization of Integration Flows - Datenbanken ...

5.2 Horizontal Queue Partitioning
denote the average selectivity of a single partitioning attribute. Thus, 1/sel(ba i ) represents
the number of distinct values of this attribute, which is equivalent to the worst-case
number of partitions at index level i. These partitions are unsorted according to the partitioning
attribute. Thus, due to the resulting linear comparison of partitions at each index
level, the enqueue operation exhibits a worst-case time complexity of O( ∑ h
i 1/sel(ba i)).
In contrast to this, the dequeue operation exhibits a constant worst-case time complexity
of O(1) because it simply removes the first top-level partition. In conclusion, there is only
moderate overhead for maintaining partition trees if the selectivity is not too low.
However, in order to ensure robustness with regard to arbitrary selectivities, we extend
the basic structure of a partition tree to the hash partition tree. Figure 5.7 illustrates
an example of its extended queue data structure. Essentially, a hash partition tree is
a partition tree with a hash table as a secondary index structure over the partitioning
attribute values (primarily applicable for attribute types value and range).
h(ba1) last first
0
1
2
3
ba1
(Customer)
partition b5 [“CustB“] partition b2 [“CustC“] partition b1 [“CustA“]
tc(b5)
> tc(b2) > tc(b1)
Figure 5.7: Example Hash Partition Tree
This hash table is used in order to probe (get) if there already exist a value of a partitioning
attribute. If so, we insert the message into the corresponding partition; otherwise,
we create a new partition, append it at the end of the list and put a reference to this
partition into the hash table as well. Accordingly, the dequeue operation still gets the
first partition from the list but additionally removes the pointer to this partition from
the hash table. As a result, we reduced the complexity for both operations, enqueue and
dequeue—for the case, where no serialized external behavior (SEB) is required—to constant
time complexity of O(1). Despite, this probe possibility, for SEB, we additionally
need to determine the number of messages that have been outrun if a related partition
already exist. As a result, for the case, where SEB is required, the worst-case complexity
is still O( ∑ h
i 1/sel(ba i)) but we benefit if the partition does not exist already.
The requirement of serialized external behavior (SEB) implies the synchronization of
messages at the outbound side. Therefore, we extended the message structure by a counter
c with c ∈ Z + to m i = (t i , c i , d i , a i ). If a message m i outruns another message m j during
the enqueue operation, its counter c i is incremented by one. The serialization at the
outbound side is then realized by comparing source message IDs similar to the serialization
concept of Chapter 4, and for each reordered message, the counter is decremented by
one. Thus, at the outbound side, we are not allowed to send message m i until c i = 0.
This counter-based serialization concept 13 is required in addition to the concept of AND
and XOR serialization operators (introduced in Chapter 4) in order to allow out-of-order
execution rather than just parallel execution of concurrent operator pipelines. Despite this
serialization concept, we cannot execute message partitions in parallel because this would
13 Counter-based serialization works also for CN:CM multiplicities between input and output messages,
where locally created messages get the counter of the input messages. For example, N:C multiplicities
arise if there are writing interactions in paths of a Switch operator. This is addressed by periodically
flushing the outbound queues, where all counters of messages that exceed lc are set to zero.
137