Cost-Based Optimization of Integration Flows - Datenbanken ...

5 Multi-Flow Optimization
mention its complexity. The A-MPR exhibits—similar to the plan vectorization algorithm
(A-PV, Subsection 4.2.2)—a cubic worst-case time complexity of O(m 3 ) according to the
number of operators m. The rationale for this is the already analyzed dependency checking.
Note that the additional inner loop over following operators do not change this asymptotic
behavior because each operator is assigned only once to an inserted Iteration operator.
The split and merge approach realizes the transparent plan rewriting and thus enables
the execution of message partitions even in the case of multiple partitioning attributes.
The rewritten plan mainly depends on the cost-based derived partitioning scheme, which
neglects any additional costs of PSlit and PMerge operators. The reason for this optimization
objective is that the ordering of partitioning attributes has a higher influence
on the overall performance (partitioned queue maintenance and benefit by partitioning)
than the additional operators, because PSplit and PMerge are low-cost operators with
linear scalability with regard to the number of messages due to the efficient hash partition
tree data structure. In addition to the throughput improvement achieved by executing
operations on partitions of messages, the inserted Iteration operators offer further optimization
potential. In detail, the technique WC3: Rewriting Iterations to Parallel Flows,
described in Subsection 3.4.1, can be applied after the A-MPR in order to additionally
achieve a higher degree of parallelism that further increases the throughput.
To summarize, we discussed the necessary preconditions in order to enable the horizontal
message queue partitioning and the execution of operations on these message partitions. In
detail, we introduced the (hash) partition tree as a message queue data structure that allows
the hierarchical partitioning of messages according to certain partitioning attributes.
Furthermore, we introduced basic algorithms (1) for deriving candidate partitioning attributes
from a plan, (2) for deriving the optimal partitioning scheme of attributes, and (3)
for rewriting the plan according to this scheme. Only minor changes of operators and the
execution environment are necessary, while all other aspects are issues of logical optimization
and therefore, fit seamlessly into our cost-based optimization framework. Multi-flow
optimization now reduces to the challenge of computing the optimal waiting time.
5.3 Periodical Re-Optimization
The cost-based decision of the multi-flow optimization technique is to compute the optimal
waiting time ∆tw in order to adjust the trade-off between message throughput and latency
times of single messages according to the current workload characteristics. In this section,
we define the formal optimization objective, we explain the extended cost model and cost
estimation, we discuss the waiting time computation and finally, show how to integrate
this optimization technique into our cost-based optimization framework.
5.3.1 Formal Problem Definition
As described in Section 2.3.1, we assume a message sequence M = {m 1 , m 2 , . . . , m n } of
incoming messages, where each message m i is modeled as a (t i , d i , a i )-tuple, where t i ∈ Z +
denotes the incoming timestamp of the message, d i denotes a semi-structured tree of namevalue
data elements, and a i denotes a list of additional atomic name-value attributes. Each
message m i is processed by an instance p i of a plan P , and t out (m i ) ∈ Z + denotes the
timestamp when the message has been successfully executed. Here, the latency of a single
message T L (m i ) is given by T L (m i ) = t out (m i ) − t i (m i ).
142