Cost-Based Optimization of Integration Flows - Datenbanken ...

5.1 Motivation and Problem Description
original queries (Q100 + Q200) and the rewritten query with non-overlapping disjunctive
predicate (Q100+200). In addition, for both types of disjunctive queries the integration
platform must post-process the received data in order to ensure correctness by assigning
only partial results to the individual messages. Second, the rewriting of queries might not
be possible at all for certain service interfaces or custom applications such that a single
query for each distinct message in the batch must be used. For example, Figure 5.3(c)
shows a service interface with a single parameter, where the service implementation always
uses the same query template such that the query cannot be rewritten by an external
client. Thus, the possible throughput improvement strongly depends on the number of distinct
items in the batch. We cannot precisely estimate this influence of rewritten queries
due to missing knowledge about data properties of involved external systems [IHW04].
In conclusion, the naïve approach can hurt performance. Furthermore, it requires that
queries to external systems can be rewritten according to the different items in the batch.
This is not always possible when integrating arbitrary systems and applications.
Batch Creation via Horizontal Queue Partitioning
Due to Problem 5.4, we propose the concept of horizontal message queue partitioning 11 as
batch creation strategy. The basic idea is to horizontally partition the inbound message
queues according to specific partitioning attributes ba. With such value-based partitioning,
all messages of a batch exhibit the same attribute value according to the partitioning
attribute. Thus, certain operators of the plan only need to access this attribute once for the
whole partition rather than for each individual message. The core steps are (1) to derive
the partitioning attributes from the integration flow, (2) to periodically collect messages
during an automatically computed waiting time ∆tw, (3) to read the first partition from
the queue and (4) to execute the messages of this partition as a batch with an instance
p ′ i of a rewritten plan P ′ . Additionally, (5) we might need to ensure the serial order of
messages at the outbound side. In order to illustrate the core idea, we revisit our example.
Example 5.3 (Partitioned Batch-Orders Processing). Figure 5.5 reconsiders the example
for partitioned multi-flow execution. The rewritten plan P 2 ′ is equivalent to the instancebased
plan (Figure 5.1(a)) because (1) external queries require no rewriting at all, and (2)
plan rewriting is only required for multiple partitioning attributes.
Partitioned Message Queue
CustA
CustB
m1 [“CustA“]
m3 [“CustA“]
m2 [“CustB“]
m5 [“CustB“]
Wait
∆tw
dequeue
b1
p’1:
o1
o2 o3 o4 o5 o6
Q’1: SELECT *
FROM s4.Credit
WHERE Customer=“CustA“
CustC
m4 [“CustC“]
m6 [“CustC“]
dequeue
b2
p’2:
o1
o2 o3 o4 o5 o6
Q’2: SELECT *
FROM s4.Credit
WHERE Customer=“CustB“
enqueue
Figure 5.5: Partitioned Message Batch Execution P ′ 2
11 Horizontal data partitioning [CNP82] is strongly applied in DBMS and distributed systems. Typically,
this is an issue of physical design [ANY04], where a table is partitioned by selection predicates (value).
133