Cost-Based Optimization of Integration Flows - Datenbanken ...

5.1 Motivation and Problem Description
incoming message as shown in Figure 5.1(b). The Receive operator (o 1 ) reads an order
message from the queue and writes it to a local variable. Then, the Assign operator (o 2 )
extracts the customer name of the received message via XPath and prepares a query with
this parameter. Subsequently, the Invoke operator (o 3 ) queries the external system s 4 in
order to load credit rating information for that customer. An isolated SQL query Q i per
plan instance (per message) is used. The Join operator (o 4 ) merges the result message
with the received message (with the customer key as join predicate). Finally, the pair of
Assign and Invoke operators (o 5 and o 6 ) sends the result to system s 3 . We see that multiple
orders from one customer (CustA: m 1 , m 3 ) cause us to pose the same query (Invoke
operator o 3 ) multiple times to the external system s 4 .
Due to the serialized execution of plan instances, we may end up with work done multiple
times, for all types of operators (interaction-oriented, data-flow-oriented as well as controlflow-oriented
operators). At this point, multi-flow optimization comes into play, where we
consider optimizing the sequence of plan instances. Our core idea is to periodically collect
incoming messages and to execute whole message batches with single plan instances. In the
following, we give an overview of the naïve, time-based approach as well as the horizontal
(value-based) message queue partitioning as batch creation strategies.
Naïve Time-Based Batch Creation
The underlying theme of the naïve (time-based) batching approach, as already proposed
in variations for distributed queries [LZL07, LX09], scan sharing [QRR + 08], operator
scheduling strategies in DSMS [Sch07], and web service interactions [SMWM06, GYSD08b,
GYSD08a], is to periodically collect messages (that would initiate plan instances p i ) using
a specific waiting time ∆tw and merge those messages to message batches b i . We then
execute a plan instance p ′ i of the modified (rewritten) plan P ′ for the message batch b i .
In the following, we revisit our example and illustrate that naïve (time-based) approach.
Example 5.2 (Batch-Orders Processing). Figure 5.2(b) shows the naïve approach, where
we wait for incoming messages during a period of time ∆tw and execute the collected
messages as a batch. For this purpose, P 2 is rewritten to P 2 ′ (see Figure 5.2(a)), where in
this particular example only the prepared query has been modified.
Receive (o1)
[service: s5, out: msg1]
Assign (o2)
[in: msg1, out: msg2]
Invoke (o3)
[service: s4, in: msg2, out: msg3]
INNER
Join (o4)
[in: msg1,msg3, out: msg4]
Qi (rewritten):
SELECT *
FROM
s4.Credit
WHERE
Customer IN()
Qi
Batch Message Queue
m1 [“CustA“]
m2 [“CustB“]
m3 [“CustA“]
m4 [“CustC“]
Wait
∆tw
dequeue
b1
p’1:
o1
o2 o3 o4 o5 o6
Q’1: SELECT *
FROM s4.Credit
WHERE Customer IN(“CustA“,
“CustB“)
Assign (o5)
[in: msg4, out: msg5]
Invoke (o6)
[service s3, in: msg5]
m5 [“CustB“]
m6 [“CustC“]
enqueue
dequeue
b2
p’2:
o1
o2 o3 o4 o5 o6
Q’2: SELECT *
FROM s4.Credit
WHERE Customer IN(“CustA“,
“CustC“)
(a) Example Plan P ′ 2
(b) Message Batch Plan Execution of P ′ 2
Figure 5.2: Example Message Batch Plan Execution
In this example, the first batch contains two messages (m 1 and m 2 ) and the second also
contains two messages (m 3 and m 4 ). In order to make use of batch processing, we extend
131