Cost-Based Optimization of Integration Flows - Datenbanken ...

5 Multi-Flow Optimization
cannot be processed as a stream such that these influences are additive components rather
than being subsumed by the most time-consuming influence.
Problem 5.2 (Cache Coherency Problem). One solution to Problem 5.1 might be the
caching of results of external queries. However, this fails due to the integration of heterogeneous
and highly distributed systems and applications (loosely coupled without any
notification mechanisms). In such distributed environments, caching is not applicable because
the central integration platform cannot ensure that the cached data is consistent with
the data in the source systems [LZL07]. A similar problem is also known for caching proxy
servers, which might break client cache directives (RFC 3143) [CD01].
Due to this problem, other projects, such as the MT-Cache, use currency bounds (maximum
time of caching certain data objects) [GLRG04]. However, they can only ensure weak
consistency, while for integration flows eventual consistency is needed as described in Subsection
2.3.2. Caching (without semantic cache invalidation) cannot ensure the properties
of (1) read-your-writes (consistency between a writing Invoke and a subsequent reading
Invoke of the same data object) and (2) session consistency (consistency between multiple
reading Invoke of the same data object) of eventual consistency [Vog08].
Problem 5.3 (Serialized External Behavior). Depending on the involved external systems,
we need to ensure the serial order of messages (see Problem 2.2 Message Outrun). For
example, this can be caused by referential integrity constraints within the target systems.
Thus, we need to guarantee monotonic reads and writes for individual data objects.
Given these problems, the optimization objective of throughput maximization has so far
only been addressed by leveraging a higher degree of parallelism, such as (1) intra-operator,
horizontal parallelism (data partitioning, see [BABO + 09]), (2) inter-operator, horizontal
parallelism (explicit parallel subflows, see Chapter 3, [LZ05, SMWM06]), and (3) interoperator,
vertical parallelism (pipelining of messages and message parts, see Chapter 4,
[PVHL09a, PVHL09b]). Although these techniques can significantly increase the resource
utilization and thus, increase the throughput, they do not reduce the executed work. We
use an example to illustrate the problem of expensive external system access and how
multi-flow optimization addresses this problem.
Example 5.1 (Instance-Based Orders Processing). Assume our example plan P 2 (Figure
5.1(a)). The instance-based execution model initiates a new plan instance p i for each
Receive (o1)
[service: s5, out: msg1]
Assign (o2)
[in: msg1, out: msg2]
m1 [“CustA“]
Q1: SELECT *
dequeue
SELECT *
m1 p1: o1 o2 o3 o4 o5 o6
FROM s4.Credit
m2 [“CustB“]
WHERE Customer=
dequeue
Qi
m2
m3 [“CustA“]
p2: o1 o2 o3 o4 o5 o6
with
= mi/Customer/Cname
dequeue
m4 [“CustC“]
m3 Q3: SELECT *
p3: o1 o2 o3 o4 o5 o6
m5 [“CustB“]
m6 [“CustC“]
enqueue
(a) Example Plan P 2
Invoke (o3)
[service: s4, in: msg2, out: msg3]
INNER
Join (o4)
[in: msg1,msg3, out: msg4]
Assign (o5)
[in: msg4, out: msg5]
Invoke (o6)
[service s3, in: msg5]
Qi:
Standard Message Queue
(b) Instance-Based Plan Execution of P 2
FROM s4.Credit
WHERE Customer=“CustA“
Q2: SELECT *
FROM s4.Credit
WHERE Customer=“CustB“
FROM s4.Credit
WHERE Customer=“CustA“
Figure 5.1: Example Instance-Based Plan Execution
130