Cost-Based Optimization of Integration Flows - Datenbanken ...

2 Preliminaries and Existing Techniques
2.1.4 Executing Integration Flows
When deploying an integration flow, we transform the logical flow into an executable plan.
Therefore, we distinguish two major plan representations. First, there are interpreted
plans, where we use an object graph of operators and interpret this object graph during
execution of a plan instance. Second, there are compiled plans, where code templates are
used in order to generate and compile physical executable plans. As a first side-effect
from the modeling perspective, (1) directed graphs are typically interpreted, while (2)
hierarchies of sequences, source code and fixed flows are commonly executed as compiled
plans.
Moreover, as a second side effect, flows with data-flow modeling semantics are typically
also executed with data-flow semantics. The same is true for control-flow semantics.
Hence, we use the flow semantics, as our major classification criterion of execution approaches.
With regard to the data granularity as the second classification criterion of plan
execution, we traditionally distinguish between two fundamental execution models:
• Iterator Model: The Volcano iterator model [Gra90, Gra94] is the typical execution
model of traditional DBMS (row stores). Each operator implements an interface
with the operations open(), next() and close(). The operators of a plan call their
predecessors, i.e., the top operator determines the execution of the whole plan (pull
principle). In addition, each operator can be executed by an individual thread (and
thus, adheres to the pipes-and-filter execution model), where each operator exhibits
a so-called iterator state (tuple buffer) [Gra90]. The advantages of this model are
extensibility with additional operators as well as the exploitation of vertical parallelism
(pipeline parallelism or data parallelism) and horizontal parallelism (parallel
pipelines). The disadvantages are the high communication overhead between operators
and the predominant applicability for row-based (tuple-based) execution.
• Materialized Intermediates: The concept of materialized intermediates is the typical
execution model of column stores [KM05, MBK00]. Operators of a plan are executed
in sequence (one operator at a time), where the result of one operator is completely
materialized (as variable) and then used as input of the next operator (push principle).
This reduces the overhead of operator communication and is particularly
advantageous for column stores, where operators work on (compressed) columns in
the form of continuous memory (arrays). This concept offers additional optimization
opportunities such as vectorized execution within a single operator, or the recycling
of intermediate results [IKNG09] across multiple plans.
Integration flows are typically executed as independent plan instances. Here, we distinguish
between data-driven integration flows, where incoming data conceptually initiates a
new plan instance, and scheduled integration flows, where such an instance is initiated by
a time-based scheduler. If strong consistency is required, data-driven integration flows are
executed synchronously, which means that the client systems are blocked during execution.
In contrast, if only weak (eventual) consistency is required, data-driven integration
flows can also be executed asynchronously using inbound queues. Note that scheduled
integration flows are per se asynchronous and thus, ensure only weak consistency. We use
this integration-flow-specific characteristic of independent instances to refine the classification
criterion of data granularity. Therefore, we introduce the notion of instance-local
(data/messages of one flow instance) and instance-global (data/messages of multiple flow
instances) data granularity.
14