Cost-Based Optimization of Integration Flows - Datenbanken ...

3 Fundamentals of Optimizing Integration Flows
Experimental Setting
The experimental setup comprises an IBM blade with two processors (each a Dual Core
AMD Opteron Processor 270 at 2 GHz) and 9 GB RAM, where we used Linux openSUSE
9.1 (32 bit) as the operating system. Our WFPE (workflow process engine) realizes the
extended reference system architecture (described in this chapter) including our optimization
component. The WFPE is implemented using Java 1.6 as the programming language
and includes approximately 36,000 lines of code. It currently includes several adapters (inbound
and outbound) for the interaction with files, databases, and Web services. However,
the external systems, used by our example integration flows, have been simulated with file
adapters in order to minimize the influence of used systems on the measured experimental
results (reproducibility). In order to use arbitrary workload scenarios with different cardinalities
and selectivities (workability), we executed all experiments on synthetic XML
data generated using our DIPBench toolsuite [BHLW08c].
There are several parameters that influence plan execution. Essentially, we analyzed
two groups of parameters. First, there is the group of optimization parameters, where we
used different sliding window sizes ∆w (default: 5 min), different optimization intervals ∆t
(default: 5 min), and different workload aggregation methods Agg (default: EMA). Second,
there is the group of workload characteristics. Here, we used different numbers of plan
instances n (executed instances), different plans with certain numbers of operators m, and
different input data sizes d (default: d = 1, which stands for 100 kB input messages) and
different selectivities. With regard to applied optimization techniques, we used all costbased
optimization techniques, except message indexing and heterogeneous load balancing
(both not discussed in this thesis) as well as vectorization (see Chapter 4) and multiflow
optimization (see Chapter 5). Furthermore, we disabled all rule-based optimization
techniques in order to focus on the benefit achieved by cost-based optimization because
these techniques either did not apply (e.g., algebraic simplifications) for the used plans or
they achieved a constant absolute improvement (e.g., static node compilation) for both
the unoptimized and the cost-based optimized execution.
End-to-End Comparison and Optimization Benefits
In a first series of experiments, we compared the end-to-end performance of no-optimization
versus the periodical re-optimization. These experiments already include all optimization
overheads (such as statistics maintenance and periodical re-optimization). As a
result, these experiments show the overall benefit achieved by periodical re-optimization.
First, we compared the periodical re-optimization with no-optimization. The periodical
re-optimization was realized as asynchronous inter-instance optimization approach. We
executed 100,000 instances of our example plan P 5 for the non-optimized plan as well as
for the optimized plan and measured re-optimization and plan execution time. For periodical
re-optimization, the plan execution time already includes the synchronous statistic
maintenance. During execution, we varied the input cardinality (see Figure 3.20(b))
and selectivities of the three selection operators (see Figure 3.20(a)). Here, the input
data was generated without correlations between different attributes. With regard to reoptimization,
there are four points (∗1, ∗2, ∗3, and ∗4) where a workload change (shown
as intersection points between selectivities) reasons the change of the optimal plan.
For periodical re-optimization, we used an optimization interval of ∆t = 5 min, a sliding
window size of ∆w = 5 min and EMA as the workload aggregation method. Figure 3.20(c)
74