Cost-Based Optimization of Integration Flows - Datenbanken ...

2 Preliminaries and Existing Techniques
2.4.2 Vertical Integration: Real-Time ETL
In contrast to horizontal integration, the use case of vertical integration addresses the
consolidation of data from the operational source systems into dispositive and strategical
systems. In this context, typically, data-centric ETL (Extraction Transformation Loading)
flows are used. However, there is a trend towards operational BI where changes in the
source systems are directly propagated to the data warehouse infrastructure in order to
achieve high up-to-dateness of analytical query results [DCSW09, O’C08, WK10]. This
is typically realized with (1) near real-time ETL flows, where data is loaded periodically
but with high frequency, or with (2) real-time ETL flows, where data is loaded based
on business transactions. As a result of both strategies, many instances of integration
flows with rather small amounts of data are executed over time. Although this is similar
to horizontal integration, we use selected ETL integration flows in order to demonstrate
their specific characteristics as well.
Example 2.8 (Real-Time Standard Orders Loading). If a new order of standard products
(not user-defined) is created using the ERP system, the data is directly propagated to
the data warehouse infrastructure. Therefore, the integration flow (plan P 5 ) shown in
Figure 2.11(a) is used. A plan instance is asynchronously initiated by receiving a data
set from the ERP system, and it executes three different Selection operators (according
to different attributes) in order to filter orders that are maintained within dedicated data
marts. Subsequently, a Switch operator routes the incoming tuples—using content-based
predicates—to specific schema mapping Translation operators (specific to the referenced
material). Finally, the result is loaded into the data warehouse s 6 .
Example 2.9 (Near-Real-Time Customer Loading). Non-disjoint (overlapping) customer
master data from the eCommerce Web shop, the CRM system as well as the ERP system
is loaded into the data warehouse in a near real-time fashion. Plan instances of the
integration flow (plan P 6 )—that is illustrated in Figure 2.11(b)—are periodically initiated
and executed. Essentially, this plan creates three parallel subflows, where the customer
master data is loaded from the ERP system s 3 , from the CRM system s 4 , and from the
eCommerce Web shop s 5 . After the subflows have been temporally joined, two subsequent
Setoperation operators (type UNION DISTINCT) are executed in order to eliminate duplicates.
Finally, the resulting customer data is loaded into the data warehouse s 6 .
Example 2.10 (Real-Time Customized Orders Loading). Incoming orders of customized
products that are registered within the central ERP system are also directly propagated
to the data warehouse architecture using the integration flow (plan P 7 ) that is shown in
Figure 2.11(c). Initiated by those incoming order messages, four parallel subflows are
executed, where we load the related supplier data from the SCM system s 1 , product information
from the Material system s 2 , customer data from the CRM system s 4 , and
transaction information from the eCommerce Web shop s 5 . Subsequently, a left-deepjoin-tree—consisting
of four Join operators (chain query type)—merges the received order
message with the loaded data. Finally, the result is sent to the data warehouse s 6 .
Example 2.11 (DSS Data Provision). The consolidated data of the data warehouse is
partially provided for strategical planning within the DSS. In order to synchronize the data
warehouse with the DSS, the integration flow (plan P 8 ) is used as shown in Figure 2.11(d).
Essentially, instances of this plan are initiated periodically. First, a procedure is called by
the Invoke operator o 2 that executes several data cleaning and aggregation operations on
30