B2B Integration : A Practical Guide to Collaborative E-commerce

56 B2B Integration — A Practical Guide to Collaborative E-commerce
redundant and poor data. It uses a staging system or a transient data hub
to extract, transform, cleanse and integrate data.
The amount of data to be integrated internally and externally in B2B
applications can be huge. Thus, the ETL engine should be capable of
providing high scalability and high data throughput for the movement
of data from the origin to the destination by supporting functionalities
such as in-memory data transformations, in-memory caching, query
optimization and multithreading.
The ETL engine can function in the following ways:
• If the data sources to be integrated are completely heterogeneous,
making local transformation impossible, the ETL engine extracts data
from the origin data source, transforms and routes and loads data to
the target data source (see Figure 3.7). In this environment,
sophisticated ETL functionalities of pulling data from multiple
operating environments, applying transformation rules and then loading
data into target applications are required.
• The load on the engine and the unnecessary use of network bandwidth
would be a lot less if aggregations, consolidations and other
transformations of data occur right at the data source level. In this
approach, the data is partially transformed right at the data source,
extracted and further transformed by the engine before routing it to
the target data source where further transformation may take place
(see Figure 3.8).
• In this approach, the complete transformation occurs at the source
and/or target data source (see Figure 3.9). There is no extraction and
Extract Load
Data Transform Data
Data
Source Target
Database Database
ETL
Engine
Figure 3.7. — Data transformation performed by the ETL engine