Cost-Based Optimization of Integration Flows - Datenbanken ...

5.3 Periodical Re-Optimization
benefit from a partitioning attribute (e.g., for writing interactions of an Invoke operator).
However, in some cases (e.g., operations on externally loaded data), all operators
can benefit from partitioning as well because they are inherently executed only once and
just expanded to the batch size if required (e.g., by the first binary operator that receives
a message partition as one of its inputs). For example, as we load data for a partition
of messages, we can execute any subsequent transformation of this loaded data also only
once.
For operators that do not benefit from partitioning, the abstract costs are computed by
C(o ′ i , k′ ) = C(o i ) · k ′ and the execution time can be computed by W (o ′ i , k′ ) = W (o i ) · k ′
or by W (o ′ i , k′ ) = W (o i ) · C(o ′ i , k′ )/C(o i ). Finally, if k ′ = 1, we get the instance-based
costs with C(o ′ i , k′ ) = C(o i ) and W (o ′ i , k′ ) = W (o i ). Thus, the instance-based execution
is a specific case of the execution of horizontally partitioned message batches. As a result,
theoretically, partitioning cannot cause any performance decrease of an operator.
In addition to the mentioned operators that can benefit from partitioning, there are
further operators that might also benefit from partitioning. Examples for these are the
Join, Selection, and Groupby operators. However, due to the partitioning of complete
messages (with tree-structured data) partitioning applies only to specific cases, where a
message has only a single tuple (to which the value of the partitioning attribute refers).
Hence, we do not consider these operators because the possible benefit is strongly limited.
Nevertheless, these operators could be included with benefit if streaming of message parts
(e.g., a part for each tuple) [PVHL09a, PVHL09b] is applied because, we could execute
Join, Selection, and Groupby operators efficiently on whole batches of these parts. We
use our example plan P 2 in order to illustrate the overall cost estimation in detail.
Example 5.7 (Extended Cost Estimation). Recall the rewritten plan P 2 ′ (Figure 5.5) and
assume a number of k ′ messages per message partition. Using the extended cost model, we
can estimate the execution time W (P 2 ′, k′ ). The monitored average execution times W (o i )
are shown in the table in Figure 5.11. Now, we compute W (P 2 ′, k′ ) as follows:
W (P ′ 2, k ′ ) =
m∑
W (o ′ i, k ′ ) = W (o 1 ) + W (o 2 ) + W (o 3 ) + W (o 4 ) · k ′ + W (o 5 ) · k ′ + W (o 6 ) · k ′
i=1
= W (o 1 ) + W (o 2 ) + W (o 3 ) + (W (o 4 ) + W (o 5 ) + W (o 6 )) · k ′
The operators o 1 , o 2 , and o 3 benefit from partitioning and hence, we assign costs that
are independent of k ′ , while costs of operators o 4 , o 5 , and o 6 increase linearly with k ′ .
Using this cost function of P 2 we can estimate the execution time for an arbitrary number
Operator o i Execution Time W (o i )
o 1
o 2
o 3
o 4
o 5
o 6
P
0.01 s
0.015 s
0.3 s
0.055 s
0.02 s
0.13 s
0.53 s
Figure 5.11: Relative Execution Time W (P ′ 2 , k′ )/k ′ 145