SESSION NOVEL ALGORIHMS AND APPLICATIONS + ...

4 Int'l Conf. Foundations of Computer Science | FCS'11 |
need for out-of-core processing for analysis over a longer
period of time. The proposed algorithms, the Periodic Partial
Result Merging and K-way Merge based Technique are
scalable, out-of-core methods, capable of process datasets
several times bigger than the main system memory.
The organization of this paper is as follows: Section 2
presents the properties of methods dealing with large datasets
and the state-of-the-art scientific approaches in this field.
Section 3 introduces two novel out-of-core approaches, their
execution time analysis compared to execution complexities of
other approaches and the execution time determining factors of
algorithms. Section 4 shows the results of algorithms measured
on real datasets, while the last section summarizes our work
and presents possible further work.
2 Related work
Although the amount of available memory has been
significantly increasing during the past decades, handling
large datasets is a still challenging problem in environments
limited by. In this paper a dataset is regarded large, if it does
not fit in the main memory at the same time. During the
evolution of computers this size is continuously varying. In
computer-related literature several approaches were published
how to handle such vast datasets.
Creating a representative subset of the original dataset
based on statistical sampling is a frequently applied method to
solve the problem of limited memory. With simple random
sampling or a more sophisticated sampling technique (e.g.
stratified sampling) a dataset can be generated which has very
similar statistical properties to the original dataset. Thus,
according to our assumption the processed datasets will share
similar statistical properties as the result of the original
dataset. But from a sample-based dataset only a partial result
can be generated; this fact makes this method inappropriate for
problems requiring results based on the whole dataset (e.g.
aggregation).
Another approach to overcome the limited memory issue is
the compression of the dataset. This technique is based on the
idea, that the redundancy of the dataset makes it
unmanageable large. According to information theory, there is
a lower bound of compression, thus in general there is no
guarantee that the compressed dataset will in fact fit in the
main memory. The compression of a dataset can be done using
specific data structures which can have other favorable
properties as well, like in [4][5][6]. Another issue related to
the compression is whether such external data structures can
be designed that have a similar I/O performance as the
uncompressed structures [7]. A compressed, external data
structure, with competitive I/O performance could further
accelerate the data preparation step.
If we have to process large datasets by orders of magnitude
larger than the size of main memory out-of-core methods can
be a well-scalable solution. Out-of-core methods make the
processing even in a memory limited environment possible by
the usage of secondary storages (e.g. hard disks). These
methods follow a partitioning principle, which means that the
original dataset is processed in smaller blocks, resulting partial
processed sets, from which the global result can be obtained.
Due to the high cardinality of dataset the principle is
applicable so that the partial results are stored on a secondary
storage, freeing the main memory for processing another
block.
In out-of-core literature we can find several techniques for
generation of global result from the partial results: the global
result of some problems can be generated as the union of the
partial result sets, as presented in [8][9][10]. For other
problems a merging can be the applicable technique to
determine the global dataset from partial results [11][12]. In
other cases a more complex algorithm has to be performed to
derive the global result [13].
We have approaches which solve the memory limited
issues using secondary storages. In this paper we discuss the
performance analysis of these methods and an essential factor
of the out-of-core methods can be observed even in conceptual
phase. If we take a look at the up-to-date computer
architecture a crucial runtime determining factor can be
pointed out: accessing a secondary storage lasts more than
accessing the same data being actually held in main memory.
This factor influences the efficiency of processing, resulting
that in an out-of-core method the number of I/O instructions
should be kept at a minimum level.
This requirement of minimal I/O instructions is essential
from another point of view as well: the raw datasets are
generated in an automated way, continuously, thus it is needed
to avoid the extrusion of the raw data. This could be done by
assuring that the procedural steps have linear time complexity,
which in general cannot be satisfied. But if we keep the I/O
instructions at the lowest level the performance of the
processing will be still efficient to avoid data extrusion.
Based on the two previous points the core-efficiency of the
out-of-core methods depends on whether they read only
constant times the input dataset or not.
Before applying an out-of-ore method the block size has to
be chosen: the successively, equal-sized partitioning is a
trivial, but working method [11][12][14]. But according to [9]
a sophisticated partitioning approach can have performance
increasing effect. The carefully chosen size is an important
performance determining factor, because with it the consumed
memory can be controlled. An eager, in-memory algorithm
will be presented in this paper to demonstrate the undesirable
behavior of the processing when the main memory reaches its
physical bounds.
3 Out-of-core processing methods
In this paragraph five different processing approaches are
presented, together with their runtime complexity analysis and
the resulting factors. First we discuss an in-memory algorithm,
which have a major shortcoming assuming that processed data
will fit in the main memory at the same time. Two different
extension ways will be presented to overcome the problem of
limited memory: dataset modification (sampling, compression)