Applying OLAP Pre-Aggregation Techniques to ... - Jacobs University

10 1. Introduction and Problem Statement
the contents of the BLOB when they wish to operate on the data. The main drawback
to this approach is that it either requires the entire array to be passed to the client, or it
requires that the client perform a large number of BLOB input/output (I/O) operations
to read only the required portions of the array. With databases growing beyond a few
tens of terabytes, the analysis of large volumes of array datasets is severely limited
by the relatively low I/O performance of most of todays computing platforms. Highperformance
numerical simulations are also increasingly feeling the I/O bottleneck.
To improve data management and analytics on large repositories of data, aggregation
has been put forward as a key process when describing high-level data. An
example of data aggregation is the computation and storage of statistical parameters,
such as count, average, median, and standard deviation. Aggregate computation has
been studied in a variety of settings [4, 21, 66]. In particular, On-Line Analytical Processing
(OLAP) technology has emerged to address the problem of efficiently computing
complex multidimensional aggregate queries on large data warehouses. Most
OLAP systems rely on the process of selecting aggregate combinations, and then precomputing
and storing their results so the database system can make use of them in
subsequent requests. Such a process is known as pre-aggregation, which has proved to
speed up aggregate queries by several orders of magnitude in business and statistical
applications [31, 41].
While considerable work has been done on the problem of efficiently computing
aggregate queries in OLAP-based applications, such computations continue to be a
data management challenge in scientific applications. A relevant example in which the
use of advanced data management and efficient query processing are highly desirable
is hyper-spectral remote-sensing imaging, in which an image spectrometer collects
hundreds or even thousands of measurements for the same area of the surface of the
Earth. The scenes provided by such sensors are often called data cubes to denote
the dimensionality of the data. Notably, efficient query processing and data mining
techniques facilitate exploration of spatio-temporal data patterns, both interactively as
well as in batch on archived data.
A significant fraction of scientific data is image-based and can be naturally represented
in multidimensional arrays. These datasets fit poorly into relational databases,
which lack efficient support for the concepts of physical proximity and order. They
are typically stored in array-friendly formats such as HDF5, netCDF, or FITS. The
extremely high computational requirements introduced by image-based scientific applications
make them an excellent case study for our research.
Since array databases and OLAP/data warehousing both deal with large multidimensional
datasets and aggregate queries, adapting OLAP pre-aggregation techniques
to the management and computation of aggregate queries in array databases may provide
a strong potential benefit. This thesis investigates the application of OLAP preaggregation
techniques in speeding up query processing in array databases. In particular,
we focus on enhancing aggregate computation in GIS and remote-sensing imaging
applications. However, the results can be generalized to other domains as well.