The Data Warehouse System of MeteoSwiss: Concept and First ...

The Data Warehouse System of MeteoSwiss: Concept and First
Experiences
Estelle Grüter*, Christian Häberli*, Dimitrios Tombros**, Nadine Tschichold*, Walter Krottendorfer †,
Rudolf Höfler †
MeteoSwiss*,STCG**, PSE†
ABSTRACT
In a world where the amount of measured data increased incredibly over the last years, an efficient
system to store and handle data and to assure its quality is of crucial importance. Large quantities
of numerical and multi medial data are collected for meteorological and climatological purposes
coming in from automatic systems as well as from observers. Data from most diverse sources have
to be integrated in one central database system where it undergoes various quality control and
consistency check procedures. At the same time the data is continuously used by meteorological
and climatological applications. This paper gives an insight into the data warehouse system and its
architecture developed for the special needs of MeteoSwiss (the national weather service of
Switzerland).
1. Introduction
Automatic meteorological measurements with high temporal resolution are carried out in
Switzerland since the late seventies. After more than two decades the need of a new efficient
system to store and treat the continually increasing amount of data using state-of-the-art
technologies became more and more urgent. Since the application of data warehouse technology in
the domain of science can be advantageous for manipulating large quantities of sensor data,
performing statistical analysis and extracting meaningful trends, this approach was chosen to meet
the various and complex requests.
2. Concept of MeteoSwiss Data Warehouse
The data to be stored in this new central system originates in different data sources such as
automatic weather stations, observations made by observers, model output, synop data delivered
by the Global Telecommunication System and others. They have to be loaded according to their
varying structure and afterwards to be checked of their plausibility using a complex system of rules
and tests. In a next step aggregations are carried out to provide customers with data of a lower
temporal resolution such as daily, monthly and yearly values. Finally homogenization of data series
takes place. Applications allow an extraction of data of different quality levels.

Following figure is supposed to give a short insight in the data process chain at MeteoSwiss:
measuring
equipment
datalogger,
onsite
QC
level 1
data
realtime data
collection and
quality control
commu
nication
and
networkmonitoring
Fig. 1: General data flow at MeteoSwiss
level 2
data
quality control,
calculation of
derived
quantities
level 3
data
Since the needs of customers ask for different requests as far as performance and applications are
concerned it was decided to part distinctively the various steps of data processing. This policy was
followed when the abstract architecture of the Data Warehouse was designed.
Data sources
Automatic Stations
Observations
GTS Synop
Sounding
Model output
...
Quality control and Loading
Process
Staging area
Database
optimized for
online
transactional
processing
4 steps of quality control
Data processing
MeteoSchweiz Data Warehouse System
Storage area
quality control &
error removal
(automatically
Fig. 2: Simplified abstract architecture of MeteoSwiss Data Warehouse System
Aggregeation
QC
Database
optimized for
online analytical
processing
Data analysis
Meta Database ( to control all processes )
Applications Clients
meta
data
level 4
data
calculation and aggregation
aggregation
meta data
Distribution
homogenization
Homog.
Applications
Privat customers
Universities
Extraction
level 5
data
Research Institutes
Companies
...

As shown in Figure 2 the conceptual architecture consists of four layers. It is a further development
of the conventional data warehouse seeing that Data Warehouse technology is combined with
classical relational database technology.
1. Layer: Data Sources: This contains a wide variety of sources for meteorological and
climatological data such as point measurements from observations and
automatic stations, profile measurements from upper-air stations, picture
data, volume data from weather radars and weather forecast models.
2. Layer: Staging Area: It comprises all databases and tools to digitize, load, check and integrate
data. Special emphasis is given to data quality control systems. All changes
made on values are logged and thus historised.
3. Layer: Storage Area: This is the classical data Warehouse which contains an integrated set of
data to support meteorologists and climatologists in weather forecasting
and climate research.
4. Layer: Analysis Area: This layer contains various analysis, visualization and data extraction
tools
4. Realisation and first experiences
The main goal of this project is to establish an efficient way of processing data and to come up with
an integrated data set of high quality. Since the complexity of this system is a huge challenge as
well as risk it was decided to have an iterative project proceeding. First the architecture of the whole
system was designed. Due to the fact that MeteoSwiss has not a core competence in realizing data
warehouses, a call of tenders (WTO) was published and a company could be found. In
collaboration with the partner the ‘backbone’ of the system was developed and implemented in a
first step for a few automatic stations to proof if the concept meets the demands. In a second step
the set of stations was enhanced to have different data sources involved. Additionally more features
were developed and integrated. The further planning of the project foresees defined releases in
which legacy systems will be removed as well as new tools made operational. The idea of the
release planning was to keep the complexity of the system under control.
5. Conclusions
In the described MeteoSwiss Data Warehouse System we have developed an architecture which
uses several elements from Data Warehouse as well as other technologies. The distinction in four
layers allows the requested flexibility as far as demands about data quality or performance are
concerned. The iterative proceeding proved to be very worthwile to keep under control the
complexity of the system and is well recommended also for other projects. The collaboration with a
partner company enforced very clear specifications of the demands which resulted in a good
documentation as well as an early determination of upcoming problems.

LITERATURE
Häberli Ch., and D. Tombros, 2001: A Data Warehouse Architecture for MeteoSwiss: An
Experience Report. International Workshop on Design and Management of Data
Warehouses (DMDW’01) in conjunction with the 13 th Conference on Advanced Information
Systems Engineering (CaiSE’01). [available from Swiss Life, IT Research&Development,
CC/ITRD, P.O. Box, 8022 Zurich, Switzerland].
Tschichold N., 2000: Konzeptbericht zum MeteoSchweiz Data Warehouse Vs. 1.1; MeteoSchweiz
Konzelmann, T. et al.,1998: New treatment of real time climate data sets from smi weather stations.
Proceedings 2 nd European conference on Applied Climatology, 19-23 Oct 1998