SEKE 2012 Proceedings - Knowledge Systems Institute

Tool Support for Anomaly Detection in Scientific
Sensor Data
Irbis Gallegos
The University of Texas at El Paso
Department of Computer Science
El Paso, USA
irbisg@utep.edu
Ann Gates
The University of Texas at El Paso
Department of Computer Science
El Paso, USA
agates@utep.edu
Abstract— Environmental scientists working on understanding
global changes and their implicatio ns for humanity, collect data in
near-real time at remote locations using a variety of instruments. As
the amount and complexity of collected data increases, so does the
amount of time and domain knowledge required to determine if the
collected data are correct and if the data collection instruments are
working correctly. The Sensor Data Verification (SDVe) tool allows
scientists to detect anom alies in sensor data. SD Ve evaluates if
scientific datasets, which can be provided at near -real time or from
file repositories, satisfy reusable data properties specified by
scientists.
Keywords- Big Data; Cyberinfrastructure; Data Quality E co-
Informatics; Software Engineering.
I. INTRODUCTION
Scientists collect and analyze large amounts of data to
determine the causes of changes in ecological ecosystems.
Scientist use Eddy covariance (EC) towers [1] to collect
measurements needed to understand such ecological changes.
In particular, scientists use m easurements taken by EC towers
to monitor carbon dioxide (CO 2 ), water balance (H 2 O), energy
balance (irradiance), and other meteorological measurements
such as temperature and atmospheric pressure.
Anomaly detection in eddy cova riance data must not only
identify instrument errors and problems with the sensors, but
also evaluate how closely conditions fulfill the theoretical
assumption underlying the method [2]. Anomaly detection
must be done in real time or s hortly after the measurements to
minimize data loss by reducing the time to detect and fix
instrument problems. The quality control procedures,
instrument malfunctions, maintenance and calibration periods
often remove 20 to 40% of the data. Efficient anomaly
detection is an outstanding pr oblem that is incom pletely
fulfilled in most of the FLUXNET networks [2].
In most cases, scientists manually evaluate eddy covariance
data by using a variety of time consuming methods and a
variety of customized software that has to be constantly
modified and recompiled. In a ddition, the data evaluation is
highly dependent on the expertise of the scientist, however,
such knowledge is typically not captured nor reused. The
sensor data verification (SD Ve) tool mitigates the
aforementioned limitations by allowing scientists to
automatically identify anomalies produced by environmental
events and equipment malfunctioning in scientific sensor
datasets collected at near-real tim e or extracted from file
repositories. SDVe evaluates whether a dataset satisfies a set
of formally specified data properties. SD Ve identifies
anomalies, i.e., a deviation from an expected value, due to
environmental variability or instrum ent malfunctioning, and
raises alarms whenever anomalies are detected. SDVe does
not require source code recompilation and allows expertdefined
data properties to be reused to verify the datasets.
Section 2 provides the background on the software
engineering techniques adapted and extended by SDVe.
Section 3 describes the data property specification approach
used by scientists to specify the data properties that are input to
SDVe. Section 4 explains the S DVe tool. Section 5 describes
the experimental setup and the re sults of using SDVe to detect
anomalies in Eddy covariance data, and section 6 discusses
some of the lessons learned from the experiment. Finally
section 7 described some efforts related to SDVe, followed by
the concluding remarks in Section 8.
II. BACKGROUND
SDVe uses software engineer ing techniques, which have
been used to provide assurance for critical softw are systems,
to detect anom alies in scientific sensor datasets. T his section
provides the background information associated to such
software engineering techniques.
A. Property Specification
The specification and pattern system (SPS) [3] was
introduced to assist practitioners to formally specify software
and hardware properties.
In the SPS, a specification consists of scopes and patterns.
Scopes define the portion of a program over which the
property holds. Patterns describe the structure of specific
behaviors and define relationships betw een propositions.
Propositions are used to represent Boolean expressions that are
evaluated over the program execution.
SPS patterns are divided into two groups: Occurrence and
Order patterns. Occurrence patterns deal with single event or
condition and specify the rate at which that condition or event
occurs. Order patterns relate two conditions or events and
specify the order at which they occur. In this context,
conditions are propositions that hold in one or m ore
consecutive states. Events are instants at w hich a proposition
changes values in two consecutive states.
678