SEKE 2012 Proceedings - Knowledge Systems Institute

V. ANOMALY DETECTION IN EDDY COVARIANCE DATA
EXPERIMENT
A. EC Data Description
A series of experiments were conducted to determine if
SDVe can be utilized to evaluate D aProS-specified data
properties to find anomalies in sensor data. Upon further
analysis of the data properties obtained from the literature
survey in [8], it w as determined that three data property types
account for approximately 72.5% of the total number of data
properties specified by scientis ts in the literature survey;
datum properties (32.5% ), i.e. properties that capture the
behavior of a single data sensor reading, datum relationship
properties (30.8%), i.e. propert ies that capture a relationship
between two or more sensors, and datum dependent
instruments (9.2%), i.e., properties that capture environmental
data behavior effect on the data collection instruments. Based
on these findings, an experiment was design to determine if
SDVe can be used to identify anom alies for the m ost
frequently specified data properties: datum, datum
relationship, and datum dependent instrument.
A data property specification expert collaborated with
expert scientists working with eddy covariance and
biomesonet towers’ CO2 data to develop a set of data
properties of interest. The contributing scientists were working
on building their first eddy covariance tower and were
interested in capturing data pr operties extracted from sensor
reference manuals [ 12] [13], climate and climatological
variations in the research site literature [2], eddy covariance
towers post-field data quality control literature [ 14] and their
own expertise.
B. Experimental Setup
An initial experiment was conducted to validate the ability
of the SDVe tool to detect anom alies. An eddy covariance
error-free data file was random ly selected from a scientistprovided
repository and seeded with a number of anomalies,
based on a 95% confidence level calculation given the number
of readings in the file, to evaluate a group of data properties of
type datum, datum-relationship, and datum dependent
instrument. The experiment identified all seeded anomalies
and all events marked as anomalies were actually anomalies.
A second experiment was conducted to determine if SDVe
can identify anomalies in Eddy C ovariance sensor data and to
illustrate how such anom alies can be identified and
documented by cross referencing the results obtained from
SDVe to existing metadata recorded during the data collection
process. The experiment does not quantify the improvement in
the overall quality of the data.
The scientists developed a m atrix of all the relationships
between sensors in the Eddy covariance tower of interest for
the specific site. T he matrix included raw sensor
measurements and derived data aggregated at different
temporal resolutions and as part of the combination of
measurements from two or more sensors. The sensor
relationship matrix consisted of approximately 118 sensor
readings along with their associated relationships. In
collaboration with scientist som e of the sensor relationships
from the matrix were used to create 23 data properties to be
evaluated over the Eddy covarian ce datasets; the scientists
specified properties of interest of type Datum, Instrument, and
Datum Relationship. The properties w ere intended to capture
anomalies in raw data at collection tim e. The sensor readings
of interest were selected based on the relationship to other
sensor readings and the deriva tion of aggregated values from
them. In collaboration w ith scientists, data properties of
interest were specified, refined, and validated using DaProS.
The numeric thresholds used in the data properties were
defined following scientific community’s algorithms and
protocols.
The Eddy Covariance (EC) data verified using SDVe were
collected from July 06, 2010 to July 13, 2010 to capture EC
summer behavior and from February 09, 2010 to February 16,
2010 to capture EC winter behavior. The sensors at the tower
collected the EC data continuously, and a scientist ma nually
split the data into 1-hour interval files to ease the verification
process. 349 data files were evaluated for this work.
Two data property specification files were created, one for
each season, and used to automatically evaluate individual data
files according to the season to which they belonged. The data
files and specification files w ere automatically inputted to
SDVe to be evaluated. For each data file, the sensor data
streams were extracted to separate data scopes according to the
data property specifications. Then, the data scopes were
evaluated by applying the specifi cation Boolean statement and
data pattern to every individual sensor reading in the scope. If
the individual sensor reading in the scope did not satisfy the
data pattern and Boolean statement, a flag was raised and
stored to the verification file. Once the verification process had
concluded, a verification summary file was generated
aggregating the number of violati ons, i.e, anomalies, identified
by each data property along with the aggregated processing
times.
C. Results
SDVe performed a total of 219,800,854 evaluation calls of
which 50,857,351 of the evaluation calls were captured as
anomalies, approximately 23% . The evaluation process took
approximately 20 hours to complete, of which approximately
1 hour was spent loading the files into the system and 19 hours
were spent verifying the data. Assum ing a data file takes 15
minutes on average to be manually processed and evaluated by
a scientist manually processing the 349 files would take
approximately 87 hours. SDVe automatically evaluates the
data in one fourth of the tim e that it w ould take a scientist to
manually evaluate the same amount of data.
For summer data, 124,958,406 evaluations calls took place,
of which 24,417,791were of the evaluation calls were captured
as anomalies, approximately 20%.
Datum properties identified the m ost anomalies
(21,429,802), followed by Data Relationship (2,639,985),
Instrument (348,004) and Data Dependent Instrument (0). The
sensor datasets with the most anomalies included water vapor
mass density (H 2 O), atmospheric pressure (atm_press), carbon
dioxide (CO 2 ), and temperature (Ts).
681