Progress with programming of QC procedures into the Matlab ...

Introduction to the
IMOS-Toolbox
Guillaume Galibert, 26 th of August 2011

Introduction to the IMOS-Toolbox
• Technical and historical background
• What does/will it do Key features
• Update on current development progress
• Future developments

Technical & historical background
• Matlab (core & GUI) + Java (access to database)
• Standalone Windows executable & source code
• MS-Access deployment database
• Project (downloads / wiki / bugs) hosted on Google
code : http://code.google.com/p/imos-toolbox/
• Developers :
– Paul McCarthy (April 2009 - February 2010 / v1.0 - v1.5)
– Laurent Besnard (August - November 2010 / v1.55)
– Guillaume Galibert (since May 2011 / v2.0 …)

IMOS-Toolbox : What does/will it do
Data file
Data file
Data file
Convert data files from ANMN sensors to
IMOS NetCDF QC’d files
Data file
PP QC
IMOS
NetCDF
file
IMOS - eMII
data center
Local
deployment
database

IMOS-Toolbox : Key features
• Read data/metadata from data files
– Seabird SBE37/39 CTD (.cnv)
– Seabird SBE19+ V2 CTD (.hex & .cnv)
– FSI NXIC CTD
– RBR XR420 CTD
– RBR DR-1050 pressure logger
– Aquatec Aqualogger 520T & 520P & 520PT
– WET Labs WQM (.raw & .dat)
– YSI 6600 V2 Water Quality Sonde
– Teledyne/RDI Workhorse ADCP
– Nortek Aquadopp / Continental / AWAC ADCPs
– Echoview (hydroacoustic processed data for fishery)
• Read metadata from deployment database
– MS-Access deployment database
– Any kind of database with minimum design compatibility
New
New

IMOS-Toolbox : Key features
• Perform pre-processing operations on data
– Variable offset/scaling
– SBE43 oxygen voltage to concentration
– Time start change
– Time offset to UTC
– Add salinity variable
– Add relative pressure variable
New
– Add depth variable for every sensor on a mooring
New

• Display data
IMOS-Toolbox : Key features
– 1D/2D time series
– Depth profile
– Lat/Lon plot
– Any Xvar/Yvar plot
• Display and edit
IMOS NetCDF
metadata
– Global attributes
– Variables attributes

IMOS-Toolbox : Key features
• Perform QC operations on data using IOC flag system
– IN / OUT water
– Range test
– Flat line detection
– Time gap detection
– Tukey 53 spike detection
– RC filter spike detection
– Teledyne workhorse ADCP QC

IMOS-Toolbox : Key features
• Export data/metadata to IMOS NetCDF or ASCII formats
• Import / PP / QC / Export steps can be operated in
interactive or batch mode
New

IMOS-Toolbox : Recent developments
• Merge contributions
– G. Keith (TAS) : Echoview parser + any kind of database connection
– C. James (SA) : Parsers improvements (vectorization, …)
– B. Morris (NSW) : Teledyne QC and batch mode improvements
• Fix critical bugs (crashes + errors in data)
– Parsers crashes (particular cases, memory issues)
– Parameters reading and identification corrected
– ADCP vertical dimension DEPTH => HEIGHT_ABOVE_SENSOR
– Absolute and relative pressure measurements identified and documented
– Time dependent DEPTH variable for every sensor on a mooring

IMOS-Toolbox : Recent developments
• NetCDF CF compliance
– T, X, Y and Z information
– IMOS parameters / CF standard_name
• Metadata auto-documentation
– Instrument serial number
– Instrument sample interval
– vertical_geospatial_min/max + comment
• Data display
– Variables (1D / 2D) have matching X axis
– Natural QC color settings
– Y axis range adapted when QC performed

IMOS-Toolbox : Recent developments
• Focus on DEPTH information (SYD100 : 2010/12 to 2011/02)
– WQM
– P & T loggers
– ADCP

IMOS-Toolbox : Future developments
• Pre-processing routines
– Back processing from drifts model
– Back processing from post calibration information
• Metadata
– Sensor serial number
– Sensor calibration information
– Flu (count) Vs Chl_a (mg.m-3) specific information

IMOS-Toolbox : Future developments
• Define IMOS standard set of QC procedure
– Which QC procedure for which instrument / variable
– Improve existing procedures
– Implement Morello et al. procedures :
o
o
o
o
o
impossible date/location test
regional range test based on climatologies
spike test used in ANFOG
stationarity test from IOC
others …
• Quantitative QC - logical set based system (Morello et al.)
– Results from QC tests
– Other information :
o time since deployment/calibration
o rate of change
o deployment log sheet information
o historical calibration information …
-Fractions (S, M, L) of uncertainty
-Qualitative flags

IMOS-Toolbox : Future developments
• QC challenges/questions
– Assessment of step functions in time series
(Timms et al.)
– Inter-observer bias (heterogeneous datasets
in long term studies)
– Different rate of change/behaviour : physical
and biological data
– Suites of special QC tests for distinct :
o
o
Instruments : ADCP, CTD, WQM, …
Variables : physical, biological, acoustic, …

IMOS-Toolbox : Future developments
• Instrument/sensor quality
management
– Overall uncertainty fraction for
each sensor after each
deployment (Timms et al.)
– Snapshots of sensors health
• Deployment database
– Update schema : calibration,
uncertainty and quality
management on sensors
– Local Vs Central

IMOS-Toolbox : Future developments
• Improve display
– Tune colorbar settings for 2D
data (ex. : ADCP)
– Zoom & axis ticks behaviour
– Legend integration
– Resizable window
• Routine maintenance
– Bug fix
– New feature request

Thank you

IMOS-Toolbox : QC on WQM
• Spikes detection

IMOS-Toolbox : QC on WQM
• Flat lines detection

IMOS-Toolbox : QC on WQM
• WQM weird noise detection
– Observation of statistical distributions

IMOS-Toolbox : QC on WQM
• WQM weird noise detection
– Filter on data from most important mode < (mode 1 – 3*standard_deviation)