H-SAF Product Validation Report (PVR) PR-OBS-3

Products Validation Report, 30 May 2010 - PVR-03 (Product PR-OBS-3) Page 1 

Products Validation Programme 

Italian Meteorological Service 

Italian Department of Civil Defence 

H-SAF Product Validation Report (PVR) 

PR-OBS-3 - Precipitation rate at ground by GEO/IR 

supported by LEO/MW 

Zentralanstalt für 

Meteorologie und 

Geodynamik 

Vienna University of Technology 

Institut für Photogrammetrie 

und Fernerkundung 

Royal Meteorological 

Institute of Belgium 

European Centre for Medium-Range 

Weather Forecasts 

Finnish Meteorological 

Institute 

Finnish Environment 

Institute 

Helsinki University 

of Technology 

Météo-France 

CNRS Laboratoire Atmosphères, 

Milieux, Observations Spatiales 

CNRS Centre d'Etudes 

Spatiales de la BIOsphere 

Bundesanstalt für 

Gewässerkunde 

Hungarian 

Meteorological Service 

CNR - Istituto Scienze 

dell’Atmosfera 

e del Clima 

Università di Ferrara 

Institute of Meteorology 

and Water Management 

Romania National 

Meteorological Administration 

Slovak Hydro-Meteorological 

Institute 

Turkish State 

Meteorological Service 

Middle East Technical 

University 

Istanbul Technical 

University 

Anadolu University 

30 May 2010


H-SAF Product Validation Report PVR-03 

Product PR-OBS-3 

Precipitation rate at ground by GEO/IR supported by LEO/MW 

INDEX 

Acronyms [not including those in the Appendix] 06 

1. The EUMETSAT Satellite Application Facilities and H-SAF 08 

2. Introduction to product PR-OBS-3 09 

2.1 Sensing principle 08 

2.2 Algorithm principle 09 

2.3 Main operational characteristics 10 

3. Validation strategy, methods and tools 11 

3.1 Validation team and work plan 11 

3.2 Validation philosophy 11 

3.2.1 Objectives and problems 11 

3.2.2 Tools to be used for validation 12 

3.2.3 Techniques to bring observations comparable 13 

3.2.4 Structuring the results of the validation activity 14 

3.3 Definition of statistical scores 16 

3.4 Inventory of validation facilities 18 

3.4.1 Facilities in Belgium (IRM) 18 

3.4.2 Facilities in Germany (BfG) 22 

3.4.3 Facilities in Hungary (OMSZ) 24 

3.4.4 Facilities in Italy (UniFe) 27 

3.4.5 Facilities in Poland (IMWM) 29 

3.4.6 Facilities in Slovakia (SHMÚ) 31 

3.4.7 Facilities in Turkey (ITU) 35 

4. Validation of the product release as at the end of the Development Phase 39 

4.1 Introduction 39 

4.2 Validation in Belgium (IRM) 40 

4.3 Validation in Germany (BfG) 42 

4.4 Validation in Hungary (OMSZ) 45 

4.5 Validation in Italy (UniFe) 48 

4.6 Validation in Poland (IMWM) 50 

4.7 Validation in Slovakia (SHMÚ) 53 

4.8 Validation in Turkey (ITU) 55 

5. Overview of findings 57 

5.1 Synopsis of validation results 57 

5.2 Summary conclusions on comparative elements 59 

Page


Appendix - Collection of validation experiment reports [low-level of editing] [89 pages] 61 

Yellow: statistical analyses over several months 

Blue: case studies in specified few days 

§ Validation experiments on PR-OBS-3 Period Institute 

2.1 Case study 17-18 January 2007 Belgium/IRM 

2.2 Case study 29-30 August 2006 Belgium/IRM 

2.3 Statistical analysis June 2007 - March 2009 Belgium/IRM 

2.4 Case study 25-26 May 2009 Belgium/IRM 

3.1 Statistical analysis September - December 2008 Germany/BfG 

3.2 Case study 17 April 2009 Germany/BfG 

4.1 Case studies: four cases August 2006 Hungary/OMSZ 

4.2 Statistical analysis December 2007 - March 2009 Hungary/OMSZ 

4.3 Case studies: five cases May-September 2008 Hungary/OMSZ 

4.4 Case studies: seven cases January-June 2009 Hungary/OMSZ 

4.5 Statistical analysis April-November 2009 Hungary/OMSZ 

5.1 Case studies: two cases 19 October and 18 November 2006 Italy/UniFe 

5.2 Statistical analysis September 2008 - March 2009 Italy/UniFe 

5.3 Case study 13 January 2009 Italy/UniFe 

6.1 Case studies: three cases May and June 2007 Poland/IMWG 

6.2 Case study 18 May 2008 Poland/IMWG 

6.3 Case study 15-16 August 2008 Poland/IMWG 

6.4 Case study 21-22 January 2009 Poland/IMWG 

6.5 Case study 11 May 2009 Poland/IMWG 

6.6 Statistical analysis December 2007 - December 2008 Poland/IMWG 

6.7 Statistical analysis January-December 2009 Poland/IMWG 

7.1 Statistical analysis June-September 2007 Slovakia/SHMÚ 

7.2 Statistical analysis August 2008 - February 2009 Slovakia/SHMÚ 

7.3 Case study 29 March 2009 Slovakia/SHMÚ 

7.4 Statistical analysis January-September 2009 Slovakia/SHMÚ 

7.5 Case study 11 April 2009 Slovakia/SHMÚ 

8.1 Case study 11 June 2007 Turkey/ITU 

8.2 Statistical analysis September 2008 - February 2009 Turkey/ITU 

8.3 Statistical analysis January-June 2009 Turkey/ITU


List of Tables 

Table 01 - List of H-SAF products 08 

Table 02 - Validation Team for precipitation products 11 

Table 03 - List of ground data used for precipitation products validation in Belgium 18 

Table 04 - Precipitation data available at BfG 22 

Table 05 - Location of the 16 meteorological radar of the DWD 22 

Table 06 - Characteristics of the three meteorological Doppler radars in Hungary 24 

Table 07 - Accuracy requirements for product PR-OBS-3 [RMSE (%)] 39 

Table 08 - Summary results of PR-OBS-3 validation in Belgium by IMR 40 

Table 09 - Summary results of PR-OBS-3 validation in Germany by BfG 42 

Table 10 - Summary results of PR-OBS-3 validation in Hungary by OMSZ 45 

Table 11 - Summary results of PR-OBS-3 validation in Italy by UniFe 48 

Table 12 - Summary results of PR-OBS-3 validation in Poland by IMWM 50 

Table 13 - Summary results of PR-OBS-3 validation in Slovakia by SHMÚ 51 

Table 14 - Summary results of PR-OBS-3 validation in Turkey by ITU and TSMS over inner 55 

land 

Table 15 - Summary results of PR-OBS-3 validation in Turkey by ITU and TSMS over coastal 55 

zones 

Table 16 - Comparative results of validation in several Countries/Teams split by season 58 

Table 17 - Simplified compliance analysis for product PR-OBS-3 59 

Table 18 - Synthesis of all validation results, including yearly average 60 

List of Figures 

Fig. 01 - Conceptual scheme of the EUMETSAT application ground segment 08 

Fig. 02 - Current composition of the EUMETSAT SAF network (in order of establishment) 08 

Fig. 03 - The H-SAF required coverage in the Meteosat projection 09 

Fig. 04 - Flow chart of the LEO/MW-GEO/IR-blending precipitation rate processing chain 09 

Fig. 05 - Structure of the Precipitation products validation team 11 

Fig. 06 - The network of 4100 rain gauges used for H-SAF precipitation products validation 12 

Fig. 07 - The network of 40 C-band radar used for H-SAF precipitation products validation 13 

Fig. 08 - Classes and sub-classes for evaluating Precipitation Rate products. Applicable to PR- 15 

OBS-1, PR-OBS-2, PR-OBS-3, PR-OBS-4 and PR-ASS-1 rate 

Fig. 09 - Classes and sub-classes for evaluating Accumulated Precipitation products. Applicable 16 

to PR-OBS-5 and PR-ASS-1 accumulated 

Fig. 10 - Meteorological radar in Belgium 18 

Fig. 11 - RMI raingauges: daily ( ) and AWS ( ) 19 

Fig. 12 - SETHY AWS network in Walloon Region 19 

Fig. 13 - Left: Gaussian filter; right: sketch of the up-scaling procedure. The circle corresponds to 20 

the range of the weather radar. The square in the middle is a common area such that it is 

entirely included in the selected PR-OBS-2 files. The grey rectangle, the tilted dark grey 

rectangle and the black ellipse are explained in the text 

Fig. 14 - Left panel: radar coverage in Germany as of 01/03/2007. Right panel: location of 23 

ombrometers for online calibration in RADOLAN; squares: hourly data provision 

(about 500), circles: event-based hourly data provision (about 800 stations): red: AMDA 

III, blue: aggregational network federal states (Bartels et al., 2004) 

Fig. 15 - Flowchart of online calibration RADOLAN (adapted from Bartels et al. 2004) 23 

Fig. 16 - The automatic rain gauge network in Hungary 24 

Fig. 17 - Location and coverage of the three meteorological Doppler radars in Hungary 24 

Fig. 18 - Map of SHMÚ raingauge stations: green – operational (98) , blue – climatological 31 

(586), red - hydrological stations in H-SAF selected test basins (37). White points show


regular grid of experimental NOAA Snow water equivalent data 

Fig. 19 - Example of Slovak radar network coverage - left circle corresponds to radar site Malý 31 

Javorník, right one corresponds to Kojšovská hoľa 

Fig. 20 - Example of 5-days cumulative precipitation constructed from raingauge measurements 32 

by means of 3D extrapolation method 

Fig. 21 - Map of SHMÚ hydrological stations: red - hydrological stations in H-SAF selected test 33 

basins (34) 

Fig. 22 - General hydrological model of HBV type 33 

Fig. 23 - Example of pick up pixels from radar measurement for selected area 34 

Fig. 24 - Example of separated ALADIN forecast pixel to smaller pixels 34 

Fig. 25 - The Susurluk and Western Black Sea catchments selected for the precipitation product 35 

validation in Turkey 

Fig. 26 - Network of Meteorological stations in Susurluk (on the left) and Western Black Sea (on 36 

the right) catchments 

Fig. 27 - Position of 193 AWOS sites used for ground truth for the precipitation product 36 

validation in western Turkey 

Fig. 28 - H02 product footprint centres with a sample footprint area as well as the AWOS ground 37 

observation sites 

Fig. 29 - Meshed structure of the sample H02 product footprint 38 

Fig. 30 - Mean Error of PR-OBS-3 (monthly values over Belgium in mm h -1 ) 41 

Fig. 31 - Root Mean Square Error of PR-OBS-3 (monthly values over Belgium in mm h -1 ) 41 

Fig. 32 - Time evolution of Mean Error and Standard Deviation for PR-OBS-3 42 

Fig. 33 - Same as Fig. 32 with stretched vertical scale 43 

Fig. 34 - Time evolution of Probability Of Detection, False Alarm Rate and Critical Success 43 

Index for PR-OBS-3 

Fig. 35 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for 45 

light and medium rain 

Fig. 36 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for 46 

heavy rain 

Fig. 37 - Time evolution of the Correlation Coefficient for the three categories of precipitation 46 


Index 

Fig. 39 - Probability Density Function for the months of January (left) and July (right) 2009 48 

Fig. 40 - Evolution of continuous statistical scores cross year 2009 49 

Fig. 41 - Mean error (ME) of PR-OBS-3 v.1.4 for the period of Jan 2009 – Mar 2010 for Poland 51 

Fig. 42 - RMSE % of PR-OBS-3 v.1.4 for the period of Jan 2009 – Mar 2010 for Poland 51 

Fig. 43 - Variabily of Probability of Detection (POD) and False Alarm Ratio (FAR) obtained for 52 

PR-OBS-3 v.1.4 using Polish RG data in the period of Jan 2009 – Mar 2010 

Fig. 44 - Time evolution of the Mean Error for the three precipitation categories 53 

Fig. 45 - Time evolution of the Mean Error for medium and light precipitation (stretched vertical 53 

scale from Fig. 44) 

Fig. 46 - Time evolution of the Root Mean Square Error (%) for the three precipitation categories 54 

Fig. 47 - Time evolution of the Correlation Coefficient for the three precipitation categories 54 


Index 

Fig. 49 - Continuous and multi-categorical statistics for inner land 56 

Fig. 50 - Continuous and multi-categorical statistics for coastal zones 56


Acronyms [not including those in the Appendix] 

ACC Fraction correct Accuracy 

AMSU Advanced Microwave Sounding Unit (on NOAA and MetOp) 

ATDD Algorithms Theoretical Definition Document 

AU Anadolu University (in Turkey) 

AWOS Automated Weather Observing Stations 

AWS Automatic Weather Station 

BfG Bundesanstalt für Gewässerkunde (in Germany) 

CAF Central Application Facility (of EUMETSAT) 

CC Correlation Coefficient 

CDOP Continuous Development-Operations Phase 

CESBIO Centre d'Etudes Spatiales de la BIOsphere (of CNRS, in France) 

CM-SAF SAF on Climate Monitoring 

CNMCA Centro Nazionale di Meteorologia e Climatologia Aeronautica (in Italy) 

CNR Consiglio Nazionale delle Ricerche (of Italy) 

CNRS Centre Nationale de la Recherche Scientifique (of France) 

CSI Critical Success Index 

DMSP Defence Meteorological Satellite Program 

DPC Dipartimento Protezione Civile (of Italy) 

DWD Deutscher Wetterdienst 

DWR Dry to Wet Ratio 

ECMWF European Centre for Medium-range Weather Forecasts 

ETS Equitable Threat Score 

EUM Short for EUMETSAT 

EUMETCast EUMETSAT‟s Broadcast System for Environmental Data 

EUMETSAT European Organisation for the Exploitation of Meteorological Satellites 

FAR False Alarm Rate 

FBI Frequency BIas 

FMI Finnish Meteorological Institute 

GEO Geostationary Earth Orbit 

GIS Geographic Information System 

GPM Global Precipitation Measurement mission 

GRAS-SAF SAF on GRAS Meteorology 

HAWK Hungarian Advanced Weather Workstation 

HBV Hydrologiska Byrans Vattenbalansavdelning (hydrological model) 

HRON Hydrological model for the Hron basin 

H-SAF SAF on Support to Operational Hydrology and Water Management 

HSS Heidke skill score 

IFOV Instantaneous Field Of View 

IMWM Institute of Meteorology and Water Management (in Poland) 

IPF Institut für Photogrammetrie und Fernerkundung (of TU-Wien, in Austria) 

IR 

Infra Red 

IRM Institut Royal Météorologique (of Belgium) (alternative of RMI) 

ISAC Istituto di Scienze dell‟Atmosfera e del Clima (of CNR, Italy) 

ITU İstanbul Technical University (in Turkey) 

LATMOS Laboratoire Atmosphères, Milieux, Observations Spatiales (of CNRS, in France) 

LEO Low Earth Orbit 

LSA-SAF SAF on Land Surface Analysis 

ME Mean Error 

Météo France National Meteorological Service of France 

METU Middle East Technical University (in Turkey) 

MHS Microwave Humidity Sounder (on NOAA 18 and 19, and on MetOp)


MSG 

MTF 

MW 

NMA 

NOAA 

NWC 

NWC-SAF 

NWP 

NWP-SAF 

O3M-SAF 

OMSZ 

ORR 

OSI-SAF 

Pixel 

POD 

POFD 

PUM 

PVR 

REP-3 

RMI 

RMS 

RMSE 

RR 

RU 

SAF 

SD 

SETHY 

SEVIRI 

SHMÚ 

SRE 

SSM/I 

SSMIS 

SYKE 

TB or T B 

T BB 

TKK 

TSMS 

TU-Wien 

UKMO 

UniFe 

UTC 

ZAMG 

Meteosat Second Generation 

Modulation Transfer Function 

Micro-Wave 

National Meteorological Administration (of Romania) 

National Oceanic and Atmospheric Administration (Agency and satellite) 

Nowcasting 

SAF in support to Nowcasting & Very Short Range Forecasting 

Numerical Weather Prediction 

SAF on Numerical Weather Prediction 

SAF on Ozone and Atmospheric Chemistry Monitoring 

Hungarian Meteorological Service 

Operations Readiness Review 

SAF on Ocean and Sea Ice 

Picture element 

Probability of Detection 

Probability Of False Detection 

Product User Manual 

Product Validation Report 

H-SAF Products Valiadation Report 

Royal Meteorological Institute (of Belgium) (alternative of IRM) 

Root Mean Square 

Root Mean Square Error 

Rain Rate 

Rapid Update 

Satellite Application Facility 

Standard Deviation 

Service d'Ètudes Hydrologiques, Walloon Ministry of Public Works - Belgium 

Spinning Enhanced Visible and Infra-Red Imager (on Meteosat from 8 onwards) 

Slovak Hydro-Meteorological Institute 

Scale Recursive Estimation 

Special Sensor Microwave / Imager (on DMSP up to F-15) 

Special Sensor Microwave Imager/Sounder (on DMSP starting with S-16) 

Suomen ympäristökeskus (Finnish Environment Institute) 

Brightness Temperature (used for MW and, inappropriately, for IR) 

Equivalent Blackbody Temperature (used for IR) 

Teknillinen korkeakoulu (Helsinki University of Technology) 

Turkish State Meteorological Service 

Technische Universität Wien (in Austria) 

United Kingdom Met Office 

University of Ferrara (in Italy) 

Universal Coordinated Time 

Zentralanstalt für Meteorologie und Geodynamik (of Austria)


1. The EUMETSAT Satellite Application Facilities and H-SAF 

The “EUMETSAT Satellite Application Facility on Support to Operational Hydrology and Water 

Management (H-SAF)” is part of the distributed application ground segment of the “European 

Organisation for the Exploitation of Meteorological Satellites (EUMETSAT)”. The application ground 

segment consists of a “Central Application Facility (CAF)” and a network of eight “Satellite 

Application Facilities (SAFs)” dedicated to development and operational activities to provide satellitederived 

data to support specific user communities. See Fig. 01. 

EUM Geostationary 

Systems 

Data Acquisition 

and Control 

Data Processing 

EUMETSAT HQ 

Systems of the 

EUM/NOAA 

Cooperation 

other data 

sources 

Application Ground Segment 

Meteorological Products 

Extraction 

EUMETSAT HQ 

Archive & Retrieval 

Facility (Data Centre) 

EUMETSAT HQ 

Satellite Application 

Facilities (SAFs) 

Centralised processing 

and generation of products 

Decentralised processing 

and generation of products 

USERS 

Fig. 01 - Conceptual scheme of the EUMETSAT application ground segment. 

Fig. 02 reminds the current composition of the EUMETSAT SAF network (in order of establishment). 

Nowcasting & Very 

Short Range Forecasting 

Ocean and Sea Ice 

Ozone & Atmospheric 

Numerical Weather 

Climate Monitoring 

Chemistry Monitoring Prediction 

GRAS Meteorology 

Land Surface Analysis 

Operational Hydrology 

& Water Management 

Fig. 02 - Current composition of the EUMETSAT SAF network (in order of establishment). 

The H-SAF was established by the EUMETSAT Council on 3 July 2005; its Development Phase started 

on 1 st September 2005 and ends on 31 August 2010. The list of H-SAF products is shown in Table 01. 

Table 01 - List of H-SAF products 

Code Acronym Product name 

H01 PR-OBS-1 Precipitation rate at ground by MW conical scanners (with indication of phase) 

H02 PR-OBS-2 Precipitation rate at ground by MW cross-track scanners (with indication of phase) 

H03 PR-OBS-3 Precipitation rate at ground by GEO/IR supported by LEO/MW 

H04 PR-OBS-4 Precipitation rate at ground by LEO/MW supported by GEO/IR (with flag for phase) 

H05 PR-OBS-5 Accumulated precipitation at ground by blended MW and IR 

H06 PR-ASS-1 Instantaneous and accumulated precipitation at ground computed by a NWP model 

H07 SM-OBS-1 Large-scale surface soil moisture by radar scatterometer 

H08 SM-OBS-2 Small-scale surface soil moisture by radar scatterometer 

H09 SM-ASS-1 Volumetric soil moisture (roots region) by scatterometer assimilation in NWP model 

H10 SN-OBS-1 Snow detection (snow mask) by VIS/IR radiometry 

H11 SN-OBS-2 Snow status (dry/wet) by MW radiometry 

H12 SN-OBS-3 Effective snow cover by VIS/IR radiometry 

H13 SN-OBS-4 Snow water equivalent by MW radiometry


2. Introduction to product PR-OBS-3 

2.1 Sensing principle 

Product PR-OBS-3 (Precipitation rate at ground by 

GEO/IR supported by LEO/MW) is based on the IR 

images from the SEVIRI instrument onboard 

Meteosat satellites. The whole H-SAF area is 

covered (see Fig. 03, same as for PR-OBS-4 and 

PR-OBS-5), but the resolution degrades with 

latitude. The equivalent blackbody temperatures 

(T BB ) are converted to precipitation rate by lookup 

tables updated at intervals by precipitation rate 

determinations generated from MW instruments (in 

H-SAF: PR-OBS-1 and PR-OBS-2). The product is 

generated at the 15-min imaging rate of SEVIRI, 

and the spatial resolution is consistent with the 

SEVIRI pixel. The processing method is called 

“Rapid Update”. 

The SEVIRI channel utilised for PR-OBS-3 is 10.8 m. The calibration of T BB ‟s in term of 

precipitation rate by means of MW measurements (supposedly accurate) implies the existence of good 

correlation between T BB and precipitation rate. This is fairly acceptable for convective precipitation, 

less for non-convective. Nevertheless, Rapid Update is currently the only operational algorithm 

enabling precipitation rate estimates with the time resolution required for nowcasting. In addition, 

frequent sampling is a prerequisite for computing accumulated precipitation (product PR-OBS-5). 

For more information, please refer to the Products User Manual (specifically, PUM-03). 

2.2 Algorithm principle 

Fig. 03 - The H-SAF required coverage in the Meteosat projection. 

The baseline algorithm for PR-OBS-3 processing is described in ATDD-03. Only essential elements are 

highlighted here. Fig. 04 shows the flow chart of the processing chain. 

SSM/I-SSMIS 

AMSU-MHS 

~ 3-hourly sequence 

of MW observations 

Morphing 

algorithm 

SEVIRI 

15-min images 

Lookup tables 

updating 

Rapid-update 

algorithm 

PRECIPITATION 

RATE 

Extraction of 

dynamical info 

Important notes: 

Fig. 04 - Flow chart of the LEO/MW-GEO/IR-blending precipitation rate processing chain. 

The PR-OBS-3 version undertaking the ORR (Operations Readiness Review) in mid-2010, and to 

continue to be pre-operationally distributed during a considerable portion of CDOP-1, does not yet 

make use of PR-OBS-1 (SSM/I-SSMIS). 

The alternative approach based on morphing (PR-OBS-4) is not being submitted to the ORR.


The blending technique adopted for PR-OBS-3 is called “Rapid Update (RU)”; see, for instance, Turk et 

al. 2000 1 . 

Key to the RU blended satellite technique is a real time, underlying collection of time and spaceintersecting 

pixels from operational geostationary IR imagers and LEO MW sensors. Rain intensity 

maps derived from MW measurements are used to create global, geo-located rain rate (RR) and T B 

(brightness temperature) relationships that are renewed as soon as new co-located data are available 

from both geostationary and MW instruments. In the software package these relationships are called 

histograms. To the end of geo-locating histogram relationships, the globe (or the study area) is 

subdivided in equally spaced lat-lon boxes (2.5°×2.5°). As new input datasets (MW and IR) are 

available in the processing chain, the MW-derived rain rate pixels are paired with their time and spacecoincident 

geostationary 10.8-µm IR equivalent blackbody temperature (T BB ) data, using a 15-minute 

maximum allowed time offset between the pixel observation times. 

2.3 Main operational characteristics 

The operational characteristics of PR-OBS-3 are discussed in PUM-03. Here are the main highlights. 

The horizontal resolution ( x) is the convolution of several features (sampling distance, degree of 

independence of the information relative to nearby samples, …). The horizontal resolution descends 

from the instrument Instantaneous Field of View (IFOV), the sampling distance (pixel), the Modulation 

Transfer Function (MTF) and number of pixels to co-process for filtering out disturbing factors (e.g. 

clouds) or improving accuracy. The IFOV of SEVIRI images is 4.8 km at nadir, and degrades moving 

away from nadir, becoming about 8 km in the H-SAF area. A figure representative of the PR-OBS-3 

resolution is: ~ 8 km. Sampling is made at ~ 5 km intervals, consistent with the SEVIRI pixel over 

Europe. Conclusion: 

resolution x ~ 8 km - sampling distance: ~ 5 km. 

The observing cycle ( t) is defined as the average time interval between two measurements over the 

same area. In the case of PR-OBS-3 the product is generated soon after each SEVIRI new acquisition, 

Thus: 

observing cycle t = 15 min - sampling time: 15 min. 

The timeliness ( ) is defined as the time between observation taking and product available at the user 

site assuming a defined dissemination mean. The timeliness depends on the satellite transmission 

facilities, the availability of acquisition stations, the processing time required to generate the product 

and the reference dissemination means. For PR-OBS-3, the time of observations is 1-5 min before each 

quarter of an hour, ending at the full hour. To this, ~ 5 min have to be added for acquisition through 

EUMETCast and ~ 5 min for processing at CNMCA, thus: 

timeliness ~ 15 min. 

The accuracy (RMS) is the convolution of several measurement features (random error, bias, sensitivity, 

precision, …). To simplify matters, it is generally agreed to quote the root-mean-square difference 

[observed - true values]. The accuracy of a satellite-derived product descends from the strength of the 

physical principle linking the satellite observation to the natural process determining the parameter. It is 

difficult to be estimated a-priori: it is generally evaluated a-posteriori by means of the validation 

activity. 

1 Turk J.F., G. Rohaly, J. Hawkins, E.A. Smith, F.S. Marzano, A. Mugnai and V. Levizzani, 2000: “Analysis and 

assimilation of rainfall from blended SSMI, TRMM and geostationary satellite data”. Proc. 10th AMS Conf. Sat. 

Meteor. and Ocean., 9, 66-69.


3. Validation strategy, methods and tools 

3.1 Validation team and work plan 

Whereas the previous operational characteristics have been evaluated on the base of system 

considerations (number of satellites, their orbits, access to the satellite) and instrument features (IFOV, 

swath, MTF and others), the evaluation of accuracy requires validation, i.e. comparison with the ground 

truth or with something assumed as “true”. PR-OBS-3, as any other H-SAF product, has been 

submitted to validation entrusted to a number of institutes (see Fig. 05). 

Precipitation products validation group 

Leader: Italy (DPC) 

Belgium 

IRM 

Germany 

BfG 

Hungary 

OMSZ 

Italy 

UniFe 

Poland 

IMWM 

Slovakia 

SHMÚ 

Turkey 

ITU 

Fig. 05 - Structure of the Precipitation products validation team. 

Table 02 lists the persons involved in the validation of H-SAF precipitation products 

Table 02 - Validation Team for precipitation products 

Silvia Puca (Leader) Dipartimento Protezione Civile (DPC) Italy silvia.puca@protezionecivile.it 

Emmanuel Roulin Institut Royal Météorologique (IRM) Belgium emmanuel.roulin@oma.be 

Angelo Rinollo Institut Royal Météorologique (IRM) Belgium angelo.rinollo@oma.be 

Thomas Maurer Bundesanstalt für Gewässerkunde (BfG) Germany thomas.maurer@bafg.de 

Peer Helmke Bundesanstalt für Gewässerkunde (BfG) Germany helmke@bafg.de 

Eszter Lábó Hungarian Meteorological Service (OMSZ) Hungary labo.e@met.hu 

Federico Porcu' Ferrara University, Department of Physics (UniFe) Italy porcu@fe.infn.it 

Bozena Lapeta Institute of Meteorology and Water Management (IMWM) Poland bozena.lapeta@imgw.pl 

Ján Kaňák Slovenský Hydrometeorologický Ústav (SHMÚ) Slovakia jan.kanak@shmu.sk 

Ľuboslav Okon Slovenský Hydrometeorologický Ústav (SHMÚ) Slovakia luboslav.okon@shmu.sk 

Ahmet Öztopal Istanbul Technical University (ITU) Turkey oztopal@itu.edu.tr 

Ibrahim Sonmez Turkish State Meteorological Service (TSMS) Turkey isonmez@meteor.gov.tr 

The Precipitation products validation programme started with a first workshop in Rome, 20-21 June 

2006, soon after the H-SAF Requirements Review (26-27 April 2006). The first activity was to lay 

down the Validation plan, that was finalised as early as 30 September 2006, i.e. about one year after the 

start of the H-SAF Development Phase. After the first Workshop, other ones followed, at roughly 

yearly intervals, often joined with the Hydrological validation group. 

At the 1 st H-SAF Workshop (Rome,16-18 October 2007), a first set of significant validation exercises 

was presented. An internal document, called REP-3 (H-SAF Products Validation Report) started being 

compiled since then. Now, moving to the end of the H-SAF Development Phase, REP-3 has been 

restructured into this Product Validation Report (PVR) split into 13 volumes, one for each H-SAF 

product. The validation experiments recorded in REP-3 constitute “Appendixes” to the various 

volumes. Because of the initial aim of REP-3 (internal document at working level) the editorial level of 

the Appendixes is of rather low standard. 

3.2 Validation philosophy 

3.2.1 Objective and problems 

The products validation activity has to serve multiple purposes: 

most urgent, to provide input to the product developers for improving calibration for better quality 

of baseline products, and for guidance in the development of more advanced products;


also urgent, to characterise the products error structure in order to enable the Hydrological 

validation programme to appropriately use the data; the Education & Training programme, part of 

the Hydrological validation programme, was particularly instrumental for this; 

to build up the background information necessary for online quality control of the products before 

distribution; 

in general, to enable attaching the necessary information on error structure to accompany H-SAF 

products distribution in an open environment, after the initial phase of distribution limited to the socalled 

“beta users”. 

Validation is obviously a hard work in the case of precipitation, both because the sensing principle from 

space is very much indirect, and because of the natural space-time variability of the precipitation field 

(sharing certain aspects with fractal fields), that places severe sampling problems. It is known that an 

absolute „ground truth‟ does not exist. For the performance evaluation of the H-SAF precipitation 

products the radar and rain gauge measurements have been assumed as „ground truth‟. This is due to the 

large use and experience of these data by the hydrologists, the main users of the products. Comparison 

with results of numerical models obviously suffer of the incompatible scales between the natural 

phenomenon and the model (for hydrostatic NWP models) or the limits of atmospheric predictability 

when entering the scale of convection (for Cloud Resolving Models). A mixture of all this techniques is 

generally used, and the results change with the climatic situation and the type of precipitation. It is 

therefore necessary a European cooperation for this programme. 

3.2.2 Tools to be used for validation 

The areas chosen for the validation task include the basins where the hydrological validation is 

performed. The data used for the validation of the satellite precipitations products are: 

Ground data: 

- automatic rain gauges with different time resolution: 5 min, 10 min, 15 min, 30 min; 

- meteorological radars with different time resolution: 5 min, 10 min, 30 min. 

Data for cloud types classification, containing information about water content in vertical column 

and for the discrimination of the synoptic situation are also foreseen. The main products used to 

derived these information are: products from the Nowcasting SAF; output from NWP models; 

SEVIRI composite images. 

Fig. 06 provides a view of the raingauge network used for precipitation products validation in H-SAF. 

Fig. 06 - The network of 4100 rain gauges used for H-SAF precipitation products validation.


Fig. 07 provides a view of the radar network utilised for precipitation products validation in H-SAF. 

Fig. 07 - The network of 40 C-band radar used for H-SAF precipitation products validation. 

3.2.3 Techniques to bring observations comparable 

Due to the time and space structure of precipitation and to the sampling characteristics of both the 

precipitation products and ground data used for validation, care has to be taken to bring data 

comparable. At a given place, precipitation occurs intermittently and at highly fluctuating rates. Over 

space, precipitation is distributed with a high variability, in cells of high intensity nested in larger area 

with lower rain rate. Aimed at observing this complex phenomenon, the satellite-based products are 

defined with a spatial resolution of several kilometres and with different sampling rate. On the other 

hand, reference ground data used to validate precipitation data from satellite are also characterized by 

their own spatial resolution ranging from point information measured on rain-gauge networks to grids 

with cells of several hundreds of meters to several kilometres for weather radar. Furthermore, none of 

these reference observations are without error. For this reason it was decided to compare the satellite 

data with ground data on the satellite product native grid. All the institutes applied the same up-scaling 

method to compare the satellite precipitation estimations with ground data. 

There are several approach to bring the observation comparable. The simplest consists in comparing 

untransformed data, e.g. comparing areal data to observations at a nearest gauge station, or 

instantaneous images with information available within a time window. Doing so, part of the error has 

to be attributed to the differences between sample volumes: this “representativeness error” may be 

estimated by using high spatial and temporal resolution gauge data (e.g. Kitchen and Blackall 1992) 2 or 

may be simulated in numerical experiments (e.g. Tustison et al. 2001) 3 . 

An alternative approach consists in upscaling reference observations to areal averages corresponding to 

the resolution of the precipitation products but in an equal-area map projection. For rain-gauge data, 

this step requires the use appropriate interpolation scheme (e.g. Thiessen polygons, kriging, etc.). For 

radar images, it requires to average the values measured at radar pixels included in each of the product 

2 Kitchen M. and R.M. Blackall, 1992: “Representativeness errors in comparisons between radar and gauge 

measurements of rainfall”. J. Hydrol., 134, 13-33. 

3 Tustison B., D. Harris and E. Foufoula-Georgiou, 2001: “Scale issues in verification of precipitation forecasts”. J. 

Geophys. Res., 106, 11,775-11,784.


pixel. For cumulate products, data from radar images have to be further integrated over the time 

intervals and using advection procedures to correct the effect of time sampling. 

The Scale Recursive Estimation (SRE) (Primus et al. 2001 4 , Tustison et al. 2003 5 , Gupta et al. 2006 6 ) 

can be used for the validation since observations are available at one ore more scales different than the 

scale of the H-SAF products. This methodology consists in filtering noisy observations taking into 

account the scale-dependant variability and the nested spatial structure of precipitation. It provides 

optimal estimates of precipitation at the desired scale i.e. unbiased and with the minimum variance as 

well as it gives information about uncertainty at that scale. Nevertheless, it may require some 

resampling of the data to make it compatible with a cascade structure. 

There is a trade-off to be found between pooling the data in space and time in order to have a validation 

sample large enough and stratifying into sub-samples with comparable situations so as to avoid that the 

performance results be biased towards the dominant regime. The validation data may be separated in 

seasons, night and day, sea and land, geographical regions, rainfall intensity and cloud or precipitation 

type. 

As mentioned before for the validation exercises inside this project the radar and rain gauge data were 

up-scaled taking into account the satellite scanning geometry and IFOV resolution of AMSU-B scan, 

SSMI and SEVIRI. Radar and rain gauge instruments provide many measurements within a single 

satellite IFOV, those measurements were averaged following the satellite antenna pattern of AMSU-B, 

SSMI and SEVIRI. This activity was developed in collaboration with the precipitation product 

developers. 

Two codes were developed by the validation group for upscaling ground data data vs AMSU-B and 

SSMI IFOV. All institutes involved in precipitation product validation activity uses these two codes 

developed by University of Ferrara and RMI 7 . 

About the SEVIRI data a common code was not developed, but all institutes involved in precipitation 

product validation activity uses the same up-scaling technique which was indicated by CNR-ISAC. A 

common code will be developed in CDOP. 

3.2.4 Structuring the results of the validation activity 

During the development phase a twofold validation strategy was applied: one based on large statistics 

(multi-categorical and continuous), and one on selected case studies. Both components were, and still 

are, considered complementary in assessing the accuracy of the implemented algorithms. Large 

statistics help in identifying existence of pathological behaviour, selected case studies are useful in 

identifying the roots of such behaviour where present. 

Common validation 

To produce a large statistical analysis of the H-SAF Precipitation Products it was necessary to define a 

„common validation methodology’ in order to make comparable the results obtained by several 

institutes and to better understand their meanings. 

To achieve these goal it was necessary: 

standardization of the up-scaling techniques of radar and rain gauge data vs AMSU, SSMI and 

SEVIRI data, 

introduction of quality filter, 

4 Primus I., D. McLaughlin and D. Enthekabi, 2001: “Scale-recursive assimilation of precipitation data”. Adv. Water 

Resour., 24, 941-953. 

5 Tustison B., E. Foufoula-Georgiou and D. Harris, 2003: “Scale-recursive estimation for multisensor Quantitative 

precipitation Forecast verification: a preliminary assessment”. J. Geophys. Res., 108, 8377-8390. 

6 Gupta R., V. Venugopal and E. Foufoula-Georgiou, 2006: “A methodology for merging multisensor precipitation 

estimates based on expectation-maximization and scale-recursive estimation”. J. Geophys. Res., 111, D02102, 

doi:10.1029/2004JD005568. 

7 Van de Vyver H. and E. Roulin, 2009: “Scale recursive estimation for merging precipitation data from radar and 

microwave cross-track scanners”. J. Geophys. Res., 114, D08104, doi: 10.1029/2008JD010709.


development and sharing of software packages. 

The Common Validation Methodology is based on comparisons with rain gauges and radar data to 

produce monthly Continuous verification and Multi-Categorical statistic scores for sea, land and coast 

area. 

The main steps are: 

all the institutes compare the national radar and rain gauge data with the precipitation values 

estimated by satellite on the satellite native grid using the same up-scaling techniques; 

all the institutes evaluate the monthly continuous scores (below reported) and contingency tables for 

the precipitation classes (below reported) producing numerical files called „CS‟ and „MC‟ files; 

all the institutes evaluate PDF producing numerical files called „DIST‟ files and plots; 

the precipitation product validation leader collect all the validation files (MC, CS and DIST files), 

verify the consistency of the results and evaluate the monthly common statistical results; 

The results obtained were: 

discussed inside the validation group and with product developers by email and two annual 

meetings, 

reported in the project document, 

published in the H-SAF web page. 

Case studies 

Each Institute, in addition to the common validation methodology, developed a more specific 

Validation Methodology based on the knowledge and experience of the Institute itself. This activity is 

focused on case studies analysis. Each institute decides whether to use ancillary data such as lightning 

data, SEVIRI images, the output of numerical weather prediction and nowcasting products. 

The main steps are: 

description of the meteorological event; 

comparison of ground data and satellite products; 

visualization of ancillary data deduced by nowcasting products or lightning network; 

discussion of the satellite product performances; 

indications to Developers; 

making the ground data (if requested) available to satellite product developers. 

The results obtained were: 

discussed inside the validation group and with product developers by email and two annual 

meetings, 

reported in the project document, 

published in the H-SAF web page. 

Subdivision in classes 

Since the accuracy of precipitation measurements depends on the type of precipitation or, to simplify 

matters, the intensity, the verification is carried out split in more classes. For intensity, user 

requirements have been expressed for three classes; however, for working purposes, finer subdivision in 

11 sub-classes is used (see Fig. 08). 

Class 

1 2 3 

< 1 mm/h (light precipitation) 1 - 10 mm/h (medium precipitation) > 10 mm/h (intense precipitation) 

Subclass 1 2 3 4 5 6 7 8 9 10 11 

(mm/h) < 0.25 0.25-0.5 0.5 - 1.0 1.0 - 2.0 2.0 - 4.0 4.0 - 8.0 8.0 - 10 10 - 16 16 - 32 32 - 64 > 64 

Fig. 08 - Classes and sub-classes for evaluating Precipitation Rate products. 

Applicable to PR-OBS-1, PR-OBS-2, PR-OBS-3, PR-OBS-4 and PR-ASS-1rate


For accumulated precipitation, user requirements are unclear in terms of dependence on amount. We 

have adopted a 5-class splitting for results presentation and a 10-subclass subdivision for working 

purpose (see Fig. 09). 

Class 

1 2 3 4 5 

< 8 mm 8 - 32 mm 32-64 mm 64-128 mm > 128 mm 

Subclass 1 2 3 4 5 6 7 8 9 10 

(mm) < 1 1 - 2 2 - 4 4 - 8 8 - 16 16 - 32 32 - 64 64 - 128 128 - 256 > 128 

Fig. 09 - Classes and sub-classes for evaluating Accumulated Precipitation products. 

Applicable to PR-OBS-5 and PR-ASS-1accumulated 

The evaluation of the statistical scores split by precipitation classes allows to analyse the product 

performances not only for precipitation mean values (light precipitation being the more frequent) but 

also for higher value, the most interesting for Hydrology. 

3.3 Definition of statistical scores 

It is appropriate to deploy the definitions of the statistical scores utilised in H-SAF product validation 

activities. Some apply to “continuous statistics”, some to “dichotomous statistics”. Although neither 

rain gauges nor radar constitute a very accurate ground truth, we assume as “true” these observations, 

thus the departures of satellite observations will be designated as “errors” 

Scores for continuous statistics: 

- Mean Error (ME) or Bias 

- Standard Deviation (SD) 

- Correlation Coefficient (CC) 

- Root Mean Square Error (RMSE) 

- Root Mean Square Error percent (RMSE %), used for precipitation since error grows with rate. 

ME or 

bias 

1 

N 

N 

k 1 

(sat 

k 

truek 

) 

SD 

1 

N 

N 

k 1 

sat 

k 

true 

k 

ME 

2 

N 

k 

k 

k 1 

CC with 

N 

N 

2 

2 

sat 

sat 

sat 

sat 

k 

k 1 

1 

true 

true 

k 

true 

true 

N 

1 

sat sat k 

and 

N 

k 1 

N 

1 

true true k 

; 

N 

k 1 

RMSE 

1 

N 

N 

k 1 

sat k 

true k 

2 

RMSE % 

1 

N 

N 

k 1 

sat 

k 

true 

2 

true k 

k 

2


Scores for dichotomous statistics 

Stemming from the contingency Table: 

Contingency Table 

Observed (ground) 

yes no total 

yes hits false alarms forecast yes 

Forecast (satellite) no misses correct negatives forecast no 

total observed yes observed no total 

where: 

- hit: event observed from the satellite, and also observed from the ground 

- miss: event not observed from the satellite, but observed from the ground 

- false alarm: event observed from the satellite, but not observed from the ground 

- correct negative: event not observed from the satellite, and also not observed from the ground. 

A large variety of scores have been defined. The following are used in H-SAF 

- Frequency BIas (FBI) 

- Probability Of Detection (POD) 

- False Alarm Rate (FAR) 

- Probability Of False Detection (POFD) 

- Fraction correct Accuracy (ACC) 

- Critical Success Index (CSI) 

- Equitable Threat Score (ETS) 

- Heidke skill score (HSS) 

- Dry-to-Wet Ratio (DWR). 

hits falsealarms forecast yes 

FBI Range: 0 to ∞. Perfect score: 1 

hits misses observed yes 

hits 

hits 

POD Range: 0 to 1. Perfect score: 1 

hits misses observed yes 

FAR falsealarms falsealarms 

hits falsealarms forecast yes 

Range: 0 to 1. Perfect score: 0 

POFD falsealarms 

falsealarms 

correctnegatives falsealarms observed no 

Range: 0 to 1. Perfect score: 0 

hits correctnegatives 

ACC Range: 0 to 1. Perfect score: 1 

total 

hits 

CSI Range: 0 to 1. Perfect score: 1 

hits misses falsealarm 

hits hits 

observed yes 

hits random 

hits misses falsealarm hitsrandom 

total 

ETS ranges from -1/3 to 1. 0 indicates no skill. Perfect score: 1. 

random 

ETS with 

(hits 

correctnegatives) (expected correct) 

N (expected correct) 

random 

HSS with 

random 

forecast yes 

1 

(ex pected correct) 

random 

(observed yes)(forecast yes) (forecast no)(observed no) 

N 

HSS ranges from -1 to 1. 0 indicates no skill. Perfect score: 1. 

false alarm correct negative observed no 

DWR Range: 0 to ∞. Perfect score: n/a. 

hits misses observed yes


3.4 Inventory of validation facilities 

In the following sections the facilities utilised in the various Institutes to perform validation of 

precipitation products are described. It is apologised that editing is not well homogenised since the 

various sections are recorded as they were contributed by the individual institutes, with minimum 

harmonisation effort in respect of length and level of detail. 

3.4.1 Facilities in Belgium (IRM) 

Ground data 

The validation results for Belgium presented in this report were obtained by comparison of the rain rates 

products with weather radar data and of the cumulated precipitation products with either cumulated 

weather radar data or rain gauge data. Table 03 summarizes the ground data used as well as the domain 

over which the validation extends. The last row has been included but refers to results to be presented in 

the report on hydrological validation. 

Table 03 - List of ground data used for precipitation products validation in Belgium 

Product Ground data Validation domain 

PR-OBS-1 MW Conical Wideumont Radar 230 km 230 km 

PR-OBS-2 MW Cross-Track Wideumont Radar 230 km 230 km 

PR-OBS-3 IR+MW Rapid Update Wideumont Radar 230 km 230 km 

PR-OBS-5 Cumulated 24h Cum. Wideumont Radar 230 km 230 km 

PR-OBS-5 Cumulated 24h SETHY Raingauges Walloon Region 

PR-OBS-5 Cumulated 24h RMI Daily Raingauges Test Catchments 

Weather Radar 

Belgium is well covered with three radars (see Fig. 10). A further radar is currently under construction 

in the coastal region. These are Doppler, C-band, single polarization radars with beam width of 1° and a 

radial resolution of 250 m. Data are available at 0.6, 0.66 and 1 km horizontal resolution for the 

Wideumont, Zaventem and Avesnois radars respectively. 

Fig. 10 - Meteorological radar in Belgium. 

In this report, only the Wideumont radar has been used. The data of this radar are controlled in three 

steps. First, a long-term verification is performed as the mean ratio between 1-month radar and gauge 

accumulation for all gauge stations at less than 120 km from the radar. The second method consists in 

fitting a second order polynomial to the mean 24 h (8 to 8 h local time) radar / gauge ratio in dB and the


range; only the stations within 120 km and where both radar and gauge values exceed 1 mm are taken 

into account. The third method is the same as the second but is performed on-line using the 90 

telemetric stations of the SETHY (Ministry of the Walloon Region). Corrected 24 h images are then 

calculated. New methods for the merging of radar and raingauge data have been recently evaluated 

(Goudenhoofdt and Delobbe 2009) 8 . 

Raingauge 

Several raingauge networks are managed in Belgium (see Fig. 11). RMI has a dense network of daily 

raingauge and an increasing network of automatic weather stations equipped with tipping bucket 

gauges. Other networks are operated by the Regional Authorities in charge of rivers. For the validation 

of the PR-OBS-5, we have used hourly data from the SETHY raingauge which are quality controlled 

daily at RMI. The daily data are gathered and checked with 1.5 to 2 month delay. These later data are 

mainly used in the hydrological validation programme. 

Fig. 11 - RMI raingauges: daily ( ) and AWS ( ). 

Fig. 12 - SETHY AWS network in Walloon Region. 

For the validation of the PR-OBS-5 cumulated rainfall product, a validation with raingauge data has 

been performed, in parallel to the radar validation. The reference data used are hourly rain gauge records 

from the SETHY (Walloon Region) network (Fig. 12). The network includes 89 automatic non-heated 

stations and 3 heated stations (in coincidence with non-heated ones). Only the non-heated stations have 

been considered, for the sake of uniformity. The data have been interpolated in onto a 5 km 5 km grid, 

following the Barnes method. The sensitivity parameter in the Barnes procedure has been set to 10 8 , 

considering the fact that the mean distance between every station and its closest neighbor is roughly 10 4 

m. The interpolation procedure is iterative. If the mean squared difference between the source field and 

the interpolated field falls below 0.01 mm h -1 , or if the improvement is below 1% between two steps, the 

procedure is stopped, otherwise it goes on for a maximum 20 iterations. The result is a series of files 

with interpolated data, one per hour. 

The quality of the interpolated data has been checked for several months in the following way: the 

interpolation is calculated taking into account all the stations except one, and the value corresponding to 

the missing station is estimated. The procedure is repeated for all the stations. A set of 89 reconstructed 

values is obtained, and compared with the measured data. The verification refers to the period from 

August to November 2008. The interpolation is first assessed in its capacity to reconstruct the rain / no 

rain field. Taking 0.01 mm h -1 as a threshold, the probability of correct rain (POD) is 0.79, the false 

reconstruction (FAR) is 0.07 and the equitable threat score (ETS) is 0.71. Then, statistical scores are 

calculated on a monthly basis. The bias ranges from 0.06 to 0.14 mm h -1 , the root mean square from 

8 Goudenhoofdt E. and L. Delobbe, 2009: “Evaluation of radar-gauge merging methods for quantitative precipitation 

estimates”. Hydrol. Earth Syst. Sci., 13, 195-203.


0.37 to 1.00 mm h -1 and the mean relative error from 0.09 to 0.19 for mean observed values of 0.50 to 

1.00 mm h -1 . 

As preliminary test for the hydrological validation of PR-OBS-5, the data of the daily raingauge stations 

have been interpolated using the Thiessen polygons method and spatially averaged over the two test 

catchments. The values obtained have been compared with the corresponding cumulated values from 

satellite. 

Miscellaneous information 

For the analysis of test cases, additional information has been used like the cloud types identified using 

the SAF-NWC tools, the expertise of weather forecasters to select and analyze the synoptic conditions, 

the SAFIR maps of lightning impacts. 

Methodology 

From a local point of view, rain rates products based on microwave sensors onboard of Low Earth Orbit 

satellites are characterized by a varying coverage and projection. To make the statistics comparables 

from one file to the other, a validation domain has been defined which is a square of 230 km 230 km 

centered on the Wideumont radar location and only the products covering entirely this common area 

have been considered. To be more precise, for every product file, a sub-set of lines and columns 

including the common square has been extracted. Then, the radar data have been up-scaled to the 

projection of the sub-set of pixels and compared with the product estimates. 

Fig. 13 - Left: Gaussian filter; right: sketch of the up-scaling procedure. The circle corresponds to the range of the 

weather radar. The square in the middle is a common area such that it is entirely included in the selected PR- 

OBS-2 files. The grey rectangle, the tilted dark grey rectangle and the black ellipse are explained in the text. 

The up-scaling of the radar data is performed taking the footprints of the microwave sensors into 

account. In Fig. 13, the Gaussian filter corresponding to the first scan position of the AMSU-B antenna 

(PR-OBS-2) is represented on the left. The filtering procedure is organized as follows (see the figure, on 

the right). First, a part of the radar image is selected (grey). Then, the radar data (0.6 km resolution are 

re-sampled onto a tilted grid (2 km resolution) where the Gaussian filter is 1% of maximum (dark 

grey). Tilting depends on the scan position and on the satellite overpass mode. Finally the Gaussian 

filter is applied. The black ellipse corresponds to half power. Additional information about the upscaling 

equations and about the tilting of the PR-OBS-2 pixels can be found in Van de Vyver and 

Roulin (2008). 

For PR-OBS-3 and PR-OBS-5, a sub-set of lines and columns has been also extracted which comprises 

the common validation area. The up-scaling has been simply performed by averaging the radar values 

included in each pixel in the SEVIRI projection. For the validation of PR-OBS-5 using raingauge data, 

the ground data have been interpolated as explained above and the comparison has been performed 

between the product estimate and the nearest interpolated grid point over a domain corresponding to the 

Walloon region in Belgium. Finally, the scores of the continuous statistics, the contingency tables and 

the probability distribution functions have been prepared on a monthly basis according to the rules 

common to all the teams involved in the precipitation products validation.


Scale Recursive Estimation 

As specific development, we have investigated an application of scale recursive estimation (SRE) to 

assimilate rainfall rates during a storm estimated from the data of two remote sensing devices. These are 

ground based weather radar and space-born microwave cross-track scanner (PR-OBS-2). Our approach 

operates directly on the data and does not require a pre-specified multi-scale model structure. We 

introduce a simple and computational efficient procedure to model the variability of the rain rate process 

in scales. The measurement noise of the radar is estimated by comparing a large number of datasets with 

rain gauge data. The noise in the microwave measurements is roughly estimated by using up-scaled 

radar data as reference. Special emphasis is placed on the specification of the multi-scale structure of 

precipitation under sparse or noisy data. The new methodology is compared with the latest SRE method 

for data fusion of multi-sensor precipitation estimates. Applications to the Belgian region show the 

relevance of the new methodology (Van de Vyver and Roulin 2009) 9 . 

9 Van de Vyver H. and E. Roulin, 2009. “Scale recursive estimation for merging precipitation data from radar and 

microwave cross-track scanners”. J. Geophys. Res., 114, D08104, doi: 10.1029/2008JD010709.


3.4.2 Facilities in Germany (BfG) 

Precipitation data 

One of the responsibilities of the Federal Institute of Hydrology (BfG) is the shipping related water level 

forecast for the river Rhine at low and middle flows. For tasks like hydrological modeling there is 

mainly a need for hourly and daily meteorological data which are provided to BfG by Germany‟s 

National Meteorological Service (Deutscher Wetterdienst DWD). 

It is intended to conduct precipitation validation activities for the territory of Germany. 

Germany has a rather dense network of raingauges and it is covered by 16 radars plus one research radar 

at Hohenpeissenberg (see Table 04, Table 05 and Fig. 14, from Bartels et al. 2004 10 ). 

Table 04 - Precipitation data available at BfG 

International) 

RW (High Resolution Calibrated 

Quantitative Composite (national) 

radar sites 

16 German 

radar sites 

4 km x 4 km 

1 hour, 

1 km x 1 km 

hourly intervals 

Near-real-time 

Table 05 - Location of the 16 meteorological radar of the DWD 

Data Number Resolution Delay Annotation 

Synoptical stations About 200 6h / 12h Near-real-time 

TTRR stations About 1000 hourly Near-real-time 

PI (Picture Composite 

European 15 min, Provided in International composite image with groundproximate 

radar reflectivity distribution 

Quantitative radar composite product from 

RADOLAN software 

Radar site Launch Model WMO No. Radar site Launch Model WMO No. 

München 1987 DWSR-88 C 10871 Rostock 1995 METEOR 360 AC 10169 

Frankfurt 1988 DWSR-88 C 10637 Ummendorf 1996 METEOR 360 AC 10356 

Hamburg 1990 DWSR-88 C 10147 Feldberg 1997 METEOR 360 AC 10908 

Berlin-Tempelhof 1991 DWSR-88 C 10384 Eisberg 1997 METEOR 360 AC 10780 

Essen 1991 DWSR-88 C 10410 Flechtdorf 1997 METEOR 360 AC 10440 

Hannover 1994 METEOR 360 AC 10338 Neuheilen-bach 1998 METEOR 360 AC 10605 

Emden 1994 METEOR 360 AC 10204 Türkheim 1998 METEOR 360 AC 10832 

Neuhaus 1994 METEOR 360 AC 10557 Dresden 2000 METEOR 360 AC 10488 

10 Bartels H. et al. / Deutscher Wetterdienst, Abteilung Hydrometeorologie, 2004: „Projekt RADOLAN - 

Routineverfahren zur Online-Aneichung der Radarniederschlagsdaten mit Hilfe von automatischen 

Bodenniederschlagsstationen (Ombrometer)“. Summary report for the project period 1997-2004.


Fig. 14 - Left panel: radar coverage in Germany as of 01/03/2007. Right panel: location of ombrometers for 

online calibration in RADOLAN; squares: hourly data provision (about 500), circles: event-based hourly data 

provision (about 800 stations): red: AMDA III, blue: aggregational network federal states (Bartels et al., 2004). 

RADOLAN 

RADOLAN (Routine procedure for an online calibration of radar precipitation data by means of 

automatic surface precipitation stations ‘ombrometers’) is a quantitative radar composite product 

provided in near-real time (via ftp) by DWD to BfG. Radar data are calibrated with hourly precipitation 

data from automatic surface precipitation stations. For a description of the radar network see 

http://www.dwd.de/de/Technik/Datengewinnung/Radarverbund/Standorte.htm . 

The process chain from the five-minute-interval radar signals to the final hourly precipitation product is 

presented in the Fig. 15. RADOLAN data of hourly precipitation (sampling period hh:51 min to 

(hh+1):50 min) have a precision of 0.1 mm/h and cover the whole territory of Germany with a spatial 

resolution of 1 km. 

Fig. 15 - Flowchart of online calibration 

RADOLAN (adapted from Bartels et al. 2004).


3.4.3 Facilities in Hungary (OMSZ) 

Ground data description (instrument characteristic and map) 

In Hungary, about 90 automatic stations work (Fig. 16), where 10-min precipitation is measured by 

tipping bucket rain gauges. This data is used to correct the accumulated precipitation radar data 

Fig. 16 - The automatic rain gauge network in Hungary. 

The main data used for validation in Hungary would be the data of meteorological radars. There are 

three C-band dual polarized Doppler weather radars operated routinely by the OMSZ-Hungarian 

Meteorological Service (see Fig. 17 and Table 06). 

Pogányvár 

Napkor 

Budapest 

Fig. 17 - Location and coverage of the three meteorological Doppler radars in Hungary. 

Table 06 - Characteristics of the three meteorological Doppler radars in Hungary 

Year of installation Location Radar type Parameters measured 

1999 Budapest Dual-polarimetric, Doppler radar Z, ZDR 

2003 Napkor Dual-polarimetric, Doppler radar Z,ZDR,KDP,ΦDP 

2004 Poganyvar Dual-polarimetric, Doppler radar Z,ZDR,KDP,ΦDP 

Ground data quality and accessibility 

Access to Hungarian radar data can be set up through contact with the responsible of the institute within 

the HSAF project. It will be provided for developers if required for case studies.


The quality of the radar measurements is influenced by several factors: 

- the accuracy of the Marshall-Palmer equation in deriving rain rates 

- the beam blockage which causes the lack of precipitation in areas behind mountains. 

The Hungarian radar data bears the consequences of both problems. The Marshall-Palmer calculations 

can result in a factor of multiplying or dividing by 2; whereas the beam blockage can result in serious 

underestimation of precipitation amounts (e.g. behind the Börzsöny mountains at the north of Budapest). 

Besides, the Hungarian radar data is filtered from WLAN signal, which is also a source of false signals 

in the radar. 

A filter to disregard signals below 7 dBz is also applied because in general, these data is not coming 

from real rain drops, but false targets. 

Ground data products 

Precipitation intensity is derived from radar reflectivity with the help of an empirical formula, the 

Marshall-Palmer equation. From the three radar images a composite image over the territory of Hungary 

is derived every 15 minutes applying the maximum method in order to make adjustments in overlapping 

regions. 

The non-corrected precipitation field can be corrected by rain gauge measurements. As recent 

researches have shown it is only possible to produce adequate precipitation fields by the correction of 

raw radar data at time scales of the order of a few hours or more, thus we do not make corrections to 15 

minutes radar data. In our institute, we only use a correction for the total precipitation over a 12 hour 

period. 

For the 3h and 6h accumulated products, we use a special method as well: we interpolate the 15-minutes 

measurements for 1-minute grid by the help of displacement vectors also measured by the radar, and 

then sum up the images which we got after the interpolation. It is more precise especially when we have 

storm cells on the radar picture, because a storm cell moves a lot during 15 minutes and thus we do not 

get continuous precipitation fields when we sum up only with 15.minutes periods. This provides 

satisfying results. However, there is still a need for rain-gauge adjustment because there are obviously 

places (behind mountains) that the radar does not see. 

The radars are corrected with rain gauge data every 12 hours. The correction method using raingauge 

data for 12 hour total precipitation consists of two kinds of corrections: the spatial correction which 

becomes dominant in the case of precipitation extended over a large area, whereas the other factor, the 

distance correction factor prevails in the case of sparse precipitation. These two factors are weighted 

according to the actual situation. The weighting factor depends on the actual effective local station 

density, and also on the variance of the differences of the bias between radar and rain gauge 

measurements. On the whole, we can say that our correction method is efficient within a radius of 100 

km from the radar. In this region, it gives a final underestimation of about 10%, while at bigger 

distances, the underestimation of precipitation fields slightly increases. Besides, we also produce 12 

hour total composite images: first the three radar data are corrected separately, and then the composite is 

made from them. The compositing technique consists of weighting the intensity of each radar at a given 

point according to the distance of the given point from the radars. 

Ground data interpolation 

No interpolation is used for the radar data. 

Institute validation methodology 

The data used for the discrimination of case studies: 

OMSZ-Hungarian Meteorological Service receives operationally the MSG data and several products 

based on satellite data are derived and sent to the weather forecasters. The SAFNWC/MSG program 

package derives cloud type product and precipitation products (convective rain rate and probability 

of precipitation) also. MSG composite images are created operationally. For the description of


meteorological environment, the Cloud Type product is the most useful as it distinguishes high-level 

and optically thick clouds from medium- and low-level clouds; and optically thin clouds. It is also 

useful for analyzing the cloud systems that have occurred at the time of satellite measurements. 

SAFIR lightning system works operationally at Hungarian Meteorological Service. We have five 

stations. The accuracy of the lightning network has been significantly increased the last two due to 

the large amount of case studies investigated during summer periods. Therefore we intend to use the 

lightning data, first of all in visual comparison as descriptor of the synoptic situation. 

The main steps of the precipitation products validation procedure are: 

Time and space alignment of the data from different sources, represented on different scales – pixel 

size of radar and satellite data is not the same. Especially in the case of microwave measurements, 

we have large footprints that need to be treated carefully. The time alignment can be solved by 

simply matching the closest data available in time to the H-SAF products. There are advanced 

methods to solve the up-scaling of ground and/or the down-scaling of satellite precipitation data. 

The OMSZ-Hungarian Meteorological Service uses the techniques discussed and proposed by the 

members of the validation group. We will contribute to the selection of the appropriate method and 

to the investigation of the accuracy of the matching method. 

The common task of the Precipitation Validation Group is the statistical validation. The up-scaling 

method has a key role in this process. We can only compare the radar, satellite, raingauge, and 

lightning if we have previously determined the coherent values from different sources. The data will 

be collected into one file containing the following information to calculate statistic values either for 

15 minutes samples, and for accumulated total precipitation information (over 3, 6, 12, 24 hours): 

- radar-derived precipitation intensity, 

- the satellite product values derived by H-SAF. 

Statistical characteristics (RMS, BIAS, etc) will be then calculated between the satellite-based 

precipitation and radar data for the different cases. These statistics will be performed for different 

periods (months, summer-winter, day-night) as well as for some satellite pass in extreme cases . 

We will also calculate multi-categorical statistics according to the classes determined by common 

agreement by the validation group and the hydrologists. We will prepare contingency tables, scatter 

plots and bias distribution functions to illustrate the performance of the precipitation products. These 

statistics will be performed for different periods (months, summer-winter, day-night) and maybe for 

some specific satellite passes. H-SAF data is going to be verified against radar data which preferably 

will be adjusted to rain gauge data. Our radar data is already gauge-adjusted in the case of 12, and 24h 

total precipitation. 

The first task to the preparation of case studies is the selection of the periods with significant rain 

amounts. This is done by the plotting of the radar and satellite rainfall estimate sums on a daily basis 

of each month. 

The second task to the preparation of case studies is the collection of all relevant data to the period 

in question. This means the collect of radar, lightning data, and of MSG images and SAFNWC 

products which correspond to the time period investigated. 

The visual comparison including both rain gauge and radar data is an inevitable tool for the 

understanding of the validation results of H-SAF products, because it: 

- provides an overall picture of the H-SAF database (resolution, territories seen, general values) 

- highlights similarities and differences in the structures and in the values 

- helps to monitor the meteorological situation as well 

To visualize the different products and to make subjective comparison we use the Hungarian Advanced 

Weather Workstation (HAWK) developed by OMSZ-Hungarian Meteorological Service which can 

interpret all kind of meteorological products together.


3.4.4 Facilities in Italy (UniFe) 

Introduction 

This report aims to set up a precipitation validation plan for Italian region. The satellite precipitation 

estimation products to be validated come from the development and operation activity. A quick 

overview of the products characteristics as request from the approved H-SAF development proposal is 

provided. Measurements from raingauge and retrievals from weather radar will be considered as 

precipitation “truth” and used to validate those products. A description of the availability of raingauge 

and radar precipitation data over Italy is provided. A validation plan at different levels is suggested. 

Satellite precipitation estimation products 

CNR-ISAC and CNMCA are in charge to develop algorithms to estimate precipitation from microwave 

(MW) and infrared (IR) sensors on board respectively of polar and geostationary satellite. The spatial 

resolution of MW precipitation product is around 15 km whereas the time resolution is about 6 hours. 

The precipitation estimation from merging/morphing MW and IR will be generated at around 5 km of 

spatial resolution and 15 minutes of time resolution. Both the products have to be validated. Both 

instantaneous and cumulated precipitation (3, 6, 12 and 24 hours) have to be validated, however, in case 

of MW precipitation estimation due to its very low time resolution only instantaneous precipitation 

estimation can be considered and therefore validated. 

Data availability and facilities 

The Dipartimento Protezione Civile (DPC) and the Centro Nazionale di Meteorologia e Climatologia 

Aeronautica (CNMCA) will provide data set to be used as precipitation “truth”. Measurements from 

1200 raingauge distributed along the Italian region are considered the main source of precipitation 

“truth”. Those data are available at 30 minutes of cumulated time. There is also the possibility to have 

some data set of raingauges with a cumulated time of 5 minutes. These data can be used to validate 

instantaneous precipitation estimation. For cumulated precipitation validation a longer cumulated time 

have to be considered. The radar network will provide precipitation estimation every 30 minutes and the 

sum of data inside longer interval is considered as cumulated radar rain. A digital elevation model of the 

Italian region is necessary to select land and sea region and also to evaluate the impact of orography on 

the precipitation data. 

For the following validation procedure the use of IDL, Fortran 90 and C++ software is foreseen. The 

first one is useful to visualise the images and therefore for a visual inspection. The second one is useful 

in case of validation of very large data set. 

Validation recommendation 

Precipitation estimation validation means to compare satellite precipitation estimation against other 

source of precipitation data assumed as “truth”, as provided by raingauge and radar, and the comparison 

procedures have to be specified. Let us to consider two rainfall maps: the first one from raingauge/radar 

data and the second one from satellite estimation, three kind of comparison are here suggested. 

VISUAL COMPARISON: the two maps are compared by visual inspection. The human eye has a natural 

and powerful ability at judging the similarity of two images. IDL software is used to produce and 

compare satellite and raingauge/radar rainfall maps, 

CONTINUOS STATISTICS COMPARISON: the rainfall rate from satellite estimation and radar/raingauge 

are provided in a continuous range and to quantify the similarity of the two images it is convenient 

to calculate some parameters considering pixel by pixel. The suggested parameters are: Mean Error 

(ME, or bias), Root Mean Square Error (RMSE) and correlation coefficient (CC). 

CATEGORICAL STATISTICS COMPARISON: the rainfall rate from satellite estimation and radar/raingauge 

data have to be organized in classes of precipitation. At the first step by fixing a rainfall threshold of 

0.1 mm/h only two classes of precipitation (rain and no-rain) can be considered. Then the 

precipitation classes can be those indicated in Fig. 08 and Fig. 09. Once the continuous data are 

organized in classes the contingency table have to be built. Then some statistical parameters have to


be calculated to quantify the similarity of two images. In case of two classes of precipitation the 

suggested parameters are: FAR, POD, BIAS, ETS and HSS. In case of more than two classes the 

useful parameters are: HSS and correlation coefficient. 

For a reliable validation plan an ensemble of cases (each of them consisting in estimated and “truth‟ 

maps) have to be considered. The larger the ensemble is the more reliable the validation is. The visual 

comparison for several pairs of images become not easy and therefore only the 2 and 3 procedures are 

suggested. However it does worth to mention that there are two ways to carry out those procedures: a) 

all the pixels from the ensemble of cases are considered together. We obtain an ensemble of pixels of 

satellite precipitation estimation and an ensemble of pixels of precipitation “truth”. The two ensemble 

are compared by calculating the previously indicated parameters. b) for each case the comparison is 

carried out by calculating the above statistics parameters. The average values of those statistics 

parameters are calculated over the ensemble of cases. 

It is also suggested to carry out the validation for two different areas: sea and land. This because the 

precipitation “truth” over the land is mainly based on raingauges whereas for the sea only radar 

estimation are available. A further distinction will be made considering orography, by using a digital 

elevation model (e.g. GTOPO30 by the U.S. Geological Survey). 

Validation: complete plan 

The precipitation “truth” data set is built for a complete year at 15 minutes of time resolution. 

Instantaneous rain validation (MW algorithm): all the polar satellite overpasses over Italy during the 

considered year can be collected and used to generate the corresponding MW-based precipitation 

estimation. All of them can be validated. MW+IR algorithm: for each time of the day and for each 

day of the year it is possible to validate the MW+IR-based precipitation estimation. In this case the 

variability of performance along the day and along the year can be assessed. Moreover the 

degradation of MW+IR performance far from the MW overpass can be evaluated. 

Cumulated rain validation (MW+IR algorithm): cumulated estimated rain at 3, 6, 12 and 24 hours 

intervals can be validated by using the corresponding cumulated “truth” rain. The variability of 

performance along the year can be assessed as well. 

Validation: minimal plan 

The precipitation “truth” data set is built for summer and winter at around 12 and 24 UTC. 

Instantaneous rain validation (MW algorithm): all the polar satellite overpasses over Italy in that 

period can be collected and used to generate the corresponding MW-based precipitation estimation. 

All of them can be validated. 

Instantaneous rain validation (MW+IR algorithm): for 12 and 24 UTC times and for each day of 

summer and winter it is possible to validate the MW+IR-based precipitation estimation. In this case 

it is expected to get information about the maximum variability of performance. 

Cumulated rain validation (MW+IR algorithm): cumulated estimated rain at 3,6,12 and 24 hours 

intervals for summer and winter can be validated by using the corresponding cumulated “truth” rain. 

Validation: specific plan 

The precipitation “truth” data set is built for particular region of Italy and/or for particular month and/or 

particular time of the day. 

Instantaneous rain validation (MW algorithm): all the polar satellite overpasses over that region and 

in that period can be collected and used to generate the corresponding MW-based precipitation 

estimation. All of them can be validated. MW+IR algorithm: for that region and that period it is 

possible to validate the MW+IR-based precipitation estimation. 

Cumulated rain validation (MW+IR algorithm): cumulated estimated rain at 3, 6, 12 and 24 hours 

intervals for that region and that period can be validated by using the corresponding cumulated 

“truth” rain.


3.4.5 Facilities in Poland (IMWM) 

Precipitation data measurements and observations in Poland 

The validation of the H-SAF precipitation products will be carried out in the IMWM on the base of the 

ground measurements networks that include: 

Telemetry network of automatic rain gauges. 

Network of standard rain gauges. 

Pluviograph network. 

Meteorological radars. 

The telemetry network encompasses 430 posts with automatic rain-gauges (2 instruments at each post). 

The posts are located all over the whole territory of Poland, however the network is more dense in the 

Southern Poland where the flood danger is very high. The measurements are available in the near-real 

time and its frequency may be configured at the telemetric network. In standard operational mode the 

rainfall and water level data are provided every 10 minutes from the post in the Southern Poland and 

every 1 hour from the other ones. The quality of the data is checked on the level of processing in 

operational mode. 

The network of standard rain gauges consists of 61 synoptic stations and 210 climatological stations, all 

equipped with rain-gauges operated by observers as well as 1027 rain-gauge posts. The network is 

operational and the data are available in real time from the stations situated in the Southern Poland and 

off-line from other parts of country. The synoptic stations provide the standard, 6 hour cumulative 

values. The data from other posts are available once per day at normal mode and every 3 hours during a 

flood. The quality of the data is checked on the level of processing in the annual mode. 

The pluviograph network includes 102 stations located all over the whole country. The network status is 

historical. The data are collected only during the period between the 1 st of May and the 31 st of October 

with 10 minute time resolution. The quality of the data is checked on the level of processing in the 

annual mode. 

The meteorological radar network consists of 8 Doppler radars located all over the country. The data are 

available in the real time with 10 minute resolution for analysis and 15 minute resolution for the 

forecast. Generally, radar measurements are processed calibrated through/in NIMROD system. 

Quality and Accessibility 

The data from telemetric posts, rain gauges and synoptic stations located in the Southern Poland are 

operationally collected in the database of the System of Hydrology SH located in the Krakow Branch of 

IMWM. The data from all standard measuring stations and post are collected in the central data base of 

the IMWM in Warsaw. The quality checked data are available from this data base with at least half of a 

year delay. The radar measurements are collected in the Radar Centre of IMWM in Warsaw. 

The databases available in the IMWM have been check in order to estimate their usefulness for H-SAF 

purposes. For data collection, the database existing in the frame of the System of Hydrology will be 

used. However, the present database capacity does not allow introducing new users. Therefore it has 

been decided that, before the further database development, the data required for H-SAF will be 

archived separately in order to simplify their use in the future. 

Validation 

The validation of the precipitation products will be carried out in the Satellite Research Department of 

IMWM in Krakow. Due to its characteristics and availability of quality checked measurements, the 10 

minute data from the telemetric network will be used for the validation. It is assumed The standard 

observations will play supporting role especially in long term analysis. 

The validation process/chain will be performed for satellite projection and include two parts. Firstly, the 

analysis, common for all valiadation partners, will be carried out. In the frame of this part of validation 

the continuous statistical analysis for all data set and separately for rainfall classes (convective,


stratiform) and different seasons will be performed. The following characteristics are to be obtained: 

correlation coefficient, STD, ME, root mean square error. For merging between satellite and ground 

measurements, the technique proposed by the validation group will be applied. 

Next, the categorical statistical analysis will be performed both for the classes proposed by the 

validation group and the classes agreed with Polish hydrologist. 

In the second part of the validation it is assumed that the satellite derived cumulative precipitation field 

will be compared with the one obtained on the base of the ground measurements. In the analysis the GIS 

techniques will be used. The works on the spatialisation methods for precipitation are being developed 

in the Krakow Branch of IMWM; 

The use of the radar data in validation heavily depends on the scheme of the Radar Operational Centre 

team. They may provide the unique methodology of satellite-radar comparison but there are still issues o 

be decided.


3.4.6 Facilities in Slovakia (SHMÚ) 

Instruments 

Slovak Hydrometeorological Institute (SHMÚ) has a large network of the ground stations able to 

measure rainfall (see Fig. 18). 684 of these raingauges will be used for validation purpose during this 

project covering the complete territory of Slovak Republic. 98 gauges are used operationally and 586 

are for research purpose (climatology). It is expected that the number of operational gauges will increase 

during the project. Data are available in 1 min, 10 min, 1, 6, 12, 24 hours interval depending on the 

gauge type. SHMÚ is performing the offline automatic and manual quality check. 

Fig. 18 - Map of SHMÚ raingauge stations: green – operational (98) , blue – climatological (586), red - hydrological stations 

in H-SAF selected test basins (37). White points show regular grid of experimental NOAA Snow water equivalent data. 

Radar data are planed to be used in this project too. The Slovak meteorological radar network consists 

of 2 meteorological radars (see Fig. 19). One is situated at Maly Javornik near city Bratislava and 

second is on top of Kojsovska hola close to the city Kosice. Both are Doppler, C-band radars with the 

beam width 1 degree and the radial resolution from 256 m (operational). The newer radar at Kojsovska 

hola is able to measure the dual polarization variables (non operational). The radar measured 

precipitation intensity is available for the whole radar network every 15 minutes, cumulative 

precipitation is done for a period of 1, 3, 6 and 24 hours. There is no quality check applied to radar data 

operationally. The radar rainfall intensity and the radar cumulative precipitation are not yet corrected or 

adjusted to the gauges observations, it will be a part of this project. 

Fig. 19 - Example of Slovak radar network coverage - left circle corresponds to radar site Malý Javorník, right one 

corresponds to Kojšovská hoľa.


Methods 

Satellite data are usually the raster data with defined projection and pixel size/resolution. Radar data are 

also raster data, but usually in a different projection and resolution from satellite data. Therefore the first 

step is to transform both data to the common projection and pixel resolution map optimal for data 

comparison. SHMÚ has good experiences how to perform necessary re-projections operationally. 

Raingauge data are point data, which can be mapped by means of 2D or 3D interpolation which use the 

method of regularised splain with tension. When 3D interpolation is used, orography is reflected in the 

outputs. This method is generally called kriging. In this way, the original point data are transformed to 

the raster file comparable with satellite and radar data (see Fig. 20). 

Fig. 20 - Example of 5-days cumulative precipitation constructed from raingauge measurements by means of 3D 

extrapolation method. 

Radar data should be validated and adjusted before being used for the validation of other products. 

SHMÚ has made calibration study on radar site Maly Javornik based on regression between set of 

raingauges and radar 24-hour precipitation products in 150 km radar range few years ago. The 

corrections on distance from radar and partial beam blockage by terrain were subject of this study. We 

plan to make similar study for the whole territory of Slovakia. Output should be used for the operational 

calibration and quality control of radar precipitation measurements. We are also planning to test method 

for precipitation estimation based on polarisation techniques for new radar in eastern part of Slovakia. 

Quality control of radar data will be based on NWC SAF products like cloud types classification and 

products containing information about water content in vertical column. 

SHMÚ is planning to use three methods to perform final validation of satellite precipitation products: 

Direct comparison of satellite data with point raingauge measurements, calculation of standard 

statistical parameters RMSE, ME and other. 

Comparison of satellite raster data with precipitation raster based data (interpolated raingauges, 

corrected radar precipitation fields,…) and with calculation of standard statistical parameters.


Dividing of precipitation amounts to the defined intervals (classes) according to precipitation 

intensity and to evaluate the following parameters: probability of detection (POD), equitable threat 

score (ETS), false alarm ratio (FAR) and critical success index (CSI). 

Hydrological validation 

Slovak Hydrometeorological Institute (SHMÚ) has a large hydrological network of the ground stations 

able to measure water level (see Fig. 21). There are 212 stations measuring in online mode. Some of 

them (on the Myjava, Kysuca, Hron and Top‟la river) will be used as input data for H-SAF application. 

The data from these stations are available in 15 min interval. SHMÚ is performing the offline automatic 

and manual quality check. 

Fig. 21 - Map of SHMÚ hydrological stations: red - hydrological stations in H-SAF selected test basins (34). 

Hydrological model HRON are conceptual (HBV type) model, semi-distributed with altitude zones 

(Fig. 22). 

Fig. 22 - General hydrological model of HBV type. 

(pictures are from http://www.smhi.se/en/index.htm)


Mask of selected area. 

Radar measurement sector with 

selected area border. 

Radar measurement in the 

selected area only. 

Fig. 23 - Example of pick up pixels from radar measurement for selected area. 

Model input data are discharge and average data of precipitation 

and temperature. We calculate average value from ground stations 

and from radar measures. Meteorological forecasted data are taken 

from weather prediction model ALADIN. 

Average catchment precipitation value from ground station is 

calculated using Thiessen polygons method and arithmetic mean 

with different station weights assigned a priori. Average 

temperature value is calculated as weighted arithmetic mean. 

Catchment average precipitation value of 1-hour precipitation 

forecast from radar product is calculated as average value of pixels 

from the selected area (mask of subcatchment) (see Fig. 24). 

Model ALADIN supplies precipitation and temperature forecasts to 

hydrological model HRON. Since the resolution of forecast is too 

Fig. 24 - Example of separated ALADIN 

low for selected areas, the large pixel is separated into smaller forecast pixel to smaller pixels. 

pixels for better average value results (see Fig. 24). SHMÚ is 

planning to use satellite precipitation products in forecasting hydrological models in similar way.


3.4.7 Facilities in Turkey (ITU) 

Locations and instruments 

For the validation of precipitation products, a first set of locations were selected as Susurluk and 

Western Black Sea catchments which represent different physiographic and climatic conditions of 

Turkey (see Fig. 25). The Susurluk catchment is influenced by Mediterranean type of climate with mildwet 

winters and hot-dry summers. The basin is covered by forest lands together with fertile alluvial 

plains. On the other hand, the Western Black-Sea catchment is characterized by high rainfall dominating 

almost two thirds of the year and humid summers. The catchment is covered by highly dense forests and 

agricultural lands. The catchment has mountains with low elevations rarely exceeding 1,500 m. 

Fig. 25 - The Susurluk and Western Black Sea catchments selected for the precipitation product validation in Turkey.


Validation of precipitation products is based mainly on the comparison with the AWOS and weather 

radar data measurements. Turkey has an extensive coverage of meteorological observations, including 

automated weather observing stations (AWOS), synoptic and climatological stations, and 4 radars. At 

present nearly 450 meteorological, 60 airport, 180 climate, and 7 upper air (radiosonde) stations are in 

operation. The meteorological observation network has been strengthened by installation of AWOS in 

recent years, and in the near future almost entire country will be covered by an AWOS network to 

replace the conventional network. 

As shown in Fig. 26, the Western Black Sea catchment has 23 AWOS and 7 synoptic stations in 

addition to radar coverage. Similarly, the Susurluk basin has 17 AWOS and 3 synoptic stations and 

radar coverage. 

Fig. 26 - Network of Meteorological stations in Susurluk (on the left) and Western Black Sea (on the right) catchments. 

Starting from January 2009, the validation area has been extended to the middle and western Turkey. 

193 Automated Weather Observation Station (AWOS) located in the western part of Turkey are used for 

the validation of the precipitation products. The locations of the AWOS sites are shown in Fig. 27. 

Fig. 27 - Position of 193 AWOS sites used for ground truth for the precipitation product validation in western Turkey. 

Validation Methodology 

Due to the time and space structure of precipitation and sampling characteristics of both the 

precipitation products and observations used for validation, care has to be taken to bring data to a 

comparable level. At a given place, precipitation occurs intermittently and at highly fluctuating rates. 

Over space, precipitation is distributed with a high variability, in cells of high intensity nested in larger 

area with lower rain rate. Aimed at observing this complex phenomenon, the satellite-based products are 

defined with a spatial resolution of several kilometres and with different sampling rate and accumulation


time. On the other hand, reference observations against which the products are going to be validated are 

also characterized by their own spatial resolution ranging from point information measured on raingauge 

networks to grids with cells of several hundreds of meters to several kilometres for weather radar. 

Furthermore, none of these reference observations are without error. 

Each precipitation product within the H-SAF project represents a foot print geometry. Among these, 

H01 and H02 products represent an elliptical geometry while H03 and H05 have a rectangular 

geometry. On the other hand, the ground observation (rain-gauge) network consists of point 

observations. The main problem in the precipitation product cal/val activities occurs in the dimension 

disagreement between the product space (area) and the ground observation space (point). To be able to 

compare both cases, either area to point (product to site) or point to area (site to product) procedure has 

to be defined. However, the first alternative seems easier. The basic assumption in such an approach is 

that the product value is homogenous within the product footprint. Fig. 28 presents satellite foot print 

(IFOV) centres of the H02 product, an elliptical footprint for the corresponding centre (area within the 

yellow dots) and AWOS ground observation sites. The comparison statistic can be performed by 

considering just the sites in the footprint area. Although this approach is reasonable on the average but it 

is less useful in spatial precipitation variability representation. The comparison is not possible when no 

site is available within the footprint area. 

Fig. 28 - H02 product footprint centres with a sample footprint area as well as the AWOS ground observation sites. 

Alternatively, the point to area approach is more appealing for the realistic comparison of the 

precipitation product and the ground observation. This approach is simply based on the determination of 

the true precipitation field underneath the product footprint area. To do so, the footprint area is meshed 

and precipitation amounts are estimated at each grid point by using the precipitation observations at the 

neighbouring AWOS sites as shown in Fig. 29. A 3x3 km grid spacing is considered for the products 

with elliptical geometry while 2x2 km spacing is considered for the products with rectangular geometry. 

At each grid point, the precipitation amount is estimated by, 

Z 

m 

n 

i 1 

n 

W ( r 

i 1 

i, 

m 

W ( r 

) Z 

i, 

m 

) 

i 

where Z m is the estimated value and W(r i,m ) is the spatially varying weighting function between the i-th 

site and the grid point m. 

(1)


Fig. 29 - Meshed structure of the sample H02 product footprint. 

Determination of the W(r i,m ) weighting function in Equation 1 is crucial. In open literature, various 

approaches are proposed for determining this function. For instance, Thiebaux and Pedder 1987 11 

suggested weightings in general as, 

2 2 

R ri 

, m 

forri 

m 

R 

2 2 

, 

W ( r ) R r 

(2) 

i, 

m 

0 

forr 

i, 

m 

i, 

m 

R 

where R is the radius of influence, r is the distance from centre to the point and is a power parameter 

that reflects the curvature of the weighting function. Another form of geometrical weighting function 

was proposed by Barnes 1964 12 as, 

ri 

, m 

W ( ri 

, m) 

exp 4 

(3) 

R 

Unfortunately none of these functions are observation dependent but suggested on the basis of the 

logical and geometrical conceptualizations only. They are based only on the configuration, i.e. geometry 

of the measurement stations and do not take into consideration the natural variability of the 

meteorological phenomenon concerned. In addition, the weighting functions are always the same from 

site to site and time to time. However, in reality, it is expected that the weights should reflect to a certain 

extent the regional and temporal dependence behaviour of the phenomenon concerned. 

For the validation activities, the point cumulative semi-variogram technique proposed by Şen and Habib 

1998 13 is used to determine the spatially varying weighting functions. In this approach, the weightings 

not only vary from site to site, but also from time to time since the observed data is used. In this way, 

the spatial and temporal variability of the parameter is introduced more realistically to the validation 

activity. 

11 Thiebaux H.J. and M.A. Pedder, 1987: “Spatial objective analysis”. Academic Press, 299 pp. 

12 Barnes S.L., 1964: “A technique for maximizing details in numerical weather map analysis”. J. App. Meteor., 3, 

pp.396-409. 

13 Şen Z. and Z. Habib, 1998: “Point cumulative semivariogram of areal precipitation in mountainous regions”. Journal 

of Hydrology 205 (1–2), 81–91.


4. Validation of the product release as at the end of the Development Phase 

4.1 Introduction 

This Chapter collects the results of the validation experiments as at the end of the H-SAF Development 

Phase. The validation is performed on the product release in force at the time of the Operations 

Readiness Review (ORR). The organisation was as follows: 

The validation period is 1 st January 2009 to 31 March 2010, in order to cover all seasons with 

margins. Granularity: one month. 

The results of the previous validation cycles, including case studies, are recorded in the Appendix to 

this PVR-03, reproduced from the so-called “REP-3 (H-SAF Products Validation Report)”, last 

issue dated 28 February 2010. The Appendix is a simple transcription of the experiments, thus it is 

characterised by a low level of editing, as it was appropriate to a project-internal working document. 

REP-3 was split in volumes, each H-SAF product being addressed by one volume. PR-OBS-3 was 

addressed by REP-3/04 (Results of validation activities for product PR-OBS-3). 

Year 2009 was addressed by REP-3/04, but the product release was not the current one. The current 

release, v1.4, was activated in March 2010, and data were re-processed back to 1 st January 2009. 

This Chapter 4 is structured by Country / Team, one section each. Each section records the main 

statistical scores across the January 2009 - March 2010 period, and provides comments, possibly 

supported by further analysis material. The next Chapter 5 (Overview of findings) provides comparative 

features among the results from the various Countries / Teams, so that the User of the product is 

informed of the variability of the performances with climatological and morphological conditions, as 

well as with seasonal effects. 

Unlike the previous validation campaigns, that placed priority to supporting the product development, 

thus made large use of case studies, this campaign focused on the objective of the ORR of assessing the 

degree of compliance of the product quality with the User requirements. For product PR-OBS-3 the 

User requirements are recorded in Table 07. 14 As a matter of fact, the original requirements do not 

specify figures for precipitation rate < 1 mm/h, since the sensing principle for PR-OBS-3 is poorly 

applicable to light (non-convective) precipitation. However, since the Institutes have performed 

validation also for light precipitation, we have added requirement figures for rate < 1 mm/h by simply 

extrapolating the values prescribed for more intense rates. 

Table 07 - Accuracy requirements for product PR-OBS-3 [RMSE (%)] 

Precipitation range threshold target optimal Remarks 

> 10 mm/h 80 40 20 

1-10 mm/h 160 80 40 

< 1 mm/h 320 120 80 This entry, originally N/A, has been extrapolated 

This formulation of the requirement implies that the main score to be evaluated is the Root Mean Square 

Error that, since depends on the precipitation type, or intensity, has to be evaluated with regard to the 

actual rate, thus RMSE (%). Supportive scores are: the ordinary RMSE (mm/h), the Mean Error (or 

bias, ME), and the Standard Deviation (SD). In addition, the Correlation Coefficient (CC), the 

Probability Of Detection (POD), the False Alarm Rate (FAR) and the Critical Success Index (CSI, 

necessary to compare POD - FAR coupled scores), also are reported, although rather unstable quantities 

for this type of geophysical parameter that does not at all comply with Gaussian characteristics. 

Each Country / Team should conclude its Section by listing the main features of the product, function of 

whatever the Team considers as a significant change of conditions associated to change of performance. 

The purpose is to characterise the applicability of the product for a correct use, especially in hydrology. 

14 There is evidence that the user requirements for precipitation observation from space adopted by authoritative bodies 

(WMO, EUMETSAT, the GPM planning board) are overstated. However, currently another reference is not available. 

The situation will be re-assessed during CDOP-1.


4.2 Validation in Belgium (IRM) 

The facilities available to IRM, and the methodology adopted for validation, are described in section 

3.4.1. The ground truth is provided by meteorological radar. 

Table 08 reports the results of the final validation campaign of the Development Phase for PR-OBS-3. 

Table 08 - Summary results of PR-OBS-3 validation in Belgium by IMR 

H03 v1.4 Belgium Land Validation period: 1 st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h 

Radar Class Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar 

Number of > 10 mm/h 593 224 195 3603 7660 11774 23062 7342 3632 4576 1850 372 198 458 3535 

comparisons 1-10 mm/h 78350 89297 123601 214815 193171 190032 238585 99197 97072 113440 354361 202536 61870 143997 165356 

(ground obs.) < 1 mm/h 136377 338700 413973 311765 235281 203011 227087 103687 105080 175380 525434 500719 268990 435240 289148 

> 10 mm/h -12.4 -12.9 -13.3 -15.3 -20.6 -19.0 -20.1 -20.4 -14.4 -17.2 -12.9 -11.9 -12.5 -12.4 -13.8 

ME (mm/h) 1-10 mm/h -2.33 -1.85 -1.82 -2.23 -2.41 -2.47 -2.85 -2.80 -2.48 -1.60 -2.07 -1.90 -2.23 -2.01 -2.29 

< 1 mm/h -0.48 -0.47 -0.49 -0.44 -0.37 -0.24 -0.33 -0.38 -0.29 -0.03 -0.39 -0.46 -0.41 -0.48 -0.47 

> 10 mm/h 2.88 3.93 3.60 7.54 18.8 12.0 14.9 15.3 6.48 10.4 4.17 2.01 3.81 2.82 4.56 

SD (mm/h) 1-10 mm/h 1.52 1.07 0.95 1.51 1.75 1.90 2.08 1.97 1.92 1.74 1.36 1.16 1.41 1.26 1.62 

< 1 mm/h 0.21 0.20 0.20 0.21 0.21 0.21 0.21 0.21 0.21 0.21 0.21 0.20 0.18 0.20 0.21 

> 10 mm/h 12.8 13.5 13.7 17.0 27.9 22.5 25.0 25.5 15.8 20.1 13.5 12.1 13.0 12.7 14.5 

RMSE (mm/h) 1-10 mm/h 2.79 2.14 2.05 2.71 3.03 3.29 3.62 3.47 3.21 4.25 2.53 2.26 2.64 2.37 2.81 

< 1 mm/h 0.55 0.52 0.54 0.59 0.72 0.95 0.81 0.66 0.73 2.11 0.66 0.57 0.49 0.53 0.56 

> 10 mm/h 100 98 100 100 99 98 99 99 97 99 99 99 100 100 99 

RMSE (%) 1-10 mm/h 99 98 100 97 96 100 98 97 93 207 96 96 99 99 97 

< 1 mm/h 103 100 99 112 154 208 173 133 155 457 132 108 103 99 104 

> 10 mm/h 0.02 0.30 - 0.04 0.06 0.00 0.03 0.04 0.11 0.08 0.02 0.07 0.07 0.05 0.02 

CC 1-10 mm/h 0.03 0.12 0.04 0.02 0.06 0.00 0.01 0.00 0.05 0.07 0.00 0.04 0.01 0.01 0.03 

< 1 mm/h 0.01 0.03 0.02 0.00 0.01 0.02 0.02 0.04 0.04 0.06 0.03 0.04 0.08 0.04 0.03 

POD ≥ 0.25 mm/h 0.04 0.03 0.01 0.11 0.15 0.20 0.15 0.14 0.23 0.19 0.14 0.07 0.06 0.02 0.08 

FAR ≥ 0.25 mm/h 0.94 0.92 0.89 0.84 0.86 0.79 0.86 0.90 0.80 0.79 0.76 0.73 0.83 0.84 0.80 

CSI ≥ 0.25 mm/h 0.02 0.02 0.01 0.07 0.08 0.11 0.08 0.06 0.12 0.11 0.10 0.06 0.05 0.02 0.06 

The monthly values of the Mean Error (ME) and of the Root Mean Square Error (RMSE) are shown in 

Fig. 30 and Fig. 31, respectively (all classes are merged). The ME shows an overall large 

underestimation with a seasonal dependence. The ME ranges from –0.8 mm h -1 in winter to –2.5 mm h -1 

in summer. It is worth noting that the ME of an earlier version of the product ranged from –0.1 mm h -1 

in winter to +0.5 mm h -1 in summer (See REP-3.04). The RMSE ranges from 1 mm h -1 to 6 mm h -1 with 

largest values during summer whereas the older version had a maximum monthly RMSE of 3 mm h -1 . 

However, during summer, the Probability of Detection (POD) is better than during winter.


1,00 

PR-OBS-3 

0,00 

-1,00 

-2,00 

-3,00 

200901 200903 200905 200907 200909 200911 201001 201003 

Fig. 30 - Mean Error of PR-OBS-3 (monthly values over Belgium in mm h -1 ). 

7,00 


6,00 

5,00 

4,00 

3,00 

2,00 

1,00 

0,00 

200901 200903 200905 200907 200909 200911 201001 201003 

Fig. 31 - Root Mean Square Error of PR-OBS-3 (monthly values over Belgium in mm h -1 ).


4.3 Validation in Germany (BfG) 

The facilities available to BfG, and the methodology adopted for validation, are described in section 



Table 09 - Summary results of PR-OBS-3 validation in Germany by BfG 

H03 v1.4 Germany Land . Validation period: 1 st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h 


Number of > 10 mm/h 773 844 2018 6312 45836 35483 90632 42970 8651 10309 1652 1054 1481 1364 7209 


(ground obs.) < 1 mm/h 1452957 3791686 4752554 1316509 3047639 2967116 3195447 1553835 1615953 3225417 4449674 4496969 3028325 3240445 3111755 

> 10 mm/h -29.9 -33.9 -34.8 -14.8 -9.8 -13.3 -13.8 -14.2 -12.2 -11.8 -14.1 -14.0 -19.8 -16.0 -11.9 

ME (mm/h) 1-10 mm/h -1.84 -1.59 -1.69 -2.00 -1.38 -1.96 -2.01 -2.26 -1.98 -1.63 -1.65 -1.71 -1.61 -1.66 -1.82 

< 1 mm/h -0.43 -0.49 -0.51 -0.37 -0.15 -0.24 -0.17 -0.24 -0.32 -0.19 -0.39 -0.47 -0.47 -0.48 -0.44 

> 10 mm/h 45.0 44.7 38.3 19.3 9.75 5.82 8.43 6.88 4.29 3.89 6.65 8.39 13.4 12.0 4.36 

SD (mm/h) 1-10 mm/h 1.14 0.81 0.87 1.57 3.20 1.85 2.11 2.05 1.55 2.53 1.13 0.98 0.83 0.89 1.26 

< 1 mm/h 0.38 0.25 0.27 0.54 1.42 0.83 0.99 0.86 0.64 1.84 0.49 0.35 0.24 0.26 0.41 

> 10 mm/h 54.0 56.1 51.7 24.3 13.8 14.6 16.2 15.7 12.9 12.5 15.6 16.3 23.9 20.0 12.7 

RMSE (mm/h) 1-10 mm/h 2.16 1.78 1.90 2.54 3.49 2.69 2.91 3.05 2.52 3.01 2.01 1.97 1.81 1.89 2.22 

< 1 mm/h 0.57 0.55 0.58 0.66 1.43 0.87 1.01 0.90 0.71 1.85 0.63 0.59 0.53 0.55 0.60 

> 10 mm/h 100 100 100 94 89 94 90 94 92 89 96 97 100 100 88 

RMSE (%) 1-10 mm/h 96 97 99 94 142 96 97 97 93 156 93 96 99 98 95 

< 1 mm/h 113 101 103 133 306 185 219 193 151 403 121 108 100 103 116 

> 10 mm/h -0.13 0.09 -0.06 -0.06 0.18 -0.01 0.12 -0.02 0.08 0.04 -0.02 -0.15 -0.05 -0.09 0.06 

CC 1-10 mm/h 0.03 0.04 0.05 0.15 0.20 0.13 0.15 0.10 0.12 0.06 0.11 0.10 -0.06 0.03 0.28 

< 1 mm/h 0.01 0.05 0.02 0.04 0.05 0.05 0.05 0.05 0.04 0.03 0.05 0.03 0.03 0.01 0.04 

POD ≥ 0.25 mm/h 0.11 0.04 0.02 0.20 0.27 0.27 0.32 0.26 0.25 0.18 0.19 0.09 0.04 0.04 0.10 

FAR ≥ 0.25 mm/h 0.81 0.75 0.76 0.81 0.65 0.66 0.63 0.69 0.71 0.69 0.65 0.61 0.80 0.82 0.67 

CSI ≥ 0.25 mm/h 0.07 0.04 0.02 0.11 0.18 0.18 0.21 0.16 0.16 0.13 0.14 0.08 0.03 0.03 0.09 

Fig. 32, Fig. 33 and Fig. 34 record the time evolution of ME, SD, POD, FAR and CSI. 

Fig. 32 - Time evolution of Mean Error and Standard Deviation for PR-OBS-3.


Fig. 33 - Same as Fig. 32 with stretched vertical scale. 

Fig. 34 - Time evolution of Probability Of Detection, False Alarm Rate and Critical Success Index for PR-OBS-3. 

The following conclusions may be drawn: 

The performance of PR-OBS-3 showed a seasonal pattern in the low and mid precipitation class 

where ME, SD and RMSE give higher errors in summer but better performance in categorical scores 

in summer and vice versa in winter. Departing from that pattern was the > 10 mm/h precipitation 

class, that showed higher errors in winter where high precipitation was not captured (ME ≤ - 30 

mm/h in January - March 2009). 

For mean error (ME), standard deviation (SD) and root mean squared error (RMSE) the dynamic 

range of the month-to-month variation is dominated by the precipitation class definition. Therefore, 

all these scores show strongest amplitude in the > 10 mm/h class. Normalized RMSE, though, 

showed extreme values in the < 1 mm/h class.


Mean error was negative all over the studied period with a total average of -0.33 (-1.81, -18.1) for 

2009 for the low (mid, high) precipitation class. 

RMSE % was not as seasonal as the other scores but exhibited two distinguished maxima (least 

negative) in the low and mid precipitation classes in May and October 2009. 

The average RMSE % for 2009 for the low (mid, high) precipitation class was 178 % (105 %, 95 

%). 

The probability of detection (POD) ranged from 0.02 to 0.32 with an average of 0.18 for 2009. False 

alarm rate ranged from 0.61 to 0.81 with an average of 0.71 resulting in a low CSI of 0.12 in 2009 

with an absolute minimum of 0.02 in March 2009.


4.4 Validation in Hungary (OMSZ) 

The facilities available to OMSZ, and the methodology adopted for validation, are described in section 



Table 10 - Summary results of PR-OBS-3 validation in Hungary by OMSZ 

H03 v1.4 Hungary Land Validation period: 1 st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h 


Number of > 10 mm/h 267 175 356 2833 14601 36757 31603 49703 8724 6749 1726 2991 517 668 1459 


(ground obs.) < 1 mm/h 1283260 1157375 1634341 820052 1394397 2091680 1015576 1008952 522458 1493153 1576114 1619110 1263888 1399820 842200 

> 10 mm/h -14.3 -33.3 -12.5 -14.1 -13.6 -16.5 -16.9 -18.4 -17.1 -14.4 -52.6 -13.9 -11.3 -10.6 -14.1 

ME (mm/h) 1-10 mm/h -1.11 -1.65 -1.74 -1.37 -0.33 -1.14 -1.33 -1.60 -1.78 -1.53 -1.44 -1.70 -1.43 -1.59 -1.78 

< 1 mm/h -0.12 -0.40 -0.40 -0.19 0.56 0.31 0.38 0.19 -0.16 -0.12 -0.18 -0.38 -0.28 -0.27 -0.32 

> 10 mm/h 24.3 104 3.98 7.75 17.5 14.0 13.4 14.3 11.4 7.91 112 8.68 2.18 2.68 7.99 

SD (mm/h) 1-10 mm/h 1.37 0.97 1.09 1.94 3.61 2.35 2.58 2.50 2.60 1.81 1.34 1.43 1.38 1.20 1.44 

< 1 mm/h 0.84 0.34 0.42 0.96 2.50 1.49 1.68 1.50 1.52 0.99 0.68 0.46 0.65 0.57 0.50 

> 10 mm/h 28.2 109 13.1 16.1 22.2 21.6 21.6 23.3 20.5 16.4 123 16.4 11.5 10.9 16.2 

RMSE (mm/h) 1-10 mm/h 1.76 1.91 2.05 2.37 3.63 2.62 2.91 2.97 3.15 2.37 1.97 2.22 1.99 1.99 2.29 

< 1 mm/h 0.85 0.52 0.58 0.98 2.55 1.52 1.72 1.51 1.53 1.00 0.70 0.59 0.71 0.63 0.59 

> 10 mm/h 96 100 100 89 81 88 90 92 96 95 96 93 96 86 96 

RMSE (%) 1-10 mm/h 100 98 97 112 201 114 123 110 138 100 87 98 95 93 95 

< 1 mm/h 198 114 126 212 564 339 376 330 347 224 158 122 162 149 130 

> 10 mm/h -0.08 -0.08 -0.08 0.07 0.02 0.00 0.02 0.03 -0.02 -0.07 -0.14 -0.02 -0.10 0.10 -0.10 

CC 1-10 mm/h -0.01 0.01 -0.04 0.11 0.15 0.14 0.07 0.10 0.06 0.04 0.06 0.18 0.10 0.13 0.11 

< 1 mm/h 0.10 0.02 0.02 0.09 0.08 0.08 0.08 0.07 0.03 0.07 0.07 0.05 0.07 0.03 0.05 

POD ≥ 0.25 mm/h 0.26 0.06 0.07 0.21 0.41 0.50 0.49 0.51 0.23 0.31 0.33 0.13 0.17 0.17 0.15 

FAR ≥ 0.25 mm/h 0.76 0.91 0.88 0.77 0.76 0.72 0.69 0.66 0.82 0.74 0.75 0.81 0.75 0.80 0.88 

CSI ≥ 0.25 mm/h 0.14 0.04 0.05 0.12 0.18 0.22 0.23 0.26 0.11 0.17 0.17 0.09 0.11 0.10 0.07 

Fig. 35, Fig. 36, Fig. 37 and Fig. 38 summarize the results, showing time trends of the various scores. 

Fig. 35 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for light and medium rain.


Fig. 36 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for heavy rain. 

Fig. 37 - Time evolution of the Correlation Coefficient for the three categories of precipitation.


Fig. 38 - Time evolution of Probability Of Detection, False Alarm Rate and Critical Success Index. 

We can conclude from the graphs about the H03 scores that: 

There is seasonal dependence in the Mean Error. For the Category 10 mm/h, there is always an underestimation, which gets very large 

in the case of higher rain rates. This means that the H03 cannot capture very well the high 

intensities. 

ME, SD and RMSE values are very high for the case of precipitation >10 mm/h in February and 

November 2009. this can be due to non-expected deficiencies of the algorithms which were 

corrected afterwards, because for other months, the scores are stable. 

The Standard Deviation and the Root Mean Square Error follow the tendencies of the Mean Error. 

Where ME is positive, the SD and RMSE increases. 

We can depict the seasonal dependence in the Correlation Coefficients and the POD, FAR, CSI 

values. Correlation and Probability of Detection is the best in summer months, when there is 

convection. So the detection of convective rain is performing better than the detection of stratiform 

rain. 

The Correlation Coefficient is stable in case of precipitation of intensity of


4.5 Validation in Italy (UniFe) 

The facilities available to the University of Ferrara, and the methodology adopted for validation, are 

described in section 3.4.4. The ground truth is provided by rain gauge networks. 


Table 11 - Summary results of PR-OBS-3 validation in Italy by UniFe 

H03 v1.4 Italy Land Validation period: 1 st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h 

Gauge Class Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar 

Number of > 10 mm/h 15688 4719 8568 8229 4778 11360 8496 5231 21935 11638 11828 8189 8678 6694 4527 


(ground obs.) < 1 mm/h 480462 339540 350497 385169 93567 185437 63285 75866 203657 210846 299523 485470 524200 533291 365103 

> 10 mm/h -15.9 -15.7 -18.5 -13.8 -16.9 -15.1 -15.5 -14.4 -14.8 -13.7 -14.2 -14.2 -14.9 -14.5 -14.7 

ME (mm/h) 1-10 mm/h -2.31 -2.30 -2.39 -2.28 -2.17 -2.04 -2.06 -2.23 -1.74 -2.01 -2.43 -2.34 -2.30 -2.24 -1.95 

< 1 mm/h -0.40 -0.48 -0.44 -0.31 -0.13 -0.04 0.17 -0.02 0.23 -0.24 -0.29 -0.44 -0.43 -0.41 -0.31 

> 10 mm/h 9.38 9.61 15.0 5.96 10.1 7.67 9.15 6.82 9.67 11.5 6.56 7.59 8.13 8.42 8.25 

SD (mm/h) 1-10 mm/h 1.77 1.56 1.77 1.85 2.26 2.36 2.76 2.44 3.50 2.89 2.04 1.67 1.71 1.69 1.68 

< 1 mm/h 0.48 0.33 0.48 0.69 1.18 1.13 1.43 1.22 2.24 1.26 0.69 0.39 0.49 0.45 0.67 

> 10 mm/h 18.5 18.4 23.9 15.0 19.7 16.9 18.0 15.9 17.7 17.9 15.6 16.1 16.9 16.8 16.8 

RMSE (mm/h) 1-10 mm/h 2.92 2.78 2.98 2.94 3.13 3.12 3.45 3.30 3.91 3.52 3.17 2.87 2.87 2.81 2.58 

< 1 mm/h 0.63 0.58 0.65 0.75 1.18 1.13 1.44 1.22 2.25 1.28 0.75 0.59 0.65 0.61 0.74 

> 10 mm/h 95 96 98 97 94 92 90 93 89 87 96 96 97 98 96 

RMSE (%) 1-10 mm/h 93 96 96 93 104 99 105 100 148 121 92 94 96 .94 92 

< 1 mm/h 119 104 123 156 272 254 325 275 505 264 159 109 127 114 152 

> 10 mm/h 0.02 0.04 -0.15 0.01 0.01 0.12 0.02 0.05 0.04 0.01 -0.03 -0.02 -0.07 -0.07 -0.10 

CC 1-10 mm/h 0.19 0.19 0.10 0.10 0.11 0.11 0.08 0.10 0.11 0.19 0.09 0.17 0.12 0.09 0.09 

< 1 mm/h 0.06 0.05 0.04 0.05 0.03 0.04 0.05 0.05 0.04 0.05 0.08 0.05 0.04 0.06 0.06 

POD ≥ 0.25 mm/h 0.19 0.10 0.12 0.24 0.26 0.37 0.43 0.35 0.39 0.254 0.27 0.15 0.13 0.17 0.24 

FAR ≥ 0.25 mm/h 0.43 0.48 0.53 0.57 0.76 0.63 0.56 0.70 0.49 0.55 0.48 0.49 0.47 0.52 0.53 

CSI ≥ 0.25 mm/h 0.17 0.09 0.11 0.19 0.14 0.23 0.28 0.19 0.28 0.194 0.22 0.13 0.11 0.14 0.19 

An overall indication of the capabilities of a satellite technique to reproduce a precipitation field is given 

by the comparison between estimated and measured probability density functions (PDF) of the rainrate 

values, sampled at 1 mm h -1 , starting from 0.25 mm h -1 . In Fig. 39, PDF for estimated and reference 

rainrates for January (left panel) and July (right panel) 2009 are shown in order to also highlight 

seasonal features. 

Fig. 39 - Probability Density Function for the months of January (left) and July (right) 2009. 

For the winter precipitation, as described by January plot, the estimated PDF is constantly below the 

measured one, indicating the relative abundance of no-rain estimates (below 0.25 mm h -1 ), not


considered in this plot. The very low POD values (see Table 11) confirms that the major problem of this 

technique is in detecting winter precipitation type. Moreover, the technique cannot estimate rainrates 

higher than 18 mm h -1 , causing the bad values of all statistical indicators for the high rainrate class in 

the Table. The summer time plots show different behaviour of the technique: estimated and measured 

PDFs are almost coincident for lower rates (below 5 mm h -1 ) indicating better capabilities of h03 in 

delineating precipitation area. The table shows a relatively higher POD value for this month. For July 

the technique could be able to estimate higher rainrates (up to 23 mm h -1 ), probably due to the fact that 

the microwave estimates included in the technique could better describe convective systems, but it is 

still evident the quantitative underestimation of highest rainrates. 

The seasonal trend of all the continuous statistical parameters reported in the Table are plotted in Fig. 

40 for the class 2 (rainrates between 1.0 and 10 mm h -1 ) for the year 2009. 

Fig. 40 - Evolution of continuous statistical scores cross year 2009. 

The correlation coefficient (blue line) slightly varies between 0.1 and 0.2 along the months, indicating 

that, despite the better matching between the PDFs, there are major problems in co-locating 

precipitation patterns. A possible reason for this is that the matching between satellite estimates and 

ground truth maps is done without parallax correction, that can be effective in reducing co-location 

errors, especially for summertime, cold-top, small-scale, convective clouds. Moreover, the number of 

the warm month samples (black dotted line on the left axis) is markedly lower than the one for the cold 

months. 

The Multiplicative Bias (yellow line) is very lower than one (no bias condition), especially for cold 

months, confirming the tendency to underestimate wintertime precipitation areas. The tendency to 

underestimate is present also in summer, but the Multiplicative Bias reaches its maximum at about 0.5 

in September. Hence, the Mean Error (black solid line) is always negative, with a value almost 2 times 

the lower limit of the considered precipitation class. The normalized (cyan line) and absolute (purple 

line) root mean square errors both increase moving from cold to warm months, reaching the peak in 

September. 

Other classes do not show different behaviour, but has to be remarked that the numerical values of the 

statistical indicators in the high precipitation class (above 10 mm h -1 ) are out of any meaningful range 

(see the table), indicating that this class is completely unresolved by this technique. Probably, as 

mentioned earlier, a parallax correction could be of some help also on this issue.


4.6 Validation in Poland (IMWM) 

The facilities available to IMWM, and the methodology adopted for validation, are described in section 

3.4.5. The ground truth is provided by rain gauge networks. 


Table 12 - Summary results of PR-OBS-3 validation in Poland by IMWM 

H03 v1.4 Poland Land Validation period: 1 st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h 


Number of > 10 mm/h 32 7 174 212 2097 4696 3862 2522 894 635 64 40 6 14 103 


(ground obs.) < 1 mm/h 31783 62299 85087 3722 52051 82127 40132 31722 18355 93534 60128 47125 47740 38561 38425 

> 10 mm/h -82.7 -16.6 -17.6 -16.8 -20.7 -19.4 -22.6 -18.2 -17.8 -33.8 -25.4 -22.0 -12.2 -14.4 -13.8 

ME (mm/h) 1-10 mm/h -0.79 -1.51 -1.70 -1.96 -1.48 -1.76 -1.98 -2.11 -2.26 -1.33 -1.52 -1.63 -1.56 -1.50 -1.85 

< 1 mm/h -0.35 -0.56 -0.56 -0.19 -0.14 -0.17 -0.14 -0.19 -0.30 -0.29 -0.45 -0.55 -0.56 -0.53 -0.56 

> 10 mm/h 167 6.15 28.6 9.74 68.1 13.9 55.1 11.9 13.4 130 39.5 15.7 2.04 3.55 3.03 

SD (mm/h) 1-10 mm/h 1.43 0.71 0.99 2.11 2.45 2.28 2.26 2.07 2.26 3.11 0.96 0.86 0.85 0.82 1.21 

< 1 mm/h 0.83 0.17 0.19 0.94 1.27 1.14 1.09 0.80 1.70 2.20 0.44 0.20 0.19 0.21 0.17 

> 10 mm/h 186 17.7 33.6 19.4 71.2 23.9 59.6 21.7 22.3 134 46.9 27.1 12.4 14.9 14.1 

RMSE (mm/h) 1-10 mm/h 1.63 1.67 1.97 2.88 2.86 2.88 3.01 2.96 3.19 3.38 1.79 1.84 1.77 1.70 2.21 

< 1 mm/h 0.90 0.58 0.59 0.96 1.28 1.15 1.10 0.83 1.73 2.22 0.63 0.59 0.59 0.58 0.58 

> 10 mm/h 100 100 100 96 90 92 93 94 96 93 100 100 100 100 100 

RMSE (%) 1-10 mm/h 112 99 98 94 116 106 95 87 117 201 93 97 98 95 98 

< 1 mm/h 157 99 100 175 222 200 190 144 334 384 108 99 100 97 99 

> 10 mm/h - - -0.05 0.08 0.00 0.10 0.02 0.12 -0.03 -0.07 -0.09 0.30 - - -0.03 

CC 1-10 mm/h 0.00 -0.01 0.00 0.14 0.10 0.13 0.09 0.09 0.08 0.05 0.04 0.10 0.04 -0.04 0.04 

< 1 mm/h 0.05 -0.03 0.02 -0.01 0.03 0.02 0.04 0.04 -0.01 0.01 0.01 -0.01 0.01 0.03 0.03 

POD ≥ 0.25 mm/h 0.17 0.03 0.05 0.30 0.30 0.33 0.41 0.48 0.14 0.20 0.16 0.07 0.04 0.08 0.05 

FAR ≥ 0.25 mm/h 0.75 0.80 0.70 0.88 0.70 0.74 0.75 0.66 0.86 0.59 0.76 0.80 0.79 0.86 0.82 

CSI ≥ 0.25 mm/h 0.11 0.03 0.05 0.09 0.18 0.17 0.18 0.25 0.07 0.15 0.11 0.06 0.03 0.05 0.04 

Reference Data 

The PR-OBS-3 v.1.4 rain rate product has been validated against automatic rain gauges data. Polish 

network of automatic rain gauges consists of 430 posts located all over the country, however, the 

network density increases in the Southern Poland, where the flood danger is very high. Each post is 

equipped with two gauges: heated and non-heated, what enables some quality control of data. For 

validation purposes, readings from both gauges were compared in order to eliminate the cases of 

clogged instruments. If both gauges worked properly, higher values was taken (automatic RG are known 

to underestimate the real precipitation). 

The measurements time resolution is 10 minutes, what allows estimating the rain rate with reasonable 

quality, especially for stratiform rainfalls. The ground rain rate (in mm/h) was calculated from 10 

minute cumulative values from the timeslot closest to the satellite overpass assuming that the real 

precipitation rate was constant within that time span. 

In order to combine satellite products with rain gauges data, the following simple method was applied. 

For each satellite pixels, the automatic posts situated within that pixel were found. If more than one rain 

gauge were found within one satellite pixel, the ground rain rate value was calculated as a mean of all 

rain gauges measurements within that pixel. 

Results 

Following the methodology agreed in the H-SAF precipitation validation group, both continuous and 

categorical statistics were calculated on the monthly mean basis for three precipitation categories: 

- category 1: >10 mm/h, 

- category 2: 1-10 mm/h, 

- category 3:

Jan'09 

Jan'09 

Feb'09 

Feb'09 

Mar'09 

Mar'09 

Apr'09 

Apr'09 

May'09 

May'09 

Jun'09 

Jun'09 

Jul'09 

Jul'09 

Aug'09 

Aug'09 

Sep'09 

Sep'09 

Oct'09 

Oct'09 

Nov'09 

Nov'09 

Dec'09 

Dec'09 

Jan'10 

Jan'10 

Feb'10 

Feb'10 

Mar'10 

Mar'10 


Fig. 41 and Fig. 42 report values of the mean error, and RMSE % calculated for the three categories and 

for each month of the analysed period are presented. It should be pointed out here that the analysis was 

performed only for situations when the ground rain rate was 0.25 mm/h 

0 

-10 

-20 

-30 

-40 

-50 

-60 

-70 

-80 

-90 

ME 

> 10 mm/h 

1-10 mm/h 

< 1 mm/h 

Fig. 41 - Mean error (ME) of PR-OBS-3 v.1.4 for the period of Jan 2009 – Mar 2010 for Poland. 

The PR-OBS-3 v.1.4 underestimates the measured rain rate values for the whole analysed period. The 

underestimation is stronger in summer and early autumn. The high negative value of ME obtained for 

category 1 in January 2009 (Fig.1) results from the rain gauge quality then from satellite product. 

During winter, only one gauge (heated) works and therefore the situation with clogged instrument are 

difficult to detect. 

450 

400 

350 

300 

250 

200 

150 

100 

50 

0 

RMSE % 

> 10 mm/h 

1-10 mm/h 

< 1 mm/h 

Fig. 42 - RMSE % of PR-OBS-3 v.1.4 for the period of Jan 2009 – Mar 2010 for Poland. 

The quality of the product in rain rate estimation in categories 1 and 2 does not reveal any seasonal 

changes and varies from 87 % to 116 %. For the light precipitation, strong seasonal variation was 

found. The RMSE % increases for the period of April – October, however, the highest values were 

obtained for September and October. These were mostly the situations with light stratiform precipitation 

that were not detected or underestimated by PR-OBS-3 v.1.4.

Jan'09 

Feb'09 

Mar'09 

Apr'09 

May'09 

Jun'09 

Jul'09 

Aug'09 

Sep'09 

Oct'09 

Nov'09 

Dec'09 

Jan'10 

Feb'10 

Mar'10 


The quality of PR-OBS-3 v.1.4 in precipitaion detection was validated using the dichotomous (yes/no) 

statistics. The 0.25 mm/h threshold was used for rain/no rain differentiation. On Fig. 43 the variability 

of Probability of Detection and False Alarm Ratio are presented. 

1,4 

1,2 

1 

0,8 

0,6 

0,4 

FAR 

POD 

0,2 

0 

Fig. 43 - Variabily of Probability of Detection (POD) and False Alarm Ratio (FAR) obtained for PR-OBS-3 v.1.4 using Polish 

RG data in the period of Jan 2009 – Mar 2010. 

The quality of PR-OBS-3 v.1.4 in precipitation detection is not very good in the whole analysed period: 

the POD exceeds 0.2 spring-summer time. Taking into account that for each month FAR values are 

always higher than POD values, it should be concluded that the ability of PR-OBS-3 of precipitation 

detection is not satisfactory. It is especially low in winter.

Mean Error [mm/h] 

Mean Error [mm/h] 


4.7 Validation in Slovakia (SHMÚ) 

The facilities available to SHMÚ, and the methodology adopted for validation, are described in section 



Table 13 - Summary results of PR-OBS-3 validation in Slovakia by SHMÚ 

H03 v1.4 Slovakia Land Validation period: 1 st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h 


Number of > 10 mm/h 2 0 181 1440 5039 11731 11712 6662 1108 243 1 3 0 0 7 


(ground obs.) < 1 mm/h 513194 638560 844472 198558 693321 840245 444706 625219 281482 683642 698513 542730 384414 305943 219294 

> 10 mm/h -16.4 - -11.7 -14.9 -15.6 -15.9 -16.0 -15.4 -15.6 -14.8 -13.1 -10.6 - - -23.0 

ME (mm/h) 1-10 mm/h -0.67 -1.20 -1.61 -1.47 -1.11 -1.11 -1.04 -1.24 -1.36 -1.09 -1.10 -1.18 -0.97 -1.27 -1.38 

< 1 mm/h 0.06 -0.36 -0.36 -0.16 0.19 0.22 0.36 0.04 -0.08 -0.04 -0.16 -0.30 -0.24 -0.30 -0.37 

> 10 mm/h 5.22 - 1.60 8.45 9.45 9.04 8.73 8.67 7.96 6.53 0.00 0.35 - - 13.7 

SD (mm/h) 1-10 mm/h 1.50 0.74 1.12 1.92 2.40 2.17 2.33 2.03 2.59 1.88 0.92 1.33 0.79 0.52 0.71 

< 1 mm/h 1.12 0.28 0.39 1.01 1.44 1.30 1.47 0.97 1.64 1.55 0.61 0.66 0.49 0.36 0.22 

> 10 mm/h 17.2 - 11.8 17.2 18.3 18.3 18.2 17.7 17.5 16.2 13.1 10.6 - - 26.7 

RMSE (mm/h) 1-10 mm/h 1.64 1.41 1.96 2.42 2.65 2.44 2.56 2.38 2.93 2.17 1.43 1.78 1.25 1.37 1.55 

< 1 mm/h 1.12 0.46 0.53 1.03 1.45 1.31 1.51 0.97 1.64 1.56 0.63 0.72 0.54 0.47 0.43 

> 10 mm/h 100 - 100 89 91 92 92 92 98 97 100 100 - - 100 

RMSE (%) 1-10 mm/h 118 99 97 107 128 105 110 101 159 119 88 109 95 96 98 

< 1 mm/h 322 105 124 238 339 305 349 225 400 429 158 167 148 120 102 

> 10 mm/h - - -0.02 0.12 -0.03 -0.05 0.03 0.01 -0.06 -0.24 - - - - - 

CC 1-10 mm/h -0.01 -0.04 -0.01 0.13 0.10 0.08 0.03 0.12 0.02 0.09 -0.01 0.09 0.02 0.08 0.05 

< 1 mm/h 0.02 0.12 0.01 0.06 0.08 0.08 0.10 0.07 0.03 0.04 0.10 0.06 0.07 -0.02 0.05 

POD ≥ 0.25 mm/h 0.28 0.05 0.09 0.22 0.37 0.49 0.55 0.50 0.27 0.27 0.26 0.15 0.12 0.11 0.04 

FAR ≥ 0.25 mm/h 0.75 0.89 0.80 0.83 0.75 0.77 0.69 0.70 0.79 0.78 0.84 0.85 0.84 0.91 0.97 

CSI ≥ 0.25 mm/h 0.15 0.04 0.06 0.11 0.18 0.19 0.25 0.23 0.13 0.14 0.11 0.08 0.07 0.05 0.02 

Seasonal dependence of product performance is shown in Figures 44 to 48. 

1 

-1 

-3 

Mean Error for H03 v1.4 

1-10mm/h 

10mm/h 

-5 

-7 

-9 

Fig. 44 - Time evolution of the Mean Error for 

the three precipitation categories. 

-11 

-13 

-15 

-17 

Jan 

2009 

Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan 

2010 

Feb 

Mar 

Mean Error for H03 v1.4 

1-10mm/h 

Scores 


450% 

400% 

RMSE [%] for H03 v1.4 

>10mm/h 

1-10mm/h 

10mm/h 

1-10mm/h 


4.8 Validation in Turkey (ITU) 

The facilities available to ITU (in collaboration with TSMS), and the methodology adopted for 

validation, are described in section 3.4.7. The ground truth is provided by rain gauge networks. 

Table 14 and Table 15 report the results of the final validation campaign of the Development Phase for 

PR-OBS-3. It is noted that ITU and TSMS performed validation over inner land and coastal zones. 

Table 14 - Summary results of PR-OBS-3 validation in Turkey by ITU and TSMS over inner land 

H03 v1.4 Turkey Land Validation period: 1 st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h 


Number of > 10 mm/h 0 0 21 158 311 387 1190 100 1841 1519 201 521 0 0 6 


(ground obs.) < 1 mm/h 240469 235039 257243 18427 6595 6664 2930 1504 7524 149644 77393 235798 1287346 1034755 1227326 

> 10 mm/h - - -10.5 -12.3 -14.3 -12.8 -23.2 -22.5 -19.0 -10.0 -11.4 -6.26 - - -10.7 

ME (mm/h) 1-10 mm/h -1.29 -1.51 -1.62 -1.72 -1.65 -1.01 -2.17 -1.17 -1.97 -1.16 -1.89 -0.69 -1.29 -1.34 -1.29 

< 1 mm/h -0.35 -0.32 -0.46 -0.46 0.11 0.70 0.11 0.10 -0.32 -0.22 -0.23 -0.24 -0.46 -0.31 -0.42 

> 10 mm/h - - 0.11 3.58 5.20 4.92 16.9 10.2 11.4 4.36 1.70 4.46 - - 0.47 

SD (mm/h) 1-10 mm/h 0.97 1.09 0.86 1.21 2.14 2.62 2.97 1.69 2.50 2.85 1.48 2.61 0.70 1.32 0.87 

< 1 mm/h 0.78 0.64 0.30 0.64 1.78 1.66 1.92 0.97 1.30 1.26 0.69 0.92 0.34 0.81 0.46 

> 10 mm/h - - 10.5 12.8 15.2 13.7 28.7 24.6 22.2 10.9 11.5 7.69 - - 10.7 

RMSE (mm/h) 1-10 mm/h 1.61 1.86 1.83 2.11 2.70 2.81 3.68 2.06 3.18 3.08 2.40 2.70 1.47 1.89 1.56 

< 1 mm/h 0.85 0.72 0.55 0.79 1.78 1.80 1.92 0.97 1.34 1.28 0.72 0.95 0.58 0.87 0.62 

> 10 mm/h - - 100 94 98 89 87 93 89 82 98 68 - - 100 

RMSE (%) 1-10 mm/h 100 91 97 94 102 110 120 75 109 151 94 122 97 102 98 

< 1 mm/h 191 151 103 128 338 311 332 172 214 237 157 204 111 180 122 

> 10 mm/h - - - -0.27 -0.16 -0.23 0.35 0.28 0.17 -0.16 0.23 -0.16 - - - 

CC 1-10 mm/h 0.12 0.03 0.01 0.19 -0.05 0.00 0.03 0.17 0.28 0.17 -0.07 0.34 0.08 0.01 0.22 

< 1 mm/h 0.01 0.07 0.06 0.01 -0.05 0.09 0.02 0.01 -0.01 0.14 0.17 0.12 0.05 0.08 0.07 

POD ≥ 0.25 mm/h 0.12 0.20 0.06 0.23 0.43 0.72 0.40 0.69 0.35 0.19 0.22 0.31 0.05 0.16 0.90 

FAR ≥ 0.25 mm/h 0.65 0.27 0.69 0.84 0.87 0.87 0.81 0.83 0.84 0.21 0.47 0.45 0.80 0.52 0.70 

CSI ≥ 0.25 mm/h 0.10 0.18 0.05 0.10 0.11 0.12 0.14 0.15 0.12 0.18 0.18 0.25 0.04 0.14 0.07 

Table 15 - Summary results of PR-OBS-3 validation in Turkey by ITU and TSMS over coastal zones 

H03 v1.4 Turkey Coast Validation period: 1 st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h 


Number of > 10 mm/h 0 0 4 16 0 9 0 0 19 67 7 37 0 0 0 


(ground obs.) < 1 mm/h 23224 24690 23878 1235 527 732 71 0 758 12403 8038 23829 33168 26679 27562 

> 10 mm/h - - -10.5 -12.6 - -4.66 - - -13.7 -11.2 -10.4 -5.00 - - - 

ME (mm/h) 1-10 mm/h -1.42 -1.48 -1.62 -2.03 -2.26 -0.39 -2.21 - -2.29 -0.55 -2.08 -0.68 -1.48 -1.25 -1.45 

< 1 mm/h -0.36 -0.27 -0.47 -0.49 1.10 0.92 -0.66 - -0.54 -0.18 -0.27 -0.22 -0.49 -0.01 -0.48 

> 10 mm/h - - 0.13 2.31 - 2.32 - - 2.88 3.89 0.67 4.26 - - - 

SD (mm/h) 1-10 mm/h 0.88 1.09 0.93 1.40 1.61 2.93 0.98 - 1.51 3.58 1.47 2.51 0.64 1.62 0.79 

< 1 mm/h 0.80 0.74 0.30 0.49 3.21 2.32 0.19 - 0.34 1.47 0.72 0.77 0.31 1.26 0.38 

> 10 mm/h - - 10.5 12.8 - 5.15 - - 14.0 11.8 10.4 6.53 - - - 

RMSE (mm/h) 1-10 mm/h 1.66 1.84 1.87 2.46 2.77 2.95 2.41 - 2.74 3.62 2.55 2.60 1.61 2.04 1.64 

< 1 mm/h 0.88 0.79 0.56 0.69 3.39 2.49 0.69 - 0.64 1.48 0.77 0.80 0.58 1.26 0.61 

> 10 mm/h - - 100 100 - 47 - - 99 93 98 62 - - - 

RMSE (%) 1-10 mm/h 103 91 97 97 101 141 100 - 95 207 92 112 98 103 96 

< 1 mm/h 193 176 102 113 677 398 100 - 96 275 175 151 105 262 110 

> 10 mm/h - - - - - -0.15 - - 0.35 -0.39 0.05 0.55 - - - 

CC 1-10 mm/h 0.06 0.08 0.02 0.05 -0.22 0.16 - - -0.07 0.21 -0.04 0.33 0.01 -0.07 0.27 

< 1 mm/h 0.00 0.05 0.08 -0.11 -0.30 0.05 - - -0.15 0.16 0.03 0.17 0.02 0.11 0.04 

POD ≥ 0.25 mm/h 0.10 0.20 0.07 0.11 0.25 0.67 0.00 0.00 0.16 0.16 0.20 0.35 0.04 0.30 0.08 

FAR ≥ 0.25 mm/h 0.68 0.19 0.68 0.95 0.96 0.92 1.00 1.00 0.94 0.25 0.45 0.40 0.86 0.25 0.61 

CSI ≥ 0.25 mm/h 0.08 0.19 0.06 0.04 0.03 0.08 0.00 0.00 0.04 0.15 0.17 0.28 0.03 0.27 0.07 

These results are shown in graphical form in Fig. 49 and Fig. 50.

Scores (mm/h) 

Scores 

Scores (mm/h) 

Scores 

Scores (mm/h) 

Scores 

Scores (mm/h) 

Scores 


4 

3 

2 

1 

0 

-1 

-2 

Jan 2009 

-3 

0.4 

0.3 

0.2 

0.1 

0 

-0.1 

-0.2 

Jan 2009 

-0.3 

-0.4 

4 

3 

2 

1 

0 

-1 

-2 

Jan 2009 

-3 

0.6 

0.4 

0.2 

0 

-0.2 

Jan 2009 

-0.4 

-0.6 

Mean Error, Standart Deviation and RMSE for H03 (Land) 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Oct 

Nov 

Dec 

Jan 2010 

Feb 

Mar 

Correlation Coefficients For H03 (Land) 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Oct 

Nov 

Dec 

Jan 2010 

Feb 

Mar 

ME 1-10 mm/h 

ME < 1 mm/h 

SD 1-10 mm/h 

SD < 1 mm/h 

RMSE < 1 mm/h 

RMSE 1-10 mm/h 

> 10 mm/h 

1-10 mm/h 

< 1 mm/h 

30 

20 

10 

0 

-10 

Jan 2009 

-20 

-30 

1 

0.9 

0.8 

0.7 

0.6 

0.5 

0.4 

0.3 

0.2 

0.1 

0 

Mean Error, Standart Deviation and RMSE for H03 (Land) 

Jan 2009 

Feb 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Oct 

Nov 

Dec 

Jan 2010 

Feb 

Mar 

Multi-Categorical Scores for H03 (Land) 

Fig. 49 - Continuous and multi-categorical statistics for inner land. 

Mean Error, Standart Deviation and RMSE for H03 (Coast) 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Oct 

Nov 

Dec 

Jan 2010 

Feb 

Mar 

Correlation Coefficients For H03 (Coast) 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Oct 

Nov 

Dec 

Jan 2010 

Feb 

Mar 

ME 1-10 mm/h 

ME < 1 mm/h 

SD 1-10 mm/h 

SD < 1 mm/h 

RMSE 1-10 mm/h 

RMSE < 1 mm/h 

> 10 mm/h 

1-10 mm/h 

< 1 mm/h 

20 

15 

10 

5 

0 

-5 

-10 

Jan 2009 

-15 

-20 

1 

0.9 

0.8 

0.7 

0.6 

0.5 

0.4 

0.3 

0.2 

0.1 

0 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Oct 

Nov 

Dec 

Jan 2010 

Feb 

Mar 

Mean Error, Standart Deviation and RMSE for H03 (Coast) 

Jan 2009 

Feb 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Oct 

Nov 

Dec 

Jan 2010 

Feb 

Mar 

Multi-Categorical Scores for H03 (Coast) 

Fig. 50 - Continuous and multi-categorical statistics for coastal zones. 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Oct 

Nov 

Dec 

Jan 2010 

Feb 

Mar 

ME > 10 mm/h 

SD > 10 mm/h 

RMSE > 10 mm/h 

ME > 10 mm/h 

SD > 10 mm/h 

POD 

FAR 

CSI 

RMSE > 10 mm/h 

The following interpretations can be drawn from graphs and tables concerning inner land and coastal 

precipitation. 

When the patterns in Fig. 49 and Fig. 50 are compared each other, very similar patterns between 

inner land and coastal zones for multi-categorical and continuous statistics can be seen. 

Mean Error (ME) is negative for the classes 10 mm/h, in other words there is always 

underestimation. 

For the class 1-10 mm/h, ME is almost zero. 

The Standard Deviation (SD) and the Root Mean Square Error (RMSE) follow the trend of the ME. 

Correlation Coefficients (CC) and Probability of Detection (POD) are the best in summer months, 

because of convection. Therefore, the detection of convective rain is performing better than the 

detection of stratiform rain. 

False Alarm Rate (FAR) is usually very high and Critical Success Index (CSI) is always very low. 

POD 

FAR 

CSI


5. Overview of findings 

5.1 Synopsis of validation results 

In the various sections of Chapter 4 the validation results have been quoted separately by each Team 

operating on a different geographic area associated to a proper climatic condition. This is correct, since 

the precipitation field is affected by orography and local climatology. In this section a synoptic 

overview is provided, of the results achieved in the different countries, and in different seasons. 

In order to reduce the volume of data to be commented, and not to overlap with the detailed reports in 

Chapter 4, the results are summarised by seasons of three months each. The phase is: 

Spring: Summer: Autumn: Winter: 

March, April and May 2009 June, July and August 2009 Sept., Oct. and Nov. 2009 Dec. 2009, Jan. and Feb. 2010 

Table 16, split in four sections, one for each season, reports the Country/Team results side to side. 

There are three sets of columns: 

one set for four Countries/Teams that compared satellite data with meteorological radar in inner land 

areas: Belgium/IMR, Germany/BfG, Hungary/OMSZ and Slovakia/SHMÚ; and their average 

weighed by the number of comparisons; 

one set for three Countries/Teams that compared satellite data with rain gauges in inner land areas: 

Italy/UniFe, Poland/IMWM and Turkey/ITU; and their average weighed by the number of 

comparisons; 

one column for Turkey/ITU that compared satellite data with rain gauges in coastal zones. 

It is reminded, from Chapter 4, Table 07, that the User requirements are: 

Table 07 - Accuracy requirements for product PR-OBS-3 [RMSE (%)] 

Precipitation range threshold target optimal Remarks 

> 10 mm/h 80 40 20 

1-10 mm/h 160 80 40 

< 1 mm/h 320 120 80 This entry, originally N/A, has been extrapolated 

The unit for accuracy specification is Root Mean Square Error percent (RMSE %), used since error 

grows with rate. In Chapter 4 further scores were recorded: Mean Error (or bias, ME), Standard 

Deviation (SD) and Correlation Coefficient (CC), Probability Of Detection (POD), False Alarm Rate 

(FAR) and Critical Success Index (CSI). In this summary Chapter 5, in order to streamline the 

discussion and minimize repetition with Chapter 4, we only focus on RMSE (%), ME (mm/h) and POD 

/ FAR. 

Table 16 highlights the RMSE (%) rows by yellow colour, the weighed average column by blue and the 

averaged RMSE (%) values (first thing to attract the attention) by green. The column for “coast”, being 

single, is not averaged.


Table 16 - Comparative results of validation in several Countries/Teams split by season 

PR-OBS-3 Spring IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast 

Version 1.4 2009 radar radar radar radar radar gauge gauge gauge gauge gauge 

> 10 mm/h 11,458 54,166 17,790 6,660 90,074 21,575 2,483 490 24,548 20 

N. of samples 1-10 mm/h 531,587 4,746,885 1,203,951 404,437 6,886,860 859,298 99,808 126,574 1,085,680 11,318 

< 1 mm/h 961,019 9,116,702 3,848,790 1,736,351 15,662,862 829,233 140,860 282,265 1,252,358 25,640 

> 10 mm/h -18.8 -11.3 -13.7 -15.3 -13.0 -16.3 -20.1 -13.5 -16.6 -12.2 

ME (mm/h) 1-10 mm/h -2.20 -1.59 -0.95 -1.29 -1.51 -2.32 -1.60 -1.64 -2.17 -1.70 

< 1 mm/h -0.44 -0.37 -0.01 -0.12 -0.26 -0.34 -0.40 -0.45 -0.37 -0.44 

> 10 mm/h 99 90 83 91 90 97 91 97 97 100 

RMSE (%) 1-10 mm/h 97 119 153 117 123 95 107 97 96 97 

< 1 mm/h 117 175 303 223 208 155 147 110 144 114 

POD ≥ 0.25 mm/h 0.09 0.14 0.24 0.23 0.17 0.19 0.17 0.10 0.17 0.08 

FAR ≥ 0.25 mm/h 0.86 0.72 0.81 0.78 0.76 0.57 0.71 0.72 0.61 0.71 

PR-OBS-3 Summer IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast 


> 10 mm/h 42,178 169,085 118,063 30,105 359,431 25,087 11,080 1,677 37,844 9 

N. of samples 1-10 mm/h 527,814 6,871,009 2,849,335 1,071,773 11,319,931 337,803 147,653 20,981 506,437 566 

< 1 mm/h 533,785 7,716,398 4,116,208 1,910,170 14,276,561 324,588 153,981 11,098 489,667 803 

> 10 mm/h -19.8 -13.8 -17.4 -15.8 -15.8 -15.1 -20.2 -20.8 -16.8 -4.66 

ME (mm/h) 1-10 mm/h -2.70 -2.04 -1.33 -1.13 -1.81 -2.08 -1.91 -1.48 -2.01 -0.84 

< 1 mm/h -0.31 -0.21 0.30 0.19 -0.01 0.01 -0.17 0.46 -0.04 0.78 

> 10 mm/h 99 92 90 92 92 91 93 88 91 47 

RMSE (%) 1-10 mm/h 98 97 115 105 102 100 98 110 100 131 

< 1 mm/h 178 201 346 289 254 273 186 298 246 372 

POD ≥ 0.25 mm/h 0.17 0.29 0.50 0.51 0.37 0.38 0.39 0.60 0.39 0.57 

FAR ≥ 0.25 mm/h 0.84 0.65 0.70 0.73 0.68 0.63 0.72 0.84 0.66 0.93 

PR-OBS-3 Autumn IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast 


> 10 mm/h 10,058 20,612 17,199 1,352 49,221 45,401 1,593 3,561 50,555 93 

N. of samples 1-10 mm/h 564,873 5,674,399 1,490,000 281,414 8,010,686 764,868 117,444 161,607 1043,919 14,203 

< 1 mm/h 805,894 9,291,044 3,591,725 1,663,637 15,352,300 714,026 172,017 234,561 1,120,604 21,199 

> 10 mm/h -15.4 -12.1 -19.6 -15.4 -15.5 -14.4 -24.5 -14.7 -14.7 -11.6 

ME (mm/h) 1-10 mm/h -2.05 -1.71 -1.54 -1.16 -1.68 -2.10 -1.51 -1.49 -1.94 -1.32 

< 1 mm/h -0.30 -0.31 -0.15 -0.10 -0.25 -0.13 -0.35 -0.23 -0.18 -0.23 

> 10 mm/h 98 91 95 98 94 90 95 86 90 95 

RMSE (%) 1-10 mm/h 118 114 102 122 112 118 153 128 123 150 

< 1 mm/h 206 224 213 310 230 289 282 210 271 231 

POD ≥ 0.25 mm/h 0.16 0.20 0.31 0.27 0.23 0.30 0.18 0.21 0.27 0.18 

FAR ≥ 0.25 mm/h 0.77 0.67 0.76 0.81 0.71 0.50 0.68 0.34 0.49 0.36 

PR-OBS-3 Winter IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast 

Version 1.4 2009/10 radar radar radar radar radar gauge gauge gauge gauge gauge 

> 10 mm/h 1,028 3,899 4,176 3 9,106 23,561 60 521 24,142 37 

N. of samples 1-10 mm/h 408,403 3,619,301 994,309 89,130 5,111,143 1,301,421 48,619 852,141 2,202,181 46,186 

< 1 mm/h 1,204,949 10,765,739 4,282,818 1,233,087 17,486,593 1,542,961 133,426 2,557,899 4,234,286 83,676 

> 10 mm/h -12.2 -16.9 -13.0 -10.6 -14.6 -14.5 -19.2 -6.26 -14.3 -5.00 

ME (mm/h) 1-10 mm/h -1.99 -1.68 -1.60 -1.18 -1.68 -2.29 -1.58 -1.23 -1.86 -1.14 

< 1 mm/h -0.46 -0.47 -0.31 -0.28 -0.42 -0.43 -0.55 -0.38 -0.40 -0.26 

> 10 mm/h 100 99 92 100 96 97 100 68 96 62 

RMSE (%) 1-10 mm/h 97 97 96 106 97 63 97 103 79 104 

< 1 mm/h 104 104 143 149 117 117 99 147 134 168 

POD ≥ 0.25 mm/h 0.05 0.06 0.15 0.13 0.08 0.15 0.06 0.13 0.14 0.23 

FAR ≥ 0.25 mm/h 0.79 0.72 0.79 0.86 0.75 0.49 0.81 0.64 0.58 0.50


5.2 Summary conclusions on comparative elements 

In the various sections of Chapter 4 the Countries/Teams have concluded with highlighting the main 

positive aspects of the product and the main failures, according to the experience on their area of 

investigation. This Section does in no way bias the original conclusions in the different sections of 

Chapter 4, but only refers to features stemming from observation of Table 16. 

Variability with geographical area 

It may be observed that the performances are rather consistent across the various geographical areas, 

especially for heavy (> 10 mm/h) and medium (1-10 mm/h) precipitation. This has been favoured by 

the adoption of a common validation methodology across the various participating Institutes. Even for 

inner lands and coastal areas the performance are rather similar. 

Variability with validation tool 

The performances resulting from validation by radar and those by rain gauges are rather similar. This is 

very important because User should not mind about which tool has been used for the validation: the 

information on the performance is regarded as a property of the product, not of the ground truth. 

Variability with season 

For heavy and medium precipitation the performance are rather similar across seasons, whereas for light 

precipitation summer is substantially worse, and winter better. The FAR is rather high in all seasons, 

whereas the POD is better in summer and worse in winter. 

Variability with precipitation type (or intensity) 

Heavy and medium precipitation have very similar performances through all seasons and geographical 

areas, whereas for light precipitation there is a substantial degradation in spring and autumn, and very 

substantial in summer. 

Overall observation on compliance with User requirements 

This final comment does not replace an in-depth analysis of the compliance between user requirements 

and satellite-derived product performance, that would deserve a much detailed discussion. However, 

with the understanding that this is just a rough overall assessment, Table 17 offers a view on the subject. 

Table 17 - Simplified compliance analysis for product PR-OBS-3 

Between target and optimal Between threshold and target Threshold exceeded by < 50 % Threshold exceeded by ≥ 50 % 

PR-OBS-3 v1.4 Spring Summer Autumn Winter 

Precipitation Requirement (RMSE %) radar gauge gauge radar gauge gauge radar gauge gauge radar gauge gauge 

class thresh target optimal land land coast land land coast land land coast land land coast 

> 10 mm/h 80 40 20 90 97 100 92 91 47 94 90 95 96 96 62 

1-10 mm/h 160 80 40 123 96 97 102 100 131 112 123 150 97 79 104 

< 1 mm/h 320 120 80 208 144 114 254 246 372 230 271 231 117 134 168 

Compact presentation of validation results 

Since it has been noted above that the variability of results with validation tool (radar or gauge), as well 

as across the various Countries, and between inner land and coastal zones, are not very pronounced, it 

may be useful to synthesis all results in a presentation that averages among the various situations only 

leaving differentiation with season. This is provided in Table 18.


Table 18 - Synthesis of all validation results, including yearly average 

PR-OBS-3 Version 1.4 Spring 2009 Summer 2009 Autumn 2009 Winter 2009/10 Yearly average 

> 10 mm/h 114,642 397,284 99,869 33,285 645,080 

N. of samples 1-10 mm/h 7,983,858 11,826,934 9,068,808 7,359,510 36,239,110 

< 1 mm/h 16,940,860 14,767,031 16,494,103 21,804,555 70,006,549 

> 10 mm/h -13.8 -15.9 -15.1 -14.4 -15.3 

ME (mm/h) 1-10 mm/h -1.60 -1.82 -1.71 -1.73 -1.73 

< 1 mm/h -0.27 -0.01 -0.25 -0.42 -0.26 

> 10 mm/h 91 92 92 96 92 

RMSE (%) 1-10 mm/h 119 102 113 92 106 

< 1 mm/h 203 254 233 120 195 

POD ≥ 0.25 mm/h 0.17 0.37 0.23 0.09 0.21 

FAR ≥ 0.25 mm/h 0.75 0.68 0.69 0.71 0.71

Products Validation Report, 30 May 2010 - PVR-03 (Product PR-OBS-3) - Appendix Page 61 

Appendix to PVR-03 (Precipitation rate at ground by GEO/IR supported by 

LEO/MW) 

Collection of validation experiment reports 

(extracted from REP-3/04 dated 28 February 2010) 

INDEX 

2. Validation exercises in Belgium 

3. Validation exercises in Germany 

4. Validation exercises in Hungary 

5. Validation exercises in Italy 

6. Validation exercises in Poland 

7. Validation exercises in Slovakia 

8. Validation exercises in Turkey


2. Validation exercises in Belgium 

2.1 Case study: 17-18 January 2007 

The figures presented in the case studies have to be put in relation with the corresponding figures for 

PR-OBS-2 (See REP-3.03, Chap. 2). The first case study refers to Fig. 2.1. 

Fig. 2.1 - PR-OBS-3 (left), up-scaled radar (middle), and SAF-NWC cloud classification (right) 

on 18 January 2007 at 06:00 UTC (top) and 12:00 UTC (bottom). 

Rainfall rates are simply represented with three classes (> 0.1 mm h -1 with light gray; > 1 mm h -1 with 

gray, and > 10. mm h -1 with black). Contrary to what was shown about PR-OBS-2, for this typical 

winter storm, the spatial pattern of the rainfall at the ground according to PR-OBS-3 is similar to the 

extension of high clouds as classified with SAF-NWC algorithm and appears to have little correlation 

with the rainfall field given by the weather radar. 

2.2 Case study: 29-30 August 2006 

This case study refers to Fig. 2.2. 

During this summer thunderstorm, the precipitation zones are better delineated than during winter 

storms of the previous case study. The correlation with high clouds is obvious and appropriate. See for 

instance the elongated cell on the upper part of the validation area of the images at 13 UTC. However, 

on the images at 17 UTC, the cell according to PR-OBS-3 is shifted compared with the radar whereas it 

has been correctly located with PR-OBS-2.


Fig. 2.2 - PR-OBS-3 (left), up-scaled radar (middle), and SAF-NWC cloud classification (right) 

on the 29 th August 2006 at 13 UTC (top) and 17 UTC (bottom). 

2.3 Statistical analysis for the period June 2007 - March 2009 

The product PR-OBS-3 has been validated by comparison with the rain rates estimated from the data of 

the radar at Wideumont. As the projection of this product remains the same, a unique window of 78 × 

42 pixels has been defined which covers the same common validation area of 230 km 230 km 

centered on the radar location as for the previous two MW products. Upscaling of the radar has been 

simply performed by averaging all the radar pixels lying inside a given PR-OBS-3 pixel. 

The monthly values of the mean error and of the root mean square error during the whole period 

available are given as example (see Fig. 2.3 and Fig. 2.4). The general trend consists in an 

overestimation during summer and underestimation during winter. The root mean square error tends to 

be larger during summer. These characteristics are much like those of PR-OBS-2 on which the merged 

product is calibrated. However, July and August 2007, and to a lesser extend August 2008 differ with 

larger errors with PR-OBS-3.

PR-OBS-1 PR-OBS-2 PR-OBS-3 


0,5 

0,4 

0,3 

0,2 

0,1 

0,0 

-0,1 

-0,2 

200706 200709 200712 200803 200806 200809 200812 200903 

Fig. 2.3 - Mean Error of PR-OBS-1, PR-OBS-2 and PR-OBS-3 (monthly values over Belgium in mm h -1 ). 

PR-OBS-1 PR-OBS-2 PR-OBS-3 

4 

3 

2 

1 

0 

200706 200709 200712 200803 200806 200809 200812 200903 

Fig. 2.4 - Root Mean Square Error of PR-OBS-1, PR-OBS-2 and PR-OBS-3 

(monthly values over Belgium in mm h -1 ). 

2.4 Case study: 25-26 May 2009 

Meteorological event description 

During these two days, sub-tropical warm air masses were unstable and three complexes of 

thunderstorm cells crossed Belgium resulting in intense precipitations, and locally, hail (up to 10 cm 

diameter) and heavy wind. The first complex was formed in Spain and characterized by a long lifetime 

(~24 h). It crossed Belgium from west to east on the 25 May between 11 and 15 UTC. The second 

complex came from France during the night and moved towards The Netherlands. The third complex 

formed from existing cells and new cells moving from France around 10 UTC; they all organized in an 

array aligned SW-NE and moved perpendicularly i.e. towards SE.


Data/Products used 

In the following figures, from Fig. 2.5 to Fig. 2.9, the PR-OBS-3 rain rates (left) are compared with upscaled 

weather radar data (right) for an area central to the radar field of view. The three levels of grey 

correspond to thresholds of 0.1, 1, and 10 mm h -1 . The images shown are taken from the three phases of 

the thunderstorm and correspond to the almost the same times as the figures presented in the validation 

sheet for PR-OBS-2 for the same case study. 

Fig. 2.5 - Rain rates (left) are compared with up-scaled weather radar data (right) - 25 May 2009 at 13:27 UTC. 

Fig. 2.6 - Rain rates (left) are compared with up-scaled weather radar data (right) - 25 May 2009 at 15:42 UTC.


Fig. 2.7 - Rain rates (left) are compared with up-scaled weather radar data (right) - 26 May 2009 at 01:42 UTC. 

Fig. 2.8 - Rain rates (left) are compared with up-scaled weather radar data (right) - 26 May 2009 at 02:42 UTC 

Fig. 2.9 - Rain rates (left) are compared with up-scaled weather radar data (right) - 26 May 2009 at 16:42 UTC.


Results of comparison 

It can be seen in the previous three figures that the cells are detected with PR-OBS-3 even if intensities 

and extension are estimated with varying accuracy. Precipitation zones are either misplaced or too large. 

The highest intensities according to radar data are not reached with the product. 

Comments 

The analysis of the same case study with a previous release of the product resulted in higher 

precipitation rates even if the precipitation zones were too large and the highest intensities according to 

radar data were not reached.


3. Validation exercises in Germany 

3.1 Statistical analysis for the period September-December 2008 

Meteorological data 

Calibrated national radar composite, merged hourly precipitation at ground level from 16 radars, 

calibrated with rain gauge data, 1 km resolution (RADOLAN). 

Outline of methodology 

After a visual inspection of radar data and exclusion of noticeably erroneous data, an inverse distance 

weighted upscaling of RADOLAN data to PR-OBS-3 data from 1 st September 2008 – 31 st December 

2008 of the concurrent hour was performed. Only data-pairs with valid satellite and RADOLAN data 

were evaluated. 

Results 

PR-OBS-3 as a product ingesting PR-OBS-1 and PR-OBS-2 is effected by both error sources a) 

mismatching of low precipitation from PR-OBS-1 and b) season-wise under and overestimating for 

winter and summer, respectively, from PR-OBS-2. 

However, September 2008 shows an overrepresentation of higher precipitation classes not observable to 

that extent in PR-OBS-2. 

Fig. 3.1 - Probability density functions of PR-OBS-3 and RADOLAN precipitation.


BIAS and standard deviation express incremental underestimation of precipitation and decreasing 

dynamic range of PR-OBS-3 towards Winter. 

Table 3.1 - Results of the continuous statistics 

2008 09 2008 10 2008 11 averages 

ME 0.026 -0.059 -0.050 -0.027 

MB 1.277 0.429 0.197 0.634 

MAE 0.180 0.125 0.071 0.125 

RMSE 0.937 0.463 0.356 0.585 

MSE 0.877 0.214 0.127 0.406 

R 0.153 0.121 0.029 0.101 

SD 0.853 0.257 0.132 0.414 

3.2 Case study: 17 April 2009 

Meteorological event 

The time-span for this study was selected on good performance of the PR-OBS-03 products in terms of 

a high probability of detection and a low false alarm rate while a high number of pixels were registered 

as „wet‟ from validation data. 

16.04.2009 (Fig. 3.2a) - Low Quirin I+II moved from the Bay of Biscay over Brittany towards 

Germany causing intense showers and scattered thunderstorms in France, while the situation in 

Germany remained warm and dry. Later Quirin split completely and the cold front of Quirin II made 

slow progress towards NE. 

17.04.2009 (Fig. 3.2b) - Quirin II was located over central Germany accompanied with intense 

precipitation (partly > 34 mm per 12h) and thunderstorms. North and East of its margins it remained 

warm and dry. In the wake of the low‟s shift towards the Czech Republic and Poland strong 

precipitation was recorded especially in SE Germany with 130 mm per 24h at the Great Arber, the 

highest peak of the Bavarian-Bohemian mountain ridge. 

18.04.2009 (Fig. 3.2c) A low pressure trough established after the dissipation of low Quirin. It reached 

from Poland over south and central Germany to Northern France. This area experienced abundant 

precipitation but no thunderstorms anymore. Northern Germany remained dry.

Products Validation Report, 30 May 2010 - PVR-03 (Product PR-OBS-3) - Appendix Page 70


Fig. 3.2 a, b, c - Surface weather charts from Freie Universität Berlin and German Weather Service for 01h MEZ 16, 17, 

18.04.2009. 

Data/products used 

Calibrated national radar composite, merged hourly precipitation at ground level from 16 radars, 

calibrated with rain gauge data, 1 km resolution (RADOLAN RW). 

The case for this study was selected on good performance of the PR-OBS-03 products in terms of a high 

probability of detection and a low false alarm rate while a high number of pixels were registered as 

„wet‟ from validation data. This methodology of case selection, rather than selecting a special 

meteorological event, was made to show the maximum potential of this precipitation product at the 

current stage of development. 

Results of comparison


Fig. 3.3 - Number of pixel counts for measured precipitation yes/no and forecast precipitation yes/no. Shown is the low 

pass filtered signal representing daily variation for the inspected period January 2009 to June 2009. 

Fig. 3.4 - Probability of detection, false alarm rate and probability of false detection computed per PR-OBS-3 data set. 

Shown is the low pass filtered signal representing daily variation for the inspected period January 2009 to June 2009.


Table 3.2 - Results of the continuous statistics in total and per precipitation class for 17.04.2009. 

Precip. class [mm] total 0 - 0.25 0.25 - 0.5 0.5 - 1.0 1.0 - 2.0 2.0 - 4.0 4.0 - 8.0 8.0 - 10 10 - 16 16 - 32 

No. Sat. 1275962 1110172 49747 51457 39872 20522 4192 0 0 0 

No. val. 1275962 946273 77587 88393 85408 55572 20632 1409 667 21 

ME -0.25 N/A -0.16 -0.44 -1.03 -2.17 -4.33 -7.39 -10.39 -15.61 

St.Dev. 0.97 N/A 0.58 0.67 0.84 1.12 1.69 1.77 1.95 1.43 

MAE 0.40 N/A 0.42 0.68 1.22 2.29 4.40 7.39 10.39 15.61 

BIAS 0.38 N/A 0.58 0.39 0.28 0.22 0.18 0.16 0.09 0.06 

R 0.33 N/A 0.03 0.04 0.04 0.10 0.08 0.02 -0.05 0.25 

RMSE 1.00 N/A 0.60 0.80 1.33 2.44 4.65 7.60 10.57 15.68 

URD-RMSE 1630.04 N/A 1.70 1.12 0.93 0.87 0.86 0.86 0.91 0.94 

Table 3.3 Results of the categorical statistics for 17.04.2009 

POD 0.35 

FC 0.74 

FAR 0.32 

POFD 0.08 

The selected period in April shows superior quality measures compared to the first half-year of 2009. 

While POD reaches a local maximum at April 17th POFD remained relatively low which is not 

inevitably the case. At the same time false alarm rate reached an absolute minimum (for rain events) of 

about 25%. 

A comparably high amount of wet pixels was recognized at the correct location by PR-OBS-3 but the 

underestimation especially of larger precipitation classes was still obvious (BIAS in Table 3.2, Fig. 3.5 

and Fig. 3.6). 

Fig. 3.5 - Multi-category contingency table for 17.04.2009. Columns hold relative abundance for precipitation classes from 

observations and rows equally for PR-OBS-3, class-breaks are 0, 0.25, 0.5, 1, 2, 4, 8, 10, 16, 32 and 64 mm/h. A perfect 

result would show 1 for all elements on the diagonal. Values are normalized to total counts of the respective row.


Comments 

Fig. 3.6 - Probability density function for 17.04.2009. 

The fact of underestimation during this period of convective influenced precipitation raises the question 

if the performance differences between winter and summer, found in earlier experiments, are to be 

attributed to the differing dominance of convective and stratiform precipitation and their signature in the 

cloud-radiation database as previously assumed or if other season-dependent error sources were 

responsible. 

Further insights might be gained by a seasonally resolved analysis of the differing detection accuracies 

under proven convective and non-convective situations.


4. Validation exercises in Hungary 

4.1 Case studies: four, in August 2006 


Radar data; from the Hungarian radar network which consists of 3 Doppler radars. It provides data every 

15 min. 

Cases’ description 

Four cases from the summer period of 2006 (12 nd August 2006 05:42 UTC, 12 th August 2006 12:12 

UTC, 12 th August 2006 15:12 UTC, 30 th August 2006 02:12 UTC) were selected to demonstrate the 

performance of the PR-OBS-3 Precipitation Product compared to the radar measurements. Also, the 

SAFNWC Cloud type product was displayed at the same time to investigate correlation between MSGderived 

Cloud type and the MSG-derived precipitatin rate. Both stratiform and convective precipitation 

was investigated. 


The cases were selected according to the first results obtained by a visual validation methodology. This 

means that H03 Precipitation rate products and radar measurements were displayed at the same time, for 

the whole day of 12 th and 29-30 th August 2006 to select the most interesting and representative cases. 

In order to display the PR-OBS-3 data, we transform it on a regular grid on the same projection as the 

radar data. Time alignment is obtained through the matching of the closest radar measurement in time. 

For the statistics calculated, the 6 closest radar pixels (2km resoolution) were attributed to each satellite 

pixel. 

Results 

The following images show the case studies carried out to characterise the differences between the PR- 

OBS-3 products and the radar measurements, but also the second version of the H02 product was 

displayed to see the differences between the two HSAF products. PR-OBS-3 is a merge between SSMI 

and MSG data, so correlations between MSG Cloud Types and H03 were investigated. Each of the 4 

images is divided into four parts. The uppermost images are the PR-OBS-3 and PR-OBS-2 products, 

subsequently. On the bottom, radar mages and SAFNWC CT images were displayed. 

12 August 2006 05:42 UTC. 12 August 2006 12:12 UTC


12 August 2006 15:12 UTC 30 th August 2006 01:12 UTC. 

Fig. 4.1 - Case studies for the comparison of H-03 Precipitation Product and radar data. 

Conclusions: 

The PR-OBS-3 products depicts well both stratiform precipitation and convective cells. 

It extends in some cases largely the area of precipitation, giving rain intensities where no rain is seen by 

the radar. 

It has a strong correlation with the SAFNWC Cloud Type Product, as it is derived from the same data as 

the CT product. Some of the false rain intensity locations can be attributed to high thick clouds. 

Intensity structures are much better depicted by PR-OBS-3 than by PR-OBS-2 v.2.


4.2 Statistical analysis for the period December 2007 - March 2009 

The PR-OBS-3 data has been validated against radar data which covering Hungary every 15-minutes. 

Validation results have been prepared for the period December 2007 – March 2009. 

The up scaling technique for radar data is a simple technique that chooses the closest radar pixels 

centers to the satellite pixel center and makes an average over these radar pixels found.The statistical 

scores are presented in three different ways, agreed within the common validation philosophy. Each of 

the scores are plotted in order to be able to receive the information more clearly. The following 

statistical scores are applied: 

1) continuous statistical scores - tables 

2) multi-categorical statistical scores - histograms 

3) rain/no-rain scores – graphics 

Results for continuous statistical scores 

The number of pixels represents the total number of satellite pixels compared to radar data within that 

month, regardless satellite passes. 

Table 4.1 – Continuous statistical scores for period December 2007 - March 2009 

H03 2007 2008 Febr March Apr May June July 

Dec Jan 

number of 

pixels 

459929 

76 

459387 

39 

447455 

25 

424314 

13 

476562 

44 

523206 

26 

512539 

65 

432088 

10 

sme 0,075 0,114 0,027 0,027 0,001 0,059 0,427 1,124 

smae 0,114 0,136 0,046 0,099 0,069 0,132 0,536 1,177 

corr 0,022 0,022 0,051 0,119 0,141 0,236 0,214 0,280 

std 0,328 0,482 0,198 0,355 0,417 0,850 3,127 5,402 

H03 2008 Sept Oct Nov Dec 2009 Febr March 

Aug 

Jan 

number of 

pixels 

531341 

81 

385805 

86 

515432 

29 

470415 

58 

515251 

50 

468607 

68 

470957 

95 

478370 

34 

sme 0,200 0,015 0,015 -0,002 -0,016 -0,003 -0,017 -0,021 

smae 0,250 0,086 0,054 0,044 0,051 0,043 0,028 0,032 

corr 0,210 0,110 0,203 0,123 0,088 0,166 0,006 0,038 

std 2,514 0,758 0,351 0,267 0,244 0,234 0,347 0,177 

Conclusions: 

standard deviation similar to H02 (July the highest) 

low correlation values (in summer only 0.2, winter close to 0) 

highest absolute and mean errors among the instantaneous products 

Results for multi-categorical statistical scores 

Multi-categorical tables are calculated and they are represented on the following graphics. The meaning 

of the colors is that if green, the same category was measured by the satellite and the radar. If blue, the 

satellite underestimated (if dark, with at least 2 categories, if light blue, with one category). If yellow, 

the satellite overestimated precipitation with one category, if orange; with more than one category (so in 

this case the satellite measurement cannot be acceptable).


Fig. 4.2 - Multi-categorical statistical scores for months March, July and November 2008, and February 2009. 

Conclusions: 

good hits only in summer (20%), but then in each categories! 

general underestimation of the cases 

lowest intensities are well captured in 10 % app., March slightly better than November.


Results for rain/no-rain scores 

Fig. 4.3 - Rain/no-rain scores for the period December 2007 - March 2009.


4.3 Case studies for spring and summer: five cases, period May-September 2008 

14 Sept 2008 15:27 UTC - Mostly cloudy weather, a front line appearing to the south of Hungary 

belonging to a Mediterranean cyclone – large humidity content. 

Features: areas of precipitation misdetected, it corresponds obviously to Cloud Type 

05 May 2008 12:27 UTC - High pressure over Hungary, but the air is unstable. Convection appeared in 

several locations due to unstable air masses. 

Features: general underestimation, convection is not detected. Location of precipitation is found. 

SAFNWC Probability of Precipitation is much more correlated to the radar image.


25 July 2008 16:27 UTC - shallow cyclone above Eastern-Europe causes lot of clouds; environment 

favorable to convection. 

Features: precipitation misdetection, convection is not detected. Location of precipitation is not found. 

SAFNWC Probability of Precipitation Product is much more correlated to the radar image. H03 is 

correlated to the Cloud Type image. 

18 June 2008 15:27 UTC - A frontal zone comes through Hungary from the North to the South. It is 

cloudy due to the front. convection appears in the early afternoon. 

Features: areas of precipitation well detected, it corresponds obviously to Cloud Type, rain rate, 

convection is not captured. Huge overestimation of rate from high opaque clouds.


01 May 2008 18:42 UTC - Cold front passing over Hungary. It causes some precipitation. some 

convection also occurred due to unstable conditions. 

Features: areas of main precipitation well detected, it corresponds obviously to Cloud Type. Some small 

rain rates are not detected in tho Southern part of Hungary.


4.4 Case studies: seven, in the period January-June 2009 

21 Jan 2009 - Europe is covered largely by clouds caused by the fronts and the cyclones being 

generated. From the South-West, a wet air is flowing over Hungary. Its temperature is moderate, 

causing rain especially on the Western part of the country. The temperature did not go under zero during 

night. 

Features: H03 gives rain at small patches. It can detect the rainfall from the high and very high level 

clouds (indicated white and brownish on the cloud Type image).


Features: H03 gives rain at small patches mainly where there very high level clouds (indicated white in 

the cloud Type image). Measurement of the H02 increases a bit the intensity of H03. But not the area. It 

is determined by the Cloud Type.


10 February 2009 - Fast-moving cyclone reached Western-Europe in the morning and this cyclone had 

warm- and cold frontal lines as well which both brought precipitation over the Carpathian Basin. This is 

why from the early afternoon, rain was detected in Hungary, even snowfall in the North. 

Features: H03 places rain where the H02 indicates. The H02 measurement time is just before the H03 

time. Rain intensity is also the same for H02 and H03. It helps for H03 to determine rain form the 

brownish high-level clouds.


20 April 2009 15:33 UTC - High-pressured area prevails from Great Britain to the Black Sea, almost no 

clouds there. But the weather in the Carpathian Basin is influenced by wet airmass coming from the 

Mediterranean. 

Features: H03 detects well rainy spots. The intensity is overestimated in the South of Hungary, probably 

due to the very high-level clouds in the SAFNWC Cloud Type image. It is the same for the 15:25 UTC 

and 16:25 UTC image, although for 16:25 UTC the intensity is diversified. Note that the shape of the 

white cloud in Cloud Type image and the yellow-green rain intensity is the same, but H03 gives it more 

to the North! It is a general feature.


23 April 2009 - While there were anti-cyclonic weather situation to the North of Hungary, a frontal zone 

just reached the county in the first hours of the day. Along this line, clouds have developed and rainy, 

stormy weather was observed.


Features: H03 overestimates rain at 00:55 UTC, while suddenly high rain intensity disappears at 01:25 

UTC. Meantime, there is H02 measurement, that can be the cause.


22 May 2009 - A strong frontal line runs from Sweden to Spain. This frontal line reached Hungary late 

afternoon, bringing heavy rain, sometimes hail. Temperature fell behind the front. 

Features: H03 rain intensity decreases from 14:55 UTC to 15:10 UTC. H02 measurement is due to 

15:20 UTC. 

H03 is similar to the cloud shapes in the Cloud Type image. The rain measured by H02 on the Western 

part of the country is not marked correctly on the H03 image.



11 June 2009 - Two fronts are shaping the weather situation over Europe. Hungary was attained by the 

front coming from the West. In the morning, shiny weather but from early afternoon cloudiness 

appeared over Hungary causing heavy rain and storms. 

Features: H03 underestimates rain intensity in both cases. It is a general feature for the H03 product. 

H02 gives the structure of the rain intensity, but H03 does not. 

H03 is similar to the cloud shapes in the Cloud Type image. There is a high correlation between the 

Cloud Type and the H03 product. It is also a general feature.


25 June 2009 - A cyclone over the Black Sea shaped the weather of Southern Europe as well as 

Carpathian Basin. It was relatively warm, but wet air-mass over the territory which produces scattered 

showers and rains all over the day. 

Features: H03 decreases in intensity when the H02 is taken into account (01:35 UTC). 

H03 underestimates rainfall (01:40 UTC), but marks well the rainy areas compared to the radar image.



Features: H03 disappears from one slot to the next one. there is H02 measurement in between. H03 

rainy area shapes are much correlated to the Cloud Type white (very high level) clouds.


4.5 Results for the H03 product for the period April -December 2009 

Results for continuous statistical scores 

The number of pixels represents the total number of satellite pixels compared to radar data within that 

month, regardless satellite passes. 

The results presented here are those from the reprocessed version of the H03 product. 

H02 2009 May June July Aug Sept Oct Nov Dec 

Apr 

number of 

pixels 

432630 

47 

424314 

13 

410574 

09 

435329 

33 

456256 

89 

448138 

44 

414221 

36 

420716 

12 

447416 

80 

sme 0.009 -0.032 0.085 -0.048 -0.064 -0.026 -0.026 -0.026 -0.026 

smae 0.049 0.059 0.255 0.062 0.079 0.039 0.067 0.056 0.046 

corr 0.164 0.082 0.001 0.148 0.166 0.17 0.146 0.013 0.142 

std 0.401 1.041 15.707 0.672 0.859 0.461 0.413 3.56 0.277 

Conclusions: 

Standard deviation is very high in June (the reason is unknown). Also std is increased in November 

just as for the H02 product. 

Low correlation values, there is no significant improvement between summer and winter. Whereas 

for the products in 2008, there was a noticeable change (in summer only 0.2, winter close to 0). 

Highest absolute and mean errors among the instantaneous products. Underestimation is general 

contrary to the overestimation of H01 and H02. 

The following Figures summarize the results obtained for the product H03 from December 2007 until 

the end of 2009. Both the reprocessed and the original versions are marked. It can be seen that there is 

not a remarkable difference between the reprocessed and the original versions, except for June (most 

probably wrong data was provided).


The correlation coefficients vary month by month in a random way, no real tendencies can be remarked. 

However, one thing can be seen: in the old versin, the correlation was better for June and July than for 

the other months. Winter time is the lowest correlation coefficient. 

Results for multi-categorical statistical scores 

Multi-categorical tables are calculated and they are represented on the following graphics. The meaning 

of the colors is that if green, the same category was measured by the satellite and the radar. If blue, the 

satellite underestimated (if dark, with at least 2 categories, if light blue, with one category). If yellow, 

the satellite overestimated precipitation with one category, if orange, with more than one category (so in 

this case the satellite measurement cannot be acceptable).


Conclusions: 

• Most of the hits only in July, but then still around 5%. It is very low. 

• Usually underestimates by more than one categories from the 3 rd category onward, which is very 

poor result. 

• The last year (2008) was characterized by much better agreements in March and July of that year. 

• Lowest intensities are well captured, April slightly better than December


Results for rain/no-rain scores 

The following Figure shows the rain/no-rain scores for the H03 products from December 2007 till end 

of 2009. The false alarm rate (FAR) is very high in all season, just a bit lowering in the summer months 

(still around 0.6). The Probability of Detection also gets better in June but very low in February. The 

monthly variations of these scores are observable also here as also in the H01 and H02 product.


5. Validation exercises in Italy 

5.1 Case studies: two, 19 October and 18 November 2006 


Rain gauges data; 30 minute cumulative values from c.a. 1398 stations 


Two cases were selected for Italy: 19-oct-2006 and 18-nov-2006. Both cases show typical autumnwinter 

pressure configuration, with a minimum around the gulf of Genoa, and a N-S cold front sweeping 

from west to east the Italian Peninsula, carrying moist air from SW. During the two cases stratified 

precipitation was present for most of the day, with embedded deep convection, forced by orography. 


First, a consistency check was carried out on the raingauge database, to screen out unrealistic values and 

missing points. The Barnes objective analysis was then performed, in order to remap the irregular 

distribution of the point measurements onto a regular 5x5 km grid over Italy, obtaining a total of about 

11130 gridpoints with data. In order to match the satellite estimates with the maps derived by the 

raingauges, a simple nearest neighbour scheme has been adopted, given the similar ground resolution of 

the two grids. No parallax correction was adopted, and this can be a possible source of problem in 

matching the two databases. For this technique we have to match temporal sequences of satellite maps 

( t=15 min), while 30 min cumulated values were available from the raingauges. Thus the satellite 

estimates were averaged every 30 min in order to compare values referred to the same time intervals. 

Results 

First, the capability of the satellite technique to reconstruct the PDF of the measured precipitation fields 

is assessed. In Fig. 5.1, the satellite product PDF is compared with the raingauges PDF considering colocated 

pixels. In this case the database is much larger, since we compared 2 maps every hour, for a total 

of 96 maps during the two days. The two PDFs show rather similar shape for the low-medium rainrates 

(0.5 to 3.5 mm h -1 ), where the satellite slightly overestimate the gauges PDF, following the behaviour 

shown in section 5.2.3.4a, since the results of the PR-OBS-2v.1 was used to feed the calibration scheme 

of this technique. From 3.5 mm h-1 on, the technique PDF goes rapidly to 0 and it is not able to estimate 

values above 7 mm h -1 with a frequency comparable to the gauges PDF. 

frequency 

1 

0.1 

0.01 

0.001 

IR+MW blended 

raingauges 

number of pixels (#) 

1000 

800 

600 

400 

200 

class 2 

class 3 

class 4 

class 5 

class 6 

class 7 

class 8 

0.0001 

0 5 10 15 20 

rainrate (mm h -1 ) 

Fig. 5.1 - Probability Density Function for AMSU.V1 (black line) 

and raingauges (red line). PDFs are sampled every 0.25 mm/h. 

0 

2 3 4 5 6 7 

satellite rainrate classes 

Fig. 5.2 - Number of pixels of raingauge rainrate classes 

as function of satellite rainrate classes. 

In Fig. 5.2 it is shown how the wet pixels are distributed through the satellite and raingauges classes. 

Class 1 is not reported because the overwhelming number of elements, while classes 9 and 10 were 

empty for the two considered case studies. Raingauge classes 3, 4 and 5 are distributed among satellite 

classes 2, 3 and 4, without clear peaks, indicating that this technique is not able to distinguish among


low-moderate rainrates. Raingauge classes 6, 7 and 8 are peaked in correspondence with satellite class 4 

(1 to 2 mm/h) are the most correlated since the occurrence peaks in the raingauges class correspond to 

the same satellite class. 

In Table 5.1, some statistical indicators are computed in order to summarize the performance of the 

technique. The rainrate of 0.25 mm/h has been chosen as threshold for rain/no-rain discrimination, 

accordingly to the rainrate classes definition. First, it has to be remarked that the DWR is rather high for 

both cases, so the ETS may be underestimated. The ETS values are very low, and the impact of the IR is 

evidenced by the increase of FAR, with respect to the values obtained for PR-OBS-2v1. Moreover, there 

are big differences form October and November cases, more marked than for PR-OBS-2v1: i.e. October 

greatly overestimates, while November underestimates. Concluding remarks can be outlined after the 

first case studies for this technique. The use of MW precipitation estimates to calibrate IR estimate is a 

challenging issue far to be fully addressed. The present algorithm, for the October case, replicates rather 

closely the performances of the calibrating MW algorithm in terms of POD, while is less precise in 

assigning precipitation classes. It has to be noted that the “calibrating algorithm” (PR-OBS-2v1) is well 

far to produce satisfying results. 

Table 5.1 - Statistical parameter values for the two events 

19 October 2006 18 November 2006 

POD 0.28 0.06 

FAR 0.89 0.85 

ETS 0.06 0.04 

BIAS 2.6 0.4 

HSS 0.18 0.04 

DWR 37.21 37.77 

A strong reduction of the number of classes (to non more than 3 or 4) would make easier this analysis. 

Very likely, a preliminary analysis of the meteorological event going on may be helpful in better drive 

the calibration, given the large difference between October and November cases. Finally, an influence 

on these poor results can derive also from bad pixel co-location, due to parallax, and from temporal 

mismatching. Temporal and spatial averaging will probably help in increasing the performances of the 

technique. 

5.2 Statistical analysis for the period September 2008 - March 2009 

Reference dataset 

The calibration and validation activity in Italy is carrying on by using the raingauge Italian network, 

made available by the Italian DPC. The dataset is described in section 2.2 of this report. 

We will consider the period from September 2008 to March 2009 as “reference period”, and most of the 

results are referred to this period with the data described above. Nevertheless, for some analysis 

requiring more data (e.g. seasonal sensitivity) older datasets with sampling time of 30 minute were used. 

Data treatment 

By using the Barnes objective analysis, the raingauges have been interpolated over a 5x5 km equispaced 

grid on a 288 lines by 240 column matrix. A total of about 11,000 grid-points over land have been made 

available for comparison with h03 products. 

Upscaling technique 

Given the similar ground resolution of h03 product and the raingauges grid, the matching between the 

two maps is done based on the “nearest neighbour” criteria: each satellite pixel is compared with the 

closest gridpoint of the reference map. The hourly rainrate measured by the raingauges is matched with 

the average value of the four estimated rainrates within each hour. 

Probability density functions


The comparison of the satellite and gauges derived rainrate probability density functions (PDFs) allows 

a general overview of the algorithm performance, highlighting the capabilities in detecting or missing 

particular precipitation classes. This is particularly useful for algorithm developers, that may benefit 

from this analysis and work on the algorithm to improve deficiencies. 

In Fig. 5.3 the PDFs for h03 and gauges are presented for the reference period: the curves are smoother 

than for h01 and h02 due to the very large number of pixels considered. For this product the seasonal 

sensitivity of the PDF is not considered because of the lack of h03 summer data. 

Fig. 5.3 - PDFs for gauges and h03 for the reference period. 

The h03 estimates overestimate very low rainrates, and this will impact the rain/no-rain statistics, as 

it was found also for h01 and h02, which is used to calibrate the h03 product. The h02 calibration of 

IR radiances is able to provide a wider range of rainrates, if compared with other IR-based 

techniques. Nevertheless, the technique is not able to estimate rainrates higher than 25 mm/h, while 

h02 it is: the impact of such highest rainrates is probably lost during the calibration process. 

Rain/no rain delimitation 

For rain/no-rain delimitation, the well known POD, FAR and ETS are considered. To build the 

contingency tables for each satellite slot, the rain/no-rain threshold is 0.25 mm/h. Only slot with at least 

100 valid wet IFOV are considered in order to have a meaningful statistics. 

In Fig. 5.4a the distribution of ETS values shows that for most of the events the ETS is between 0.0 and 

0.1, while for very few cases it reaches high values. In Fig. 5.4b, the scatterplot between FAR and POD 

is presented. 

a) b) 

Fig. 5.4 - Distribution of ETS values a) and POD values plotted vs corresponding FAR values b).


The main problem of h03 seems to be the detection of precipitation as it also was found for both h01 

and h02, since POD values are rather low, while a significant number of slots present very low FAR 

values (Fig. 5.4b). Nevertheless, in contrast with microwave products, relatively high POD values can 

be found, given the capability of IR to generally overestimate the extent of the precipitation area and 

then, sometime, to detect light precipitation. 

Quantitative rainrate estimation 

A measure of the ability of the h03 product to quantitatively estimate rainrate is the categorical Heidke 

Skill Score, computed over the 11 rainrate classes defined in this project. First, it has to be mentioned 

that such a high number of classes is probably not appropriated in our case, given the reference dataset 

used. While it is of certain interest to compare satellite areal instantaneous precipitation with reference 

precipitation of the same nature (such as the one derived by radar data), the comparison with point-like, 

hourly cumulated rain amount (such as the one obtained by raingauges) is much more questionable, 

given the discontinuous character of precipitation fields. Moreover, radar derived rainrate is a 

continuous variable, while commonly used tipping-bucket raingauges measures rainrate at steps of 0.1 

or 0.2 mm/h, so that classes like 1, 2 and 3, collecting precipitation within the intervals 0.0-0.25, 0.25- 

0.5, 0.5-1.0, respectively, cannot be properly defined in the raingauges dataset. 

Nevertheless, to keep homogeneity with other results in this project, we used the defined rainrate 

classification, bearing in mind that the results may not be fully significant. In Fig. 5.5 the distribution of 

the HSS values for the 11 rainrate classes is shown. The frequency peak is between 0.0 and 0.1, as it 

was for both h01 and h02. This histogram shows a distribution of HSS very similar to the one obtained 

for h02, indicating the high impact of h02 estimates in determining the h03 values. 

Fig. 5.5 - Distribution of HSS values. 

The HSS gives the ability of h03 to assign a given rainrate in the right class; it also measures how far is 

the estimate from the correct class, if there is no perfect matching. To evaluate the specific contribution 

of each class to the HSS value, we computed the distribution of the satellite IFOVs as classified by h03 

on the gauges classes, and plotted in Fig. 5.6.


Fig. 5.6 - Estimated rainrate distribution as function of gauge classes. 

Class 1 and 11 have been omitted in the graph, because first class has an incomparably high number of 

pixels and the last because it is empty. Satellite classes are decreasingly populated from the first to the 

last. Satellite class 2 is peaked over gauges classes 4, satellite classes 3 and 4 over gauges class 5, and 5 

and 6 over class 6. This is rather encouraging if compared with the distribution of h02, where the more 

populated satellite classes were all peaked over gauges class 5. The h03 algorithm takes advantage of 

the calibration by h02, but the IR provides an added value in distributing precipitation among the classes 

more similar to what is found with the raingauges. 

Coastline impact over the estimates 

Coastal areas deserve a peculiar attention while validating satellite precipitation products, especially if 

derived by microwave sensors. The product h03 makes use of h02 estimates for calibration: an impact of 

First, the brightness temperature of an IFOV partially covered by cold sea and warm land can be misinterpreted 

by the retrieval algorithm. This does not affect the IR retrieval, because of the high 

emissivity of both land and sea. Second, the differential heating properties of sea and land may induce 

additional lift to the cloud overpassing a coastline, triggering the precipitation formation process. 

Generally speaking, it is expected that PMW and blended PMW-IR techniques performs better over land 

than over coastlines. Validating satellite estimates over coastal areas with raingauges may induce a 

further source of error. As a matter of fact, the reference precipitation rate over coastal areas is measured 

over land, while the precipitation over the sea is never measured, thus, for a coast IFOV, we match the 

estimated rainrate with the rainrate measured only over land. 

In Fig. 5.7 scatterplots for September 2008 are reported, where the HSS and ETS (Fig. 5.7a) and POD 

and FAR (Fig. 5.7b) are compared. The regression lines for HSS and ETS run parallel, with slope higher 

than 1, indicating better performances for coast pixels for both indicators. This is in contrast to what 

expected and found for h01: unfortunately, we did not performed the same analysis for h02. Numerical 

values are very low (below 0.3 for both indicators). Fig. 5.7b shows that the better performances over 

the coast are reached because of higher POD values.


a) b) 

Fig. 5.7 - Scatterplot of HSS and ETS over land and coast for September 2008 a); and scatterplots of POD and FAR over 

land and coastlines for September 2008 b). 

The same is computed for January 2009 and reported in Fig. 5.8. The behavior is rather different: both 

HSS and ETS values are generally lower than the September cases, and the slope of the regression lines 

is lower than 1, indicating better performances over land than over coasts. Fig. 5.8b shows different 

behavior also for POD and FAR: the two regression lines run parallel each other, indicating that better 

results over land are due to improvements in detection of rain and detection of no-rain. The 

discrepancies between September and January have to better analyzed in the next future. 

a) b) 

Fig. 5.8 - The same of Fig. 5.7, but for January 2009.


Impact of rainrate values over the performances 

The quality of the estimate depends rather strongly by the nature of the precipitating event. The quality 

of the products depends, more specifically, on the cloud structure and type, the rainfall regime, the 

background and the environmental conditions (moisture, wind speed, ecc…). 

In Fig. 5.9 the HSS values are plotted as function of the averaged rainrate measured over the scene. This 

is computed only for the slots with at least 100 wet pixels, in order to have significant statistics. The 

tendency of h03 to perform better in case of higher averaged rainrate is less evident than for h01 and 

h02 from Fig. 5.9: most of the negative HSS values (i.e. estimate worst than random) are still achieved 

for low rainrates, but also higher HSS are reached in cases with lower rainrate. 

Fig. 5.9 - Scatterplot between HSS and averaged rainrates for the reference period. 

5.3 Case study: 13 January 2009 

Meteorological event description 

Fig. 5.10 - On January 12-13 a deep low 

pressure structure established over 

north Africa. The instability in southern 

Mediterranean makes the whole area 

prone to convective development. The 

strong high pressure over Turkey 

makes slower the eastward motion of 

the low, causing a blocking situation.


Fig. 5.11 - SEVIRI IR image at 06:00 (on 

Jan 13) shows a huge cloud shield 

covering southern Italy and 

surrounding areas. It was formed from 

merging of long lasting structures 

moving from the south in the previous 

day. The cloud top is very cold, 

attesting embedded convection, that 

can also be triggered by orography 

over Sicily and southern Italy. The IR is 

not able to show the structure of the 

cloud system. 

Data/Products used 

Reference data: Italian hourly raingauges network (provided by DPC) 

Ancillary data (used for case analysis): 

SEVIRI images (courtesy of University of Dundee – NEODAAS) 

Weather charts (courtesy of Wetterzentrale) 

Result of comparisons 

The main features of this product is, if compared to h01 and h02, is the capability of providing a 

continuous monitoring of the full area of interest every 15 minutes, at a much higher ground resolution. 

We considered for this case one full day of observation, and compared the gauges hourly cumulated 

rainfall with the average of the satellite estimates in the four slots in the given hour, at 12 th , 27 th , 42 th and 

57 th minutes. 

This event can be divided in two parts: in the first half of the day convective precipitation was found 

over Sicily and Calabria, while in the afternoon more stratified and moderate rainfall reached also 

central Italy. 

These two phases were also differently described by the h03 maps. The first (convective) part was fairly 

good understood, with averaged POD around 0.6 and a rather low FAR (0.2), with corresponding ETS 

ranging between 0.22 and 0.57. The corresponding HSS show values between 0.16 and 0.35, indicating 

acceptable skill in detecting high precipitation areas, even if the estimate is markedly lower than the 

measure. 

For the second part of the event, the performances strongly decrease, showing average POD and FAR of 

about 0.2 and 0.35, respectively, while ETS and HSS show values of few cents above the no-skill value 

(zero for both indicators). 

A clear picture of this different behaviour is given by the four panels of Fig. 5.12: in the top panels 

estimates (left) and raingauges (right) precipitation from 03:00 to 04:00 UTC are shown, while in the 

bottom panels the same is presented for the 18:00-19:00 hour. The top panel shows the best 

performances and the bottom panel the worse for the whole event. 

Even for the best matching, however, the estimation technique show its main weakness, largely 

overestimating the precipitation area over southern Italy. The precipitation over Sicily is also partially 

overestimated, while smaller area over Sardinia are detected, but a little misplaced. The estimated 

rainrates are very low and is not well represented the structure of the precipitation fields in terms of 

intensity over Calabria, while over Sicily, more precipitation is correctly estimated on the eastern part of 

the Island rather on the western one, in qualitative agreement with raingauges.


The figures in the bottom panels show that, for the second part of the event, some underestimation of 

precipitating areas is also significant: no precipitation is estimated over Sicily and Sardinia, while a 

large false alarm area is present over central Italy. 

Fig. 12 - Top panel: h03 averaged precipitation map between 03:12 and 3:57 UTC (left) and raingauges hourly precipitation 

cumulated at 04:00 UTC (right) of 13 January 2009. Bottom panel: h03 averaged precipitation map between 18:12 and 19:57 

UTC (left) and raingauges hourly precipitation cumulated at 19:00 UTC (right) of 13 January 2009. Please note different 

colour scales for estimates and measure.


Comments 

This event was characterized by both convective and stratified precipitation, and h03 performed 

differently for the two regimes. The calibrating passive microwave estimate is effective in detecting and 

resolving convective structures, and errors are mainly due to the IR. For stratified precipitation, very 

likely the quality of the microwave calibration is lower, and this results in worse performance of the 

technique, given the weak relation between IR radiance and precipitation at the ground. 

Improvement of this technique have to go in two directions: to choose the best possible microwave 

estimate, and improve the IR contribution. For the first issue, it has to be considered that different 

microwave algorithms perform differently, with different pros and cons, depending on cloud system, 

precipitation type, geographical area, background. The IR could take benefit from the multispectral 

capability of the last generation of geostationary satellite sensors.


6. Validation exercises in Poland 

6.1 Case studies: three, in May and June 2007 


Rain gauges data; 10 minute cumulative values from c.a. 360 posts. 


Three selected days (22 May, 04 and 05 June 2007) with convective precipitation. Convective clouds, 

developed during those days, caused heavy rainfall occurred usually in the afternoon. 


In order to compare satellite derived rain rate with ground measurements the following procedure was 

applied. For each satellite SEVIRI / MSG pixel, the automatic rain gauges situated within that pixel 

were found. The pixel size was assumed to be 8 km over Poland and its shape to be rectangular. That 

procedure was applied for each grid. If more than one rain gauge were found within one satellite pixel, 

the ground rain rate value was calculated as the mean of all rain gauges measurements. 

Results 

The obtained results are presented on the figures below. The histograms obtained for satellite derived 

and ground measured convective precipitation intensity are quite similar for light and moderate rainfall. 

For very light and heavy rainfall, the differences are observed (Fig. 6.1, left panel). The first two classes 

(0.25-1) are clearly over numbered and very heavy rainfall cases were not recognised at all (Fig. 6.1, left 

panel). This resulted in strong underestimation of convective precipitation intensity by PR-OBS-3 

product (Fig. 6.1, right panel). 

Fig. 6.1 - (left) Histograms of satellite derived and ground rain rate values. (right) Differences between satellite derived and 

ground (rain gauges) rain rate values in relation to the ground rain rate values. 

The small value of hit rate and high false alarm rate (Table 6.1, left panel) were obtained. Also the odds 

ratio value is not too high, what would suggest that PR-OBS-3 has some problem with proper 

recognition of convective precipitation. One can also notice that the correlation between two dataset is 

almost 0, however so low value of correlation coefficient is determined by few situations with extremely 

heavy precipitation that were missed by PR-OBS-3 product. The moderate convective rainfall cases 

were properly estimated.


Table 6.1 - (left) Results of dichotomous (rain/no rain) analysis. (right) Results of the continuous statistics 



Hit rate 0.26 Mean error (mm/h) 0.08 

False-alarm rate 0.89 Mean absolute error (mm/h) 0.23 

Odds ratio 4.92 MSE (mm/h) 2 3.19 

Accuracy 0.91 RMSE (mm/h) 1.79 

Frequency Bias 2.40 Multiplicative bias 1.89 

Standard deviation of diff. (mm/h) 1.78 

Correlation coeff. 0.04 

6.2 Case study: 18 May 2008 

The main objective of the validation with rain gauges was to assess the H-SAF precipitation products 

ability to recognize and estimate the heavy convective and stratiform precipitation. 

Meteorological situation 

Using data from Polish lightning detection system, the 18 th of May 2008 was selected for detailed 

analysis. Fig. 6.2 presents the lightning activity map on the 18 th May 2008 during the time interval of 

09:00 UTC - 21:00 UTC. 

Fig. 6.2 - Lightning activity map, 09:00-21:00UTC, 18.05.2008; yellow points correspond to inter-cloud discharges, red – 

positive cloud-to-ground discharges and blue – negative cloud-to-ground discharges 

That day, the cold front, connected with low pressure from above the Baltic Sea, lingered over the 

Northwester part of Poland, while the Southeastern part of the country was under the influence of 

slowly moving cold frontal wave (Fig. 6.3). The reasonably cold maritime polar air was coming over 

Northwestern Poland and the worm tropical air was coming to the Southeastern part of the country. Big 

thermal contrast (horizontal temperature gradient over country was of 10 C ), high relative humidity in 

the frontal zones, unstable thermodynamic equilibrium of the lingering air masses and finally the 

collision of two frontal zones resulted in the intensification of meteorological phenomena in front of and 

within the frontal zones.


Fig. 6.3 - Meteorological situation: DWD analysis at surface, 18.05.2008 12 UTC. 

The most hazardous atmospheric events occurred over Southern part of Poland. There were numerous 

heavy storms accompanied by hurricane wind (gusts up to 180 km/h), downpours, hail falls, and 

tornados. In many places, the convective and stratiform precipitation that occurred in the cold front zone 

caused flooding. The strong convective clouds are easily seem on the satellite image (Fig. 6.4). 

Fig. 6.4 - False colour image composite (0.6 m, 1.6 m, 10.8 m), SEVIRI/METEOSAT-9, 18.05.2008, 13:00 UTC.


Ground data 

The H-SAF precipitation products were validated using data from the automatic rain gauges. Polish 


network density increases in the Southern Poland, where the flood danger is very high. The 

measurements time resolution is 10 minutes, what allows estimating the rain rate with reasonable 

quality. 

Each post is equipped with two gauges: heated and non-heated, what enables some quality control of 

data. For validation purposes, readings from both gauges were compared in order to eliminate the cases 

of clogged instruments (rain rate increases continuously). If both gauges worked properly, higher values 

was taken (automatic RG are known to underestimate the real precipitation). However, during winter 

(November-March) data only from heated RG were available, so the quality of ground data was lower 

than during other months. 

Results 

In order to combine satellite products with rain gauges data, for each satellite pixels, the automatic posts 

situated within that pixel were found. The pixel size was take into account, however, its shape was 

assumed to be rectangular. If more than one rain gauge were found within one satellite pixel, the ground 

rain rate value was calculated as a mean of all rain gauges measurements. 

The statistical analysis was performed for all overpasses available for the 18 th of May 2008. The ability 

of PR-OBS-05 product to recognize the precipitation was analysed using dichotomous statistics 

parameters. The 1 mm threshold was used to discriminate rain and no-rain cases. In the Table 6.2 the 

values of Probability of Detection (POD), Proportion correct, False Alarm Rate (FAR), Probability of 

False Detection, Bias, HSS are presented. As those parameters depend on the distribution of the 

numerical force of rain, no-rain classes, the odds ration was also calculated. Odds ratio is defined as a 

ratio of the probability of an event occurring to the probability of the event not occurring and therefore 

the numbering forces of classes do not influence its value. Odds ratio ranges from 0 to infinity. The 

perfect score is infinity. 

Table 6.2 - Results of categorical statistics obtained for PR-OBS-3 on the base of all data available on 18 May 2008 

Parameter 


POD 0.42 

Proportion correct 0.34 

FAR 0.49 

Probability of False Detection 0.82 

Bias 0.82 

HSS -0.37 

Odds Ratio 0.16 

The product ability to recognize the precipitation is rather poor: the values of POD and PC are lower 

than the values of FAR and Probability of False Detection. The low values of odds ratio that is not 

influenced by numbering forces of classes also confirm this conclusion. 

The quality of each product in estimating the rain rate was estimated using the continuous statistics 

parameters were calculated and the results are presented in the Table 6.3. The product tends to 

underestimate the measured rain.


Table 6.3 - Results of continuous statistics obtained for PR-OBS-3 on the base of all data available on 18 May 2008 

Parameter PR-OBS-03 

Mean RG 1.18 

Mean satellite 1.12 

ME -0.06 

MAE 1.54 

RMSE 2.74 

In the next step, the analysis for rain classes was performed. Three categories were selected: no rain: 

rain rate (RR) 0.25 mm/h, light: 0.25 7.5 

Rain rate from RG [mm/h] 

Rain rate 

PR-OBS-3 v 1.1 

> 7.5 

(2.8, 7.5] 

(0.25, 2.8] 

[0, 0.25] 

Fig. 6.5 - Percentage distribution of PR-OBS-3 precipitation classes in the rain classes defined using rain gauges (RG) data. 

One can easily notice that only 18 % non precipitation grids were properly recognized by PR-OBS-3 

products and 69.6% of them were classified as light rain. For the grids with light precipitation the 

situation was opposite: most of them were classified as non-precipitating and only 11.1% of these grids 

were properly recognized by PR-OBS-3. For the moderate and heavy classes the percentage number of 

properly recognized pixels was 23.3% and 1.8% and respectively. 

On the other hand, temporal resolution of PR-OBS-3, that is calculated using SEVIRI IR data, is good 

enough to monitor the rain fall variability. On the Fig. 6.6, the temporal variability of measured (RG) 

and PR-OBS-3 rain rates is presented for two selected points at which heavy stratiform and convective 

precipitation occurred.

Rain rate [mm/h] 


0:12 

1:27 

2:42 

3:57 

5:12 

6:27 

7:42 

8:57 

10:12 

11:27 

12:42 

13:57 

15:12 

16:27 

17:42 

18:57 

20:12 

21:27 

22:42 

23:57 

0:12 

1:27 

2:42 

3:57 

5:12 

6:27 

7:42 

8:57 

10:12 

11:27 

12:42 

13:57 

15:12 

16:27 

17:42 

18:57 

20:12 

21:27 

22:42 

23:57 


a) 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

0 

Point 1 - 

Stratiform 

precipitation 


v.1.1 

RG 

Time UTC 

b) 

70 

60 

50 

Point 2 - 

Convective 

precipitation 

40 

30 


v.1.1 

RG 

20 

10 

0 

Time UTC 

Fig. 6.6 - Temporal variability of rain rate at selected points as deduced from PR-OBS-3 product and rain gauges data (RG) 

for the 18th of May 2008; a) stratiform precipitation event; b) convective precipitation event. 

The heavy precipitation events that occurred at analyzed points were recognized by satellite product but 

not their magnitude and duration. The stratiform event, as deduced from PR-OBS-3 lasted shorter then 

according to rain gauges data and the maximum rain rate measured around 10:00 a.m. was strongly 

underestimated by PR-OBS-3. Additionally, light precipitation observed early in morning and in the 

late afternoon was not recognized at all (Fig. 6.6a). This might be caused by difficulties in low stratus 

clouds detection on the satellite images. 

The heavy convective precipitation was also properly recognized by PR-OBS-3 but again strongly 

underestimated in the magnitude (Fig. 6.6b). 

The following conclusions can be made on the base of the above analysis: 

poor ability of H−03 to recognize precipitating grids: values of POD and PC are lower than FAR 

and PFD. Those problems may be caused by parallax effect (no correction at the moment) or applied 

cloud mask quality. 

moderate and high stratiform precipitation cases were properly recognized but not properly 

estimated;


light stratiform precipitation were not recognized, what can be connected with difficulties in recognition 

of low level, warm stratus clouds; 

convection precipitation was properly recognized but significantly underestimated. 

6.3 Case study: 15-16 August 2008 


ability to recognize and estimate the heavy convective precipitation. 


That day, South-Eastern Europe and Iberian Peninsula were under the influence of high pressure 

systems, while the rest part of Europe was under the low pressure systems (Fig. 6.7). Firstly, Poland 

was within the warm frontal zone but in the afternoon, the wavy cold front moved over Southern Poland 

and the cold air got over the Western Poland. The rest part of Poland was within hot and wet air mass 

coming from Mediterranean Sea. 

Fig. 6.7 - Surface synoptic map at 15 UTC and the map at 700 hPa level at 12 UTC. 

The temperature difference between North and South Poland was of 12°C at 6 UTC and c.a. 18°C in the 

afternoon. This big thermal contrast, high relative humidity in the frontal zones, resulted in the 

intensification of meteorological phenomena within the frontal zone. The satellite image presents the 

heavy convective system developing within the frontal zone in the afternoon 15 th of August 2008 (Fig. 

6.8). The overshooting tops products confirmed its high vertical extent.


Fig. 6.8 - False colour image composite (0.6 m, 1.6 m, 10.8 m), SEVIRI/METEOSAT-9, 15.08.2008, 16:00UTC (upper panel) 

and the OverShootingTop product (6.2 m – 10.8 m) (lower panel). 

The convective precipitation was accompanied by strong lightning activity. Almost 242 000 lightning 

were detected during 15 August 2008, what counted for 58 % of monthly sum. On the Fig. 6.9, the 

lightning activity map on the 15 th August 2008 was presented. The map was constructed on the base of 

data from Polish Lighting Detection System, PERUN.


Fig. 6.9 - Lightning activity map on 15.08.2008. 

The precipitation occurred over whole Poland except for the South-Eastern Poland. Heavy and intensive 

precipitation was observed within the frontal zone of 400km width. In many places, 24 hour cumulative 

precipitation exceeded 60 mm. In the South Poland, in Pielgrzymowo, the 24 hour cumulative 

precipitation reached 135.7 mm 

Ground data 





quality. 




was taken (automatic RG are known to underestimate the real precipitation). 

Results 





On Fig. 6.10 and Fig. 6.11 the PR-OBS-2 product is visualized for two selected overpasses. For 

comparison, the distribution of 10 minute precipitation obtained from RG data is presented. The RG 

derived precipitation map was obtained using Near Neighbor method.


Fig. 6.10 - PR-OBS-3 at 03:57 UTC on the 15 th August 2008 (left panel) and 10 minute precipitation interpolated from RG data 

from 04:00 UTC (right panel). 

Fig. 6.11 - PR-OBS-3 at 15:27 UTC on the 15 th August 2008 (left panel) and 10 minute precipitation interpolated from RG data 

from 15:30 UTC (right panel). 

From the above examples it can be concluded that PR-OBS-3 rather properly recognizes the 

precipitation however, the moderate precipitation in the North Poland was not detected. It could be also 

noticed that PR-OBS-3 tends to overestimate its intensity (Fig. 6.11). 

Further analysis was performed for all overpasses available for the 15-16 th of May 2008. The ability of 

PR-OBS-03 product to recognize the precipitation was analysed using dichotomous statistics 

parameters. The 0.25mm/h threshold was used to discriminate rain and no-rain cases. In the Table 6.4 

the values of Probability of Detection (POD), False Alarm Rate (FAR), Accuracy, Bias are presented. 

As those parameters depend on the distribution of the numerical force of rain, no-rain classes, the odds 

ration was also calculated. 

Table 6.4 - Results of categorical statistics obtained for PR-OBS-3 on the base of all data available on 15-16 August 2008 

Parameter 


POD 0.43 

FAR 0.17 

Accuracy 0.74 

Bias 0.95 

Odds Ratio 3.84

Precentage contribution 


Higher value of POD than the value of FAR indicate that the product ability to recognize 

the precipitation is quite good. 

The quality of each product in estimating the rain rate was studied using the continuous statistics 

parameters. The results are presented in the Table 6.5. 

Table 6.5 - Results of continuous statistics obtained for PR-OBS-3 on the base of all data available on 15-16 May 2008 

Parameter PR-OBS-03 

Mean RG 1.68 

Mean satellite 2.54 

ME 0.86 

MAE 3.02 

RMSE 5.64 

In the next step, the analysis for rain classes was performed. The categories were selected in accordance 

with the common validation. Fig. 6.12 shows the percentage distribution of satellite derived 

precipitation categories within each precipitation class defined using ground measurements. 

100 

90 

80 

70 

60 

50 

40 

30 

20 

10 

0 

1.7 3.8 

0.8 4.4 

6.2 

13.2 

8.6 

12.1 

28.4 

18.6 24.0 

48.1 

30.0 

23.1 

83.4 

65.4 

19.2 

21.5 

39.6 

11.9 

21.6 

14.1 

[0 0,25) [0,25 2) [2 8) [8 32) >=32 

Rain rate from RG [mm/h] 


>=32 

[8 32) 

[2 8) 

[0,25 2) 

[0 0,25) 

Fig. 6.12 - Percentage distribution of PR-OBS-3 precipitation classes in the rain classes defined using rain gauges (RG) 

data. 

One can easily notice very good ability of PR-OBS-3 to recognize no-rain pixels: 83 % of nonprecipitating 

pixels were properly recognized by PR-OBS-3. On the other hand, most of the pixels with 

light precipitation were classified as non-precipitating ones and only 18.6% were properly recognized. 

The moderate, heavy and extreme precipitation was properly classified in 24%, 28.4% and 4.4%, 

respectively. 

On the other hand, temporal resolution of PR-OBS-3, that is calculated using SEVIRI IR data, is good 

enough to monitor the rain fall variability. On the Fig. 6.13, the temporal variability of measured (RG) 

and PR-OBS-3 rain rates is presented for two selected points at which heavy convective precipitation 

occurred.

12 

257 

412 

527 

642 

757 

1512 

1627 

1742 

1857 

2012 

12 

127 

242 

442 

912 

1042 

1227 

1342 

1457 

1742 


42 

427 

557 

727 

857 

1027 

1212 

1342 

1512 

1642 

1812 

1942 

2212 

27 

157 

612 

942 

1112 

1627 



a) Katowice 

70 

60 

50 

40 

30 

20 

10 

0 

RG H-03 

Time 

80 

70 

60 

50 

40 

30 

20 

10 

0 

b) Łódź 

RG H-03 

Fig. 6.13 - Temporal variability of rain rate at two selected points as deduced from PR-OBS-3 product and rain gauges data 

(RG) for the 15-16th of August 2008. 

Time 

The heavy precipitation events that occurred at analyzed points were recognized by satellite product but 

not their magnitude and duration. The duration of events with heavy convective precipitation was 

overestimated by PR-OBS-3 and its intensity usually underestimated. 

To sum it up, the analysis performed for situation with heavy convective precipitation showed very 

good ability of PR-OBS-3 product in recognition of heavy convective rain events. It has been also found 

that heavy convective precipitation events were underestimated while moderate and light convective 

precipitation events were often overestimated.


6.4 Case study: 21-22 January 2009 


ability to recognize and estimate precipitation. 


That day, W circulation resulted in thaw (temperatures over 0°C day and night) and snowfall. Low 

pressure centre over N Atlantic (970 hPa) tends to deepen during next hours (see Fig. 6.14). Intense 

rainfall over Sicily and S Italy, isotherm 12°C reaches Croatia and plume of cirrus clouds covers 

Slovakia and Poland. The today frontline over Poland is an introduction to an intense and prolong 

precipitation period of snow and snow with rain in S Poland (see Fig. 6.15). 

Fig. 6.14 - 500 hPa geopotential map of Europe on 21 st January 2009 at 0000 UTC.


Fig. 6.15 - Synoptic chart of Poland on 22 nd January 2009 at 0000 UTC.


Fig. 6.16 - SEVIRI/METEOSAT-8 10.8 

m image, 21.01.2009, 16:15 UTC 

The convective precipitation was accompanied by faint lightning activity (stable startiform precipitation 

with no convective updraft, see Fig. 6.16). On the Fig. 6.17, the lightning activity map on the 21-22 nd 

January 2009 was presented. The map was constructed on the base of data from Polish Lighting 

Detection System, PERUN.


Fig. 6.17 - Lightning activity map on 21-22.01.2009 

Ground data 





quality. 





Results 





On the Fig. 6.18 PR-OBS-3 product is visualized for two selected overpasses. For comparison, the 

distribution of 10 minute precipitation obtained from RG data measured at the closest time slot is 

presented. The RG derived precipitation map was obtained using Near Neighbor method.


55 

54 

53 

52 

51 

50 

H03 rain rate 21.01.2009 ,1612 UTC 

49 

14 15 16 17 18 19 20 21 22 23 24 

2.1 

2 

1.9 

1.8 

1.7 

1.6 

1.5 

1.4 

1.3 

1.2 

1.1 

1 

0.9 

0.8 

0.7 

0.6 

0.5 

0.4 

0.3 

0.2 

0.1 

0 

55 

54 

53 

52 

51 

50 

RG rain rate 21.01.2009, 1612 UTC 

49 

14 15 16 17 18 19 20 21 22 23 24 

Fig. 6.18 - PR-OBS-3 at 16:12 UTC on the 21 st January 2009 (left panel) and 10 minute 

precipitation interpolated from RG data from 16:20 UTC (right panel) 

2.1 

2 

1.9 

1.8 

1.7 

1.6 

1.5 

1.4 

1.3 

1.2 

1.1 

1 

0.9 

0.8 

0.7 

0.6 

0.5 

0.4 

0.3 

0.2 

0.1 

0 

From the above examples it can be concluded that most of the precipitation was not recognized by PR- 

OBS-3 or its intensity was estimated to be smaller than the assumed in the frame of WP2300 threshold 

value, 0.25 mm/h, used to discriminate rain from no-rain cases. 

Further analysis was performed for all overpasses available for the 21-22 nd of January 2009. The ability 

of PR-OBS-03 product to recognize the precipitation was analysed using dichotomous statistics 

parameters. The 0.25mm/h threshold was used to discriminate rain and no-rain cases. In the Table 6.6 

the values of Probability of Detection (POD), False Alarm Rate (FAR), Accuracy, Bias are presented. 

Table 6.6 - Results of the categorical statistics obtained for PR-OBS-3 on 

the base of all data available on the 21-22 nd of January 2009 

Parameter 

Scores 

POD 0.26 

FAR 0.45 

Accuracy 0.75 

Bias 0.47 

Higher value of FAR than the value of POD indicate that the product skill to estimate the stratiform 

precipitation spatial distributions is not very good. 


parameters. The results are presented in the Table 6.7. Although maximum values of rain rate estimated 

by PR-OBS-3 is higher than the measured one, the mean values of rain rate differ – the satellite derived 

product tends to underestimate the measured precipitation.


Table 6.7 - Results of the continuous statistics obtained for PR-OBS-3 

product on the base of all data available on the 21-22 nd of January 2009 

Parameter 

Scores [mm/h] 

Max RG 5.4 

Max H03 2.48 

Mean RG 0.86 

Mean H03 0.29 

ME -0.56 

MAE 0.74 

RMSE 0.86 

St.Dev 0.66 

The rain rate intensities, measured and derived from satellite data, for 21-22 nd of January 2009 were 

presented on the scatter plot (Fig. 6.19). The graphs was created using all pixels for which either 

satellite derived or measured precipitation was found. First of all, one can easily notice that measured 

values of precipitation intensity are discrete, what indicates that the measured precipitation intensity did 

not change in time. Then, it can be concluded that moderate precipitation (intensity higher than 2 mm/h) 

is underestimated by PR-OBS 3. 

. 

Fig. 6.19 - PR-OBS-3 and RG rain rate scatter plot obtained for all overpasses dated for 21-22 nd January 2009. 

To sum it up, the analysis performed for situation with light stratiform precipitation showed that the 

ability of PR-OBS-3 product to recognise is not very good. While no clear tendency cannot be found for 

light stratiform precipitation, for the moderate rain, the products underestimates the its intensity.


6.5 Case study: 11 May 2009 

The main objective of the validation with rain gauges was to assess the H-SAF PR-OBS-3 (H-03) 

precipitation products ability to recognize and estimate the heavy convective precipitation. 


On the 11 th of May the dominant frontline moved from Poland to Ukraine and Bielarus leaving ground 

for new group of clouds translocating to Poland from Germany. Low and medium clouds are moving 

over S Poland bringing convective precipitation (see Fig. 6.21 and also Fig. 6.22). This rainfall was 

magnified by small low pressure centre moving from Austria over Slovakia (see Fig. 6.20). The main 

wind direction over Middle Europe was N but clouds moving from Germany to Poland are directed by 

SW circulation. Ground frosts were possible in Poland due to N cold flow. 

Fig. 6.20 - Surface synoptic map at 12 UTC on 11 th May 2009.


Fig. 6.21 - HRV imagery SEVIRI/METEOSAT-9, 11.05.2009, 16:30 UTC 

showing huge convective cell that caused meaningful precipitation. 

Fig. 6.22 - Lightning activity map on 11.05.2009 

The meteorological situation resulted in heavy convective precipitation that occurred in the afternoon, at 

the South of Poland. The 6 hour cumulated precipitation measured at the SYNOP stations exceeded 60 

mm. The precipitation was accompanied by strong lightning activity within clouds convective cores. On


the Fig. 6.22, the lightning activity map on the 11 th May 2009 is presented. The map was constructed on 

the base of data from Polish Lighting Detection System, PERUN. 

Ground data 


network of automatic rain gauges consists of 430 posts alocated all over the country, however, the 


measurements time resolution was set for 10 minutes, what allows estimating the rain rate with 

reasonable quality. 





The analysis was limited to the area where convective precipitation had been detected. The area was 

defined using the lightning activity map coordinates (Fig. 6.22). 

Results 


located within that pixel were found. The pixel size was take into account, however, its shape was 

assumed to be rectangular. If more than one rain gauge was found within one satellite pixel, the ground 

rain rate value was calculated as a mean of all rain gauges measurements within that pixel. 

On the Fig. 6.23 the PR-OBS-3 product is visualized for the afternoon overpasses. For comparison, the 

distribution of 10 minute precipitation obtained from RG data is presented. The RG derived 

precipitation map was obtained using Natural Neighbor method. The 0.25 mm/h threshold was used for 

rain/no rain differentiation (all values below 0.25 mm/h were marked with white). 

H03 rain rate 11.05.2009, 1627 UTC 

54 

53 

52 

51 

50 

14 15 16 17 18 19 20 21 22 23 24 

32 

RG rain rate 11.05.2009, 1627 UTC 

30 

28 

2654 

24 

2253 

20 

18 

52 

16 

14 

1251 

10 

8 

50 

6 

4 

2 

14 15 16 17 18 19 20 21 22 23 24 

0 

32 

30 

28 

26 

24 

22 

20 

18 

16 

14 

12 

10 

8 

6 

4 

2 

0 

Fig. 6.23 - PR-OBS-3 rain rate at 1627 UTC on the 11 th May 2009 (left panel) and 

precipitation interpolated from RG data from 1630 UTC (right panel), [mm/h]. 

One can easily notice the differences in distribution of measured and estimated by PR-OBS-3 

precipitation. The PR-OBS-3 precipitation is more uniformly distributed over the South Poland than the 

measured one. Moreover, none of the events with heavy precipitation seen in measurements (Fig. 6.23 

right panel) was recognized by PR-OBS-3 (Fig. 6.23 left panel). 

Further analysis was performed for all overpasses available for the 11 th of May 2009. The ability of PR- 

OBS-03 product to recognize the precipitation was analysed using dichotomous statistics parameters.


The 0.25 mm/h threshold was used to discriminate rain and no-rain cases. In the Table 6.7 the values of 

Probability of Detection (POD), False Alarm Rate (FAR), Accuracy, Bias are presented. 

Table 6.7 - Results of the categorical statistics obtained for PR-OBS-3 

on the base of all data available on the 11 th of May 2009 

Parameter 

Scores 

POD 0.51 

FAR 0.53 

Accuracy 0.89 

Bias 1.07 

The POD and FAR values are almost equal what indicates that the product ability to recognize 

the convective precipitation is not very good. 


parameters. The results are presented in the Table 6.8. It is easy to notice that in this case, the product 

tends to underestimate the measured rain. 

Table 6.8 - Results of the continuous statistics obtained for PR-OBS-3 

product on the base of all data available on the 11 th of May 2009 

Parameter Scores [mm/h] 

Max RG 58.8 

Max H03 2.3 

Mean RG 3.56 

Mean H03 0.59 

ME -2.96 

MAE 3.11 

RMSE 6.94 

St.Dev 6.78 

The rain rate intensities, measured and derived from satellite data, for 11 th of May 2009 were presented 

on the scatter plot (Fig. 6.24). 

Fig. 6.24 - PR-OBS-3 and RG rain rate scatter plot obtained for all overpasses dated on 11 th May 2009.

Rain Rate (H-03-RG) [mm/h] 

Dec'07 

Jan'08 

Feb'08 

Mar'08 

Apr'08 

May'08 

Jun'08 

Jul'08 

Aug'08 

Sep'08 

Oct'08 

Nov'08 

Dec'08 


6.6 Statistical analysis for the period December 2007-December 2008 


The PR-OBS-3 rain rate product has been validated against automatic rain gauges data. Polish network 

of automatic rain gauges consists of 430 posts located all over the country, however, the network 

density increases in the Southern Poland, where the flood danger is very high. Each post is equipped 

with two gauges: heated and non-heated, what enables some quality control of data. For validation 

purposes, readings from both gauges were compared in order to eliminate the cases of clogged 

instruments. If both gauges worked properly, higher values was taken (automatic RG are known to 

underestimate the real precipitation). 









Results 

Following the methodology agreed in WP 2300, both continuous and categorical statistics were 

calculated on the monthly mean basis. 

Continuous statistics 

On the 5, values of the mean error, mean absolute error and RMSE calculated on the base of all PR- 

OBS-3 data are presented for each month of the analysed period. It should be pointed out here that the 

analysis was performed only for situations when the ground rain rate was 0.25 mm/h. 

14 

12 

10 

8 

6 

4 

2 

0 

-2 

-4 

ME 

MAE 

RMSE 

Fig. 6.25 - Mean error (ME), mean absolute error (MAE) and RSME of PR-OBS-3 for period Dec 2007 - Dec 2008 for Poland. 

For the most part of the analysed period, the PR-OBS-3 overestimates the measured rain rate values, 

only in November and December 2008 the ME is negative. The overestimation is significant in summer 

(Fig. 6.25).


Categorical statistics 

The quality of PR-OBS-3 in precipitaion detection was validated using the dichotomous (yes/no) 

statistics. The 0.25 mm/h threshold was used for rain/no rain differentiation. On the Fig. 6.26 and Fig. 

6.27 the variability of Probability of Detection, False Alarm Ratio, Accuracy and Probability of False 

Detetction is presented. 

2 

1,5 

H-03 

1 

0,5 

0 

FAR 

POD 

Fig. 6.26 - Variabily of Probability of Detection (POD), False Alarm Ratio (FAR) obtained for PR-OBS-3 using Polish RG data. 

H-03 

1,05 

1 

0,95 

0,9 

0,85 

0,8 

0,75 

POFD 

ACC 

Fig. 6.27 - Variabily of Accuracy (ACC) and Probability of False Detetction (POFD) obtained for PR-OBS-3 using Polish RG 

data. 

For the most of the analysed period, the POD is lower than 0.5 – only in the first part of analysed period 

the POD of 0.6 was obtained. Taking into account high values of False Alarm Ratio, higher than 0.8 for 

the whole period as well as the fact that for each month FAR values are always higher than POD values, 

it should be concluded that the ability of PR-OBS-3 of precipitation detection is not satisfactory. On the 

other hand, high values of ACC and low values of POFD indicate that PR-OBS-3 properly recognises 

non-precipitating areas. The quality of precipitation detection depends on the season – it is better at the 

beginning of the analysed period and in the summer.


6.7 Statistical analysis for the period January-December 2009 


The PR-OBS-3 v.1.2 rain rate product has been validated against automatic rain gauges data. Polish 


network density increases in the Southern Poland, where the flood danger is very high. Each post is 

equipped with two gauges: heated and non-heated, what enables some quality control of data. For 

validation purposes, readings from both gauges were compared in order to eliminate the cases of 

clogged instruments. If both gauges worked properly, higher values was taken (automatic RG are known 

to underestimate the real precipitation). 









Results 

Following the methodology agreed in WP 2300, both continuous and categorical statistics were 

calculated on the monthly mean basis. 

Continuous statistics 

On the 6.28, values of the mean error, mean absolute error and RMSE calculated on the base of all PR- 

OBS-3 v.1.2 data are presented for each month of the analysed period. It should be pointed out here that 

the analysis was performed only for situations when the ground rain rate was 0.25 mm/h 

Fig. 6.28 - Mean error (ME), mean absolute error (MAE) and RSME of PR-OBS-3 v.1.2 for period Jan - Dec 2009 for Poland. 

The PR-OBS-3 v.1.2 underestimates the measured rain rate values for the whole analysed period. 

The underestimation is stronger in summer and late spring. The quality of the product in rain rate 

estimation decreases in summer an increases in winter (Fig. 6.28). 

Categorical statistics 

The quality of PR-OBS-3 v.1.2 in precipitaion detection was validated using the dichotomous 

(yes/no) statistics. The 0.25 mm/h threshold was used for rain/no rain differentiation. On the Fig.


6.29 and Fig. 6.30 the variability of Probability of Detection, False Alarm Ratio, Accuracy and 

Probability of False Detetction is presented. 

H-03 

1,4 

1,2 

1 

0,8 

0,6 

0,4 

0,2 

0 

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 

FAR 

POD 

Fig. 6.29 - Variabily of Probability of Detection (POD), False Alarm Ratio (FAR) obtained for PR-OBS-3 v.1.2 using Polish RG 

data in 2009. 

Fig. 6.30 - Variabily of Accuracy (ACC) and Probability of False Detetction (POFD) obtained for PR-OBS-3 v.1.2 using Polish 

RG data in 2009. 

The quality of PR-OBS-3 v.1.2 in precipitation detection is very poor, the POD exceeds 0.2 only in 

April and May. Taking into account that for each month FAR values are always higher than POD 

values, it should be concluded that the ability of PR-OBS-3 of precipitation detection is not satisfactory. 

It is especially low in winter.

GEO/IR+LEO/MW 


7. Validation exercises in Slovakia 

7.1 Statistical analysis for the period June-September 2007 


Slovak radar network precipitation intensities at ground level 


PR-OBS-3 GEO/IR supported by LEO/MW v1.0 data representing precipitation rate at ground were 

compared to radar precipitation intensities. Data for the period from 1 June 2007 to 30 September 2007 

were available and processed. Time frequences of both radar and satellite data were 15 minutes. 

Upscaling method to reproject satellite data into radar projection map was developed and used. Testing 

was based on comparison of precipitation averaged over 5x5 radar pixels. 

Results 

Visual inspection of validated products showed excessively large areas of low precipitation intensities, 

especially connected with storms anvils. Therefore class 0-0,25 mm/h was excluded from the following 

chart (Fig. 7.1) showing distribution of PR-OBS-03 data in comparison to radar measurements. In 

principle, propriety of the algorithm adjustment can be detected from the class 4,0-8,0 mm/h which is 

matching the radar values. 

0,2 

Distribution of GEO/IR+LEO/MW in comparison to Radar rain intensities 

0,18 

0,16 

0,14 

0,12 

0,1 

0,08 

0,06 

0,25-0,5 

0,5-1,0 

1,0-2,0 

2,0-4,0 

4,0-8,0 

8,0-10,0 

10,0-16,0 

16,0-32,0 

0,04 

0,02 

0 

0,25-0,5 0,5-1,0 1,0-2,0 2,0-4,0 4,0-8,0 8,0-10,0 10,0-16,0 16,0-32,0 

Radar 

Fig. 7.1 - Distribution of PR-OBS-03 data in comparison to radar measurements 

Parameters of continuous statistics (Table 7.1) show the wide spreading of errors and generally very low 

correlation in comparison to correlation of PR-OBS-2 data (which reaches 0,6). Blending process of 

GEO/IR and LEO/MW can be considered as one of the main progenitors of errors growth. 

Table 7.1 - Scores of continuous statistics per the period June- September 2007 

GEO/IR+LEO/MW - Radar 200706 200707 200708 200709 

Mean error 0,771 1,356 1,045 0,746 

(Multiplicative) bias 9,094 16,408 8,868 5,172 

Mean absolute error 0,866 1,458 1,227 0,94 

Root mean square error 3,418 6,238 7,526 2,645 

Mean squared error 11,68 38,916 56,639 6,994 

Correlation coefficient 0,223 0,144 0,007 0,248 

Anomaly correlation -99 -99 -99 -99 

Skill score -99 -99 -99 -99 

Standard deviation 11,518 37,514 20,157 6,808


Fig. 7.2 - Comparison of radar cappi product, 7 September 2007 20:00 UTC (top left), PR-OBS-03 (top right), Airmass RGB 

(bottom left) and Microphysics RGB (bottom right). Dark red cloud over eastern Slovakia in the top right image was not 

detected by the radar. According to MSG RGB products it is evident that the cloud top temperature and microphysics affect 

substantially the rain at ground detection from the satellite.

percentage of pixels (log) 


7.2 Statistical analysis for the period August 2008 - February 2009 

After the first step (radar data availability and quality) all available data were checked visually. Smaller 

period with nodata occurred in September due to upgrade of the radar software. Very good data 

availability was during December 2008 and January 2009. 

Distribution of radar and PR-OBS03 (h03-v1.2 product) precipitation intensities are shown in Table 7.2. 

Precipitation intensities up to class 64 mm/h were observed over Slovakia radar network, but in 

coincidence with PR-OBS-3 data were found only classes up to 16 mm/h. 

Table 7.2 - Distribution of radar and PR-OBS-3 precipitation data over Slovak radar network 

Rain 

class 

[mm/h] 

Number of 

radar pixels 

Number of 


pixels 

Percentage 

of radar 

pixels 

Percentage 

of PR-OBS- 

2 pixels 

0 35199440 37472244 90,26% 96,08% 

0,25 2371837 537459 6,08% 1,38% 

0,5 958978 509665 2,46% 1,31% 

1 337868 302971 0,87% 0,78% 

2 101760 143270 0,26% 0,37% 

4 24572 28386 0,06% 0,07% 

8 2261 1686 0,01% 0,00% 

10 1784 3139 0,00% 0,01% 

16 661 520 0,00% 0,00% 

32 153 0 0,00% 0,00% 

64 26 0 0,00% 0,00% 

38999340 38999340 100,00% 100,00% 

0 

-1 

-2 

-3 

-4 

-5 

SK radar network / PR-OBS-3 (h03 v1.2) 

Period 200809-200902 

0 0,25 0,5 1 2 4 8 10 16 32 64 

rainrate mm/h 

SK radar 


Table 7.3 - Results of continuous statistics performed on PR-OBS-2 (h02-v2.0) against SHMÚ radar data 

Year/Month 2008 09 2008 10 2008 11 2008 12 2009 01 2009 02 

Mean error -0,41 -0,50 -0,60 -0,59 -0,31 -0,60 

Mean Abs.error 0,64 0,70 0,67 0,61 0,66 0,65 

RMSE 0,81 0,77 0,73 0,64 0,82 0,67 

Correlation coefficient 0,04 0,01 0,03 0,05 0,02 0,07 

Standard deviation 0,70 0,58 0,43 0,25 0,76 0,30 

According to correlation coefficient between radar precipitation intensities and PR-OBS-3 values the 

performance of PR-OBS-3 was not satisfactory and therefore it is necessary repeatidly to check 

manually input data and processing procedures for hidden failures. Also case studies can help us to 

understand better the performance of PR-OBS-3 data over Slovak radar network.


7.3 Case study: 29 March 2009 

Each measurement method of precipitation brings specific problems and errors. To avoid these errors 

and to receive more certain precipitation ground truth SHMU is adapting and developing new tools and 

algorithms for integration of precipitation from various sources of precipitation measrurements – 

raingauge network, radar network and NWP outputs. INCA is a numerical system under adaptation in 

SHMU for integration of these data and purpose is to have comprehensive analysis of precipitation data 

which can be considered as the best guess of ground truth. Inca is now defined in the domain over the 

teritory of Slovakia and neighbouring countries. As input data for precipitation analysis are used: 

Automatic weather stations 

Automatic precipitation stations 

Automatic hydrological stations 

Climatological precipitation measurements 

Radar nework precipitation measurements 

Satellite derived precipitation data (instantaneous precipitation PR-OBS-03) can be validated against 

comprehensive INCA analysis or also used as additional input to this analysis in the future. 

Currently INCA outputs are used only for validation purposes in the context of H-SAF, therefore only 

comprehensive analysis of raingauges and radar data was used in this case study. The following case 29 

March 2009 shows the problems with catching the precipitation fields by PR-OBS-3 and over/underestimation 

of the precipitation intensities over Slovakia. On the Fig. 7.2 SEVIRI Airmass and HRV+IR 

clouds RGB-products show the distribution of clouds over central European region. There were few 

isolated rain fields with different microphysical characteristics over Slovakia which were detectable by 

radar signal: Field A over west part, field B over north-east, field C in the central and D over south-east 

part of Slovakia. 

Fig. 7.2 - SEVIRI Airmass and HRV+IR clouds products show distribution 

of precipitating clouds over Slovakia on 29 March 2009 13:00 UTC 

.


Fig. 7.3 - 29 March 2009 13:00 UTC: PR-OBS-3 precipitation intensity field (top-left), INCA comprehensive analysis based on 

raingauge and radar measurements (bottom-left), overlaping areas of PR-OBS-3 and INCA (top-right), over/underestimation 

of PR-OBS-3 against INCA (bottom-right). Meaning of colors: Green – precip detected only by INCA, blue – only 

by PR-OBS-3, orange – by INCA and PR-OBS-3, light blue – overestimation, dark blue - underestimation of precip by PR- 

OBS-3 

While precipitation fields detected by PR-OBS-3 show only small intensities lower then 4 mm/h 

and extra in the region B there is no signal, ground based data catched local heavy precipitation 

higher then 15mm/hour near to the west Slovakian border in region A and also precipitation of 

medium intensities in regions B, C and D. 

After visual inspection of time sequence of imagery for this case in the period 00:45-23:45 UTC we 

found number of regions for which PR-OBS-3 data did not detect any precipitation but there was 

precipitation localised by INCA analysis. We can considere this as one of reasons of statistical 

disturbances observed in continuous statistics which can lead to results of poor performance of PR- 

OBS-3 in validation period 200809-200902 over Slovakia.

0,0 

2,0 

4,0 

6,0 

8,0 

10,0 

12,0 

14,0 

16,0 

18,0 

20,0 

22,0 

24,0 

26,0 

28,0 

0,5 

2,5 

4,5 

6,5 

8,5 

10,5 

12,5 

14,5 

16,5 

18,5 

20,5 

22,5 

24,5 

26,5 

28,5 

0,5 

2,5 

4,5 

6,5 

8,5 

10,5 

12,5 

14,5 

16,5 

18,5 

20,5 

22,5 

24,5 

26,5 

28,5 

0,0 

2,0 

4,0 

6,0 

8,0 

10,0 

12,0 

14,0 

16,0 

18,0 

20,0 

22,0 

24,0 

26,0 

28,0 

0,0 

2,0 

4,0 

6,0 

8,0 

10,0 

12,0 

14,0 

16,0 

18,0 

20,0 

22,0 

24,0 

26,0 

28,0 

0,0 

2,0 

4,0 

6,0 

8,0 

10,0 

12,0 

14,0 

16,0 

18,0 

20,0 

22,0 

24,0 

26,0 

28,0 

0,0 

2,0 

4,0 

6,0 

8,0 

10,0 

12,0 

14,0 

16,0 

18,0 

20,0 

22,0 

24,0 

26,0 

28,0 

0,0 

2,0 

4,0 

6,0 

8,0 

10,0 

12,0 

14,0 

16,0 

18,0 

20,0 

22,0 

24,0 

26,0 

28,0 

0,0 

2,0 

4,0 

6,0 

8,0 

10,0 

12,0 

14,0 

16,0 

18,0 

20,0 

22,0 

24,0 

26,0 

28,0 


7.4 Statistical analysis for the period January-September 2009 

Distribution of radar and PR-OBS03 (h03-v1.2 product) precipitation intensities were produced for each 

month and Probability Density Functions (PDF) and scatter plots were created to perform comparison of 

both data types. PDF functions are shown in charts below (Fig. 7.4). Pink colour represents number of 

satellite precipitation pixels and yellow radar precipitation pixels. Vertical scale is logarithmic. 

Distribution of H03 and SK Radar 200901 

Thresholds 

Sat200901 


Thresholds 

Sat200902 


Thresholds 

Sat200903 

7 

Rad200901 

7 

Rad200902 

7 

Rad200903 

6 

6 

6 

5 

5 

5 

4 

4 

4 

3 

3 

3 

2 

2 

2 

1 

1 

1 

0 

0 

0 

7 


Thresholds 

Sat200904 

Rad200904 

7 


Thresholds 

Sat200905 

Rad200905 

7 


Thresholds 

Sat200906 

Rad200906 

6 

6 

6 

5 

5 

5 

4 

4 

4 

3 

3 

3 

2 

2 

2 

1 

1 

1 

0 

0 

0 

7 


Thresholds 

Sat200907 

Rad200907 

7 


Thresholds 

Sat200908 

Rad200908 

7,0 


Thresholds 

Sat200909 

Rad200909 

6 

6 

6,0 

5 

5 

5,0 

4 

4 

4,0 

3 

3 

3,0 

2 

2 

2,0 

1 

1 

1,0 

0 

0 

0,0 

Fig. 7.4 - PDF’s of PR-OBS-3 and radar measurements in the period January-September 2009. 

As can be observed from charts above generally satellite signal detects lower number of precipitation 

pixels than radar. But very good coincidence was found for February 2009 and April 2009. No 

coincidence was found only for September 2009 where H03 product practicaly didn‟t detect rain. In 

other months from validation period the PDFs can be splitted into two parts: 1. for lower rain intensities 

there is a good coincidence or small decreasing of satellite signal on precipitation pixels and 2. for 

higher rain intensities there is loss of the coincidence or no pixels were observed in satellite signal. Also 

the threshold values can be determined. This threshold varies from month to month and is highlited in 

the charts by vertical blue lines. It can be stated that decreasing of number satellite pixels can be 

partially caused by upscaling technique but partially by precipitation algorithms insufficiency. 

Series of scatter plots in Fig. 7.5 show additional information about relation between h03 and radar 

precipitation estimates.


H03v1.2-Rad January 2009 H03v1.2-Rad February 2009 H03v1.2-Rad March 2009 

H03v1.2-Rad April 2009 H03v1.2-Rad May 2009 H03v1.2-Rad June 2009 

H03v1.2-Rad July 2009 H03v1.2-Rad August 2009 H03v1.2-Rad September 2009 

Fig. 7.5 - Scatter plots PR-OBS-3 v/s radar for the period January-September 2009. 

Results of continuous statistics are shown in Table 7.4. Number of radar derived precipitation cases 

belonging to certain precipitation class over land and the number of satellite precipitation cases 

belonging to the same precipitation class was counted and stored into the first two lines of the partial 

table. Each partial table contains also the following statistical parameters: mean error, standard 

deviation, mean absolute error, mean bias, correlation coefficient and root mean square error. These 

statistical results in comparison to similar results for products H01 and H02 show higher causality on 

stochastic characteristics of precipitation. While products H01 and H02 are based on physical retrieval, 

in the case of H03 there is strong impact of additional algorithms dealing with matching the IR MSG 

SEVIRI data and MW precipitation estimations. 

For selected validation period January-September 2009 the results of validation over Slovakia region are 

showing that H03 product did not detect higher precipitation intensities coming especially from 

convective clouds. This fact is demonstrated by mean error which is acceptable for rain intensities lower


then 4 mm/h (highlighted in Table 7.4 by yellow) but not acceptable for higher precipitation intensities 

(highlighted in Table 7.4 by red colour). This result is valid through the whole validation period. 

Table 7.4 - Results of continuous statistics performed on h03-v12 against SHMÚ radar data for the period 200901-200909 

Precip.class Month 200901 200902 200903 200904 200905 200906 200907 200908 200909 

0,25-0,5mm/h Number of SAT 36965 9654 14010 5504 38156 44881 20482 33260 3390 

Number of RAD 376395 465501 589298 150678 422422 456567 310250 371508 213806 

Mean error -0,15 -0,31 -0,30 -0,11 -0,24 -0,20 -0,18 -0,23 -0,34 

SD 0,49 0,20 0,27 0,90 0,28 0,36 0,51 0,33 0,11 

MAE 0,38 0,34 0,36 0,48 0,33 0,34 0,38 0,33 0,34 

MB 0,57 0,11 0,15 0,70 0,34 0,44 0,50 0,36 0,05 

CC -0,01 0,03 0,00 0,03 0,05 0,04 0,05 0,05 0,02 

RMSE 0,51 0,37 0,40 0,91 0,37 0,41 0,54 0,40 0,35 

URD_RMSE 6,55 7,65 7,17 54,60 10,57 15,11 18,60 15,92 5,67 

0,5-1,0mm/h Number of SAT 6304 2760 4488 5040 28716 30896 19861 18864 1179 

Number of RAD 98470 126103 253589 90417 271562 337445 231803 268777 107660 

Mean error -0,45 -0,58 -0,63 -0,34 -0,53 -0,50 -0,46 -0,53 -0,67 

SD 0,46 0,39 0,28 1,10 0,35 0,44 0,61 0,42 0,17 

MAE 0,59 0,65 0,67 0,78 0,59 0,61 0,65 0,61 0,67 

MB 0,32 0,12 0,08 0,51 0,25 0,30 0,36 0,25 0,03 

CC 0,03 0,07 0,01 0,03 0,02 0,04 0,04 0,03 0,02 

RMSE 0,65 0,70 0,69 1,15 0,63 0,67 0,76 0,68 0,69 

URD_RMSE 12,85 14,74 10,98 70,51 13,21 17,60 21,54 18,74 8,05 

1-2mm/h Number of SAT 1142 339 870 3394 6126 17370 15677 5531 34 

Number of RAD 15924 19676 71858 44646 128646 226458 169944 162164 46806 

Mean error -1,02 -1,11 -1,26 -0,85 -1,21 -1,13 -1,09 -1,16 -1,34 

SD 0,57 0,69 0,39 1,50 0,45 0,55 0,71 0,59 0,29 

MAE 1,09 1,26 1,28 1,34 1,22 1,17 1,20 1,23 1,34 

MB 0,22 0,13 0,05 0,39 0,13 0,20 0,23 0,16 0,02 

CC 0,01 0,00 0,01 0,06 -0,01 0,05 0,02 0,04 0,04 

RMSE 1,17 1,31 1,32 1,72 1,29 1,25 1,30 1,30 1,38 

URD_RMSE 31,97 37,32 20,64 100,35 19,21 21,51 25,18 24,15 12,25 


Number of RAD 1884 1406 12921 17212 53550 115444 93902 68378 17551 

Mean error -2,26 -2,39 -2,57 -1,84 -2,57 -2,41 -2,38 -2,43 -2,66 

SD 0,65 0,63 0,59 2,34 0,67 0,75 0,90 0,95 0,55 

MAE 2,26 2,43 2,58 2,51 2,57 2,42 2,43 2,51 2,66 

MB 0,11 0,03 0,02 0,32 0,06 0,12 0,14 0,11 0,02 

CC -0,03 -0,06 -0,04 0,03 0,01 -0,01 0,02 0,02 0,00 

RMSE 2,35 2,47 2,64 2,98 2,66 2,53 2,55 2,61 2,72 

URD_RMSE 92,94 139,60 48,69 161,62 29,80 30,13 33,88 37,20 20,03 


Number of RAD 127 45 3050 4895 17793 34666 29513 18179 4590 

Mean error -4,61 -4,71 -5,27 -4,14 -5,17 -5,04 -4,92 -5,01 -5,27 

SD 0,78 0,66 1,06 2,75 1,13 1,16 1,37 1,39 1,07 

MAE 4,61 4,71 5,27 4,62 5,17 5,04 4,94 5,05 5,27 

MB 0,07 0,00 0,02 0,22 0,03 0,05 0,08 0,06 0,01 

CC 0,12 -0,24 0,09 0,04 0,00 -0,06 0,01 -0,01 -0,07 

RMSE 4,68 4,75 5,37 4,97 5,29 5,17 5,11 5,20 5,37 

URD_RMSE 357,95 780,32 100,22 303,06 51,70 55,00 60,44 72,15 39,19 


Number of RAD 4 0 260 584 2123 3915 3966 2264 460 

Mean error -8,97 -9999,00 -8,80 -7,54 -8,71 -8,68 -8,41 -8,63 -8,88 

SD 0,32 -9999,00 0,60 2,43 0,71 0,68 1,24 1,03 0,57 

MAE 8,97 -9999,00 8,80 7,68 8,71 8,68 8,41 8,63 8,88 

MB 0,00 -9999,00 0,01 0,16 0,02 0,03 0,06 0,03 0,00 

CC -9999,00 -9999,00 0,02 -0,04 0,00 -0,01 0,02 0,01 -0,03 

RMSE 8,98 -9999,00 8,82 7,92 8,74 8,71 8,50 8,69 8,89 

URD_RMSE 2016,94 -9999,00 343,26 877,41 149,67 163,65 164,88 204,45 123,79 


Number of RAD 3 0 140 633 2433 4288 5070 2782 501 

Mean error -14,02 -9999,00 -11,36 -10,87 -12,19 -12,14 -11,83 -12,08 -12,20 

SD 2,34 -9999,00 1,09 2,90 1,71 1,73 2,04 1,86 1,65 

MAE 14,02 -9999,00 11,36 10,96 12,19 12,14 11,83 12,08 12,20 

MB 0,00 -9999,00 0,01 0,12 0,02 0,02 0,04 0,02 0,00 

CC -9999,00 -9999,00 0,07 -0,05 0,03 -0,05 0,00 -0,01 -0,07 

RMSE 14,21 -9999,00 11,41 11,25 12,31 12,27 12,01 12,22 12,31 

URD_RMSE 2328,97 -9999,00 467,79 842,77 139,81 156,38 145,83 184,44 118,62


7.5 Case study: 11 April 2009 

This case study is aimed at comparison of precipitation field of H03 product with radar precipitation 

field in sense of their spatial consistency. 

Fig. 7.6 shows precipitation fields observed at 11:45 UTC consisting of several significant precipitation 

cells (marked A, B, C and D). By this time, the precipitation producing convective clouds reached at 

most the mature stage of their life-cycle as is indicated by relatively small cloud top areas in echo top 

radar product (Fig. 7.6 bottom-right). Similarly, the dimensions of H03 precipitation cells are relatively 

small and comparable to the radar echoes dimensions. In the radar cappi product superimposed on the 

H03 precipitation field (Fig. 7.6 bottom-left), distinct dislocation of corresponding precipitation cells 

can be seen. 

A 

B 

C 

D 

A 

B 

C 

D 

D 

B 

C 

B 

C 

D 

A 

A 

Fig. 7.6 -11 April 2009 11:45 UTC: Dislocation of precipitation field observed by satellite and radar during mature stage of 

the convective clouds; Top-left - PR-OBS-3 precipitation intensity field, Top-right – radar cappi2km product, Bottom-left – 

radar cappi2km product (shown in black colour) superimposed on PR-OBS-3 field, Bottom-right – radar echo top height 

product. 

Although the dislocation could be partially ascribed to the quite common shift between coldest areas of 

cloud tops and true precipitation locations, magnitude and direction of the dislocation (especially in case 

of cells A, B and C) indicates the parallax error of H03 product as a probable root cause. In order to 

improve the spatial accuracy of the H03 precipitation field, parallax correction of the H03 product 

should be performed. It should also be noted that the parallax error negatively affects the results of 

statistical verification. 

In Fig. 7.7 from 14:00 UTC, the convective clouds near the Ukrainian border developed to dissipating 

stage and formed anvils. The cold high-level anvil clouds were detected by H03 algorithm as 

precipitation areas and thus the precipitation cells enlarged and covered the corresponding radar echoes


(see Fig. 7.7 bottom-left). Therefore, the cell shift is not such evident as in the previous situation from 

11:45 UTC. 

Fig. 7.7 - 11 April 2009 14:00 UTC: Dislocation of precipitation field observed by satellite and radar during mostly 

dissipating stage of the convective clouds; Top-left - PR-OBS-3 precipitation intensity field, Top-right – radar cappi2km 

product, Bottom-left – radar cappi2km product (shown in black colour) superimposed on PR-OBS-3 field, Bottom-right – 

radar echo top height product .


8. Validation exercises in Turkey 

8.1 Case study: 11 June 2007 

The study area 

The validation studies in Turkey are carried out at two catchments each representing different climate 

region (Fig. 8.1 and Fig. 8.2). The Susurluk catchment is within the Marmara Sea region, which has 

modified Mediterranean climate with hot summer and cold winter periods. However, the western Black 

Sea catchment falls along the Black Sea coast, which is subject to weather and climatic patterns from 

the northeastern Europe and Balkan area. 

Rain rate prototype products derived from PR-OBS-3 have been validated using rain-gauge data at 

AWOS sites in the Susurluk and Western Black Sea catchments. Both catchments have adequate 

number of rainfall measurements for the validation with approximately 40 AWOS scattered all over the 

study areas. 

Fig. 8.1 - Susurluk catchment 

Fig. 8.2 - Western Black Sea catchment 

Methodological Basis 

The satellite derived precipitation measurements represent an areal (rectangular) average of the product, 

and therefore a statistical approach has been adopted to determine the rain rate average for the 

corresponding area by using the ground observations. By using the Point Cumulative Semi-Variogram 

(PCSV), the spatial dependence function (SDF) for each site is determined. Parameter optimization for 

Barnes equation is performed to determine the relationship between distance and weighting coefficients 

by training with Genetic Algorithms (GA) which was rendered that it can represent these spatial 

dependence functions (SDFs). A 2 km by 2 km grid is defined inside the rectangular area and the rain 

rate amounts at each node are estimated geo-statistically by considering the observed rain rate amounts 

and the SDF functions of the nearby AWOS sites. Then, the average of the all node is compared with 

the satellite product for the validation purpose. 

The degree of spatial variability is mostly quantified by variance and correlation techniques. 

Determination of the spatial variability at locations where parameter is not observed depends very much 

on the weighting functions. Among the various weighting functions, Cressman (1951) and Barnes 

(1964) are the most well known and world wide used. However, the assumptions of isotropy and 

homogeneity are the major limitations of these techniques in determining the spatial variability. In 

addition, never-changing weighting values respect to distance, regardless of the time or even 

observation, is the other major restriction of these techniques. For this reason, point cumulative semivariogram 

(PCSV), proposed by Şen (1995), is used for the validation process for the rain rate (RR) 

products of PR-OBS-3 data. PCSV measures the spatial variability around a site to provide the regional 

effect of all other sites within the area on the site of concern by taking the observations at the 

neighboring sites into account for determination of the weighting functions.

Observation (mm/h) 



AWOS data; 1 minute cumulative values 


One selected day (11 June 2007) with convective precipitation 












Fig. 8.1 - OBS 3 v1.0 product. 

OBS-03_v1.0 

35 

30 

Fig. 8.1 shows OBS 3 v1.0 product. There is a 

convective precipitation in selected day and the 

product is characterized with high rain rates (RR). 

Scattered diagram of this product with 

observation can be seen in Fig. 8.2. Relatively 

higher RR amounts (overestimation) are observed 

for this product when compared with the ground 

observations. 

25 

20 

15 

10 

5 

0 

0 5 10 15 20 25 30 35 

Product (mm/h) 

Fig. 8.2 - Scatter diagram between observation 

and product. 

Fig. 8.3 shows the frequency data in each class, which are no rain on both (1), rain on AWOS (2), rain 

on product (3) and rain on both (4), for OBS 3 V1.0 product. According to this figure, this product has 

31 data in class 1, 1920 data in class 2, 1507 data in class 3 and 1204 data in class 4. 

2500 

2000 

1500 

1000 

500 

OBS-3_v1.0 

1: No rain on both 

2: Rain on AWOS 

3: Rain on product 

4: Rain on both 

0 

1 2 3 4 

OBS-3_v1.0 31 1920 1507 1204 

Fig. 8.3 - Classes and frequencies for each Class.


8.2 Statistical analysis for the period September 2008 - February 2009 




Sep 2008-Feb 2009 period monthly statistics. 












Results 

Table 8.1 - Statistical Scores for PR – OBS 3 product 

Sep 08 Oct 08 Nov 08 Dec 08 Jan 09 Feb 09 

ME -5.62 -5.38 -6.81 -6.39 -5.36 -5.66 

MAE 6.08 5.91 6.92 6.46 5.51 5.73 

MSE 100.62 80.70 91.45 87.61 73.48 74.75 

RMSE 10.03 8.98 9.56 9.36 8.57 8.65 

SD 8.31 7.19 6.71 6.84 6.69 6.54 

R 0.07 -0.02 -0.07 -0.10 -0.16 -0.09 

Bias 0.07 0.08 0.01 0.01 0.02 0.01 

Fig. 8.4 - Correlation variation respect to land & coast. 

Fig. 8.5 - SD comparison respect to precipitation classes.


Comments 

Fig. 8.4: Not any significant difference is observed when the correlation variation over land and cost 

areas are considered. 

Fig. 8.5: Statistical score variations respect to precipitation classes are also considered. Not a 

significant trend is observed with increasing class number. Monotonically increasing statandard 

deviation amount is observed in Fig. 8.5 respect to de decreasing sample amount with incresing 

class number. 

8.3 Statistical analysis for the period January-June 2009 

The study area 

The validation studies in Turkey are carried out by using the AWOS sites located in the north and 

western part of Turkey (Fig. 8.6). The areas where ground observations are located are under the effect 

of the Mediterranean climate type characterized with hot summer and relatively cooler winter periods. 

In addition, the area is affected by weather and climatic patterns originating from northeastern Europe 

and Balkan region. 

Rain rate prototype products derived from PR-OBS-3 have been validated using rain-gauge data at 

AWOS sites. Total number of 193 AWOS sites scattered all over the study area is used for the 

validation activities. 

Methodological Basis 

Fig 8.6 - Awos ground observation sites. 

The satellite derived precipitation measurements represent an areal (elliptical) average of the product, 



(PCSV), the spatial dependence function (SDF) for each site is determined. A 2 km by 2 km grid is 

defined inside the rectangular area and the rain rate amounts at each node are estimated geo-statistically 

by considering the observed rain rate amounts and the SDF functions of the nearby AWOS sites. Then, 

the average of the all node is compared with the satellite product for the validation purpose. 

The degree of spatial variability is mostly quantified by variance and correlation techniques. 

Determination of the spatial variability at locations where parameter is not observed depends very much 

on the weighting functions. Among the various weighting functions, Cressman (1951) and Barnes 

(1964) are the most well known and world wide used. However, the assumptions of isotropy and 

homogeneity are the major limitations of these techniques in determining the spatial variability. In 

addition, never-changing weighting values respect to distance, regardless of the time or even


observation, is the other major restriction of these techniques. For this reason, point cumulative semivariogram 

(PCSV), proposed by Şen (1995), is used for the validation process for the rain rate (RR) 

products of PR-OBS-3 data. PCSV measures the spatial variability around a site to provide the regional 

effect of all other sites within the area on the site of concern by taking the observations at the 

neighbouring sites into account for determination of the weighting functions. 




Jan 2009-Jun 2009 period monthly statistics. 




corresponding footprint area, which is the rectangle with 6kmX6km over Turkey, by using the ground 

observations. By using the Point Cumulative Semi-Variogram (PCSV), the spatial dependence function 

(SDF) for each site is determined. A 2km by 2 km grid is defined inside the rectangle and the rain rate 

amounts at each node are estimated geo-statistically by considering the observed rain rate amounts and 

the SDF functions of the nearby AWOS sites. 

Results 

The statistical scores for the Jan-Jun 2009 are presented in Table 8.2. An increasing error amount trend 

is observed from January to June. From the correlation perspective, negative trend presence is clear for 

the same time period. 

Table 8.2 - Statistical Scores for PR–OBS 3 product (over land) 

Jan 09 Feb 09 Mar 09 Apr 09 May 09 Jun 09 

ME -0.53 -0.32 -0.37 0.29 -1.23 -1.52 

MAE 0.56 0.50 0.43 1.25 1.34 1.89 

MSE 0.76 0.58 0.41 3.91 6.30 18.57 

RMSE 0.87 0.76 0.64 1.98 2.51 4.31 

SD 0.70 0.69 0.52 1.96 2.19 4.03 

R 0.05 0.02 -0.13 -0.03 -0.15 -0.04 

Bias 0.05 0.30 0.08 1.48 0.06 0.13 

Fig. 8.1 - Scatter diagram for March 2009. 

Comments 

For overall product validation: 

underestimation of the product is 

dominant (e.g., Fig. 8.7) 

no significant difference in the 

magnitude of the product rainrate 

over land & coast 

relatively lower rainrate amounts are 

observed in respect to H01 & H02 

products.

H-SAF Product Validation Report (PVR) PR-OBS-3

Create successful ePaper yourself

Delete template?

Save as template?