Zoran Pasaric: Polychoric correlation coefficient in forecast ... - FMI

Polychoric correlation coefficient inforecast verification based onKxK contingency tablesZoran Pasarić and Josip JurasGeophysical Institute, Faculty of ScienceUniversity of Zagreb, CROATIAFourth International Verification Methods WorkshopHelsinki, 8 -10 June 2009

Outline The bivariate normal distribution (BND) and KxK table Example:- PCC for 11x11 tables of temperature change forecasts- Additional information: Biases and base rates- Reconstruction- Residual Summary of PCC More examples- QPF for the United States (6x6 tables)

CC = 0.9Bivariate normal distribution (BND)and K x K contingency table

Bivariate normal distribution (BND)and K x K contingency tableCC = 0.9 From CC towards the table

Bivariate normal distribution (BND)and K x K contingency tableCC = 0.9 From CC towards the table• From table towards the CC(ML method)• Polychoric Corelation Coefficient,PCC (Ritchie-Scott, 1918,Pearson,1922)

ExampleBrooks & Doswell (W&F,1996): Four 11 x 11 tables of temperature changesObservationCOLDER N.C. WARMERForecastWARMER N.C. COLDERNWSFOLFM-MOSNGM-MOSCONNWSFOCONForecastingsystemForecastingsystemLFM-MOSNGM-MOSCC(from B&D)0.910.870.880.90TCC, 5x5(ML method)0.9030.8480.8560.884TCC(ML method)0.9020.8560.8720.896PCC, 3x3(ML method)0.8830.8450.8250.862

ExampleBrooks & Doswell (W&F,1996): Four 11 x 11 tables of temperature changesObservationCOLDER N.C. WARMERForecastWARMER N.C. COLDERNWSFOLFM-MOSNGM-MOSCONNWSFOCONForecastingsystemForecastingsystemLFM-MOSNGM-MOSCC(from B&D)0.910.870.880.90TCC, 5x5(ML method)0.9030.8480.8560.884TCC(ML method)0.9020.8560.8720.896PCC, 3x3(ML method)0.8830.8450.8250.862

Differences: NWSFO - CONCOLDER N.C. WARMERWARMER N.C. COLDERTCC: 0.896 → 0.902

TCC measuresthe association, onlyAdditional information:Biases and marginal frequencies of observations

ReconstructionForecastThe KxK contingency table:C 1C 2… … …C KP K1P K2P O,1P O,2ObservationC 1P 11P 12… P 1KP F,1C 2…P 21P 22…………C KP 2KP F,2… …P KKP F,KP O,K• Consider the table obtained by partitioninga normalized BND according to some thresholds• From CC and marginal frequencies it ispossible to reconstruct the whole table!Bias = (P F,1/P 1., …, P F,K-1/P O,K-1),P O= (P O,1, …, P O,K-1)K x K table (TCC, Bias, P OBS)+ residualK 2 1 + (K-1) + (K-1) + 1Total no. of elements

The residuals: OverallResidual table = Original minus theoretical (BND) tableNWSFONWSFOSums of absolute differences [%]LFMMOSNGMMOSCONNWSFO11 x 11 20.3 20.2 17.4 21.25 x 5 15.0 13.5 10.1 14.23 x 3 8.6 5.5 4.5 10.8

The residuals, cont.CON: TCC=0.896,resid=17.4%,N=590COLDER N.C. WARMERNWSFO: TCC=0.902,resid=21.2%COLDER N.C. WARMERWARMER N.C. COLDERWARMER N.C. COLDERPCC: 0.896 → 0.902Correction of 2 three-class errors improves the association ascorrection of 20, or so, one-class errors

Question• Sampling variability due to insufficient sample size?or• Real features of the prognostic system ? Measure oriented Distribution oriented}Complementary approaches

Summary of PCC• Partition of informationK x K table (PCC, Bias, P OBS) + residual• Reduction in dimensionalityK 2 2*K• The PCC, Biases and P OBSare independent of each other• Using them, the table could be essentially reconstructed• The distribution oriented approach could be applied to (usuallysmall) residual

Monte Carlo, cc=PCCMore examplesQPF, USA CONUShttp://www.hpc.ncep.noaa.gov/npvu/qpfv/PCC = 0.883

QPF, USA CONUS 2008Biases, frequencies of observations, residualPCC = 0.883sum(abs(differences)) = 1.15 %< 0.01 0.01 –0.10.1 –0.250.25 –0.50.5 – 1 1

Seasonal variation Slowly but constantly increasing trend Year to year variationsQPF, USA CONUS monthlyTime evolution of PCC for 6x6 tables

QPF, USA monthly tables, 2005-2009PCC-s and residualsAll 12 RFCs, togetherPCC Residual sum [%]

QPF, USA CONUS 2008Various scores for 2x2 tables fromDependence on the base rateORSSEDSTCCHSSPSSETS

GE 0.01 inchUSA CONUS 2001-2008Trends of various scoresGE 1 inchThe TCC approximations (Pearson,1900)

Many details not mentioned here, and especially so for the TCC, could be find in:J. Juras and Z. Pasarić (2006):Application of tetrachoric and polychoric correlationcoefficients to forecast verification. Geofizika, 23, 59-82.(http://geofizika-journal.gfz.hr)Thanks for your attention!!