Slide (460 kb) - Met Office

Research to Improve Forecasting ofProbability of Precipitation (PoP)Ken MylneEnsemble Forecasting Applications Manager© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Outline• Summary of Conclusions from Written Paper• Results since September upgrades• The Stornoway problem• Potential solutions• Improved site-specific processing• Hedging• Bayesian updating• The Way forward• The future• Potential for an Ensemble Index• Ensemble Upgrades• Post-processing strategy© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Summary of Conclusions –PoP Score• PoP Score is volatiledue to• Weather sensitivity(convective/dynamic)• Sampling errors© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Summary of Conclusions –MOGREPS Changes• Localization of ETKF hasgiven:• Improved distribution ofperturbations• Reduced land-oceancontrast in perturbationscaling• Reduced overall meanperturbation size• Small improvements inprobabilistic resolution atcost of slightly reducedreliability© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Summary of Conclusions –Post-processing24hr Accumulation Brier Skill Scores• Old updating “KFMOS”approach did not performwell for precipitation• Simple constant correctionperformed better mostmonths0.80.70.60.50.40.30.20.10FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN JUL TOTALmonth 2007-2008• Average ~+0.05 per month• Calibration furtherimproved• Average ~+0.02 per month• But note: BSS is not asimple average of months© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 20080.60.50.40.30.20.10RAW Bias Log Const antJAN FEB M AR APR M AY JUN JUL TOTALMonth 2008RAW RECAL KFM OS (-0.2) RAW_RECAL

Recent Performance• Update of Table1 from paper• Sep good• Oct very poor• PoP KPT Score:• Aug 0.425• Sep 0.428• Oct 0.416• Why so Bad?Month 2005 2006 2007 2008Jan 0.174 0.365 0.462 0.347Feb 0.149 0.518 0.586 0.586Mar 0.321 0.556 0.537 0.428Apr 0.441 0.396 0.694 0.356May 0.337 0.448 0.498 0.383Jun 0.404 0.362 0.463 0.164Jul 0.466 0.276 0.368 0.351Aug 0.385 0.288 0.433 0.469Sep 0.408 0.400 0.215 0.532Oct 0.594 0.297 0.465 0.1720.417Nov 0.391 0.499 0.304(Wed 12 th )Dec 0.175 0.494 0.610© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

The Stornoway EffectPoor October score largely due to one site• Blue –new postprocess• Pink – rawMOGREPS10.503026 3091 3134 3257 3348 3544 3590 3716 3772 3827 3917• Improvementat severalsitesBSS(Site)-0.5-1EU_MODELEU_RAW-1.5-2-2.5Site ID© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Stornoway daily Brier scoresOnly two days without rain –• Blue – new post-process• Pink – raw MOGREPS• Yellow – climatology0.90.8p-p improved f/c these days0.7• Climatology PoP81% for Stornowayin October• Rained every daybut 2• BSS Reference ofclimatologyscored well• New p-p reducedf/c probs and gavelarge BSBS(contrib)0.60.50.40.30.20.10No data1-Oct-08 6-Oct-08 11-Oct-08 16-Oct-08 21-Oct-08 26-Oct-08 31-Oct-08EU_MODELEU_RAWClimatology© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Stornoway –Maritime convection not advected inlandStornoway© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Stornoway EffectImpact• BSS can give very severe penalty whenclimatology is frequent rain, and it happens• (Low PoP forecast) + (Rain) gives huge penalty• (High PoP forecast) + (Rain) gives only tiny benefit• (Low PoP forecast) + (No rain) gives potential largescore• BUT – it rained every day except two in October• Formulation of score makes it particularlyweather-sensitive in extreme climates© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Stornoway EffectWeaknesses of PoP Score• Stornoway represents tiny proportion of UK population• Extreme climate leads to disproportionate sensitivity ofoverall score• Single rain/no-rain (0.4mm) threshold means smallforecast errors heavily penalised• Negligible effect on any issued forecasts• KPT target leads to disproportionate effort to solveStornoway problem• Target driving bad behaviour.© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Can we “solve” Stornoway byHedging?© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Possibilities of Hedging StornowaySummaryOctober Stornoway Scores by HedgingHedging Approach Forecast BSSMOGREPS (No hedge) PoP -2.00Climatology Climatology (=81%) 0.0Always Rain 100% 0.046Limit min prob to 50% max(PoP, 50%): -0.838Half-way betweenclimatology andMOGREPS(PoP+Climatology)/2 -0.567• Almost any crude hedging strategy can improve theStornoway PoP, but we should not be hedging….© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Improved site-specific interpolation© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

IGPSIntelligent GridPoint SelectionModel landpoint• Bilinear Interpolation• Nearest Neighbour• Nearest Land Point• Nearest sea-point could helpStornoway precipitation• Minimum Height ErrorSiteWindModel seapoint© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

CDP – Convective DiagnosisProduct© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

The “new CDP”Additionally – advectionof maritime showersinland (on, and workswell).Model profile oftemperature (T),humidity Q andwind W selected byIntelligent GridPoint Selection inSSPSParcel tops CCT at Wwhenever temperature V-C> 1.0KCAPE between CCB andCCT“Slice tops” at C whereSALR of parcel equals ELR50 metre mixed layer potentialtemperature and dewpointabove actual site orographyConvective cloud base CCBat CCL (U) +50m© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Introducing CDPwith MOGREPSoffers potential tomeet many morecustomer needs,e.g.• Lightning• ConvectivegustsUsing the derived CCB, CCT andCAPE, together with T, Q and Wthe following are derived:• Peak shower precipitation rate• Mean shower precipitation rate (rain & snow)• Lightning flash rate• Potential hail size• Significant tornado parameter• Max. possible convective gust• Convective cloud amount estimate at site© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

How good is the CDP?Last summer independent tests at T+0 12Z fromselected UK stations (23/7/07 – 19/8/07)• CCB (from 158 surface reports)• Mean error = 31 feet• Standard deviation = 625 feet• Max. error = 2146 feet© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Verification of CDPAll UK stations T+0 to T+36 at 6 hourly intervals in period22/1/08 to 14/4/08• Shower/no shower (from 99249 surface reports, 18% of whichobserved a shower at VT)• BIAS: CDP = 29% NAE = 141%• Percent Correct: CDP = 81% NAE = 70%• Hit Rate: CDP = 13% NAE = 38%• False Alarm Ratio: CDP = 54% NAE = 73%• ETS: CDP = 0.071 NAE = 0.068• Odds Ratio CDP = 4.2 NAE = 2.0• Low Hit Rate and Bias:• Use max(CDP, NAE) for each MOGREPS member© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updating© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updating• Philosophy is to emulate forecaster behaviourby using latest forecast to update an existingprediction• Mylne (2002) ECMWF Seminar on Predictability:• Start from climatology and refine probabilities withupdated information• Start from PoP climatology• Could use a naive algorithm…© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updatingnaïve algorithmClimateForecastWeightingUpdate© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updatingnaïve algorithmClimateForecastWeightingUpdateForecastWeightingUpdate© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updatingnaïve algorithmClimate0.8ForecastWeightingUpdateForecastWeightingUpdate© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updatingnaïve algorithmClimate0.8WeightingUpdateForecastEqual0.6WeightingUpdate0.4Forecast© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updatingnaïve algorithmClimate0.8WeightingUpdateForecast0.4Equal0.6ForecastWeightingEqualUpdate0.40.2© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updating• Philosophy is to emulate forecaster behaviourby using latest forecast to update an existingprediction• Start from POP climatology• Could use a naive algorithm• Proposed method uses finite-counting statistics© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updatingnaïve vs finite counting methodsAssumed probability ofobserved frequencyInitialforecast© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 20080.8 Forecast Probability/Observed Frequency

Bayesian Updatingnaïve vs finite counting methodsAssumed probability ofobserved frequencyNewforecastInitialforecast© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 20080.4 0.8 Forecast Probability/Observed Frequency

Bayesian Updatingnaïve vs finite counting methodsAssumed probability ofobserved frequencyNewforecastEqual weightforecastInitialforecast© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 20080.4 0.6 0.8 Forecast Probability/Observed Frequency

Bayesian Updatingnaïve vs finite counting methodsAssumed probability ofobserved frequencyInitialforecast© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 20080.8 Forecast Probability/Observed Frequency

Bayesian Updatingnaïve vs finite counting methodsAssumed probability ofobserved frequencyNewforecastInitialforecast© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 20080.4 0.8 Forecast Probability/Observed Frequency

Bayesian Updatingnaïve vs finite counting methodsAssumed probability ofobserved frequencyNewforecastBayesian updateforecastEqual weightforecastInitialforecast0.4 0.80.62© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 20080.6 Forecast Probability/Observed Frequency

Bayesian Updatingnaïve vs finite counting methodsAssumed probability ofobserved frequencyNewforecastBayesian updateforecastEqual weightforecastInitialforecast0.4 0.80.54© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 20080.6 Forecast Probability/Observed Frequency

Bayesian Updating• Philosophy is to emulate forecaster behaviourby using latest forecast to update an existingprediction• Start from PoP climatology• Could use a naive algorithm• Proposed method uses finite-counting statistics• Currently using binomial distribution• Compare performance of these two options© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updatingnaïve vs finite counting methods• Applied to FY 07-08 POP for KPT sites• Inputs are• ECMWF(RECAL) at 5, 4, 3, and 2 days ahead• MOGREPS-R (old-KFMOS) at 2 days ahead• Brier Skill Scores:• Equal weighting 0.529• Bayesian updating 0.531• MOGREPS-R (KFMOS) 0.477• ECMWF (RECAL) 0.509© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updatingnaïve vs finite counting methods0.750.70.650.60.550.50.45NaïveBayesECMWF-Day2MOGREPS-Day20.40.350.30.250.2Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb MarMonth (07-08)© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updatingnaïve vs finite counting methods0.030-0.03-0.06-0.09NaïveBayesECMWF-Day2MOGREPS-Day2-0.12-0.15-0.18Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb MarMonth (07-08)© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updatingfinite counting - sharpnessECMWF-Day2 MOGREPS-Day2 Bayesian-Day2© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updatingfinite counting - reliabilityECMWF-Day2 MOGREPS-Day2 Bayesian-Day2Error bars from sampling of binomial distribution© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian UpdatingWhat does it do for KPT in October 2008?Site ID MOGREPS ECMWFRECALBayesianUpdating03026 -2.52 0.44 0.1103091 0.44 0.26 0.3703134 0.02 0.47 0.3303257 0.46 0.23 0.3803348 -0.004 0.32 0.2903544 0.36 0.56 0.5003590 0.72 0.66 0.7203716 0.40 0.29 0.4803772 0.41 0.53 0.5103827 -0.01 0.10 0.1503917 0.24 0.17 0.24• Combining all sites:• MOGREPS 0.23• EC-RECAL 0.36•B-U 0.40© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updating• Philosophy is to emulate forecaster behaviourby using latest forecast to update an existingprediction• Start from POP climatology• Could use a naive algorithm• Proposed method uses finite-counting statistics• Compare performance of these two options• Finite counting method marginally better in BSS• It also has sensible forecast reliability and sharpness• ROC area is also improved over component systems© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updatingwhy not be naïve?• Simple weighting performs well• We could try to optimise the weights© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Bayesian Updatingwhy not be naïve?• Simple weighting performs well• We could try to optimise the weights• The Bayesian approach could be refined• Test sensitivity to assumed sample sizes• Move away from using the binomial distribution• Select distribution parameters using verification data• Update parameters in real-time• Possibility of generalising;• Generate parameters without reference to longhistory of forecasts© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Planned upgraded process© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Combining improvementsFallbackClimatologyECMWFDay 5 f/cECMWFDay 4 f/cKFMOS &RECALECMWFDay 3 f/cKFMOS &RECALMOGREPSDay 2 f/cKFMOS &RECALBayesianUpdateIGPS &CDPBayesianUpdateBayesian update propagates benefits of other post-processingdevelopments like CDP to PoPBayesianUpdateBayesianUpdatePoP© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Future strategy© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Ensemble Index• Development of a new Ensemble Index shouldallow better all-round assessment ofMOGREPS performance• Ideas so far• Basket of user-oriented variables (as UK Index)• Global and Regional using different variables• RPS (or CRPS)• Ideally, capable of being calculated for Raw and PostprocessedProbability Forecasts (ie probably RPS)• Full project required to define and test© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Post-processing strategy• Anders Persson contracted to develop genericKFMOS for deterministic and ensembles• Aim to develop strategy for ensembles in UKPP• Gridded calibration• Calibration for full range of thresholds – not just 0.4mm• Dedicated calibration for severe (extreme) events• But note post-processing for extreme events is verydifficult due to shortage of training data• Training too much on case studies leads to excessfalse alarms© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Ensemble Upgrades• Some key ensemble plans will advance PoP:• Vertical localisation• Better control of perturbations at different levels• Stratosphere to prevent instability• Boundary layer for improved spread in weatherparameters• Surface perturbations• Ensembles with convection-allowing model (1.5km)• Resolved convection• Improved topographic forcing© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Summary and Conclusions• Recent performance dominated by the Stornoway Effect• Measures brought in early (with minimal testing) to try torecover KPT• Intelligent Grid-point Selection• Convective Diagnosis• Additional benefit of forecasts of other convectiveparameters of interest to customers (e.g. lightning)• Bayesian Updating• Planned MOGREPS and Post-Processing developmentsoffer hope of improved PoP in future© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008

Questions?© Crown copyright Met Office MOSAC Paper 13.9, 14 Nov 2008