C - MUCM

Calibration of ecosystem models:

problems with imbalanced datasets and

model discrepancy

David Cameron, Marcel van Oijen

(Centre for Ecology and Hydrology, Edinburgh)

Mat Williams (University. of Edinburgh)

Beverly Law (Oregon State University)

dcam@ceh.ac.uk

Overview

Introduction to ecological process based models

Models currently disagree

New plentiful sources of data could help however

observational datasets are often imbalanced

Imbalanced datasets can cause problems in the

calibration of imperfect models

How can observations be used most effectively given

model discrepancy? Two pragmatic ideas proposed.

Observations used in calibration should take account of a

model’s believable scale.

Lessen over-fitting by avoiding physically improbable system

values in the likelihood.

Ecological forest models: Water, Carbon and

Nitrogen cycles.

Net tree C

gain

Photosynthesis

(P)

Respiration

(R)

dW/dt = P – R

P = k 1 x radiation

R = k 2 x W

radiation = input = f(t)

{k 1, k 2} = parameters

Ecological forest models: Water, Carbon and

Nitrogen cycles.

Net tree C

gain

Photosynthesis

(P)

Respiration

(R)

dW/dt = P – R

P = k 1 x radiation

R = k 2 x W

radiation = input = f(t)

{k 1, k 2} = parameters

N

N

N

N

Atmosphere

C

Tree

C

Soil

C

Subsoil

H 2O

H 2O

H 2O

H 2O

Time series

(2-200 y)

1. Deterministic

How do the models work?

Soil: Carbon

N-deposition

CO 2

Nitrogen

Water

Weather: Radiation

Temperature

Precipitation

Humidity

Wind

Model

2. Complex: many differential equations encoded

• Many parameters (30-500)

• Multiple inputs: soil conditions, time series of weather, …

• Multiple outputs

• Highly non-linear with respect to parameters and inputs

3. Dynamic: solved by iteration with timestep 1h - 1d

• Slow !

4. Physiological processes at point or plot-scale

• Upscaling to grid cells required

C in vegetation, C in soil

1900 2000 2100

Year

Frequency

Frequency

Frequency Frequency

0.16

0.12

0.08

0.04

J max

0.00

-100 0 100 200 300 400 500

0.30

0.25

0.20

0.15

0.10

0.05

u max,root

0.00

-0.05 0.00 0.05 0.10 0.15 0.20 0.25 0.30

0.20

0.15

0.10

0.05

Initial C soluble

0.00

-0.05 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35

0.5

0.4

0.3

0.2

0.1

Value

Initial W total

0.0

-0.01 0.00 0.01 0.02 0.03 0.04 0.05 0.06

Parameter

value

0.25

Photosynthesis V max

0.20

0.15

0.10

0.05

0.00

-50 0 50 100 150 200 250 300

0.25

0.20

0.15

0.10

0.05

Models Disagree

f root

0.00

-0.5 0.0 0.5 1.0 1.5

0.20

0.15

0.10

0.05

Initial Cstarch

0.00

-0.05 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35

0.20

0.15

0.10

0.05

Allocation

C-pools

N-pools

Value

Initial Nsoluble

0.00

-0.01 0.00 0.01 0.02 0.03 0.04 0.05 0.06

Parameter

value

H 2O

Atmosphere

Nutr.

C H2O Nutr.

Nutr.

Soil

C

Trees

C

Nutr.

Subsoil (or run-off)

Model

H 2O

H 2O

[Levy et al, 2004]

Frequency

Frequency

Frequency Frequency

0.16

0.12

0.08

0.04

J max

0.00

-100 0 100 200 300 400 500

0.30

0.25

0.20

0.15

0.10

0.05

u max,root

0.00

-0.05 0.00 0.05 0.10 0.15 0.20 0.25 0.30

0.20

0.15

0.10

0.05

Initial C soluble

0.00

-0.05 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35

0.5

0.4

0.3

0.2

0.1

Value

Initial W total

0.0

-0.01 0.00 0.01 0.02 0.03 0.04 0.05 0.06

Parameter

value

0.25

Photosynthesis V max

0.20

0.15

0.10

0.05

0.00

-50 0 50 100 150 200 250 300

0.25

0.20

0.15

0.10

0.05

f root

0.00

-0.5 0.0 0.5 1.0 1.5

0.20

0.15

0.10

0.05

Initial Cstarch

0.00

-0.05 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35

0.20

0.15

0.10

0.05

Allocation

C-pools

N-pools

Value

Initial Nsoluble

0.00

-0.01 0.00 0.01 0.02 0.03 0.04 0.05 0.06

Parameter

value

Models Disagree

bgc

century

hybrid

Frequency

0.4

0.3

0.2

0.1

0.0

0.4

0.3

0.2

0.1

0.0

0.4

0.3

0.2

0.1

bgc

century

hybrid

0.0

-40 -20 0 20 40 60 80

N depUE (kg C kg -1 N)

C total / N deposited kg C (kg N) -1

[Levy et al, 2004]

Increasing amounts of observational data

available

2 examples

CarboEurope http://www.carboeurope.org/

NitroEurope http://www.nitroeurope.eu

Flux towers:

Large coverage and high temporal resolution

Just a few ecosystem variables (NEE, N2O, NO fluxes)

Many observations still sparse and patchy (Soil Carbon)

Remote sensing

Good temporal and spacial coverage

But could have large errors

Key Question: How do we make best use of these

new but imbalanced datasets

How do we make best use of these new but

imbalanced datasets

MODELS

OBSERVATIONS

MODELS OBSERVATIONS

-Capable of interpolation

& forecasts

-Subjective & inaccurate?

FUSION

Complete with a quantification

of uncertainty

& capable of forecasts

-Clear confidence limits

-Incomplete, patchy

Bayesian Calibration using MCMC

Not fast (10 4 - 10 5 model runs)

Might not be a problem as many ecosystem models are

computationally efficient

Conceptually simple, grounded in probability theory

Algorithmically simple (eg Metropolis)

Works on all models, and provides well-defined

measures of uncertainty (in parameters, with

correlations, and output)

DALEC – Data Assimilation Linked Ecosystem Carbon

Model (Williams et. al. GCB 2005)

R a

C foliage

GPP C root

6 model pools

10 model fluxes

9 parameters

10 data time series

Williams et. al. GCB 2005

R total & Net Ecosystem Exchange of CO 2

A f

A r

A w

C wood

L f

L r

L w

C = carbon pools

A = allocation

L = litter fall

R = respiration (auto- & heterotrophic)

C litter

D

C SOM/CWD

R h

Temperature controlled

Observations – Ponderosa Pine, Oregon (Bev Law)

Flux tower (2000-2)

Sap flow

Soil/stem/leaf respiration

LAI, stem, root biomass

Litter fall measurements

Pool/Flux Observation Number of Observations

Total Respiration Rate 401

Net Ecosystem Exchange Flux 684

Foliar C Mass 4

Wood C Mass 3

Fine Root C Mass 2

Fresh Litter C Mass 1

Soil Organic Matter 1

Autotrophic Respiration Rate 4

Calibrating DALEC using MCMC

Used MCMC to calibrate model with observations from

the Oregon site

19 parameters and 5 initial values

Gaussian used in the Likelihood to represent

observational errors

Chains of ~ 400000 required to converge on the

posterior

How do imbalances in the observations available

influence the calibration?

Pool/Flux Observation Number of Observations

Total Respiration Rate 401

Net Ecosystem Exchange Flux 684

Foliar C Mass 4

Wood C Mass 3

Fine Root C Mass 2

Fresh Litter C Mass 1

Soil Organic Matter 1

Autotrophic Respiration Rate 4

Resp. R 2 Resp. NRMSE NEE R 2 NEE NRMSE

All Observations 0.84 0.26 0.55 1.13

NEE Only 0.50 0.34 0.66 0.95

Pool/Flux Observation Number of Observations

Total Respiration Rate 401

Net Ecosystem Exchange Flux 684

Foliar C Mass 4

Wood C Mass 3

Fine Root C Mass 2

Fresh Litter C Mass 1

Soil Organic Matter 1

Autotrophic Respiration Rate 4

Resp. R 2 Resp. NRMSE NEE R 2 NEE NRMSE

All Observations 0.84 0.26 0.55 1.13

NEE Only 0.50 0.34 0.66 0.95

NEE + Stocks and Autotrophic Resp. 0.58 0.32 0.66 0.96

Given model discrepancy how can observations be used

most effectively to inform models?

Model believable scale (1)

At which scales (time/space) does the model have good

predictability.

Concept of a model’s “believable scale” (Lander and

Hoskins 1997).

This is often unknown/uncertain.

Observations at a higher temporal spacial scale may not

be beneficial to the calibration.

Danger of over fitting if higher resolution data used.

Model outputs for which there is few or no observations can

become poorly represented after calibration.

Given model discrepancy how can observations be used

most effectively to inform models?

Model believable scale (1)

NEE averaged over a number or timescales

1,2,4,8,16,32 and 64 days

Parameter vector with maximum likelihood at each

averaging period – using MCMC.

R squared and normalised RMSE calculated for model

output against NEE and also respiration data (not used

in the above optimisation) at the averaging period.

Given model discrepancy how can observations be used

most effectively to inform models?

Problem with Gaussian Likelihood (2)

• Gaussian likelihood allows values out to infinity.

• Thus allows model outputs with few observations

to be pulled far from data if large quantities of data

exist for another output (allows model over-fitting)

Given model discrepancy how can observations be used

most effectively to inform models?

Problem with Gaussian Likelihood (2)

• Gaussian likelihood allows values out to infinity.

• Thus allows model outputs with few observations

to be pulled far from data if large quantities of data

exist for another output (allows model over-fitting)

One potential solution could be to use a

truncated likelihood for model outputs with

few observations to protect against

this over-fitting.

Resp. R 2 Resp. NRMSE NEE R 2 NEE NRMSE

All Observations 0.84 0.26 0.55 1.13

NEE Only 0.50 0.34 0.66 0.95

NEE + Stocks and Autotrophic Resp. 0.58 0.32 0.66 0.96

NEE + Stocks and Autotrophic Resp. with truncation 0.66 0.29 0.61 1.06

Conclusions

Imbalances in available observations can give problems

in model calibration because of model discrepancy

Sparse observations ignored leading to over-fitting

In view of this. How can observations be used most

effectively given model discrepancy?

Calibrate models with data at scales at which the models have

predictive skill (believable scale)

Lessen over-fitting: Avoid likelihood functions that allow

physically improbable values.

Quantify observational systematic errors and autocorrelations.

and …

Is it the overall structure or the

component processes that matters?

Structure 1:

Structure 3:

F

F

P L R

X Y

Structure 2: F X Ya P

U

L

Pa Xa La Pb Xb Lb Y b

Y

M

R

R

Rastetter 2003

Finding the posterior: MCMC

MCMC: walk through parameter-space →set of visited points

approaches the posterior parameter distribution.

[e.g. using Metropolis-Hastings random walk]

Sample of 10 4 -10 5 parameter vectors from the

posterior distribution for the parameters

3.1 Process-based forest models

Environmental

scenarios

Initial

values

Parameters

H 2O

Atmosphere

Nutr.

C H2O Nutr.

Nutr.

Soil

C

Trees

C

Nutr.

Subsoil (or run-off)

Model

H 2O

H 2O

Height

NPP

Soil C

Process-based modelling

Changes in

CO 2, N,

weather

Parameters

(44, 119, 249,

261)

1900 2000

Increased

growth

2100

Change in annual

mean Temperature

UKCIP

BASFOR: forest C-sequestration 2005-2076

Change in potential Cseq.

Uncertainty in change

of potential C-seq.

- Uncertainty due to model

parameters only, NOT uncertainty

in inputs / upscaling

Integrating Remote Sensing Data (Patenaude et al.)

Model 3-PG

RS-data:

Hyperspectral,

LiDAR,

SAR

BC

Model Uncertainty

INPUT MODEL

OUTPUT

Uncertain as a representation of reality:

• may not be known

• may be inadequate

— parameter or initial value uncertainty

— model inadequacy

Model parameter uncertainty - Bayes Idea

Eg experimental determination of a model constant

Data (plus

quantification

of errors)

Prior ideas

about

Posterior ideas

about

Bayes Rule:

prob. parameter is correct given data prob. getting data given parameter x prior

model parameter/initial condition

observations

posterior likelihood prior

Finding the posterior: MCMC

Post(p|D) Pr(p) x L(D|p)

MCMC: walk through parameter-space →set of visited points

approaches the posterior parameter distribution.

[e.g. using Metropolis-Hastings random walk]

Metropolis (1953) algorithm

1. Start anywhere in parameter-space: p 1..9(i=0)

2. Randomly choose p(i+1) = p(i) + δ

3. IF: [ Pr(p(i+1)) x L(D|p(i+1)) ] / [ Pr(p(i)) x L(D|p(i)) ] > Random[0,1]

THEN: accept p(i+1)

ELSE: reject p(i+1)

i=i+1

4. IF i < 10 4 GOTO 2

3.10 Calculating the posterior using

0 5000 10000

MCMC

0 5000 10000

15

P( |D) 0 5000 P( 10000 ) P(D|f( x 10 ))

-3

0.012

0.12

0.01

1. Start anywhere in parameter-space: 0.008

0.25

0.1 p 0.08 1..39(i=0)

2.5 KDLITT

2

2. Randomly choose p(i+1) = p(i) + δ

0.6

0.55

FW 0.55

0.5

0.45

0

14

GAMMA 12

10

8

6

4

5000 10000 0

KCA

0.45

KCAEXP

0.4

0.35

0.8

5000 10000 0 5000 10000 0 5000 10000

1

x 10

14

12

10

8

6

1.8

1.6 KDL

1.4

4

1.2

2

0

-5

KDW 6

4

0

0.32

KH 0.3

0.28

0.26

0.24

0.22

5000 10000 0

x 10

KHEXP 7

6

5

4

3

5000 10000 0 5000 10000

-3

KLAIMAX

0.8

0.6

0 5000 10000

1

x 10

1.8

1.6

1.4

1.2

-3

KNMIN

0.8

0.6

0

1

0.035

0.03

0.025

KTA

25

20

KTB 0.6

0.5

0.4

x

1.8

1.6

1.4

1.2

x 10

KTREE

2.5

2

1.5

0 5000 10000 0 5000 10000 0 5000 10000

-3

0.025

LUE0

NLCONMIN0.045

0.02

0.04

0.015

0.035

0 5000 10000 0

0.035

0.03

0.025

0

NRCON

0.8

0.6

5000 10000 0 5000 10000

1

1.8

1.6

1.4

1.2

NWCON

14

12

10

8

6

0

SLA

5000

CLITT0

0.4

8

0.2

6

10000 0x 10 5000 10000 0

CSOMF0 2.5

2

1.5

5000 10000 0

0.018

0.016 NLITT0 0.35 NSOMF0 0.18

0.16 NSOMS0 1.5

0.014

0.3

0.14

1

0.5

0.006

0.06

0 5000 10000 0 5000 10000 0 5000 10000 0 5000 10000

-3

x 10

NMIN0 0.7

0.6

0.5

0

FLITTSOMF 0.08

0.06

0.04

0.02

5000 10000 0

-3

x 10

10

-5

x 10

KDSOMF 10

-6

MCMC trace plots

KDSOMS

1.5

1

0 5000 10000 0 5000 10000 0 5000 10000

3. IF: [ P(p(i+1)) P(D|f(p(i+1))) Iteration]

/ [ P(p(i)) IterationP(D|f(p(i)))

Iteration]

>

Random[0,1]

THEN: accept p(i+1)

ELSE: reject p(i+1)

i=i+1

4. IF i < 10 4 GOTO 2 Metropolis et al (1953)

Sample of 10 4 -10 5 parameter vectors from the

posterior distribution P( |D) for the parameters

5

5

Iteration

Iteration

Should we measure the “sensitive

Yes, because the parameters”?

sensitive parameters:

• are obviously important for prediction ?

No, because model parameters:

• are correlated with each other, which we do not measure

• cannot really be measured at all

So, it may be better to measure output variables, because they:

• are what we are interested in

• are better defined, in models and measurements

• help determine parameter correlations if used in Bayesian

calibration

Key question: what data are most informative?

2.4 Bayesian updating of probabilities for process-based models

Bayes’ Theorem: Prior probability → Posterior prob.

Model parameterization: P(params) →

P(params|data)

Model selection: P(models) →

P(model|data)

Some points to note:

If the model is too poor the data could tend to

take the posterior to the edge of the prior.

This can lead to very small steps in the MCMC and

therefore very long chains before good sampling of

posterior

Some points to note:

If the model is too poor the data could tend to

take the posterior to the edge of the prior.

This can lead to very small steps in the MCMC and

therefore very long chains before good sampling of

posterior

Don’t throw away your scientific understanding

in applying the method!

Some points to note:

If the model is too poor the data could tend to

take the posterior to the edge of the prior.

This can lead to very small steps in the MCMC and

therefore very long chains before good sampling of

posterior

Don’t throw away your scientific understanding

in applying the method!

Bayesian methods can be useful for developing

models that are consistent with data.

By swapping in and out observational data and

adding and subtracting model structural innovations.

How will climate change affect the biosphere?

Uncertainty whether regions/landscapes are/will be

sources or sinks for carbon.:

Amazonian dieback (Cox et. al. Nature 2000)

Droughts in Europe leading to reduced carbon sink (Cias et. al.

Nature 2005)

Clear value in trying to predict impacts/feedbacks but

predictions are of little value without a quantitative

assessment of uncertainty or how probable prediction is.

Thus we need to consider the mathematics of

probabilities.

In order to make predictions and quantify uncertainties

about the future of the biosphere we need both

mathematical models and observations.

Simulating forest growth across Europe

Recognition project (M. van Oijen)

20

15

10

5

0

-5

Simulated changes in

Net Primary Productivity (NPP)

NPP

(% increase)

Causal

Factors:

25 N-dep.

Latitude

Climate

CO 2

Evaluation

How robust is this result ?

Compare results

of 4 models

RECOGNITION: 4 models

EFIMOD

EFM

FinnFor

Q

119 261 249 44 # Parameters

6 10 5 5 # Tree C pools

6 10 5 5 # Tree N pools

5 3 14 - # Surface litter pools

16 10 10 24 # Soil pools

25

20

15

10

5

0

-5

40

30

20

10

0

Changes in NPP (%)

EFM

FinnFor

Latitude

Latitude

Causal

Factors:

N-dep.

Climate

CO 2

20

15

10

5

0

-5

-10

20

10

0

-10

Q

EFIMOD

Latitude