Using Derivative Information in the Statistical Analysis of ... - MUCM

mucm.ac.uk

Using Derivative Information in the Statistical Analysis of ... - MUCM

Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Using Derivative Information in the Statistical

Analysis of Computer Models

Gemma Stephenson, Peter Challenor and Bob Marsh

08 June 09

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Outline

1 Complex Models and Emulators

Background

Example

Use of derivatives

2 Emulating function output with derivative information

Toy Models

Results

3 Emulating Derivatives

Introduction

Examples

Borehole Model

4 C-GOLDSTEIN

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Outline

1 Complex Models and Emulators

Background

Example

Use of derivatives

2 Emulating function output with derivative information

Toy Models

Results

3 Emulating Derivatives

Introduction

Examples

Borehole Model

4 C-GOLDSTEIN

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Outline

1 Complex Models and Emulators

Background

Example

Use of derivatives

2 Emulating function output with derivative information

Toy Models

Results

3 Emulating Derivatives

Introduction

Examples

Borehole Model

4 C-GOLDSTEIN

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Outline

1 Complex Models and Emulators

Background

Example

Use of derivatives

2 Emulating function output with derivative information

Toy Models

Results

3 Emulating Derivatives

Introduction

Examples

Borehole Model

4 C-GOLDSTEIN

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Complex Models

Background

Example

Use of derivatives

Simulate the behaviour of real-world systems

Simulator: deterministic function, y = η(x)

x are the inputs

y are the predictions of the true real-world system being

modelled

Computationally expensive to run

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Emulator

Background

Example

Use of derivatives

Statistical approximation of the simulator

Approximation is simpler and quicker than original function

Emulator can then be used for any analysis

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Building an emulator

Background

Example

Use of derivatives

Consider the model, η(.), as an unknown function.

Represent the model as a Gaussian process.

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Building an Emulator

Prior Knowledge

Specify mean.

Background

Example

Use of derivatives

E[η(x)| β] = h(x) T β

h(x) T is a vector of known functions of x

β is a vector comprising of unknown coefficients.

Specify covariance function.

Cov(η(x), η(x ′ )|σ 2 ) = σ 2 c(x, x ′ )

c(x, x ′

) = exp −(x − x ′ ) T B (x − x ′

)

B is a diagonal matrix of positive roughness parameters

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Building an Emulator

Prior Knowledge

Specify mean.

Background

Example

Use of derivatives

E[η(x)| β] = h(x) T β

h(x) T is a vector of known functions of x

β is a vector comprising of unknown coefficients.

Specify covariance function.

Cov(η(x), η(x ′ )|σ 2 ) = σ 2 c(x, x ′ )

c(x, x ′

) = exp −(x − x ′ ) T B (x − x ′

)

B is a diagonal matrix of positive roughness parameters

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Building an Emulator

Training data

Background

Example

Use of derivatives

Choose design to obtain inputs (x1, . . . , xn)

Latin Hypercube sample

Sobol’ sequence

Run code at inputs to obtain training data:

y T = {y1 = η(x1), . . . , yn = η(xn)}

Derive the distribution of η(.) conditional only on y

Integrate out β and σ 2

Estimate B by MLE

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Building an Emulator

Training data

Background

Example

Use of derivatives

Choose design to obtain inputs (x1, . . . , xn)

Latin Hypercube sample

Sobol’ sequence

Run code at inputs to obtain training data:

y T = {y1 = η(x1), . . . , yn = η(xn)}

Derive the distribution of η(.) conditional only on y

Integrate out β and σ 2

Estimate B by MLE

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Building an Emulator

Training data

Background

Example

Use of derivatives

Choose design to obtain inputs (x1, . . . , xn)

Latin Hypercube sample

Sobol’ sequence

Run code at inputs to obtain training data:

y T = {y1 = η(x1), . . . , yn = η(xn)}

Derive the distribution of η(.) conditional only on y

Integrate out β and σ 2

Estimate B by MLE

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Building an Emulator

Posterior Process

Background

Example

Use of derivatives

The posterior process is then used as a fast approximation

to η(x).

The standard deviation gives a measure of the uncertainty

in this approximation.

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

1 Dimensional Example

Background

Example

Use of derivatives

True function = x + cos(x) + sin(x)

Chose linear form for prior mean

h(x) T = (1 x)

Specify covariance function

c(x, x ′ ) = exp{−b (x − x ′ ) 2 }

Choose n = 5 design points.

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

1 Dimensional Example

Output

−4 −2 0 2 4 6

Background

Example

Use of derivatives

−6 −4 −2 0 2 4 6

Input

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Background

Example

Use of derivatives

How can derivative information help?

How much information is gained using derivatives?

At what cost does this information come?

Can additional efficiency be achieved through use of

derivative information?

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Background

Example

Use of derivatives

Methods of obtaining derivatives

1 Differentiate the simulator analytically

Not very easy for complex models!

2 Divided difference approach

Only approximate

Computationally expensive

3 Adjoint

Obtained via automatic differentiation

4 Emulate derivatives

Work in progress

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Background

Example

Use of derivatives

Methods of obtaining derivatives

1 Differentiate the simulator analytically

Not very easy for complex models!

2 Divided difference approach

Only approximate

Computationally expensive

3 Adjoint

Obtained via automatic differentiation

4 Emulate derivatives

Work in progress

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Background

Example

Use of derivatives

Methods of obtaining derivatives

1 Differentiate the simulator analytically

Not very easy for complex models!

2 Divided difference approach

Only approximate

Computationally expensive

3 Adjoint

Obtained via automatic differentiation

4 Emulate derivatives

Work in progress

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Background

Example

Use of derivatives

Methods of obtaining derivatives

1 Differentiate the simulator analytically

Not very easy for complex models!

2 Divided difference approach

Only approximate

Computationally expensive

3 Adjoint

Obtained via automatic differentiation

4 Emulate derivatives

Work in progress

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Methodology

Inclusion of Derivatives

yD =




Background

Example

Use of derivatives

η(x1)

.

η(xn)


∂x η(x1)

.


∂x η(xn)




Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Methodology

Inclusion of Derivatives


Cov η(x), ∂

∂xj



Cov η(x),

∂xi


∂xj

Background

Example

Use of derivatives

η(x ′


)

β, σ2

, B

η(x ′


)

β, σ2

, B

2 ∂

= σ c(x, x

∂xj

′ )

2 ∂2

= σ c(x, x

∂xi∂xj

′ )

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Background

Example

Use of derivatives

Covariance matrix × training data

2 input dimensions example



⎢ ↓

⎢ ↑

⎢ ↓

⎢ ↑




←− n −→ ←− n −→ ←− n −→

n c{η(x, z), η(x, z)} c{η(x, z), ∂


η(x, z)} c{η(x, z), η(x, z)}

∂x ∂z

n c{ ∂





η(x, z), η(x, z)} c{ η(x, z), η(x, z)} c{ η(x, z), η(x, z)}

∂x ∂x ∂x ∂x ∂z

n c{ ∂





η(x, z), η(x, z)} c{ η(x, z), η(x, z)} c{ η(x, z), η(x, z)}

∂z ∂z ∂x ∂z ∂z

⎤ ⎡

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎥ ⎢

⎦ ⎣

Gemma Stephenson Use of Derivative Information in Computer Models

η(x1, z1))

.

η(xn, zn)


∂xη(x1, z1)

.


∂xη(xn, zn)


∂z η(x1, z1)

.


∂z η(xn, zn)




Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Derivatives at 5 points

Output

−4 −2 0 2 4 6

Background

Example

Use of derivatives

−6 −4 −2 0 2 4 6

Input

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Derivatives at 3 points

Output

−4 −2 0 2 4 6

Background

Example

Use of derivatives

−6 −4 −2 0 2 4 6

Input

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

No Derivatives

Output

−4 −2 0 2 4 6

Background

Example

Use of derivatives

−6 −4 −2 0 2 4 6

Input

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

The Models

Output

Output

100 150 200 250 300

1 2 3 4 5 6 7

−4 −2 0 2 4

Input

(a) Model 1

5 10 15 20

Input

(c) Model 3

Output

Output

Toy Models

Results

−4 −2 0 2 4 6

−4 −2 0 2 4 6

−6 −4 −2 0 2 4 6

Input

(b) Model 2

2 4 6 8 10

Input

(d) Model 4

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

The Models

Model 5

Toy Models

Results

Model 5 simulates the flow of water through a borehole, taken

from Morris et al (1993):

flow rate (m 3 /yr) =

ln( r

rw )

2πTu(Hu − Hl)


1 + 2LTu

ln( r

rw )r 2 w Kw

+ Tu

T l

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

The Models

Model 5

Toy Models

Results

Input Description Range

rw radius of borehole 0.05 – 0.15

r radius of influence 100 – 50,000

Tu transmissivity of upper aquifer 63,070 – 115,600

Hu potentiometric head of upper aquifer 990 – 1,110

Tl transmissivity of lower aquifer 63.1 – 116

Hl potentiometric head of lower aquifer 700 – 820

L length of borehole 1,120 – 1,680

Kw hydraulic conductivity of borehole 1,500 – 15,000

Vary three of the input parameters: rw, L and Kw

Fix all others at lower end of range

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Toy Model Investigation

5 models

Toy Models

Results

Build emulators with and without derivatives

Designs are chosen with a Latin hypercube sample

Predict function output at 1000 new points, chosen by a

LHS. Compare:

1 Prediction error

Prediction Error = 1 P (|True Value−Predicted Value|)

1000

|True Value|

2 Measure of uncertainty

Mean Standard Deviation = 1

1000

P ˆσ p c ∗∗ (xi, xi)

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Toy Model Investigation

5 models

Toy Models

Results

Build emulators with and without derivatives

Designs are chosen with a Latin hypercube sample

Predict function output at 1000 new points, chosen by a

LHS. Compare:

1 Prediction error

Prediction Error = 1 P (|True Value−Predicted Value|)

1000

|True Value|

2 Measure of uncertainty

Mean Standard Deviation = 1

1000

P ˆσ p c ∗∗ (xi, xi)

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Prediction Error

Prediction error

Prediction error

0.00 0.02 0.04

1 2 3 4

Model 1

3 4 5 6

Number of simulator runs

Model 4

5 10 15 20 25 30 35 40

Number of simulator runs

Prediction error

Prediction error

0 2 4 6 8

0.0 0.2 0.4 0.6

Model 2

Toy Models

Results

3 4 5 6 7 8

Number of simulator runs

Model 5

2 4 6 8 10 12 14

Number of simulator runs

Prediction error

0.00 0.10 0.20 0.30

0.0 0.2 0.4 0.6

Model 3

3 4 5 6 7 8 9 10

Number of simulator runs

With derivatives

Without derivatives

2 4 6 8 10 12 14

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Measure of uncertainty

Mean Standard Deviation

Mean Standard Deviation

0 5 10 15 20

1 2 3 4 5 6 7 8

Model 1

3 4 5 6

Number of simulator runs

Model 4

5 10 15 20 25 30 35 40

Number of simulator runs

Mean Standard Deviation

Mean Standard Deviation

0.0 0.5 1.0 1.5

0 2 4 6 8 10

Model 2

Toy Models

Results

3 4 5 6 7 8

Number of simulator runs

Model 5

2 4 6 8 10 12 14

Number of simulator runs

Mean Standard Deviation

0.0 0.5 1.0 1.5

0.0 0.2 0.4 0.6

Model 3

3 4 5 6 7 8 9 10

Number of simulator runs

With derivatives

Without derivatives

2 4 6 8 10 12 14

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Introduction

Examples

Borehole Model

Methods of obtaining derivatives

1 Differentiate the simulator analytically.

Not very easy for complex models!

2 Divided difference approach.

Only approximate.

Computationally very expensive.

3 Adjoint

Obtained via automatic differentiation.

More efficient than FD.

Still computationally expensive to run.

Time consuming to write (despite AD software)

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Introduction

Examples

Borehole Model

Can we use an emulator to predict the derivatives of

the simulator?

Extend theory to model the derivatives of the simulator,

η(.)

Derivatives of η(.) also described by Gaussian process

Could reduce the demand for adjoints

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Introduction

Examples

Borehole Model

With derivatives in training data

True function = x + cos(x) + sin(x)

True derivatives = 1 − sin(x) + cos(x)

Chose linear form for prior mean

h(x) T = (1 x)

Specify covariance function

c(x, x ′ ) = exp{−b (x − x ′ ) 2 }

Choose n = 5 design points.

Training data is then:

y T = {y1 = η(x1), . . . , y5 = η(x5), y6 = ∂

∂x η(x1), . . . , y10 =


∂x η(x10)}

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Introduction

Examples

Borehole Model

With derivatives in training data

True function = x + cos(x) + sin(x)

True derivatives = 1 − sin(x) + cos(x)

Chose linear form for prior mean

h(x) T = (1 x)

Specify covariance function

c(x, x ′ ) = exp{−b (x − x ′ ) 2 }

Choose n = 5 design points.

Training data is then:

y T = {y1 = η(x1), . . . , y5 = η(x5), y6 = ∂

∂x η(x1), . . . , y10 =


∂x η(x10)}

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Introduction

Examples

Borehole Model

With derivatives in training data

True function = x + cos(x) + sin(x)

True derivatives = 1 − sin(x) + cos(x)

Chose linear form for prior mean

h(x) T = (1 x)

Specify covariance function

c(x, x ′ ) = exp{−b (x − x ′ ) 2 }

Choose n = 5 design points.

Training data is then:

y T = {y1 = η(x1), . . . , y5 = η(x5), y6 = ∂

∂x η(x1), . . . , y10 =


∂x η(x10)}

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Introduction

Examples

Borehole Model

With derivatives in training data

Derivatives

-4 -2 0 2 4

-6 -4 -2 0 2 4 6

Input

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Introduction

Examples

Borehole Model

Without derivatives in training data

Derivatives

-4 -2 0 2 4

-6 -4 -2 0 2 4 6

Input

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Introduction

Examples

Borehole Model

Without derivatives in training data (6 points)

Derivatives

-4 -2 0 2 4

-6 -4 -2 0 2 4 6

Input

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Introduction

Examples

Borehole Model

Without derivatives in training data (7 points)

Derivatives

-4 -2 0 2 4

-6 -4 -2 0 2 4 6

Input

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Introduction

Examples

Borehole Model

Without derivatives in training data (8 points)

Derivatives

-4 -2 0 2 4

-6 -4 -2 0 2 4 6

Input

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Emulating borehole derivatives

Vary 3 input dimensions

Introduction

Examples

Borehole Model

Build emulator with 20 simulator runs.

Include only function output (i.e flow of water) in training

data.

Emulate partial derivatives of flow of water w.r.t. the 3

inputs.

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Introduction

Examples

Borehole Model

Comparison of derivatives w.r.t rw

Emulated derivatives w.r.t radius of borehole

500 1000 1500 2000 2500 3000

0 500 1000 1500 2000 2500 3000

True derivatives w.r.t radius of borehole

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Introduction

Examples

Borehole Model

Comparison of derivatives w.r.t L

Emulated derivatives w.r.t length of borehole

-0.15 -0.10 -0.05 0.00

-0.20 -0.15 -0.10 -0.05 0.00

True derivatives w.r.t length of borehole

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Introduction

Examples

Borehole Model

Comparison of derivatives w.r.t Kw

Emulated derivatives w.r.t. hydraulic conductivity of borehole

0.005 0.010 0.015

0.005 0.010 0.015

True derivatives w.r.t. hydraulic conductivity of borehole

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Summary

Emulation can provide a practical solution to running

computationally expensive models.

Results from toy models suggest using derivative

information in building emulators might improve efficiency.

Dependent on the computational cost of derivatives.

Emulating derivatives could reduce the demand for writing

and running adjoints.

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Summary

Emulation can provide a practical solution to running

computationally expensive models.

Results from toy models suggest using derivative

information in building emulators might improve efficiency.

Dependent on the computational cost of derivatives.

Emulating derivatives could reduce the demand for writing

and running adjoints.

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Summary

Emulation can provide a practical solution to running

computationally expensive models.

Results from toy models suggest using derivative

information in building emulators might improve efficiency.

Dependent on the computational cost of derivatives.

Emulating derivatives could reduce the demand for writing

and running adjoints.

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

C-GOLDSTEIN

C-GOLDSTEIN =

Global Ocean-Linear Drag Salt and

Temperature Equation Integrator

+ Energy Moisture

Balance Model

Gemma Stephenson Use of Derivative Information in Computer Models

+ sea ice

model


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Surface air temperature

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

C-GOLDSTEIN

Differentiated using AD (Dan Zachary and Neil Edwards)

TAPENADE Requires

Source code in Fortran or C.

Variables whose derivatives are required.

Variables with respect to which need to be differentiated.

A mode of differentiation

TAPENADE Returns

Tangent (forward) mode differentiated source code.

Tangent multidirectional (forward) mode differentiated source

code.

Reverse mode differentiated source code.

For C-GOLDSTEIN forward mode adopted -

multidirectional (12) tangent mode

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Validation of C-GOLDSTEIN adjoint

Automatic Differentiation

Run C-GOLDSTEIN adjoint for 5000 years.

Save derivatives of global mean air temperature with

respect to 12 input parameters output every 100 years.

Finite Differences (Centre Divided Difference Equation)

∂ η(x)

∂x

≈ η(x+ɛx)−η(x−ɛx)


Take ɛ = 1 × 10 −7

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

FD vs AD

Partial Derivative

-0.5 -0.4 -0.3 -0.2 -0.1 0.0 0.1

d(airt)/d(scf)

0 1000 2000 3000 4000 5000

Year

FDairt

ADairt

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

FD vs AD

Partial Derivative

-2.5e-07 -2.0e-07 -1.5e-07 -1.0e-07

d(airt)/d(diffamp1)

0 1000 2000 3000 4000 5000

Year

FD

AD

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

FD vs AD

Partial Derivative

-0.00005 0.00005 0.00015

d(airt)/d(diff1)

0 1000 2000 3000 4000 5000

Year

FD

AD

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

FD vs AD

Partial Derivative

-0.5 -0.4 -0.3 -0.2 -0.1 0.0

d(airt)/d(adrag)

0 1000 2000 3000 4000 5000

Year

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Emulating partial derivatives of global mean air

temperature

Vary one input parameter: wind stress scale

Run C-GOLDSTEIN adjoint at 30 points

10 runs for training data

30 validation derivatives

Build emulator with only function output (i.e global mean

air temperature)

Emulate derivatives of global mean air temperature w.r.t.

wind stress scale at validation points

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Initial data

airt

12.0328 12.0332 12.0336 12.0340

1.90 1.95 2.00 2.05 2.10

scf

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Initial data

AD derivs

-0.020 -0.010 0.000 0.010

1.90 1.95 2.00 2.05 2.10

scf

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Initial data

airt

AD derivs

12.0328 12.0332 12.0336 12.0340

-0.020 -0.010 0.000 0.010

1.90 1.95 2.00 2.05 2.10

scf

1.90 1.95 2.00 2.05 2.10

scf

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Emulating air temperature

airt

12.0328 12.0332 12.0336 12.0340

1.90 1.95 2.00 2.05 2.10

scf

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Emulating derivatives

deriv

-0.020 -0.010 0.000 0.010

AD

Emulator

1.90 1.95 2.00 2.05 2.10

scf

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Further runs and narrow the input space

Perform an additional 50 runs of C-GOLDSTEIN across

the whole input space.

Restrict the training data to 11 points between scf = 2.02

and scf = 2.10.

Rebuild emulators.

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Further runs

airt

12.0328 12.0332 12.0336 12.0340

1.90 1.95 2.00 2.05 2.10

scf

Gemma Stephenson Use of Derivative Information in Computer Models


12.0328 12.0332 12.0336 12.0340

Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Rebuilt emulators

1.90 1.95 2.00 2.05 2.10

scf

0 1

2 3

4 5

6 7

8 9 1

deriv

-0.04 -0.03 -0.02 -0.01 0.00 0.01

AD

Emulator

FD5

1

0 1

0

2

4

3

5

4 5

5

4

7

8

1

6

8 9

6 8 9 1

1.90 1.95 2.00 2.05 2.10

Gemma Stephenson Use of Derivative Information in Computer Models

scf

3

3

7

6 7


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

Intensive runs required

Much less conflict now between the emulator and the

adjoint.

In particular the high peak at derivative point 2

Emulator still not satisfactory though.

Run C-GOLDSTEIN intensively in the narrowed region.

Further 134 points in narrowed region

214 points altogether

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

airt

12.0328 12.0332 12.0336 12.0340

1.90 1.95 2.00 2.05 2.10

scf

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

grandy

12.0328 12.0332 12.0336 12.0340

1.90 1.95 2.00 2.05 2.10

grandx

0 1 2 3

4 5

6 7

8 9 1

Gemma Stephenson Use of Derivative Information in Computer Models


Complex Models and Emulators

Emulating function output with derivative information

Emulating Derivatives

Summary

C-GOLDSTEIN

www.mucm.group.shef.ac.uk

Thanks to all in the MUCM team.

Gemma Stephenson Use of Derivative Information in Computer Models

More magazines by this user
Similar magazines