# Bounding rare event probabilities in computer experiments - MUCM

Bounding rare event probabilities in computer experiments - MUCM

Πρ distribution

Importance sampling

Applications

Discussion

Bounding rare event probabilities in computer experiments

Pierre BARBILLON 1 , Yves AUFFRAY 2 , Jean-Michel MARIN 3

1 AgroParisTech

2 Dassault Aviation

3 Université Montpellier 2

UCM 2012, Sheffield

P. Barbillon Bounding rare event probabilities

Context

Goal

Πρ distribution

Importance sampling

Applications

Discussion

A computer experiment is an evaluation of a determinist expensive

black-box function describing a physical system:

f : x ∈ E ⊂ R d ↦→ R .

Ref. : Fang et al. (2006), Koehler et Owen (1996), Santner et al. (2003) .

Uncertainties on inputs ⇒ modelled by a random variable X (known

distribution).

Failure of the system: Rρ = {f (X) < ρ} with ρ given threshold.

Upper bounding:

πρ = P(f (X) < ρ) = P(X ∈ Rρ) = PX(Rρ) ,

under the constraint of a limited number N of calls to f .

P. Barbillon Bounding rare event probabilities

Crude Monte Carlo

Πρ distribution

Importance sampling

Applications

Discussion

Monte Carlo estimator of πρ: X1:L = X1, . . . , XL a L-sample of X,

1

L

L

i=1

I]−∞,ρ[(f (Xi)) = 1

L Γ(f , X1:L, ρ) .

Confidence Bound, Γ(f , X1:L, ρ) ∼ B(L, πρ) ⇒ For α ∈]0, 1[,

If πρ small (⇔ {f (X) < ρ} rare event) :

P(πρ ≤ b(Γ(f , X1:L, ρ), L, α)) = 1 − α .

Γ(f , X1:L, ρ) = 0 with high probability,

then, L ≥ 230, 000 to obtain a 0.9 confidence bound:

⇒ More than 230 000 calls to f .

b(0, L, 0.1) ∼ 10 −5 .

P. Barbillon Bounding rare event probabilities

Πρ distribution

Importance sampling

Applications

Discussion

Metamodelling: prior distribution on f

Sacks et al. (1989).

f realization of a Gaussian process F:

∀x ∈ E,

Q

F(x) = βkhk(x) + ζ(x) = H(x) T β + ζ(x) ,

where

k=1

h1, . . . , hQ regression functions and β parameters vector,

ζ centred Gaussians process with covariance function:

where K is a correlation kernel.

Hypotheses

Cov(ζ(x), ζ(x ′ )) = σ 2 K (x, x ′ ) ,

parameters β, σ 2 and those of K assumed fixed;

process F independent of X.

P. Barbillon Bounding rare event probabilities

Πρ distribution

Importance sampling

Applications

Discussion

Metamodelling: posterior

y1 = f (x1), . . . , yn = f (xn) evaluations of f on a design Dn (n ≤ N).

Process F Dn : Conditioning F to F(x1) = y1, . . . , F (xn) = yn.

Gaussian Process with mean mDn(x) and covariance CDn(x, x ′ ) ∀x, x ′ .

For all x ∈ E,

m Dn (x) approximates f (x),

C Dn (x, x) uncertainty on this approximation.

For all xi ∈ Dn,

m Dn (x i ) = f (x i ),

C Dn (x i , x i ) = 0.

P. Barbillon Bounding rare event probabilities

Consequence on πρ

The probability to estimate

Πρ distribution

Importance sampling

Applications

Discussion

πρ = P(f (X) < ρ) = E

is a realization of the random variable :

Πρ = E

I]−∞,ρ[(F Dn (X))|F Dn

= f

I]−∞,ρ[(F Dn (X))|F Dn

.

P. Barbillon Bounding rare event probabilities

Simulations of F Dn

Πρ distribution

Importance sampling

Applications

Discussion

P. Barbillon Bounding rare event probabilities

Outline

1 Πρ distribution

2 Importance sampling

Πρ distribution

Importance sampling

Applications

Discussion

3 Applications

Toy example

A real case study: release enveloppe clearance

P. Barbillon Bounding rare event probabilities

Outline

1 Πρ distribution

2 Importance sampling

Πρ distribution

Importance sampling

Applications

Discussion

3 Applications

Toy example

A real case study: release enveloppe clearance

P. Barbillon Bounding rare event probabilities

Bayesian estimator

Proposition

Πρ distribution

Importance sampling

Applications

Discussion

E(Πρ) = E Φ

ρ − mDn(X)

CDn(X, X)

,

Φ N (0, 1)-cumulative distribution function.

No more needed calls to f ⇒ computed by Monte Carlo integration.

E(Πρ) ⇒ estimates πρ.

Markov’s inequality: α ∈ [0, 1],

P Πρ ≤ E(Πρ)

≥ 1 − α .

α

P. Barbillon Bounding rare event probabilities

Realizations of Πρ

Πρ distribution

Importance sampling

Applications

Discussion

Simulations of F Dn similar to Oakley (2004)

⇒ Quantiles estimates of the distribution of Πρ.

P. Barbillon Bounding rare event probabilities

Outline

1 Πρ distribution

2 Importance sampling

Πρ distribution

Importance sampling

Applications

Discussion

3 Applications

Toy example

A real case study: release enveloppe clearance

P. Barbillon Bounding rare event probabilities

Πρ distribution

Importance sampling

Applications

Discussion

Importance distribution

Make the rare event happens more frequently:

PZ : A ⊂ E ↦→ PX(A| Rρ) ,

where Rρ ⊂ E aims at being close to Rρ = {x ∈ E : f (x) < ρ}.

Rρ based on metamodelling (n calls to f ):

Rρ =

Rρ,κ = x : mDn(x) < ρ + κ

CDn(x, x) .

Checking if x ∈ Rρ is free.

P. Barbillon Bounding rare event probabilities

Πρ distribution

Importance sampling

Applications

Discussion

Importance sampling estimator

Computation:

importance sample: Z1:m = (Z1, . . . , Zm),

estimator of πρ :

PX( Rρ)

m Γ(f , Z1:m, ρ) = PX( Rρ)

m

m

I]−∞,ρ[(f (Zk)) ,

PX( Rρ) computed by numerical integration (does not depend on f ),

m more calls to f necessary for computing I]−∞,ρ[(f (Zk)) ⇒ n + m ≤ N.

Properties

Unbiased estimator ⇔ Rρ ⊂ Rρ.

Unbiased estimator of P(X ∈ Rρ ∩ Rρ) = EX(I]−∞,ρ[(f (X))Iˆ Rρ (X)).

k=1

P. Barbillon Bounding rare event probabilities

Πρ distribution

Importance sampling

Applications

Discussion

Upper confidence bound, Γ(f , Z1:m, ρ) ∼ B

m, P(X∈Rρ∩ Rρ)

P(X∈ Rρ)

P P(X ∈ Rρ ∩ Rρ) ≤ b(Γ(f , Z1:m, ρ), m, α)P(X ∈

Rρ)

Upper confidence bound given by crude Monte Carlo:

Decomposition:

P(P(X ∈ Rρ) ≤ b(Γ(f , X1:L, ρ), L, α)) = 1 − α .

πρ = P(X ∈ Rρ) = P(X ∈ Rρ ∩ Rρ) + P(X ∈ Rρ ∩ c

Rρ )

, ⇒ ∀α ∈]0; 1[,

> 1 − α ,

= E(I]−∞,ρ[(f (X))I Rρ (X)) + E(I]−∞,ρ[(f (X))(1 − I Rρ (X))) .

Πρ = E(I]−∞,ρ[(F Dn (X))I Rρ (X)|F Dn ) + E(I]−∞,ρ[(F Dn (X))(1 − I Rρ (X))|F Dn ) .

P. Barbillon Bounding rare event probabilities

Stochastic bound on Πρ

Proposition

Πρ distribution

Importance sampling

Applications

Discussion

For α, β ∈]0, 1[ such that α + β < 1,

P Πρ ≤ bPX( Rρ) + c

≥ 1 − (α + β) ,

β

where

b = b(Γ(F Dn , Z1:m, ρ), m, α),

c = E

Φ

ρ−mDn (X)

√ CDn (X,X)

(1 − I Rρ (X))

.

P. Barbillon Bounding rare event probabilities

Outline

1 Πρ distribution

2 Importance sampling

Πρ distribution

Importance sampling

Applications

Discussion

3 Applications

Toy example

A real case study: release enveloppe clearance

Toy example

A real case study: release enveloppe clearance

P. Barbillon Bounding rare event probabilities

Toy computer model

f : [−10, 10] 2 → R+ :

X ∼ U([−10, 10] 2 ).

Πρ distribution

Importance sampling

Applications

Discussion

Figure: Modèle

Toy example

A real case study: release enveloppe clearance

ρ = 0.01 given threshold ⇒ P (f (X) < ρ) = 4.72 · 10 −4 .

Constraint:

No more than N = 100 calls to f

P. Barbillon Bounding rare event probabilities

Comparing two strategies:

Πρ distribution

Importance sampling

Applications

Discussion

Toy example

A real case study: release enveloppe clearance

1 Bayesian strategy with a maximin design with n = 100 points,

2 MBIS (Metamodel Based Importance Sampling) with a maximin design

with n = 50 points. m = 50 calls to f to check if Z1:m ∈ Rρ.

Metamodelling concerns: h0 ≡ 1 et K (x, x ′ ) = exp(−θx − x ′ 2 ).

κ = 3 fixed in

Rρ,κ =

x : mDn(x) < ρ + κ

CDn(x, x) .

100 repetitions × 2 strategies × 3 sampling methods:

Full: LHS-maximin (Morris & Mitchell, 1995) .

JSUR: 80 % LHS-maximin, 20 % adaptive (Bect et al., 2011) .

tIMSE: 80 % LHS-maximin, 20 % adaptive (Picheny et al., 2011) .

Crude Monte Carlo with N = 100 calls,

estimators equals 0 with a probability greater than 0.95,

upper confidence bound 0.038 at level 98%, in this case.

P. Barbillon Bounding rare event probabilities

Πρ distribution

Importance sampling

Applications

Discussion

Toy example

A real case study: release enveloppe clearance

Figure: 98% Stochastic bounds on πρ

P. Barbillon Bounding rare event probabilities

Πρ distribution

Importance sampling

Applications

Discussion

Toy example

A real case study: release enveloppe clearance

Full LHS-maximin J SUR tIMSE

Minimum 4.55 5.50 4.30

1 st quartile 6.55 6.20 6.30

Mean 7.78 7.58 52.2

Median 7.10 6.30 6.45

3 rd quartile 7.97 6.52 6.87

Maximum 35.3 55.4 3027

Table: 98%-“Bayesian” bounds on πρ (multiplied by 10 4 )

Full LHS-maximin J SUR tIMSE

Minimum 6.82 4.98 5.88

1 st quartile 12.58 5.71 8.00

Mean 16.8 6.43 9.67

Median 16.4 6.28 9.39

3 rd quartile 20.1 6.81 11.3

Maximum 36.3 10.4 16.7

Table: 98%-“MBIS” bounds on πρ (multiplied by 10 4 )

P. Barbillon Bounding rare event probabilities

Context

Πρ distribution

Importance sampling

Applications

Discussion

Risk of collision with the airbone load.

Toy example

A real case study: release enveloppe clearance

Measure on the risk of collision (“algebraic distance") f : C × E ↦→ R

collision ⇔ f (xC, xE) < 0 .

x C controled parameters: speed, altitude....

x E uncontroled parameters: aerodynamics, humidity...

Here, C ⊂ R 5 and E ⊂ R 26

P. Barbillon Bounding rare event probabilities

Πρ distribution

Importance sampling

Applications

Discussion

Goal: classification of points in C

Formalization

x E realizations of the random vector X E ,

Risk of x C : π(x C) = P(f (x C, X E ) < 0).

Goal: give to xC ∈ C a label:

Toy example

A real case study: release enveloppe clearance

safe if π(xC) ≤ 10 −5

harmful if π(xC) ≥ 10 −2

relatively safe if 10 −5 < π(xC) < 10 −2 ⇒ sharply upper bound π(xC)

For a given xC, no more than 2000 calls to f available

f : x ∈ E ↦→ f (x C, x), X = X E ,

R = {x ∈ E : f (x) < 0},

π = P(f (X) < 0) = P(X ∈ R).

P. Barbillon Bounding rare event probabilities

Πρ distribution

Importance sampling

Applications

Discussion

Combining the two strategies

1 Dn space-filling design with n = N/2, giving F Dn .

Toy example

A real case study: release enveloppe clearance

−mDn (X)

2 Computation of E(Π) = E Φ √ by numerical integration.

KDn (X,X)

a Label safe if E(Π) ≤ 10−10

4

b Label harmful if E(Π) ≥ 10 −2

c otherwise ( 10−10

4

3 Importance sampling (only in case c)

(based on P(Π ≤ 10−5

2 ) ≥ 1 − 10−5

2 ).

< E(Π) < 10−2 ), second stage procedure...

(Z 1, . . . , Z m=N/2) sample with distribution PX(.| ˆ Rρ)

Bound on πρ with a 1 − 2α confidence level. (m more calls to f are done).

P. Barbillon Bounding rare event probabilities

To conclude

Πρ distribution

Importance sampling

Applications

Discussion

Bayesian strategy is natural in this context.

⊲ Difficulties to simulate conditioned Gaussian process in “high dimension”

(space-filling concern, numerical stability...),

⊲ Discretization induced by simulations is NOT taken into account.

Metamodel based Importance sampling (MBIS) strategy is more fluent.

⊲ More sensitive to the design of experiments,

⊲ Achieves similar performances than the Bayesian strategy with an adaptive

design.

P. Barbillon Bounding rare event probabilities

Future work

Πρ distribution

Importance sampling

Applications

Discussion

Cross validation to check metamodelling assumptions,

⇒ (if necessary) increasing σ 2 (∼ less informative prior).

Tune κ, n/m in MBIS ?

Rρ depending on several outputs ?

P. Barbillon Bounding rare event probabilities

References

Πρ distribution

Importance sampling

Applications

Discussion

Auffray, Y., Barbillon, P. et Marin, J.-M. (2011).

Bounding rare event probabilities in computer experiments.

ArXiv, http://arxiv.org/abs/1105.0871

Bect, J., Ginsbourger, D., Li, L., Picheny, V., and Vazquez, E. (2011).

Sequential design of computer experiments for the estimation of a

probability of failure.

Statistics and Computing.

Oakley, J. (2004).

Estimating percentiles of uncertain computer code outputs.

Applied Statistics, 53:83–93.

Picheny, V., Ginsbourger, D., Roustant, O., and Haftka, R. (2010).

Adaptive designs of experiments for accurate approximation of a target

region.

Journal of Mechanical Design, 132(7).

P. Barbillon Bounding rare event probabilities

More magazines by this user
Similar magazines