AOH 0510 Emulating a.. - MUCM

mucm.ac.uk

AOH 0510 Emulating a.. - MUCM

Emulating a computationally intensive

simulation model of cost-effectiveness

Tony O’Hagan, Jeremy Oakley and John Stevens


Outline

Introduction

RADDM, BCTS, CEA, patient-level simulation

The problem

Infeasibility of running the economic model enough times

The solution

Emulators, emulating the economic model, efficiency

Some results

Conclusions

2 SSC,

Quebec City, May 2010


3 SSC,

Introduction

Quebec City, May 2010


RADDM

The Rheumatoid Arthritis Drug Development Tool is a

decision-support tool developed for AstraZeneca

AZ are researching drugs for RA and expect to bring

forward a series of promising new compounds for testing

For each new compound RADDM will predict chances of

it being developed into a successful drug given:

Results of early trials

Design proposals for future trials

Rules for stopping development

Facilitates optimisation of development process

By exploring different designs and termination rules

4 SSC,

Quebec City, May 2010


BCTS

RADDM uses Bayesian Clinical Trial Simulation

Sample uncertain parameters

for drug performance

Simulation based on trial

design and sampled parameters

Simulated trial outcomes

determine regulatory success

Also cost-effectiveness

Report probability of success

Simulate parameters

Simulate patients

and outcomes

Determine trial

results

Collate results

Report assurances.

5 SSC,

Quebec City, May 2010


CEA

Cost-Effectiveness Analysis is increasingly important in

healthcare decision-making

Healthcare providers choosing what drugs will be paid for on

the basis of their cost-effectiveness

Cost of all healthcare, not just the drug itself

Effectiveness in terms of quality and quantity of life

QALY = Quality-Adjusted Life Years

Willingness to pay relates the two

Net benefit is average over the treated population of

( WTP x QALYs ) – Cost

If net benefit is positive, drug is cost-effective

6 SSC,

Quebec City, May 2010


Economic modelling

All three components of the formula are uncertain

Decision-makers usually want to consider a range of WTP

Both population mean costs and QALYs are uncertain

Evidence for costs and QALYs is often messy and involves

considering treatment and disease development over time

Economic models represent this process in order to deliver

population mean costs and QALYs

Increasingly, such models involve simulating large numbers of

individual patients

The models contain uncertain parameters

Typically run many times to evaluate distributions of mean costs and

QALYs

Known in CEA as probabilistic sensitivity analysis

But uncertainty analysis in other fields

7 SSC,

Quebec City, May 2010


8 SSC,

The problem

Quebec City, May 2010


Too much simulation

The economic model in RADDM for evaluating costeffectiveness

involves three nested levels of simulation

BCTS simulates final trial

outcomes

PSA simulates true drug

performance parameters

Economic model

simulates patients

Computation time becomes

enormous

BCTS

PSA

Thousands of iterations needed at each level

9 SSC,

Patients

Quebec City, May 2010


And to make things worse

There is a fourth level at which the company explores

different trial designs and decision rules

Any kind of exploration will be computationally infeasible

Design

parameters

and

termination

rules

BCTS

PSA

Patients

10 SSC,

Quebec City, May 2010


11 SSC,

The solution

Quebec City, May 2010


Computer models

In almost all fields of science, technology, industry and

policy making, people use mechanistic models to describe

complex real-world processes

For understanding, prediction, control

There is a growing realisation of the importance of

uncertainty in model predictions

Can we trust them?

Without any quantification of output uncertainty, it’s easy to

dismiss them

12

SSC,

Quebec City, May 2010


Examples

Climate

prediction

Molecular

dynamics

Nuclear waste

disposal

Oil fields

Engineering design

Hydrology

13

SSC,

Quebec City, May 2010


MUCM

The Managing Uncertainty in Complex Models project is

addressing the whole area of quantifying and analysing the

uncertainty in predictions of complex simulation models

http://mucm.group.shef.ac.uk

A key tool is emulation

The simulator is a function whose arguments are the simulator

inputs and whose results are the simulator outputs

When simulator runs take a long time, various approaches have

been proposed to approximate the function

Often called metamodels

An emulator is, in our opinion, the best and most efficient kind

of metamodel

14 SSC,

Quebec City, May 2010


Emulation

We use Bayesian statistics

Based on a training sample of model runs, we estimate what

the model output would be at all untried input configurations

The result is a statistical representation of the model

In the form of a stochastic process over input space

Posterior distribution of the output as a function of the inputs

The process mean is our best estimate of what the output would be

at any input configuration

Uncertainty is captured by variances and covariances

It correctly returns what we know

At any training sample point, the mean is the observed value

With zero variance

15 SSC,

Quebec City, May 2010


2 code runs

Consider one input and one output

Emulator estimate interpolates data

Emulator uncertainty grows between data points

16

dat2

10

5

0

0

1

2

3

x

4

5

6

SSC, Quebec City, May 2010


3 code runs

Adding another point changes estimate and reduces

uncertainty

17

dat3

10

5

0

0

1

2

3

x

4

5

6

SSC, Quebec City, May 2010


5 code runs

And so on

18

dat5

9

8

7

6

5

4

3

2

1

0

0

1

2

3

x

4

5

6

SSC, Quebec City, May 2010


Emulating the RADDM PSA

We emulated the combination of the two inner

simulation levels (PSA applied to the economic model)

Inputs: results of the final clinical trial

Estimate and standard error of drug effect

Outputs: PSA summaries

PSA expectations of mean costs and QALYs

PSA variances and covariance of mean costs and QALYs

Emulator built using fewer than 100 PSA runs

Each taking about an hour

The emulator is then used for the BCTS simulations

5000 simulations for each set of design/decision parameters

19 SSC,

Quebec City, May 2010


Application

New AstraZeneca compound that had completed a Phase

2a study

A small number of patients followed up for 1 month

They now needed to design a Phase 2b study, to be

followed if results were encouraging by a full Phase 3

study

A larger and/or longer 2b study would have a better chance of

ditching the new compound if ineffective

But would be more costly if it proved to be effective.

RADDM was used to search 72 possible design options

4 (Phase 2b sample size) x 3 (Phase 3 sample size) x 3 (noninferiority

margin) x 2 (duration of the Phase 2b trial)

20 SSC,

Quebec City, May 2010


Chance of passing regulatory approval

Chance of success

21

35.0%

30.0%

25.0%

20.0%

15.0%

10.0%

5.0%

0.0%

Development Programme

SSC,

Pass Phase 2b, Fail

Phase 3

Pass Phase 3

Quebec City, May 2010


Trial results from two different designs

5000

Simulations

Fail Phase 2b:

Ex. 1 Ex.2

Total Failures: 4066 4171

(81.3%) (83.4%)

non-inferiority 4035 3904

superiority 2243 2125

adverse events 169 1347

22

Pass Phase 2b:

Ex. 1 Ex.2

934 (18.7%) 829 (16.6%)

Fail Phase 3:

SSC,

Pass Phase 3:

Ex. 1 Ex.2

716 (14.3% ) 750 (15.0%)

Ex. 1 Ex.2

Total Failures: 218 79

(4.4%) (1.6%)

non-inferiority 26 30

adverse events 202 51

Quebec City, May 2010


PSA results

For the best trial designs, in every simulation that

produced trial results leading to regulatory approval we

also had a high probability of cost-effectiveness

At the anticipated drug price and likely WTP values

However, this compound was dropped without going

into Phase 2b

The low probability of regulatory success made it not worth

pursuing

Even though cost-effectiveness and reimbursement approval

would almost certainly have followed regulatory approval

23 SSC,

Quebec City, May 2010


24 SSC,

Conclusions

Quebec City, May 2010


The power of emulation

Emulation makes all kinds of investigations of complex

computer models possible, even when individual model

runs are very expensive/computer-intensive

Sensitivity analysis, calibration, data assimilation etc

In the case of RADDM, it would have been impossible to

determine cost-effectiveness by brute force multi-level

simulation

Emulation was crucial to providing this level of decision

support information

25 SSC,

Quebec City, May 2010

More magazines by this user
Similar magazines