Bayes - Medreonet

Background

Deductive logical schemes
Result 1
Result 2
cause
Result 3
Result 4

Inductive logical schemes
cause
cause
cause
Result 1
Result 2
Result 3
Result 4
Given a series of outcomes, it is possible to hypothesize the
most likely cause(s). The more the prior information we have,
the better the reliability of our estimates

Two rules are important to apply Bayes'
theorem:
RULE OF SUM:
P
( A) + P( A)
= 1
where: P ( A) = P( A not occurring)

Two rules are important to apply Bayes'
theorem:
RULE OF PRODUCT:
P
P
( A∩
B) = P( B) ⋅ P( A B)
( B ∩ A) = P( A)
⋅ P(
B / A)
P
( B A
)
=
P
( B
) ⋅
P
( A B
)
P( A)

Bayes'
theorem

P
( B A)
=
P
( B) ⋅ P( A B)
P( A)
If we consider B the hypothesis and A the set of collected data:
P
where:
( hypothesis | data)
=
P
( hypothesis)
⋅ P(
data / hypothesis)
P
( data)
P(hypothesis) = Prior probability of the hypothesis is the probability of the
hypothesis being true before we collect and analyze the set of data, that is
the state of knowledge before the analysis of data
P(data | hypothesis) = Likelihood function obtained after the experiment
P(hypothesis | data) = Posterior probability of the hypothesis after the
analysis of data
P(data) = Probability of all possible cases giving those data

Example:
Predictive value of a positive result to a diagnostic
test.
We take an animal belonging to a population with
prevalence of infection PR to a certain disease
The hypothesis is "the animal is infected"

P(hypothesis) = probability of the hypothesis being true before
we collect and analyze the set of data = PR prevalence of
infection
We test the animal and get a positive result. This is the data
collection.
P(data | hypothesis) = Likelihood function obtained after the
experiment = probability of a positive test result (the data we
got) given that the animal is infected (that's our hypothesis) = it
is the Sensitivity of our diagnostic test!
P(data) = Probability of all possible cases giving those data. A
positive animal can be a either True positive or a False positive.
Probability of being a True positive is PR times Sensitivity of the test
Probability of being a False positive is (1-PR) times (1-Specificity of the
test)
so, the Predictive value of a positive result to a diagnostic
test, i.e. the probability of our positive animal being infected is:
P
( )
( hypothesis)
⋅ P(
data / hypothesis)
PR ⋅ Sensitivity
P hypothesis | data =
=
P( data) PR ⋅ Sensitivity + ( 1−
PR) ⋅( 1−
Specificity)

Bayesian inference

Bayesian inference
Bayesian inference is a useful, powerful technique whereby
newly acquired empirical data can be combined with
existing information,
whether that information is itself based on pre-existing
empirical data or on expert opinion,
to improve an estimate of the parameter(s) used to
characterise a distribution.
Note: Bayesian inference is based on subjective probability (q.v.)

Bayesian inference
Bayesian inference is a natural extension of Bayes’
theorem
It provides a powerful and flexible means of learning
from experience
As new information becomes available it enables our
existing knowledge to be easily and logically updated
It explicitly acknowledges subjectivity and describes
the learning process mathematically
We begin with an opinion, however vague, and modify
it as new information becomes available

Bayesian inference
Bayesian inference involves three steps:
1. Determining a prior estimate of a parameter in the form of
a probability distribution that expresses our state of
knowledge (or ignorance) before any observations are
made.
2. Finding an appropriate likelihood function for the
2. Finding an appropriate likelihood function for the
observed data. The likelihood function calculates the
probability of observing the data for a given value of the
prior estimate of the parameter
3. Calculating the posterior (i.e. revised) estimate of the
parameter in form of a probability distribution of all
possible values of the parameter

Step 1. Prior distributions
A prior distribution expresses our state of knowledge
before any new observations are made
The prior distribution is not necessarily dependent on
data and may be purely subjective
Depending on the circumstances there are several
options available:
1. Uninformed priors
2. Informed priors

Uninformed priors
An uninformed prior does not provide any
additional information to a Bayesian inference
other than establishing a possible range.
For example, in some circumstances we may not have any
information about the likely prevalence of infection within a
herd.
We might assume that, for a particular disease, the prevalence
is likely to range from 0% to 30% and that any value within
this range is equally as likely as any other value.
This constitutes a uniform prior, Uniform(0,0.3), and has no
influence on the Bayesian inference calculation, apart from
establishing a range

Other examples of uninformed priors
We might want to estimate the average number of
disease outbreaks per year (λ)
If we assume that each outbreak is independent of
every other outbreak and that there is a constant and
continuous probability of a disease outbreak occurring
throughout the year, then the outbreaks follow a
Poisson process.

Other examples of uninformed priors
The average number of outbreaks per year can also be
expressed as 1/β, where β is the mean interval between
events
We might think it is reasonable to assign an
We might think it is reasonable to assign an
uninformed prior in the form of a uniform
distribution, Uniform(0,x), to λ.

Other examples of uninformed priors
However, we could have just as easily parameterised the
problem in terms of β.
Since β=1/λ our prior distribution would be 1/Uniform(0,x)
which is clearly not uninformed with respect to β.
probability
0.35
0.30
0.25
0.20
0.15
0.10
prior distribution for β
expressed as
1/λ = 1/Uniform(0,x)
0.05
0.00
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Other examples of uninformed priors
A useful technique in these circumstances to minimise the
effects of re-parameterisation is
to set up the prior distribution for λ as 1/λ and for β as 1/β,
that is we are using β as a prior for λ and vice versa.
As a result, the prior distribution is transformation
invariant.

Other examples of uninformed priors
While such a distribution still does not appear to be
uninformed, it is the best that can be achieved in the
circumstance
and gives the same answer whether we undertake an
analysis from the point of view of λ or β.
probability
0.35
0.30
0.25
0.20
0.15
0.10
0.05
0.00
prior distribution for λ
1
prior( λ)
∝
λ
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
lambda

Informed priors
An informed prior may be based on actual data or be
purely subjective
A conjugate prior has the same functional form as the
likelihood function and leads to a posterior
distribution belonging to the same distribution as the
prior

Conjugate priors and their associated likelihoods
and posterior distributions

Step 2. Likelihood functions
The likelihood function calculates the probability of
observing the data for a given value of the prior
estimate of a parameter
The shape of the likelihood function embodies the
amount of information contained in the data
If the information is limited, the likelihood function
will be broadly distributed, whereas if the information
is significant, the likelihood function will be tightly
focused around a particular parameter value

Likelihood functions
There are a number of useful probability distribution
functions that can be used as likelihood functions,
depending on the circumstances
These include the Binomial, Poisson, hypergeometric and
These include the Binomial, Poisson, hypergeometric and
negative binomial

Step 3. Posterior distributions
The posterior distribution is the revised estimate
of the parameter we are investigating and is
obtained simply by multiplying the prior
distribution and the likelihood function
Since the individual probabilities calculated by the
likelihood function are independent of each other,
the resulting posterior probabilities need to be
normalised
This ensures that the area under the curve of a
continuous distribution equals one and that the
probabilities for a discrete distribution all add up
to one

What form does Bayes’ theorem assume
when dealing with distributions
where:
pr(H i |D) = Posterior probability of the hypothesis after the analysis of
data. We have a distribution of posterior probabilities (one for each of the i
values of the distribution)
pr(H i ) = Prior probability of the hypothesis (one for each of the i values of
the distribution)
Σ j pr(H j )pr(D|H j ) = Sum of all probabilities of the data given all the
hypotheses. (*)

What form does Bayes’ theorem assume
when dealing with distributions
Note:
Σ j pr(H j )pr(D|H j ) = Sum of all probabilities of the data given all the
hypotheses. It is very difficult to know all probabilities of data given all the
hypotheses.
Nevertheless, since this expression is the same denominator for each
pr(H i ǀD):
pr(H i ǀD)∝pr(H i )pr(DǀH i )
and Σ j pr(H j )pr(DǀH j ) becomes a normalization constant to make the sum of
all numerators = 1

How to apply bayesian inference
Let’s imagine that we test a flock of 50 sheep with a
test with 93% sensitivity and 95% specificity
We get no positive results
What is the probability distribution of the number of
What is the probability distribution of the number of
(undetected) infected animals in the flock?

How to apply bayesian inference
Tested= 50
Positive = 0
Sensitivity of the test= 93%
Specificity of the test= 95%
=Uniform(0,50)
→Pr(H i )=1/51
Infected animals
in the sample Prior Binomial likelihood Posterior Nurmalized posterior
0 0,019608 1 0,019608 0,93
1 0,019608 0,07 0,001373 0,0651
2 0,019608 0,0049 9,61E-05 0,004557
3 0,019608 0,000343 6,73E-06 0,00031899
49 0,019608 2,56924E-57 5,04E-59 2,38939E-57
50 0,019608 1,79847E-58 3,53E-60 1,67257E-58

How to apply bayesian inference
Tested= 50
Positive = 0
Sensitivity of the test= 93%
Specificity of the test= 95%
=Binomial(0,n,93%,false)
Infected animals
in the sample Prior Binomial likelihood Posterior Nurmalized posterior
0 0,019608 1 0,019608 0,93
1 0,019608 0,07 0,001373 0,0651
2 0,019608 0,0049 9,61E-05 0,004557
3 0,019608 0,000343 6,73E-06 0,00031899
49 0,019608 2,56924E-57 5,04E-59 2,38939E-57
50 0,019608 1,79847E-58 3,53E-60 1,67257E-58

How to apply bayesian inference
Tested= 50
Positive = 0
Posterior=Prior x Likelihood
Sensitivity of the test= 93%
Specificity of the test= 95%
Infected animals
in the sample Prior Binomial likelihood Posterior Nurmalized posterior
0 0,019608 1 0,019608 0,93
1 0,019608 0,07 0,001373 0,0651
2 0,019608 0,0049 9,61E-05 0,004557
3 0,019608 0,000343 6,73E-06 0,00031899
49 0,019608 2,56924E-57 5,04E-59 2,38939E-57
50 0,019608 1,79847E-58 3,53E-60 1,67257E-58

How to apply bayesian inference
Tested= 50
Positive = 0
Sensitivity of the test= 93%
Specificity of the test= 95%
Normalized=Posterior/Sum
Infected animals
in the sample Prior Binomial likelihood Posterior Nurmalized posterior
0 0,019608 1 0,019608 0,93
1 0,019608 0,07 0,001373 0,0651
2 0,019608 0,0049 9,61E-05 0,004557
3 0,019608 0,000343 6,73E-06 0,00031899
49 0,019608 2,56924E-57 5,04E-59 2,38939E-57
50 0,019608 1,79847E-58 3,53E-60 1,67257E-58

How to apply bayesian inference
Tested= 50
Positive = 0
Sensitivity of the test= 93%
Specificity of the test= 95%
Normalized=Posterior/Sum
Infected animals
in the sample Prior Binomial likelihood Posterior Nurmalized posterior
0 0,019608 1 0,019608 0,93
1 0,019608 0,07 0,001373 0,0651
2 0,019608 0,0049 9,61E-05 0,004557
3 0,019608 0,000343 6,73E-06 0,00031899
49 0,019608 2,56924E-57 5,04E-59 2,38939E-57
50 0,019608 1,79847E-58 3,53E-60 1,67257E-58

The result is:
Bayesian inference
1
0,9
0,8
0,7
Probability
0,6
0,5
0,4
0,3
0,2
0,1
0
0 1 2 3 4 5
Infected animals in the sample

If we have
positives?

Second order bayesian inference
Let’s imagine to test the same flock and to get three
positives
What is the probability of having 0, 1, 2, 3 true
positives and the remaining positives as false positives?
What is the probability of having 0, 1, 2, 3, 4, ...
infected animals in the flock?

Second order bayesian inference
Tested= 50
Positive = 3
Sensitivity of the test= 93%
Specificity of the test= 95%
Uniform(0,50)
Uniform(0,3)
Binomial likelihood
Posterior
True positives
True positives
Infected animals 0 1 2 3 0 1 2 3
in the sample Prior 0,25 0,25 0,25 0,25 0,25 0,25 0,25 0,25 Normalized posterior
0 0,019608 0,219875 0 0 0 0,001078 0 0 0 0,258188132
1 0,019608 0,015229 0,24538 0 0 7,47E-05 0,001203 0 0 0,306020881
2 0,019608 0,001053 0,034685 0,186289 0 5,16E-06 0,00017 0,000913 0 0,260715921
3 0,019608 7,28E-05 0,003674 0,040322 0,072187 3,57E-07 1,8E-05 0,000198 0,000354 0,136512804
4 0,019608 5,02E-06 0,000346 0,005816 0,021276 2,46E-08 1,69E-06 2,85E-05 0,000104 0,032224319
49 0,019608 0 0 2,67E-53 1,05E-49 0 0 1,31E-55 5,17E-52 1,23862E-49
50 0,019608 0 0 0 8,27E-51 0 0 0 4,05E-53 9,70674E-51

Second order bayesian inference
Binomial(TP,Inf,Se,False) x
Tested= 50
Positive = 3
Sensitivity of the test= 93%
Specificity of the test= 95%
Binomial(Tested-Inf-(Pos-TP),Tested-Inf,Sp,False)
Binomial likelihood
Posterior
True positives
True positives
Infected animals 0 1 2 3 0 1 2 3
in the sample Prior 0,25 0,25 0,25 0,25 0,25 0,25 0,25 0,25 Normalized posterior
0 0,019608 0,219875 0 0 0 0,001078 0 0 0 0,258188132
1 0,019608 0,015229 0,24538 0 0 7,47E-05 0,001203 0 0 0,306020881
2 0,019608 0,001053 0,034685 0,186289 0 5,16E-06 0,00017 0,000913 0 0,260715921
3 0,019608 7,28E-05 0,003674 0,040322 0,072187 3,57E-07 1,8E-05 0,000198 0,000354 0,136512804
4 0,019608 5,02E-06 0,000346 0,005816 0,021276 2,46E-08 1,69E-06 2,85E-05 0,000104 0,032224319
49 0,019608 0 0 2,67E-53 1,05E-49 0 0 1,31E-55 5,17E-52 1,23862E-49
50 0,019608 0 0 0 8,27E-51 0 0 0 4,05E-53 9,70674E-51

Second order bayesian inference
Tested= 50
Positive = 3
Sensitivity of the test= 93%
Specificity of the test= 95%
Prior 1 x Prior 2 x Likelihood
Binomial likelihood
Posterior
True positives
True positives
Infected animals 0 1 2 3 0 1 2 3
in the sample Prior 0,25 0,25 0,25 0,25 0,25 0,25 0,25 0,25 Normalized posterior
0 0,019608 0,219875 0 0 0 0,001078 0 0 0 0,258188132
1 0,019608 0,015229 0,24538 0 0 7,47E-05 0,001203 0 0 0,306020881
2 0,019608 0,001053 0,034685 0,186289 0 5,16E-06 0,00017 0,000913 0 0,260715921
3 0,019608 7,28E-05 0,003674 0,040322 0,072187 3,57E-07 1,8E-05 0,000198 0,000354 0,136512804
4 0,019608 5,02E-06 0,000346 0,005816 0,021276 2,46E-08 1,69E-06 2,85E-05 0,000104 0,032224319
49 0,019608 0 0 2,67E-53 1,05E-49 0 0 1,31E-55 5,17E-52 1,23862E-49
50 0,019608 0 0 0 8,27E-51 0 0 0 4,05E-53 9,70674E-51

Second order bayesian inference
Tested= 50
Positive = 3
Sensitivity of the test= 93%
Sum(Posterior) / Sum
Specificity of the test= 95%
Binomial likelihood
Posterior
True positives
True positives
Infected animals 0 1 2 3 0 1 2 3
in the sample Prior 0,25 0,25 0,25 0,25 0,25 0,25 0,25 0,25 Normalized posterior
0 0,019608 0,219875 0 0 0 0,001078 0 0 0 0,258188132
1 0,019608 0,015229 0,24538 0 0 7,47E-05 0,001203 0 0 0,306020881
2 0,019608 0,001053 0,034685 0,186289 0 5,16E-06 0,00017 0,000913 0 0,260715921
3 0,019608 7,28E-05 0,003674 0,040322 0,072187 3,57E-07 1,8E-05 0,000198 0,000354 0,136512804
4 0,019608 5,02E-06 0,000346 0,005816 0,021276 2,46E-08 1,69E-06 2,85E-05 0,000104 0,032224319
49 0,019608 0 0 2,67E-53 1,05E-49 0 0 1,31E-55 5,17E-52 1,23862E-49
50 0,019608 0 0 0 8,27E-51 0 0 0 4,05E-53 9,70674E-51

Second order bayesian inference
Tested= 50
Positive = 3
Sensitivity of the test= 93%
Sum(Posterior) / Sum
Specificity of the test= 95%
Binomial likelihood
Posterior
True positives
True positives
Infected animals 0 1 2 3 0 1 2 3
in the sample Prior 0,25 0,25 0,25 0,25 0,25 0,25 0,25 0,25 Normalized posterior
0 0,019608 0,219875 0 0 0 0,001078 0 0 0 0,258188132
1 0,019608 0,015229 0,24538 0 0 7,47E-05 0,001203 0 0 0,306020881
2 0,019608 0,001053 0,034685 0,186289 0 5,16E-06 0,00017 0,000913 0 0,260715921
3 0,019608 7,28E-05 0,003674 0,040322 0,072187 3,57E-07 1,8E-05 0,000198 0,000354 0,136512804
4 0,019608 5,02E-06 0,000346 0,005816 0,021276 2,46E-08 1,69E-06 2,85E-05 0,000104 0,032224319
49 0,019608 0 0 2,67E-53 1,05E-49 0 0 1,31E-55 5,17E-52 1,23862E-49
50 0,019608 0 0 0 8,27E-51 0 0 0 4,05E-53 9,70674E-51

And the result is:
Second order bayesian inference
0,35
0,3
0,25
Probability
0,2
0,15
0,1
0,05
0
0 1 2 3 4 5 6 7 8 9 10
Number of infected animals