STA 36-786: Bayesian Theoretical Statistics I

STA 36-786: Bayesian Theoretical Statistics I
Assignment 3: Spring 2013
Due: Tuesday February 26 at 10:30 a.m.
Show all your work to obtain full/partial credit. You are not to consult outside sources other
than your class notes/slides/reference books for this assignment (except for the instructor
or TA). No late assignments will be accepted. Please follow the instructions for writing up
solutions given out on 1.12.13. Start each problem on a new page.
1. Let
Y |θ ∼ Exp(θ)
θ ∼ Gamma(a, b).
Suppose we have a new observation Ỹ |θ ∼ Exp(θ), where conditional on θ, Y and Ỹ
are independent. Show that
p(ỹ|y) =
b(a + 1)(by + 1)a+1
(bỹ + by + 1) a+2 ,
where a is an integer. (Note that this is a valid density function that integrates to 1).
Solution: Observe that
p(θ|y) ∝ p(θ)p(y|θ) ∝
(
θ a−1 e −b−1 θ ) ( θe −θy) = θ a e −(b−1 +y)θ
Thus θ|y ∼ Gamma(a + 1, (b −1 + y) −1 ). Next, recall that
∫
=
Observe that
∫
p(ỹ|y) =
p(ỹ|θ)p(θ|y)dθ =
θe −θỹ (b−1 + y) a+1
θ a e −(b−1 +y)θ dθ = (b−1 + y) a+1 ∫
Γ(a + 1)
Γ(a + 1)
= (b−1 + y) a+1
Γ(a + 1)
∫ ∞
0
p(ỹ|y)dỹ =
Γ(a + 2) b(a + 1)(1 + by)a+1
(b −1 =
+ y + ỹ) a+2 (1 + by + bỹ) a+2
∫ ∞
0
b(a + 1)(1 + by) a+1
(1 + by + bỹ) a+2 dỹ =
∣
(1 + by)a+1 ∣∣∣
∞
= −
(1 + by + bỹ) a+1 = 1
0
θ a+1 e −(b−1 +y+ỹ)θ dθ =

2. Suppose
X 1 , . . . , X n |θ iid ∼ Poisson(θ).
(a) Find Jeffreys’ prior. Is it proper or improper?
(b) Find p(θ|x 1 , . . . , x n ) under Jeffreys’ prior.
Solution:
(a) Since (X 1 , . . . , X n )|θ are iid, I X1 ,...,X n
(θ) = nI X1 (θ) ∝ I X1 (θ). Hence, in order to
determine Jeffrey’s prior, it is sufficient to compute the Fisher Information for a
single observation . Since X 1 |θ ∼ Poisson(θ),
Hence,
Thus, p(θ) ∝ I(θ) 1 2 = θ − 1 2 .
log(f(X|θ)) = −θ + X 1 log(θ) − log(Γ(X 1 + 1))
[ ∣ ]
d log f(X1 |θ) ∣∣∣
I(θ) = −E
dθ 2 θ = E[X 1 θ −2 |θ] = θ −1
(b)
∫ ∞
0
θ − 1 2 dθ = 2 −1 θ 1 2 ∣ ∞ 0
= ∞
Since p(θ) is not integrable, the Jeffrey’s prior in this case is improper.
p(θ|x 1 , . . . , x n ) ∝ p(θ)p(x 1 , . . . , x n |θ) ∝
n∏
∝ θ − 1 e −θ θ x i
2
Γ(x i + 1) ∝
i=1
∝ θ n¯x− 1 2 e
−nθ
Conclude that θ|x 1 , . . . , x n ∼ Gamma ( n¯x + 1 2 , n−1) .
3. Consider dose response models. The setup is the following: animals are tested for
development of drugs or other chemical compounds. Someone administers various
levels of doses to k batches of animals. The response variable is a dichotomous (binary)
outcome. So, it might be alive or dead or maybe tumor or no tumor. Let x i represent
the data, n i represent the number of animals receiving the ith dose, and y i the number
of positive outcomes for n i animals.
ind
(a) Suppose that y i ∼ Binomial(n i , θ i ), where θ i is the probability of death (or
tumor) for the ith animal that receives dose x i . The typical modeling the prior
on θ i is a logistic regression. That is, we suppose that logit(θ i ) = α + βx i . Write
out the likelihood in a simple form (it will contain a product).
2

(b) Find Jeffreys’ prior for (α, β). Also, write down the equations you need to solve
for finding the posterior modes α and β under the uniform prior for α and β.
Solution:
(a) It follows from logit(θ i ) = α + βx i , that θ i = exp(α+βx i)
(
problem description, y i |(α, β) ∼ Binomial n i ,
p(y 1 , . . . , y k |(α, β)) =
=
i=1
1+exp(α+βx i ) .
exp(α+βx i )
1+exp(α+βx i )
Hence, from the
)
. Conclude that
k∏
( ) ( )
ni exp(α + βxi )
yi
(
1
y i 1 + exp(α + βx i ) 1 + exp(α + βx i )
k∏
( )
ni
(exp(α + βx i )) y i
(1 + exp(α + βx i )) −n i
y i
i=1
(b) Using the previous item, observe that:
) ni −y i
log(p(y 1 , . . . , y k |(α, β))) =
−
k∑
i=1
( )
ni
log +
y i
k∑
y i (α + βx i )+
i=1
k∑
n i log(1 + exp(α + βx i ))
i=1
Hence, the gradient of the log-likelihood is:
d log(p(y 1 , . . . , y k |(α, β)))
dα
d log(p(y 1 , . . . , y k |(α, β)))
dβ
= kȳ −
=
k∑
i=1
k∑
x i y i −
i=1
n i exp(α + βx i )
1 + exp(α + βx i )
k∑
i=1
n i x i exp(α + βx i )
1 + exp(α + βx i )
Taking a uniform prior on (α, β), the posterior is proportional to the likelihood.
Thus, the modes of the posterior correspond to the MLE. In order to find the
MLE, we must solve the system of equations obtained setting the gradient to 0.
The Hessian matrix of the log-likelihood is:
[ ∑ − k
i=1
H log(p(y 1 , . . . , y k |(α, β))) =
− ∑ k
i=1
n i exp(α+βx i )
(1+exp(α+βx i )) 2
n i x i exp(α+βx i )
(1+exp(α+βx i )) 2
− ∑ k
i=1
− ∑ k
i=1
n i x i exp(α+βx i )
]
(1+exp(α+βx i )) 2
n i x 2 i exp(α+βx i)
(1+exp(α+βx i )) 2
Since the Hessian matrix is constant on y i ,
I(α, β) = −H(α, β) =
[ ∑ k
i=1
n i exp(α+βx i ) ∑ k n i x i exp(α+βx i )
]
(1+exp(α+βx i )) 2 i=1 (1+exp(α+βx i ))
∑ 2
k n i x i exp(α+βx i ) ∑ k n i x 2
i=1 (1+exp(α+βx i )) 2 i exp(α+βx i)
i=1 (1+exp(α+βx i )) 2
3

Conclude that the Jeffreys prior is
p(α, β) = |I(α, β| 1 2
=
⎛
⎝ ∑ i,j
⎞
n i exp(α + βx i )n j exp(α + βx j )(x 2 j − x ix j )
⎠
(1 + exp(α + βx i )) 2 (1 + exp(α + βx j )) 2
1
2
4. Consider the Galenshore distribution (it’s just a transformed Gamma density). That
is, let Y |θ ∼ Galenshore(a, θ). Then
(a) Consider
p(y|θ) = 2
Γ(a) θ2a y 2a−1 e −θ2 y 2 , y > 0, a > 0, θ > 0.
Y |θ ∼ Galenshore(a, θ), a known, θ unknown
θ ∼ Galenshore(c, d).
Find the posterior distribution of θ|y.
Solution:
p(θ|y) ∝ 2
Γ(a) θ2a y 2a−1 e −θ2 y 2 2
Γ(c) d2c θ 2c−1 e −d2 θ 2
∝ θ 2a e −θ2 y 2 θ 2c−1 e −d2 θ 2
= θ 2(a+c)−1 e −θ2 (y 2 +d 2) .
Thus, θ|y ∼ Galenshore(a + c, √ y 2 + d 2 ).
4