STA 36-786: Bayesian Theoretical Statistics I

STA 36-786: Bayesian Theoretical Statistics I
Assignment 1: Spring 2013
Due: 1/31/2013 at 10:30 a.m.
Show all your work to obtain full/partial credit. You are not to consult outside sources
other than your class notes/slides for this assignment (except for the instructor or TA). No
late assignments will be accepted. Please follow the instructions for writing up solutions
that were given out in class on 1.12.13.
1. (a) Find the Bayes estimator when the loss function is L(θ, δ(x)) = c (θ − δ(x)) 2 ,
where c is a constant.
(b) Derive the Bayes estimator when L(θ, δ(x)) = w(θ)(g(θ) − δ(x)) 2 . Do so without
writing any integrals. Note that you can write ρ(π, δ(x)) = E[L(θ, δ(x))|X].
Solution: When L(θ, δ) = c(θ − δ) 2 , the Bayes estimator is simply the posterior mean.
When L(θ, δ) = w(θ)(g(θ) − δ(x)) 2 , we want to minimize
ρ = E[w(θ){g(θ) − δ(x)} 2 | X] = E[w(θ){g(θ) 2 + δ(x) 2 − 2δ(x)g(θ)} | X].
Then
∂ρ
E[w(θ)g(θ) | X]
= 2E[w(θ)δ(x) | X] − 2E[w(θ)g(θ) | X] = 0 =⇒ δ(x) =
∂δ(x) E[w(θ) | X]
Then verify that this is indeed a minimum.
2. Consider the decision problem in which Θ = {θ 1 , θ 2 }, A = {a 1 , a 2 , a 3 , a 4 , a 5 }, and the
loss function is given as follows:
L(θ 1 , a 1 ) = 0, L(θ 1 , a 2 ) = 3, L(θ 1 , a 3 ) = 1, L(θ 1 , a 4 ) = 3, L(θ 1 , a 5 ) = 4;
L(θ 2 , a 1 ) = 4, L(θ 2 , a 2 ) = 6, L(θ 2 , a 3 ) = 0, L(θ 2 , a 4 ) = 0, L(θ 2 , a 5 ) = 1.
Consider the prior π under which π(θ 1 ) = 4/5 and π(θ 2 ) = 1/5. Find the Bayes
action(s) under this prior.
Solution: Note that this is a no data decision problem in that we don’t observe X|θ.
Thus, simply find the δ = a i such that E θ [L(θ, a i )] is minimized.
Recall that the posterior risk ρ(π, a i ) = ∑ i L(θ, a i)π(θ i ). Then
ρ(π, a 1 ) = 4/5,
Thus, a 1 and a 3 are Bayes actions.
ρ(π, a 2 ) = 4/5 × 3 + 1/5 × 6 = 18/5,
ρ(π, a 3 ) = 4/5 × 1 = 4/5,
ρ(π, a 4 ) = 4/5 × 3 = 12/5,
ρ(π, a 5 ) = 4/5 × 4 + 1/5 × 1 = 17/5.

3. Suppose that both the parameter space and the action space are the [0, 1] interval and
the loss L(θ, a) = 100(a − θ) 2 . Consider the prior π(θ) = 2θ, 0 ≤ θ ≤ 1.
(a) Show that the Bayes action under the prior π is given by a = 2/3 and that the
corresponding Bayes risk is 50/9.
(b) Next, suppose that one changes the prior to π(θ) = 3θ 2 , 0 ≤ θ ≤ 1, but all else
remains the same. What do you know about the Bayes action under squared
error loss for this prior? What is the Bayes action now?
Solution: Note that this is a no data problem so R(θ, δ) = L(θ, δ). Then
∫
ρ(π, a) = R(θ, δ) π(θ) dθ
= 100
= 200
= 200
∫ 1
0
∫ 1
0
∫ 1
0
(a − θ) 2 2θ dθ
(a − θ) 2 θ dθ
(a 2 − 2θ + θ 2 ) θ dθ
= 200(a 2 /2 − 2a/3 + 1/4).
Then ∂r
∂a = 200(a − 2/3) and ∂2 r
= 200 > 0. Thus, r(π, a) is minimized at a = 2/3.
∂a2 The corresponding Bayes risk is
r(π, 2/3) = 200(2/9 − 4/9 + 1/4) = 200/36 = 50/9.
Since the weight is constant, the Bayes rule is the same as under squared error loss.
Thus, the Bayes action is E(θ) = ∫ 1
0 2θ2 dθ = 2/3.
Under the other prior, the Bayes action is simply ∫ 1
0 3θ3 dθ = 3/4.
4. Suppose that the parameter space Θ = (θ 1 , θ 2 ) and that the action space A = (a 1 , a 2 ).
Also suppose that
L(θ 1 , a 1 ) = L(θ 2 , a 2 ) = 0; L(θ 1 , a 2 ) = 5, L(θ 2 , a 1 ) = 10.
The statistician observes a random variable X such that conditional on θ = θ 1 ,
X ∼ N(0, 1), and conditional on θ = θ 2 , X ∼ N(1, 1). Now consider the prior
under which P (θ = θ 1 ) = η. Show that the Bayes rule under this prior is to take
action a 1 if and only if
x ≤ 1 2 + log η
2(1 − η) .
Solution: Consider
p(θ = θ 1 |x) =
p(x|θ 1 )p(θ = θ 1 )
p(x|θ 1 )p(θ = θ 1 ) + p(x|θ 2 )p(θ = θ 2 )
= 1 − p(θ = θ 2 |x).
2

Now the posterior risk under action a 1 is
L(θ 1 , a 1 ) p(θ = θ 1 |x) + L(θ 2 , a 1 ) p(θ = θ 2 |x) = 10 p(θ = θ 2 |x)
while the posterior risk under action a 2 is
L(θ 1 , a 2 ) p(θ = θ 1 |x) + L(θ 2 , a 2 ) p(θ = θ 2 |x) = 5 p(θ = θ 1 |x).
Hence, a 1 is preferred to a 2 if and only if
This implies
10 p(θ = θ 2 |x) ≤ 5 p(θ = θ 1 |x).
p(x|θ 2 ) p(θ = θ 2 ) ≤ (1/2) p(x|θ 1 ) p(θ = θ 1 ) =⇒
1
(1 − η) √ e −1/2(x−1)2 ≤ η 1
√ e −1/2(x)2 =⇒
2π 2 2π
e x−1/2 η
≤
2(1 − η) =⇒
x ≤ 1/2 + log
η
2(1 − η) .
3