ST5223 ASSESSMENT SHEET 1 SOLUTIONS Question 1 (a) We ...

ST5223 ASSESSMENT SHEET 1 SOLUTIONS
(a) We have, assuming t ∈ (−λ, λ),
Question 1
MX(t) := E[exp{Xt}]
= λ
� � 0
2
= λ
��
1
2 λ + t ex(λ+t)
�0 = λ
� �
1 1
+
2 λ + t λ − t
=
e
−∞
x(λ+t) dx +
1
[1 − (t 2 /λ 2 )] .
−∞
� ∞
0
e −x(λ−t) �
dx
�
+ − 1
λ − t e−x(λ−t)
�∞� Note that we have used t ∈ (−λ, λ), to ensure that the coefficients of x in the
exponents in the integrands on the second line are positive; this ensures that the
integrals are finite. [3 Marks]
(b) To solve this question we will use moment generating functions. We have
MY (t) = E[exp{Y t}]
= E[E[exp{Y t}|X]]
= E[e t2 X
2 ]
=
� ∞
0
λe −x(λ−t2 /2) dx.
On the third line, we have used that the moment generating function of a N (0, σ 2 )
is exp{t 2 σ 2 /2}. Then for t ∈ (− √ 2λ, √ 2λ), we have
i.e. Y ∼ La( √ 2λ). [2 Marks]
MY (t) =
=
λ
[λ − t2 /2]
1
[1 − (t2 /2λ)]
(c) From part b), we know that the marginal prior for θj is La(1/ √ τj). Hence the
marginal posterior on θ is, up-to proportionality:
exp{− 1
2 (Y − Xθ)′ �
(Y − Xθ)} exp
�p−1
−
j=0
|θj|
�
√ .
τj
As maximizing the un-normalized posterior (w.r.t. θ) is the same as maximizing
the posterior and minimzing the minus log-unormalized posterior is the same as
1
0

2 ST5223 ASSESSMENT SOLUTIONS
maximizing the un-normalized posterior, we have that the maximum-aposteriori
estimate is equivalent to the minimization problem:
min
θ∈Rp �
1
2 (Y − Xθ)′ �p−1
�
|θj|
(Y − Xθ) + √ .
τj
j=0
This minimization problem is similar to least squares estimation, except there is an
additional factor
�p−1
j=0
|θj|
√ τj
this penalizes very large (in some sense) values of the parameters and generally
(dependent on the τ0:p−1) encourages shrinking the coefficients towards zero. [5
Marks]
Question 2
(a) Since there is independence across data-points, we can consider a single i ∈
{1, . . . , n}. We have:
p(yi|θ1:k) =
k�
p(yi|zi, θ1:k)P(zi = j) =
j=1
which completes the question. [2 Marks]
k�
f(yi|θj)wj
(b) The main difference of this model against the standard normal regression model
is that it allows each response data to be explained by one of k possible regression
curves. One might prefer to use this model against standard normal regression if
the data are subject to different groups (e.g. male and female) which may lead to
very different regression curves between the groups. [2 Marks]
(c) The joint density is:
p(y1:n, z1:n, θ1:k) =
i=1
j=1
�
�n
ϕ(yi; x ′ �
�k
iθzi , 1)wzi ϕp(θj; µ, Σ).
where ϕp(θj; µ, Σ) is the p−dimensional normal density of mean µ and covariance
matrix Σ.
To obtain the conditional densities, we start with zi. For any i ∈ {1, . . . , n}:
hence
p(zi| · · · ) ∝ ϕ(yi; x ′ iθzi , 1)wzi
j=1
p(zi| · · · ) = ϕ(yi; x ′ θzi i , 1)wzi
�k j=1 ϕ(yi; x ′ iθj, .
1)wj
Now, for j ∈ {1, . . . , k}, we have for θj
�
�n
p(θj| · · · ) ∝ I {j}(zi)ϕ(yi; x ′ �
iθzi , 1)wzi ϕp(θj; µ, Σ).
i=1
If no zi = j then p(θj| · · · ) = ϕp(θj; µ, Σ). Consider the case where at least one
zi = j (write this number nj). Now write Yj as all the concatenated vector of

ST5223 ASSESSMENT SOLUTIONS 3
response variables with zi = j and write Xj as the associated design matrix. Then
we have
p(θj| · · · ) ∝ ϕnj (Yj; X ′ jθj, Inj×nj )ϕp(θj; µ, Σ).
Recalling from problem sheet 1 that:
(Yj − Xjθj) ′ (Yj − Xjθj) + (θj − µ) ′ Σ −1 (θj − µ) = (θj − µ ∗ j ) ′ Σ ∗−1
j (θj − µ ∗ j ) + b ∗
where
µ ∗ j = Σ ∗ j (Σ ∗−1
j µj + X ′ jYj)
Σ ∗ j = (Σ −1
j + X′ jXj)
and b∗ is a constant that doesn’t depend upon θ. Hence we have that
θj| · · · ∼ Np(µ ∗ j , Σ ∗ j ).
Thus a Gibbs sampler, for a single iteration (to move from z1:n, θ1:k to z ′ 1:n, θ ′ 1:k ) is:
• Sample p(z ′ 1|z2:n, θ1:k, y1:n), p(z ′ 2|z ′ 1, z3:n, θ1:k, y1:n), . . . , p(z ′ n|z ′ 1:n−1, θ1:k, y1:n).
• Sample p(θ ′ 1|z ′ 1:n, θ2:k, y1:n), p(θ ′ 2|z ′ 1:n, θ ′ 1, θ2:k, y1:n), . . . , p(θ ′ k |z′ 1:n, θ ′ 1:k−1 , y1:n).
[6 Marks]