ST5223 ASSESSMENT SHEET 1 SOLUTIONS Question 1 (a) We ...

2 ST5223 ASSESSMENT SOLUTIONS
maximizing the un-normalized posterior, we have that the maximum-aposteriori
estimate is equivalent to the minimization problem:
min
θ∈Rp �
1
2 (Y − Xθ)′ �p−1
�
|θj|
(Y − Xθ) + √ .
τj
j=0
This minimization problem is similar to least squares estimation, except there is an
additional factor
�p−1
j=0
|θj|
√ τj
this penalizes very large (in some sense) values of the parameters and generally
(dependent on the τ0:p−1) encourages shrinking the coefficients towards zero. [5
Marks]
Question 2
(a) Since there is independence across data-points, we can consider a single i ∈
{1, . . . , n}. We have:
p(yi|θ1:k) =
k�
p(yi|zi, θ1:k)P(zi = j) =
j=1
which completes the question. [2 Marks]
k�
f(yi|θj)wj
(b) The main difference of this model against the standard normal regression model
is that it allows each response data to be explained by one of k possible regression
curves. One might prefer to use this model against standard normal regression if
the data are subject to different groups (e.g. male and female) which may lead to
very different regression curves between the groups. [2 Marks]
(c) The joint density is:
p(y1:n, z1:n, θ1:k) =
i=1
j=1
�
�n
ϕ(yi; x ′ �
�k
iθzi , 1)wzi ϕp(θj; µ, Σ).
where ϕp(θj; µ, Σ) is the p−dimensional normal density of mean µ and covariance
matrix Σ.
To obtain the conditional densities, we start with zi. For any i ∈ {1, . . . , n}:
hence
p(zi| · · · ) ∝ ϕ(yi; x ′ iθzi , 1)wzi
j=1
p(zi| · · · ) = ϕ(yi; x ′ θzi i , 1)wzi
�k j=1 ϕ(yi; x ′ iθj, .
1)wj
Now, for j ∈ {1, . . . , k}, we have for θj
�
�n
p(θj| · · · ) ∝ I {j}(zi)ϕ(yi; x ′ �
iθzi , 1)wzi ϕp(θj; µ, Σ).
i=1
If no zi = j then p(θj| · · · ) = ϕp(θj; µ, Σ). Consider the case where at least one
zi = j (write this number nj). Now write Yj as all the concatenated vector of