Answers to Homework #5 - Statistics

Homework #5 Solutions
1. (a) No; the one-vector is not in the column space of the design matrix. (b) Yes, the
one-vector is in the column space of the design matrix. (c) Yes (d) Yes (e) No
2. We have
X ′ X =
(
5 0
0 50
)
; and (X ′ X) −1 =
(
1/5 0
0 1/50
We can compute the projection matrix using H = X(X ′ X) −1 X ′ , to get
⎛
H =
⎜
⎝
19/50 16/50 13/50 10/50 −8/50
16/50 14/50 12/50 10/50 −2/50
13/50 13/50 11/50 4/50 10/50
10/50 10/50 10/50 10/50 10/50
−8/50 −2/50 4/50 10/50 46/50
The covariance matrix for the residual vector is (I − H)σ 2 , so the variance for each
individual residual can be taken from the diagonal of the hat matrix: var(e 1 ) =
31/50σ 2 ; var(e 2 ) = 36/50σ 2 ; var(e 3 ) = 39/50σ 2 ; var(e 4 ) = 40/50σ 2 ; and var(e 5 ) =
14/50σ 2 . The 5th residual has substanially smaller variance than the others.
The leverages can be taken from the hat matrix as well: 19/50, 14/50, 11/50, 10/50,
and 46/50. Note that the 5th observation has a leverage more than twice as big as
any of the others.
Now, we find the actual residual vector given the data. We get X ′ y = (36, 25) ′ , and
ˆβ = (7.2, 0.5)’. The residual vector is e = (4.3, −0.2, −1.7, .4.2, 1.8) ′ . The leaveone-out
residuals are: e (−1) = 6.94, e (−2) = −0.78, e (−3) = −2.18, e (−4) = −5.25,
and e (−5) = 22.5.
The 5th observation has a large leverage and is very influential. If we fit a line to
the first four observations, we get an intercept estimate of 2.7 and a slope estimate
of -2.2. When we plug in x = 6, the predicted value of y is -10.5. Note that e (−5) is
the difference between the observed value at x 5 and the predicted value at x 5 , with
that observation not used in the prediction.
3. The 95th percentile of the simulated distribution is about 7, and the test statistic is
only 2.57, so we do not reject the null hypothesis. There is no evidence of violation
of the equal variances assumption.
Here is the R code I used:
⎞
⎟
⎠
)
.

# code to simulate the null distribution of Fmax when the
# sample sizes are 12, 8, 10, 10, 14.
svec=1:5
nloop=100000
fmax=1:nloop*0
nj=c(12,8,10,10,14)
for(i in 1:nloop){
for(j in 1:5){
x=rnorm(nj[j])
svec[j]=var(x)
}
fmax[i]=max(svec)/min(svec)
}
0 10 20 30 40
4. The fit using two parabolas with vertex at the origin is shown in (a), with the
residuals in (b). The hypothesis test gives p = .044, which is significant at α = .05
but just barely. Because the residual variance seems to be growing in time, and
because we expect the rust measurements to be more variable if the metal sits
longer, we decide to use a weighted model. If we assume that the variance of y is
proportional to x, the fit is shown in (c). It is not noticeably different than the
unweighted fit; however, the p-value is now p = .074. The residuals look better, but
now let’s assume that the variance of y is proportional to x 2 . The fit again looks
very similar, but now we get p = .141.
(a)
res
-5 0 5
(b)
y
0 10 20 30 40
(c)
restr
-4 -2 0 2 4
(d)
restr
-2 -1 0 1 2
(e)
1.0 1.5 2.0 2.5 3.0 3.5 4.0
1.0 1.5 2.0 2.5 3.0 3.5 4.0
1.0 1.5 2.0 2.5 3.0 3.5 4.0
1.0 1.5 2.0 2.5 3.0 3.5 4.0
1.0 1.5 2.0 2.5 3.0 3.5 4.0
To explain why the results are not significant after we weight the analysis, we point
out to the scientists that most of the difference in the data is seen at month 4, where
the paint A measurements (green dots) are smaller than the paint B measurements.
If we don’t account for the fact that the variance is also higher here, these differences
seem more significant. Notice that for the first month’s measurements, the means
are about the same. When we weight these values more, the differences in paint are
not significant.

Here is the R code for the analysis and the plots. The variable x is the time (months),
a is the indicator for paint A, and y is the measurement of the rust.
plot(x-1/20+a/10,y,col=a+2)
title("(a)")
# try the IID model:
x1=x^2;x2=x^2*a
xmat=cbind(x1,x2)
xpxinv=solve(t(xmat)%*%xmat)
betahat=xpxinv%*%t(xmat)%*%y
yhat=xmat%*%betahat
sse=sum((y-yhat)^2)
tstat=betahat[2]/sqrt(sse/38*xpxinv[2,2])
# two-sided p-value
pval=(1-pt(abs(tstat),38))*2
lines(8:42/10,betahat[1]*(8:42/10)^2,col=2)
lines(8:42/10,sum(betahat)*(8:42/10)^2,col=3)
# Plot the residuals and notice "fanning out"
res=y-xmat%*%betahat
plot(x,res)
lines(c(.6,4.4),c(0,0),lty=3)
title("(b)")
# Now assume variance is proportional to x;
# get t-stat in transformed model
ytr=y/sqrt(x)
x1tr=x^2/sqrt(x)
x2tr=x^2*a/sqrt(x)
xmattr=cbind(x1tr,x2tr)
xpxinvtr=solve(t(xmattr)%*%xmattr)
betatr=xpxinvtr%*%t(xmattr)%*%ytr
yhattr=xmattr%*%betatr
ssetr=sum((ytr-yhattr)^2)
tstattr=betatr[2]/sqrt(xpxinvtr[2,2]*ssetr/38)

# two-sided p-value
pvaltr=(1-pt(abs(tstattr),38))*2
plot(x-1/20+a/10,y,col=a+2)
lines(8:42/10,betatr[1]*(8:42/10)^2,col=2)
lines(8:42/10,sum(betatr)*(8:42/10)^2,col=3)
title("(c)")
# plot residuals again -- in transformed model
restr=ytr-xmattr%*%betatr
plot(x,restr)
lines(c(.6,4.4),c(0,0),lty=3)
title("(d)")
# Now assume variance is proportional to x^2;
# get t-stat in transformed model
ytr=y/x
x1tr=x^2/x
x2tr=x^2*a/x
xmattr=cbind(x1tr,x2tr)
xpxinvtr=solve(t(xmattr)%*%xmattr)
betatr=xpxinvtr%*%t(xmattr)%*%ytr
yhattr=xmattr%*%betatr
ssetr=sum((ytr-yhattr)^2)
tstattr=betatr[2]/sqrt(xpxinvtr[2,2]*ssetr/38)
# two-sided p-value
pvaltr=(1-pt(abs(tstattr),38))*2
# plot residuals again -- in transformed model
restr=ytr-xmattr%*%betatr
plot(x,restr)
lines(c(.6,4.4),c(0,0),lty=3)
title("(e)")
5. (a) We can assume that the water evaporates linearly with time. Let y i be the
amount of evaporation for the one-inch beaker on the ith day, and let y 5+i be
the amount of evaporation for the two-inch beaker removed on the ith day, for
i = 1, . . . , 5. Let β 1 be the evaporation rate (ml per day) from the one-inch beakers,

and let β 2 be the evaporation rate (ml per day) from the two-inch beakers. Then
we can write y = Xβ + ɛ, where
⎛ ⎞
1 0
2 0
3 0
4 0
X =
5 0
0 1
.
0 2
0 3
⎜ ⎟
⎝ 0 4 ⎠
0 5
(b) The null hypothesis is H 0 : β 2 = 2β 1 , and the alternative is β 2 ≠ 2β 1 . We can
use c = (−2, 1) and write H 0 : c ′ β = 0. The test concerning linear combinations of
parameters tells us:
T =
c ′ ˆβ
√
c ′ (X ′ X) −1 cSSE/(n − k)
has a t(n−k) density under H 0 . Plugging things in to simplify, we get c ′ (X ′ X) −1 c =
1/11, n − k = 8, and
T = [y 6 + y 7 + y 8 + y 9 + y 10 − 2(y 1 + y 2 + y 3 + y 4 + y 5 )]/55
√
.
SSE/88
6. Let β 0 be the level of white blood cells before the injection, and let β 1 be the rate of
change (per hour) after the injection, and ˆβ = (β 0 , β 1 ). If x is the number of hours
after 9:00, then we have the design matrix
and the model is y = Xβ + ɛ.
⎛
X =
⎜
⎝
1 0
1 0
1 0
1 0
1 1
1 2
1 3
1 4
⎞
⎟
⎠

7. If β = (β 1 , β 2 , β 3 , β 4 ), then we can make
⎛
⎞
1 0 1 0
1 0 1 0
1 0 0 1
1 0 0 1
X =
.
0 1 0 1
0 1 0 1
⎜
⎟
⎝ 0 1 1 0 ⎠
0 1 1 0
(b) There are three dimensions in the row space. We can use rows to find estimable
linear combinations! Two are: β 1 +β 3 , and β 1 +β 4 . We can not estimate β 1 because
(1, 0, 0, 0) is not in the row space. Similarly, we can’t estimate β 1 + β 2 .
(c) We can simply use the first three dummies in the model, so that ˜β = (β 1 , β 2 , β 3 ),
and
⎛ ⎞
1 0 1
1 0 1
1 0 0
1 0 0
˜X =
.
0 1 0
0 1 0
⎜ ⎟
⎝ 0 1 1 ⎠
0 1 1
(d) β 1 is the expected tumor size after two weeks of Treatment A, low dose; β 2 is the
expected tumor size after two weeks of Treatment B, low dose; β 3 is the expected
increase in tumor size after two weeks of treatment using high dose compared to
low dose, for either A or B.
8. Let x = (1, 2, 3, 2, 3, 4) ′ and d a = (1, 1, 1, 0, 0, 0) and 1 = (1, 1, 1, 1, 1, 1). Then we
want to project the response y onto the linear space L(1, d a , x). We know that we
will have “confounding” because the pot B plants are given more fertilizer. Let’s
compute the VIF for fertilizer. To do this, we get the R 2 for the model using x
as the response and d a (and 1) as predictors. We get ˆx = (2, 2, 2, 3, 3, 3) ′ , so the
SSE = 4. Also, SST = 5.5, and R 2 = 3/11. So V = 11/8.
9. Note that the design matrix is not full rank; we expect to get a zero eigenvalue. We
solve
⎛
⎞
4 − λ 2 2
⎜
⎟
det ⎝ 2 2 − λ 0 ⎠ = λ(λ − 2)(λ − 6).
2 0 2 − λ

to get ordered eigenvalues 6, 2, and 0. To get the eigenvector associated with λ = 6,
we solve
⎛ ⎞ ⎛ ⎞ ⎛ ⎞
4 2 2 x 1 x 1
⎜ ⎟ ⎜ ⎟ ⎜ ⎟
⎝ 2 2 0 ⎠ ⎝ x 2 ⎠ = 6 ⎝ x 2 ⎠ .
2 0 2 x 3 x 3
We get v 1 = (2/ √ 6, 1/ √ 6, 1/ √ 6) ′ . Similarly, v 2 = (0, 1/ √ 2, −1/ √ 2) ′ , and v 3 =
(1/ √ 3, −1/ √ 3, −1/ √ 3) ′ .
We have chosen length one for all of the eigenvectors; also note that they form an
orthogonal set. The spectral decomposition is:
⎛
⎜
⎝
4 2 2
2 2 0
2 0 2
⎞
⎟
⎠ =
⎛
⎜
⎝
2/ √ 6 0 1/ √ 3
1/ √ 6 1/ √ 2 −1/ √ 3
1/ √ 6 −1/ √ 2 −1/ √ 3
⎞ ⎛
⎟ ⎜
⎠ ⎝
6 0 0
0 2 0
0 0 0
⎞ ⎛
⎟ ⎜
⎠ ⎝
2/ √ 6 1/ √ 6 1/ √ 6
0 1/ √ 2 −1/ √ 2
1/ √ 3 −1/ √ 3 −1/ √ 3
⎞
⎟
⎠ .