From Algorithms to Z-Scores - matloff - University of California, Davis

15.12. PARAMETRIC ESTIMATION OF LINEAR REGRESSION FUNCTIONS 301
Cov( ˆ β) = Cov(BQ ′ V ) ((15.30)) (15.38)
= BQ ′ Cov(V )(BQ ′ ) ′ (7.50) (15.39)
= BQ ′ σ 2 I(BQ ′ ) ′ (15.37) (15.40)
= σ 2 BQ ′ QB (lin. alg.) (15.41)
= σ 2 (Q ′ Q) −1 (def. of B) (15.42)
Whew! That’s a lot of work for you, if your linear algebra is rusty. But it’s worth it, because
(15.42) now gives us what we need for confidence intervals. Here’s how:
First, we need to estimate σ 2 . Recall first that for any random variable U, V ar(U) = E[(U −EU) 2 ],
we have
σ 2 = V ar(Y |X = t) (15.43)
= V ar(Y |X (1) = t1, ..., X (r) = tr) (15.44)
= E {Y − mY ;X(t)} 2
(15.45)
= E (Y − β0 − β1t1 − ... − βrtr) 2
(15.46)
Thus, a natural estimate for σ 2 would be the sample analog, where we replace E() by averaging
over our sample, and replace population quantities by sample estimates:
s 2 = 1
n
n
i=1
(Yi − ˆ β0 − ˆ β1X (1)
i − ... − ˆ βrX (r)
i ) 2
(15.47)
As in Chapter 12, this estimate of σ 2 is biased, and classicly one divides by n-(r+1) instead of n.
But again, it’s not an issue unless r+1 is a substantial fraction of n, in which case you are overfitting
and shouldn’t be using a model with so large a value of r.
So, the estimated covariance matrix for ˆ β is
Cov( ˆ β) = s 2 (Q ′ Q) −1
(15.48)
The diagonal elements here are the squared standard errors (recall that the standard error of an
estimator is its estimated standard deviation) of the βi. (And the off-diagonal elements are the
estimated covariances between the βi.) Since the first standard errors you ever saw, in Section 10.5,
included factors like 1/ √ n, you might wonder why you don’t see such a factor in (15.48).