Applied Statistics Using SPSS, STATISTICA, MATLAB and R

7.1.2 Estimating the Regression Function
7.1 Simple Linear Regression 273
A popular method of estimating the regression function parameters is to use a least
square error (LSE) approach, by minimising the total sum of the squares of the
errors (deviations) between the observed values yi and the estimated values
b0 + b1xi:
E =
n
∑
i=
1
2
i
n
∑
ε = ( y − b − b x ) . 7.2
i=
1
i
0
1
i
2
where b0 and b1 are estimates of β0 and β1, respectively.
In order to apply the LSE method one starts by differentiating E in order to b0
and b1 and equalising to zero, obtaining the so-called normal equations:
∑ ∑
⎪⎧
yi
= nb0
+ b1
xi
⎨
⎪⎩ ∑xiyi= b0
∑ xi
+ b1∑x
2
i
, 7.3
where the summations, from now on, are always assumed to be for the n predictor
values. By solving the normal equations, the following parameter estimates, b0 and
b1, are derived:
b
1
=
∑ ( xi
∑
b0 1
− x)(
yi
− y)
. 7.4
2
( x − x)
i
= y − b x . 7.5
The least square estimates of the linear regression parameters enjoy a number of
desirable properties:
i. The parameters b0 and b1 are unbiased estimates of the true parameters β0
and β1 ( E[ b 0 ] = β 0 , E[
b1
] = β1
), and have minimum variance among all
unbiased linear estimates.
ii. The predicted (or fitted) values yˆ i = b0
+ b1x
i are point estimates of the true,
observed values, yi. The same is valid for the whole relationYˆ = b0
+ b1
X ,
which is the point estimate of the mean response E[Y ].
iii. The regression line always goes through the point ( x , y ).
iv. The computed errors ei = yi
− yˆ
i = yi
− b0
− b1x
i , called the residuals, are
point estimates of the error values εi. The sum of the residuals is zero:
∑ e i = 0 .
v. The residuals are uncorrelated with the predictor and the predicted values:
∑ e i xi
= 0; ∑ ei
yˆ
i = 0 .
vi. ∑ yi =∑ yˆ
i ⇒ y = yˆ
, i.e., the predicted values have the same mean as
the observed values.