ST217: Mathematical Statistics B - Of the Clux

MSB
ST217: Mathematical Statistics B
Aim
To review, expand & apply the ideas from MSA.
In particular, MSA mainly studied one unknown quantity at once. In MSB we’ll study interrelationships.
Lectures & Classes
Monday 12–1 R0.21
Wednesday 10–11 R0.21
Friday 3–4 ACCR
Examples classes will begin in week 3.
Contents
1. Overview of MSA.
2. Bivariate & Multivariate Probability Distributions.
Joint distributions, conditional distributions, marginal distributions; conditional expectation. The
χ 2 , t, F and multivariate Normal distributions and their interrelationships.
3. Inference for Multiparameter Models.
Likelihood, frequentist and Bayesian inference, prediction and decision-making. Comparison between
various approaches. Point and interval estimation. Classical simple and composite hypothesis testing,
likelihood ratio tests, asymptotic results.
4. Linear Statistical Models.
Linear regression, multiple regression & analysis of variance models. Model choice, model checking
and residuals.
5. Further Topics (time permitting).
Nonlinear models, problems & paradoxes, etc.
Books
The books recommended for MSA are also useful for MSB. Excellent books on mathematical statistics are:
1. ‘Statistical Inference’ by George Casella & Roger L. Berger [C&B], Duxbury Press (2001),
2. ‘Probability and Statistics’ by Morris DeGroot, Addison-Wesley (2 nd edition 1989; 3 rd edition by
DeGroot & Schervish, 2002).
A good book discussing the application and interpretation of statistical methods is ‘Introduction to the
Practice of Statistics’ by Moore & McCabe [M&M], Freeman (4 th edition 2002). Many of the data sets
considered below come from the ‘Handbook of Small Data Sets’ [HSDS] by Hand et al., Chapman & Hall,
London (1994).
There are many other useful references on mathematical statistics available in the library, including books
by Hogg & Craig [H&C], Lindgren, Mood, Graybill & Boes [MG&B], and Rice.
1

Chapter 1
Overview of MSA
1.1 Basic Ideas
1.1.1 What is ‘Statistics’
Statistics may be defined as:
‘The study of how information should be employed to reflect on, and give guidance for action
in, a practical situation involving uncertainty.’ [italics by JEHS]
Vic Barnett, Comparative Statistical Inference
Figure 1.1: A practical situation involving uncertainty
2

1.1.2 Statistical Modelling
The emphasis of modern statistics is on modelling the patterns and interrelationships in the existing data,
and then applying the chosen model(s) to predict future data.
Typically there is a measurable response (for example, reduction Y in patient’s blood pressure) that is
thought to be related to explanatory variables x j (for example, treatment applied, dose, patient’s age,
weight, etc.) We seek a formula that relates the observed responses to the corresponding explanatory
variables, and that can be used to predict future responses in terms of their corresponding explanatory
variables:
Observed Response = Fitted Value + Residual,
Future Response = Predicted value + Error.
Here the fitted values should take account of all the consistent patterns in the data, and the residuals
represent the remaining random variation.
1.1.3 Prediction and Decision-Making
Always remember that the main aim in modelling as above is to predict (for example) the effects of different
medical treatments, and hence to decide which treatment to use, and in what circumstances.
The fundamental assumption is that the future data will be in some sense similar to existing data. The
ideas of exchangeability and conditional independence are crucial.
The following notation is useful:
X ⊥ Y ‘X is independent of Y ’, i.e. Y gives you no information about X,
X ⊥ Y |Z
‘X is conditionally independent of Y given Z’, i.e. if you know the value taken by the
RV Z, then Y gives you no further information about X.
Most methods of statistical inference proceed indirectly from what we know (the observed data and any
other relevant information) to what we really want to know (future, as yet unobserved, data), by assuming
that the random variation in the observed data can be thought of as a sample from an underlying population,
and learning about the properties of this population.
1.1.4 Known and Unknown Features of a Statistical Problem
A statistic is a property of a sample, whereas a parameter is a property of a population. Often it’s natural
to estimate a parameter θ (such as the population mean µ) by the corresponding property of the sample
(here the sample mean X). Note that θ may be a vector or more complicated object.
Unobserved quantities are treated mathematically as random variables. Potentially observable quantities
are usually denoted by capital letters (X i , X, Y etc.) Once the data have been observed, the values taken
by these random variables are known (X i = x i , X = x etc.) Unobservable or hypothetical quantities are
usually denoted by Greek letters (θ, µ, σ 2 etc.), and estimators are often denoted by putting a hat on the
corresponding symbol (̂θ, ̂µ, ̂σ 2 etc.)
Nearly all statistics books use the above style of notation, so it will be adopted in these notes. However,
sometimes I shall wish to distinguish carefully between knowns and unknowns, and shall denote all
unknowns by capitals. Thus Θ represents an unknown parameter vector, and θ represents a particular
assumed value of Θ. This is especially useful when considering probability distributions for parameters;
one can then write f Θ (θ) and P(Θ = θ) (the probability that Θ = θ) by exact analogy with f X (x) and
P(X = x).
The set of possible values for a RV X is called its sample space Ω X . Similarly the parameter space Ω Θ is
the set of possible values for the parameter Θ.
3

1.1.5 Likelihood
In general, we can infer properties θ of the population by comparing how compatible are the various possible
values of θ with the observed data. This motivates the idea of likelihood (equivalently, log-likelihood or
support). We need a probability model for the data, in which the probability distribution of the random
variation is a member of a (realistic but mathematically tractable) family of probability distributions,
indexed by a parameter θ.
Likelihood-based approaches have both advantages and disadvantages—
Advantages
Unified theory (many practical problems can be
tackled in essentially the same way).
Often get simple sufficient statistics (hence we can
summarise a huge data set by a few simple properties).
CLT suggests likelihood methods work well when
there’s loads of data.
Disadvantages
Is the theory directly relevant (is likelihood alone
enough and how do we balance realism and
tractability)
If the probability model is wrong, then results can
be misleading (e.g. if one assumes a Normal distribution
when the true distribution is Cauchy).
One seldom has loads of data!
1.1.6 Where Will We Go from Here
• MSA provided the mathematical toolbox (e.g. probability theory and the idea of random variables)
for studying random variation.
• MSB will add to this toolbox and study interrelationships between (random) variables.
• We shall also consider some important general forms for the fitted/predicted values, in particular
linear models and their generalizations.
1.2 Sampling Distributions
Statistical analysis involves calculating various statistics from the data, for example the maximum likelihood
estimator (MLE) ̂θ for θ. We want to understand the properties of these statistics; hence the importance
of the central limit theorem (CLT) & its generalizations, and of studying the probability distributions of
transformed random variables.
If we have a formula for a summary statistic S, e.g. S = ∑ X i /n = X, and are prepared to make certain
assumptions about the original random variables X i , then we can say things about the probability
distribution of S.
The probability distribution of a statistic S, i.e. the pattern of values S would take if it were calculated in
successive samples similar to the one we actually have, is called its sampling distribution.
4

1.2.1 Typical Assumptions
1. Standard Assumption (IID RVs):
X i are IID (independent and identically distributed) with (unknown) mean µ and variance σ 2 .
This implies
(a) E[X] = E[X i ] = µ, and
(b) Var[X] = 1 n Var[X i] = σ 2 /n.
(c) If we define the standardised random variables
Z n = X − µ ,
σ
then as n → ∞, the distribution of Z n tends to the standard Normal N(0, 1) distribution.
2. Additional Assumption (Normality):
IID
The X i are IID Normal: X i ∼ N (µ, σ 2 ).
This implies that X ∼ N(µ, σ 2 /n).
1.2.2 Further Uses of Sampling Distributions
We can also
• compare various plausible estimators (e.g. to estimate the centre of symmetry of a supposedly symmetric
distribution we might use the sample mean, median, or something more exotic),
• obtain interval estimates for unknown quantities (e.g. 95% confidence intervals, HPD intervals, support
intervals),
• test hypotheses about unknown quantities.
Comments
1. Note the importance of expectations of (possibly transformed) random variables:
E[X] = µ (measure of location)
E[(X − µ) 2 ] = σ 2 (measure of scale)
E[e sX ] = moment generating function
E[e itX ] = characteristic function
2. We must always consider whether the assumptions made are reasonable, both from general considerations
(e.g.: is independence reasonable is the assumption of identical distributions reasonable
is it reasonable to assume that the data follow a Poisson distribution etc.) and with reference to
the observed set of data (e.g. are there any ‘outliers’—unreasonably extreme values—or unexpected
patterns)
3. Likelihood and other methods suggest estimators for unknown quantities of interest (parameters etc.)
under certain specified assumptions.
Even if these assumptions are invalid (and in practice they always will be to some extent!) we may still
want to use summary statistics as estimators of properties of the underlying population. Therefore
(a) We’ll want to investigate the properties of estimators under various relaxed assumptions, for
example partially specified models that use only the first and second moments of the unknown
quantities.
(b) It’s useful if the calculated statistics (e.g. MLEs) have an intuitive interpretation (like ‘sample
mean’ or ‘sample variance’).
5

1.3 (Revision) Problems
1. First-year students attending a statistics course were asked to carry out the following procedure:
Toss two coins, without showing anyone else the results.
If the first coin showed ‘Heads’ then answer the following question:
“Did the second coin show ‘Heads’ (Yes or No)”
If the first coin showed ‘Tails’ then answer the following question:
“Have you ever watched a complete episode of ‘Teletubbies’ (Yes or No)”
The following results were recorded:
Yes No
Males 84 48
Females 23 24
For each sex, and for both sexes combined, estimate the proportion who have watched a complete
episode of ‘Teletubbies’.
Using a chi-squared test, or otherwise, test whether the proportions differ between the sexes.
Discuss the assumptions you have made in carrying out your analysis.
2. Let X and Y be IID RVs with a standard Normal N(0, 1) distribution, and define Z = X/Y .
(a) Write down the lower quartile, median and upper quartile of Z, i.e. the points z 25 , z 50 & z 75
such that P(Z < z k ) = k/100.
(b) Show that Z has a Cauchy distribution, with PDF 1/π(z 2 + 1).
HINT : consider the transformation Z = X/Y and W = |Y |.
3. Let X 1 , . . . X n be mutually independent RVs, with respective MGFs (moment generating functions)
M X1 (t), . . . , M Xn (t), and let a 1 , . . . , a n and b 1 , . . . , b n be fixed constants.
Show that the MGF of Z = (a 1 X 1 + b 1 ) + (a 2 X 2 + b 2 ) + · · · + (a n X n + b n ) is
M Z (t) = exp ( t ∑ b i
)
MX1 (a 1 t) × · · · × M Xn (a n t).
Hence or otherwise show that any linear combination of independent Normal RVs is itself Normally
distributed.
4. A workman has to move a rectangular stone block a short distance, but doesn’t want to strain himself.
He rapidly estimates:
• height of block = 10 cm, with standard deviation 1 cm.
• width of block = 20 cm, with standard deviation 3 cm.
• length of block = 25 cm, with standard deviation 4 cm.
• density of block = 4.0 g/cc, with standard deviation 0.5 g/cc.
Assuming these estimates are mutually independent, calculate his estimates of the volume V (cc)
and total weight W (Kg) of the block, and their standard deviations.
The workman fears that he might hurt his back if W ≥ 30.
Using Chebyshev’s inequality, give an upper bound for his probability P(W ≥ 30).
[Chebyshev’s inequality states that if X has mean µ & variance σ 2 , then P(|X − µ| ≥ c) ≤
σ 2 /c 2 —see MSA].
What is the workman’s value for P(W > 30) under the additional assumption that W is Normally
distributed Compare this value with the bound found earlier.
How reasonable are the independence and Normality assumptions used in the above analysis
6

5. Calculate the MLE of the centre of symmetry θ, given IID RVs X 1 , X 2 , . . . , X n , where the common
PDF f X (x) of the X i s is
(a) Normal (or Gaussian):
f X (x|θ, σ) = √ 1 ( ( ) )
exp − 1 2
2πσ
2 (x − θ)/σ
(b) Laplacian (or Double Exponential):
f X (x|θ, σ) = 1
2σ exp( |x − θ|/σ )
(c) Uniform (or Rectangular):
{ ( ) ( )
1 if θ −
1
f X (x|θ) =
2 < x < θ +
1
2
0 otherwise.
Do you consider these MLEs to be intuitively reasonable
6. Calculate E[X], E[X 2 ], E[X 3 ] and E[X 4 ] under each of the following assumptions:
(a) X ∼ Poi(λ), i.e. X has PMF (probability mass function)
P(X = x|λ) = λx exp(−λ)
x!
(x = 0, 1, 2, . . . )
(b) X ∼ Exp(β), i.e. X has PDF (probability density function)
{
βe
−βx
if x > 0
f X (x|β) =
0 otherwise.
(c) X ∼ N ( µ, σ 2) , i.e. X has PDF
f X (x|µ, σ) = √ 1 ( ( ) )
exp − 1 2
2πσ
2 (x − µ)/σ
7. Describe briefly how, and under what circumstances, you might approximate
(a) a binomial distribution by a Normal distribution,
(b) a binomial distribution by a Poisson distribution,
(c) a Poisson distribution by a Normal distribution.
Suppose X ∼ Bin(100, 0.1), Y ∼ Poi(10), and Z ∼ N ( 10, 3 2) . Calculate, or look up in tables,
(i) P(X ≥ 6), (ii) P(Y ≥ 6), (iii) P(Z > 5.5),
(iv) P(X > 16), (v) P(Y > 16), (vi) P(Z > 16.5),
and comment on the accuracy of the approximations here.
8. The t distribution with n degrees of freedom, denoted t n or t(n), has the PDF
f(t) = Γ( 1
2
(n + 1))
Γ ( 1 1
1
2 n) √ nπ
(
1 + t2 /n ) , −∞ < t < ∞,
(n+1)/2
and the F distribution with m and n degrees of freedom, denoted F m,n or F (m, n), has PDF
with f(x) = 0 for x ≤ 0.
f(x) = Γ( 1
2
(m + n))
Γ ( 1
2 m) Γ ( 1
2 n) mm/2 n n/2 x (m/2)−1
, 0 < x < ∞,
(mx + n)
(m+n)/2
Show that if T ∼ t n and X ∼ F m,n , then T 2 and X −1 both have F distributions.
7

9. Table 1.1 shows the estimated total resident population (thousands) of England and Wales at
30 June 1993:
Age Persons Males Females
< 1 669.6 343.1 326.5
1–14 9,268.0 4,756.9 4,511.1
15–44 21,875.0 11,115.6 10,759.4
45–64 11,435.8 5,676.6 5,759.2
65–74 4,595.9 2,081.7 2,514.2
≥ 75 3,594.9 1,224.5 2,370.4
Total 51,439.2 25,198.4 26,240.8
Table 1.1: Estimated resident population of England & Wales, mid 1993, by sex and
age-group (simplified from Table 1 of the 1993 mortality tables )
Table 1.2, also extracted from the published 1993 Mortality Statistics, shows the number of deaths
in 1993 among the resident population of England and Wales, categorised by sex, age-group and
underlying cause of death.
Assume that the rates observed in Tables 1.1 and 1.2 hold exactly, and suppose that an individual
I is chosen at random from the population. Define the random variables S (sex), A (age group), D
(death) and C (cause) as follows:
For example,
S = 0 if I is male,
1 if I is female,
A = 1 if I is under 1 year old, 4 if I is aged 45–64,
2 if I is aged 1–14, 5 if I is aged 65–74,
3 if I is aged 15–44, 6 if I is 75 years old or over,
D = 0 if I survives the year,
1 if I dies,
C = cause of death (0–17).
P(S=0) = 25198.4/51439.2,
P(S=0 & A=6) = 1224.5/51439.2,
P(D=0|S=0 & A=6) = 1 − 138.239/1224.5,
P(C=8|S=0 & A=6) = 28.645/1224.5, etc.
(a) Calculate P(D=1|S=0), and P(D=1|S=0 & A=a) for a = 1, 2, 3, 4, 5, 6.
Also calculate P(S=0|D=1), and P(S=0|D=1 & A=a) for a = 1, 2, 3, 4, 5, 6.
If you were an actuary, and were asked by a non-expert “is the death rate for males higher or
lower than that for females”, how would you respond based on the above calculations Justify
your answer.
(b) Similarly, explain how you would respond to the questions
i. “is the death rate from neoplasms higher for males or for females”
ii. “is the death rate from mental disorders higher for males or for females”
iii. “is the death rate from diseases of the circulatory system higher for males or for females”
iv. “is the death rate from diseases of the respiratory system higher for males or for females”
8

All
Age at death (years)
Cause of death Sex ages < 1 1–14 15–44 45–64 65–74 ≥ 75
0 Deaths below 28 days M 1,603 1,603 − − − − −
(no cause specified) F 1,192 1,192 − − − − −
1 Infectious & parasitic M 1,954 60 79 565 390 346 514
diseases F 1,452 46 44 169 193 283 717
2 Neoplasms M 74,480 16 195 2,000 16,372 25,644 30,253
F 67,966 8 138 2,551 15,026 19,141 31,102
3 Endocrine, nutritional M 3,515 28 43 208 639 959 1,638
& metabolic diseases and F 4,403 17 37 153 474 901 2,821
immunity disorders
4 Diseases of blood and M 897 5 12 62 106 204 508
blood-forming organs F 1,084 3 14 28 73 163 803
5 Mental disorders M 2,530 − 8 281 169 334 1,738
F 5,189 − 1 83 99 297 4,709
6 Diseases of the nervous M 4,403 59 136 530 675 890 2,113
system and sense organs F 4,717 42 118 313 546 809 2,889
7 Diseases of the M 123,717 41 66 1,997 20,682 37,195 63,736
circulatory system F 134,439 44 45 834 7,783 23,185 102,548
8 Diseases of the M 41,802 86 79 608 3,157 9,227 28,645
respiratory system F 49,068 59 74 322 2,145 6,602 39,866
9 Diseases of the M 7,848 10 27 511 1,706 2,058 3,536
digestive system F 10,574 20 14 298 1,193 1,921 7,128
10 Diseases of the M 3,008 4 6 57 215 676 2,050
genitourinary system F 3,710 4 7 55 219 535 2,890
11 Complications of pregnancy, M − − − − − − −
childbirth and the puerperium F 27 − − 27 − − −
12 Diseases of the skin M 269 1 1 7 22 62 176
and subcutaneous tissue F 748 − − 15 30 80 623
13 Diseases of the musculoskeletal M 785 1 5 28 106 151 494
system and connective tissue F 2,639 − 5 43 173 385 2,033
14 Congenital anomalies M 660 131 114 158 118 58 81
F 675 136 116 133 101 87 102
15 Certain conditions originating M 186 93 8 13 18 16 38
in the perinatal period F 114 60 5 3 4 10 32
16 Signs, symptoms and M 1,642 238 17 126 111 72 1,078
ill-defined conditions F 5,146 171 17 50 53 75 4,780
17 External causes of M 9,859 34 311 4,749 2,183 941 1,641
injury and poisoning F 5,869 30 162 1,240 882 731 2,824
Total M 279,158 2,410 1,107 11,900 46,669 78,833 138,239
F 299,012 1,832 797 6,317 28,994 55,205 205,867
Table 1.2: Deaths in England & Wales, 1993, by underlying cause, sex and age-group
(extracted from Table 2 of the 1993 mortality tables )
9

(c) Now treat the data in Tables 1.1 & 1.2 as subject to statistical fluctuations.
estimate
One can still
psac = P(S=s & A=a & C=c), p·ac = P(A=a & C=c), ps · · = P(S=s) etc.
from the data, for example ̂p 0,·,14 = 660/25198400 = 2.62×10 −5 . Similarly estimate p 1,·,14 and
p·,a,14 for a = 1 . . . 6. Using a chi-squared test or otherwise, investigate whether the relative
risk of death from a congenital anomaly between males and females is the same at all ages,
i.e. whether it reasonable to assume that
ps, a,14 = ps, ·,14 × p·, a,14.
10. Data were collected on litter size and sex ratios for a large number of litters of piglets. The following
table gives the data for all litters of size between four and twelve:
Number
Litter size
of males 4 5 6 7 8 9 10 11 12
0 1 2 3 0 1 0 0 0 0
1 14 20 16 21 8 2 7 1 0
2 23 41 53 63 37 23 8 3 1
3 14 35 78 117 81 72 19 15 8
4 1 14 53 104 162 101 79 15 4
5 4 18 46 77 83 82 33 9
6 0 21 30 46 48 13 18
7 2 5 12 24 12 11
8 1 7 10 8 15
9 0 0 1 4
10 0 1 0
11 0 0
12 0
Total 53 116 221 374 402 346 277 102 70
(a) Discuss briefly what sort of probability distributions it might be reasonable to assume for the
total size N of a litter, and for the number M of males in a litter of size N = n.
(b) Suppose now that the litter size N follows a Poisson distribution with mean λ. Write down
an expression for P(N = n|4 ≤ N ≤ 12). Hence or otherwise give an expression for the loglikelihood
l(λ; . . .) given the above table of data.
(c) Evaluate l(λ; . . .) at λ = 7.5, 8 and 8.5. By fitting a quadratic to these values, provide point
and interval estimates of λ.
(d) Using a chi-squared test or otherwise, check how well your model fits the data.
(e) Comment on the following argument: ‘Provided λ isn’t too small, we could approximate the
Poisson distribution Poi(λ) by the Normal distribution N (λ, λ). This is symmetric, so we may
simply estimate the mean λ by the mode of the data (8 in our case). The standard deviation is
therefore nearly 3, and so we would expect the counts at litter size 8 ± 3 to be nearly 60% the
count at 8 (note that for a standard Normal, φ(1)/φ(0) = exp(−0.5) ≏ 0.6). Since there are far
fewer litters of size 5 & 11 than this, the Poisson distribution must be a poor fit.’
Data from HSDS, set 176
Education is what survives when what has been learnt has been forgotten.
Burrhus Frederoc Skinner
10

Chapter 2
Bivariate & Multivariate
Distributions
MSA largely concerned IID (independent & identically distributed) random variables.
However in practice we are usually most interested in several random variables simultaneously, and their
interrelationships. Therefore we need to consider the probability distributions of random vectors, i.e. the
joint distribution of the individual random variables.
Bivariate Examples
A. (X 1 , X 2 ), the number of male & female pigs in a litter.
B. (X, Y ), the systolic and diastolic blood pressure of an individual.
C. (X, Y ), the age and height of an individual.
D. (X, Y ), the height and weight of an individual.
E. (̂µ, ̂σ 2 ), the estimated common mean and variance of n IID random variables X 1 , . . . , X n .
F. (Θ, X) where Θ ∼ U (0, 1) and X|Θ ∼ Bin(n, Θ), i.e.
f Θ (θ) =
{ 1 if 0 < x < 1
0 otherwise,
f X (x|Θ = θ) = ( n
x)
θ x (1 − θ) n−x x = 0, 1, . . . , n.
Definition 2.1 (Bivariate CDF)
The joint cumulative distribution function of 2 RVs X & Y is the function
F X,Y (x, y) = P(X ≤ x & Y ≤ y), (x, y) ∈ R 2 . (2.1)
Comments
1. The joint cumulative distribution function (or joint CDF) may also be called the ‘joint distribution
function’ or ‘joint DF’.
2. If there’s no ambiguity, then we may simply write F (x, y) for F X,Y (x, y).
11

2.1 Discrete Bivariate Distributions
If RVs X & Y are discrete, then they have a discrete joint distribution and a probability mass function
(PMF) that, similarly to the univariate case, is usually written f X,Y (x, y) or more simply f(x, y):
Definition 2.2 (Bivariate PMF)
The joint probability mass function of discrete RVs X and Y is
f(x, y) = P(X = x & Y = y).
Exercise 2.1
Suppose that the numbers X 1 and X 2 of male and female piglets follow independent Poisson distributions
with means λ 1 & λ 2 respectively. Find the joint PMF.
‖
Exercise 2.2
Now assume the model N ∼ Poi(λ), (X 1 |N) ∼ Bin(N, θ), i.e. the total number N of piglets follows a
Poisson distribution, and, conditional on N = n, X has a Bin(n, θ) distribution (in particular θ = 0.5 if
the sexes are equally likely). Again find the joint PMF.
‖
Exercise 2.3
Verify that the two models given in Exercises 2.1 & 2.2 give identical fitted values, and are therefore in
practice indistinguishable.
‖
2.1.1 Manipulation
A discrete RV has a countable sample space, which without loss of generality can be represented as
N = {0, 1, 2, . . .}. Values of a discrete joint distribution f(x, y) can therefore be tabulated:
Y
0 1 2 3 . . .
0 f 00 f 01 f 02 . . .
X 1 f 10 f 11 f 12 . . .
. . . .
. ..
and the probability of any event E obtained by simple summation:
P ( (X, Y ) ∈ E ) =
∑
f(x i , y i ).
(x i,y i)∈E
Exercise 2.4
Continuing Exercise 2.2, find the PMF of X 1 , and hence identify the distribution of X 1 .
‖
Exercise 2.5
The RV Q is defined on the rational numbers in [0, 1] by Q = X/Y , where f(x, y) = (1 − α)α y−1 /(y + 1),
0 < α < 1, y = {1, 2, . . .}, x = {0, 1, . . . , y}.
Show that P(Q = 0) = (α − 1) ( α + log(1 − α) ) /α 2 .
‖
12

2.2 Continuous Bivariate Distributions
Definition 2.3 (Continuous bivariate distribution)
Random variables X & Y have a continuous joint distribution if there exists a function f from R 2 to
[0, ∞) such that
P ( (X, Y ) ∈ A ) ∫∫
= f(x, y) dx dy ∀A ⊆ R 2 . (2.2)
A
Definition 2.4 (Bivariate PDF)
The function f(x, y) defined by Equation 2.2 is called the joint probability density function of X & Y .
Comments
1. f(x, y) may be written more explicitly as f X,Y (x, y).
2.
∫ ∞ ∫ ∞
−∞
−∞
f(x, y) dx dy = 1.
3. f(x, y) is not unique—it could be arbitrarily defined at a countable set of points (x i , y i ) (more
generally, any ‘set with measure zero’) without changing the value of ∫∫ f(x, y) dx dy for any set A.
A
4. f(x, y) ≥ 0 at all continuity points (x, y) ∈ R 2 .
Examples
1. As in Example E from page 11, we will want to know properties of the joint distribution of (̂µ, ̂σ 2 ),
the MLEs of µ and σ 2 respectively given X 1 , . . . , X n
IID
∼ N (µ, σ 2 ).
2. In the situation of Example B from page 11, where X is the systolic blood pressure and Y the diastolic
blood pressure of an individual, it might be reasonable to assume that
and hence obtain
X ∼ N (µ S , σ 2 S),
Y |X ∼ N (α + βX, σ 2 D),
f X,Y (x, y) = f X (x) f Y |X (y|x).
Comment
As in Exercise 2.2, a family of multivariate distributions is most easily built up hierarchically using simple
univariate distributions and conditional distributions like that of Y |X. Conditional distributions are
considered formally in Section 2.4.
2.2.1 Visualising and Displaying a Continuous Joint Distribution
A continuous bivariate distribution can be represented by a contour or other plot of its joint PDF (Fig. 2.1).
Comments
1. The joint distribution of X and Y may be neither discrete nor continuous, for example:
• Either X or Y may have both continuous and discrete components,
• One of X and Y may have a continuous distribution, the other discrete (like Example F on
page 11).
2. Higher dimensional joint distributions are obviously much more difficult to interpret and to represent
graphically, with or without computer help.
13

Figure 2.1: Contour and perspective plots of a bivariate distribution
2.3 Marginal Distributions
Given a joint CDF F X,Y (x, y), the distributions defined by the CDFs F X (x) = lim y→∞ F X,Y (x, y) and
F Y (y) = lim x→∞ F X,Y (x, y) are called the marginal distributions of X and Y respectively:
Definition 2.5 (Marginal CDF, PMF and PDF—bivariate case)
F X (x) = lim y→∞ F X,Y (x, y) is the marginal CDF of X.
If X has a discrete distribution, then f X (x) = P(X = x) is the marginal PMF of X.
If X has a continuous distribution, then f X (x) = d
dx F X(x) is the marginal PDF of X.
Marginal CDFs and PDFs of Y , and of other RVs for higher-dimensional joint distributions, are defined
similarly.
Exercise 2.6
Suppose that you are given a bag containing five coins:
1 double-tailed, 1 with P(head) = 1/4, 2 fair, 1 double-headed.
You pick one coin at random (each with probability 1/5), then toss it twice.
By finding the joint distribution of Θ = P(head) and X = number of heads, or otherwise, calculate the
distribution of the number of heads obtained.
‖
Comments
1. If you’ve tabulated P(Θ = θ & X = x), then it’s simple to find F Θ (θ) and F X (x) by writing the row
sums and column sums in the margins of the table of P(Θ = θ & X = x)—hence the name ‘marginal
distribution’.
2. Although the most satisfactory general definition of marginal distributions is in terms of their CDFs,
in practice it’s usually easiest to work with PMFs or PDFs
2.4 Conditional Distributions
2.4.1 Discrete Case
If X and Y are discrete RVs then, by definition,
P(Y =y|X=x) = P(X=x & Y =y)/P(X=x). (2.3)
14

In other words (or, more accurately, in other symbols):
Definition 2.6 (Conditional PMF—bivariate case)
If X and Y have a discrete joint distribution with PMF f X,Y (x, y), then the conditional PMF f Y |X
of Y given X = x is
f Y |X (y|x) = f X,Y (x, y)
(2.4)
f X (x)
where f X (x) = ∑ y f X,Y (x, y) is the marginal PMF of X.
Exercise 2.7
Continuing Exercise 2.6, what are the conditional distributions of [X |Θ = 1/4] and [Θ|X = 0]
‖
2.4.2 Continuous Case
Now suppose that X and Y have a continuous joint distribution. If we observe X = x, then we will want
to know the conditional CDF F Y |X (y|X = x). But we CAN’T use Equation 2.3 directly, which would
entail dividing by zero. Therefore, by analogy with Equation 2.4, we adopt the following definition:
Definition 2.7 (Conditional PDF—bivariate case)
If X and Y have a continuous joint distribution with PDF f X,Y (x, y), then the conditional PDF f Y |X
of Y given that X = x is
f Y |X (y|x) = f X,Y (x, y)
, (2.5)
f X (x)
defined for all x ∈ R such that f X (x) > 0.
2.4.3 Independence
Recall that two RVs X and Y are independent (X ⊥ Y ) if, for any two sets A, B ∈ R,
P(X∈A & Y ∈B) = P(X ∈ A)P(Y ∈ B) (2.6)
Exercise 2.8
Show that X and Y are independent according to Formula 2.6 if and only if
or equivalently if and only if
F X,Y (x, y) = F X (x)F Y (y) − ∞ < x, y < ∞, (2.7)
f X,Y (x, y) = f X (x)f Y (y) − ∞ < x, y < ∞, (2.8)
(where the functions f are interpreted as PMFs or PDFs in the discrete or continuous case respectively).
‖
15

2.5 Problems
1. Let the function f(x, y) be defined by
{ 6xy
2
if 0 < x < 1 and 0 < y < 1,
f(x, y) =
0 otherwise.
(a) Show that f(x, y) is a probability density function.
(b) If X and Y have the joint PDF f(x, y) above, show that P(X + Y ≥ 1) = 9/10.
(c) Find the marginal PDF f X (x) of X.
(d) Show that P(0.5 < X < 0.75) = 5/16.
2. Suppose that the random vector (X, Y ) takes values in the region A = {(x, y)|0 ≤ x ≤ 2, 0 ≤ y ≤ 2},
and that its CDF within A is given by F X,Y (x, y) = xy(x + y)/16.
(a) Find F X,Y (x, y) for values of (X, Y ) outside A.
(b) Find the marginal CDF F X (x) of X.
(c) Find the joint PDF f X,Y (x, y).
3. Suppose that X and Y are RVs with joint PDF
{ cx
f(x, y) =
2 y if x 2 ≤ y ≤ 1,
0 otherwise.
(a) Find the value of c.
(b) Find P(X ≥ Y ).
(c) Find the marginal PDFs f X (x) & f Y (y)
4. For each of the following joint PDFs f of X and Y , determine the constant c, find the marginal PDFs
of X and Y , and determine whether or not X and Y are independent.
(a)
(b)
(c)
(d)
f(x, y) =
{
ce −(x+2y) , for x, y ≥ 0,
0 otherwise.
{ cy
f(x, y) =
2 /2, for 0 ≤ x ≤ 2 and 0 ≤ y ≤ 1,
0 otherwise.
{ cxe
f(x, y) =
−y , for 0 ≤ x ≤ 1 and 0 ≤ y < ∞,
0 otherwise.
{ cxy, for x, y ≥ 0 and x + y ≤ 1,
f(x, y) =
0 otherwise.
5. Suppose that X and Y are continuous RVs with joint PDF f(x, y) = e −y on 0 < x < y < ∞.
(a) Find P(X + Y ≥ 1) [HINT : write this as 1 − P(X + Y < 1)].
(b) Find the marginal distribution of X.
(c) Find the conditional distribution of Y given that X = x.
16

6. Assume that X and Y are random variables each taking values in [0, 1]. For each of the following
CDFs, show that the marginal distribution of X and Y are both uniform U(0, 1), and determine the
conditional CDF F X|Y (x|Y = 0.5) in each case:
(a) F (x, y) = xy,
(b) F (x, y) = min(x, y),
{ 0, if x + y < 1,
(c) F (x, y) =
x + y − 1 if x + y ≥ 1.
7. Suppose that Θ is a random variable uniformly distributed on (0, 1), i.e. Θ ∼ U (0, 1), and that, once
Θ = θ has been observed, the random variable X is drawn from a binomial distribution [X|θ] ∼
Bin(2, θ).
(a) Find the joint CDF F (θ, x).
(b) How might you display the joint distribution of Θ and X graphically
(c) What (as simply as you can express them) are the marginal CDFs F 1 (θ) of Θ and F 2 (x) of X
8. Suppose that X and Y are two RVs having a continuous joint distribution. Show that X and Y are
independent if and only if f X|Y (x|y) = f X (x) for each value of y such that f Y (y) > 0, and for all x.
9. Suppose that X ∼ U (0, 1) and [Y |X = x] ∼ U (0, x). Find the marginal PDFs of X and Y .
2.6 Multivariate Distributions
2.6.1 Introduction
Given a random vector X = (X 1 , X 2 , . . . , X n ) T , the joint distribution of the random variables X 1 , X 2 , . . . , X n
is called a multivariate distribution.
Definition 2.8 (Joint CDF)
The joint cumulative distribution function of RVs X 1 , X 2 , . . . , X n is the function
F X (x 1 , x 2 , . . . , x n ) = P(X k ≤ x k ∀ k = 1, 2, . . . , n). (2.9)
Comments
1. Formula 2.9 can be written succinctly as F X (x) = P(X ≤ x), in an ‘obvious’ vector notation.
2. F X (x) can be called simply the CDF of the random vector X.
3. Properties of F X are similar to the bivariate case. Unfortunately the notation is messier, particularly
for the things we’re generally most interested in for statistical inference, such as
(a) marginal distributions of unknown quantities and vectors,
(b) conditional distributions of unknown quantities and vectors, given what we know.
4. It’s often simpler to blur the distinction between row and column vectors, i.e. to let X denote either
(X 1 , X 2 , . . . , X n ) or (X 1 , X 2 , . . . , X n ) T , depending on context.
17

Definition 2.9 (Discrete multivariate distribution)
The RV X ∈ R n has a discrete distribution if it can take only a countable number of possible values.
Definition 2.10 (Multivariate PMF)
If X has a discrete distribution, then its probability mass function (PMF) is
f(x) = P(X = x), x ∈ R n (2.10)
[i.e. the RVs X 1 . . . X n have joint PMF f(x 1 . . . x n ) = P(X 1 = x 1 & · · · &X n = x n )].
Definition 2.11 (Continuous multivariate distribution)
The RV X = (X 1 , X 2 , . . . , X n ) has a continuous distribution if there is a nonnegative function f(x),
where x = (x 1 , x 2 , . . . , x n ), such that for any subset A ⊂ R n ,
P ( (X 1 , X 2 , . . . , X n ) ∈ A ) ∫ ∫
= . . . f(x 1 , x 2 , . . . x n ) dx 1 dx 2 . . . dx n . (2.11)
Definition 2.12 (Multivariate PDF)
The function f in 2.11 is the (joint) probability density function of X.
A
Comments
1. Without loss of generality, if X is discrete, then we can take its possible values to be N n (i.e. each
coordinate X i of X is a nonnegative integer).
2. Equation 2.11 could be simply written
P ( ∫
X ∈ A) =
3. As usual, f(·) may be written more explicitly f X (·), etc.
4. By the fundamental theorem of calculus,
A
f(x)dx (2.12)
f X (x 1 , . . . , x n ) = ∂n F X (x 1 , . . . , x n )
∂x 1 · · · ∂x n
(2.13)
at all points (x 1 , . . . , x n ) where this derivative exists ( i.e. f X (x) = ∂ n F X (x)/∂x ) .
5. Mixed distributions (neither continuous nor discrete) can be handled using appropriate combinations
of summation and integration.
2.6.2 Useful Notation for Marginal & Conditional Distributions
We’ll sometimes adopt the following notation from DeGroot, particularly when the components X i of X
are in some way similar, as in the multivariate Normal distribution (see later).
F (x) denotes the CDF of X = (X 1 , X 2 , . . . , X n ) at x = (x 1 , x 2 , . . . , x n ),
f(x) denotes the corresponding joint PMF (discrete case) or PDF (continuous case),
f j (x j ) denotes the marginal PMF (PDF) of X j (integrating over x 1 . . . x j−1 , x j+1 . . . x n ),
f jk (x j , x k ) denotes the marginal joint PDF of X j & X k (integrating over the remaining x i s),
g j (x j |x 1 . . . x j−1 , x j+1 . . . x n ) denotes the conditional PMF (PDF) of X j given X i = x i , i ≠ j,
F j (x j ) denotes the marginal CDF of X j ,
G jk
denotes the conditional CDF of (X j , X k ) given the values x i of all X i , i ≠ j, k, etc.
18

2.7 Expectation
2.7.1 Introduction
The following are important definitions and properties involving expectations, variances and covariances:
Var(X) = E [ (X − µ) 2] where µ = EX
= E [ X 2] − µ 2 ,
E[aX + b] = aEX + b where a and b are constants,
E [ (aX + b) 2] = a 2 E [ X 2] + 2abEX + b 2 ,
Var(aX + b) = a 2 Var(X),
E[X 1 X 2 ] = (EX 1 )(EX 2 ) if X 1 ⊥ X 2 ,
Cov(X 1 , X 2 ) = E(X 1 − µ 1 )(X 2 − µ 2 ) = E[X 1 X 2 ] − µ 1 µ 2 ,
SD(X) = √ Var(X),
corr(X 1 , X 2 ) = ρ(X 1 , X 2 ) =
Cov(X 1 , X 2 )
SD(X 1 )SD(X 2 ) .
Note that the definition of expectation applies directly in the multivariate case:
Definition 2.13 (Multivariate expectation)
∑
x
⎧
⎪⎨
h(x) f(x)
E[h(X)] = ∫
⎪⎩ h(x) f(x) dx
R n
if X is discrete,
if X is continuous.
For example, if X = (X 1 , X 2 , X 3 ) has a continuous distribution, then
E[X 1 ] =
∫ ∞ ∫ ∞ ∫ ∞
−∞
−∞
−∞
x 1 f(x 1 , x 2 , x 3 ) dx 1 dx 2 dx 3
Exercise 2.9
Let X and Y be independent continuous RVs. Prove that, for arbitrary functions g(·) and h(·),
E [ g(X)h(Y ) ] = ( E g(X) )( E h(Y ) ) .
‖
Exercise 2.10
Let X, Y and Z have independent Poisson distributions with means λ, µ, ν respectively. Find E[X 2 Y Z].
‖
Exercise 2.11
[Cauchy-Schwartz] By considering E ( (tX − Y ) 2) , or otherwise, prove the Cauchy Schwartz inequality for
expectations, i.e. for any two RVs X and Y with finite second moments, ( E(XY ) ) 2
≤ E
(
X
2 ) E ( Y 2) , with
equality if and only if P(Y = cX) = 1 for some constant c.
Hence or otherwise prove that the correlation ρ X,Y between X and Y satisfies |ρ X,Y | ≤ 1.
Under what circumstances does ρ X,Y = 1
‖
19

2.8 Approximate Moments of Transformed Distributions
The moments of a transformed RV g(X) can often be well approximated via a Taylor series:
Exercise 2.12
[delta method] Let X 1 , X 2 , . . . , X n be independent, each with mean µ and variance σ 2 , and let g(·) be a
function with a continuous derivative g ′ (·).
By considering a Taylor series expansion involving
Z n = X − µ √
σ2 /n ,
show that
E [ g(X) ] = g(µ) + O(n −1 ), (2.14)
Var [ g(X) ] = n −1 σ 2 g ′ (µ) 2 + O(n −3/2 ). (2.15)
‖
Comments
1. There is similarly a multivariate delta method, outside the scope of this course.
2. Important uses of expansions like the delta method include identifying useful transformations g(·), for
example to remove skewness or, when Var(X) is a function of µ, to make Var [ g(X) ] (approximately)
independent of µ.
3. A useful transformation g(X) is sometimes in practice applied to the original RVs on the (often
reasonable) assumption that the properties of (∑ g(X i ) ) /n will be similar to those of g (∑ X i )/n ) .
Exercise 2.13
[Variance stabilising transformations]
Suppose that X 1 , X 2 , . . . , X n are IID and that the (common) variance of each X i is a function of the
(common) mean µ = EX i .
Show that the variance of g(X) is approximately constant if
g ′ (µ) = 1/ √ Var(µ).
If X ∼ Poi(µ), show that Y = √ X has approximately constant variance.
‖
20

2.9 Problems
1. The discrete random vector (X 1 , X 2 , X 3 ) has the following PMF:
(X 1 = 1) X 3 (X 1 = 2) X 3
1 2 3 1 2 3
1 .02 .03 .05 1 .08 .04 .03
X 2 2 .04 .06 .10 X 2 2 .12 .11 .07
3 .02 .03 .05 3 .05 .05 .05
(a) Calculate the marginal PMFs: f 1 (x 1 ), f 2 (x 2 ), f 3 (x 3 ) and f 12 (x 1 , x 2 ).
(b) Are X 1 and X 2 independent
(c) What are the conditional PMFs: g 1 (x 1 |X 2 = 1, X 3 = 3), g 2 (x 2 |X 1 = 1, X 3 = 3),
g 3 (x 3 |X 1 = 1, X 2 = 3), and g 12 (x 1 , x 2 |X 3 = 3)
2. The RVs A, B, C etc. count the number of times the corresponding letter appears when a word is
chosen at random from the following list (each being chosen with probability 1/16):
MASCARA, MASK, MERCY, MONSTER,
MOVIE, PREY, REPLICA, REPTILES,
RITE, SEAT, SNAKE, SOMBRE,
SQUID, TENDER, TIME, TROUT.
(a) Complete the following table of the joint distribution of E, M and R:
E = 0 E = 1 E = 2
M=0 M=1 M=0 M=1 M=0 M=1
R=0 1/16 1/16 R=0 2/16 R=0
R=1 R=1 R=1
(b) Calculate all three bivariate marginal distributions, and hence find which of the following statements
are true:
(i) E ⊥ M, (ii) E ⊥ R, (iii) M ⊥ R.
(c) Similarly discover which of the following statements are true:
3. Find variance stabilizing transformations for
(a) the exponential distribution,
(b) the binomial distribution.
4. Let Z ∼ N (0, 1) and define the RV X by
(iv) M ⊥ R|E=0, (v) M ⊥ R|E=1, (vi) M ⊥ R|E=2,
(vii) M ⊥ R|E, (viii) E ⊥ R|M, (ix) E ⊥ M |R.
P(X = − √ 3) = 1/6, P(X = 0) = 4/6, P(X = + √ 3) = 1/6.
(a) Show that X has the same mean and variance as Z, and that X 2 has the same mean and
variance as Z 2 .
(b) Suppose the RV Y has mean µ and variance σ 2 . Compare the delta method for estimating the
mean and variance of the RV T = g(Y ) with the alternative estimates ̂µ(T ) ≏ E ( g(µ + σX) ) ,
̂Var(T ) ≏ Var ( g(µ + σX) ) . [Try a few simple distributions for Y and transformations g(·)].
21

2.10 Conditional Expectation
2.10.1 Introduction
A common practical problem arises when X 1 and X 2 aren’t independent, we observe X 2 = x 2 , and we
want to know the mean of the resulting conditional distribution of X 1 .
Definition 2.14 (Conditional expectation)
The conditional expectation of X 1 given X 2 is denoted E[X 1 |X 2 ]. If X 2 = x 2 then
E[X 1 |x 2 ] =
∫ ∞
−∞
x 1 g 1 (x 1 |x 2 ) dx 1 (continuous case) (2.16)
= ∑ x 1
x 1 g 1 (x 1 |x 2 ) (discrete case) (2.17)
where g 1 (x 1 |x 2 ) is the conditional PDF or PMF respectively.
Comment
Note that before X 2 is known to take the value x 2 , E[X 1 |X 2 ] is itself a random variable, being a function
of the RV X 2 . We’ll be interested in the distribution of the RV E[X 1 |X 2 ], and (for example) comparing it
with the unconditional expectation EX 1 . The following is an important result:
Theorem 2.1 (Marginal expectation)
For any two RVs X 1 & X 2 ,
E [ E[X 1 |X 2 ] ] = EX 1 . (2.18)
Exercise 2.14
Prove Equation 2.18 (i) for continuous RVs X 1 and X 2 , (ii) for discrete RVs X 1 and X 2 .
‖
Exercise 2.15
Suppose that the RV X has a uniform distribution, X ∼ U (0, 1), and that, once X = x has been observed,
the conditional distribution of Y is [Y |X = x] ∼ U (x, 1).
Find E[Y |x] and hence, or otherwise, show that EY = 3/4.
‖
Exercise 2.16
Suppose that Θ ∼ U (0, 1) and (X|Θ) ∼ Bin(2, Θ).
Find E[X |Θ] and hence or otherwise show that EX = 1.
‖
2.10.2 Conditional Expectations of Functions of RVs
By extending Theorem 2.1, we can relate the conditional and marginal expectations of functions of RVs
(in particular, their variances).
Theorem 2.2 (Marginal expectation of a transformed RV)
For any RVs X 1 & X 2 , and for any function h(·),
E [ E[h(X 1 )|X 2 ] ] = E[h(X 1 )]. (2.19)
22

Exercise 2.17
Prove Equation 2.19 (i) for discrete RVs X 1 and X 2 , (ii) for continuous RVs X 1 and X 2 .
An important consequence of Equation 2.19 is the following theorem relating marginal variance to conditional
variance and conditional expectation:
‖
Theorem 2.3 (Marginal variance)
For any RVs X 1 & X 2 ,
Var(X 1 ) = E [ Var(X 1 |X 2 ) ] + Var ( E[X 1 |X 2 ] ) . (2.20)
Comments
1. Equation 2.20 is easiest to remember in English:
‘marginal variance = expectation of conditional variance
+ variance of conditional expectation’.
2. A useful interpretation of Equation 2.20 is:
Var(X 1 ) = average random variation inherent in X 1 even if X 2 were known
+ random variation due to not knowing X 2 and hence not knowing EX 1 .
i.e. the uncertainty involved in predicting the value x 1 taken by a random variable X 1 splits into two
components. One component is the unavoidable uncertainty due to random variation in X 1 , but the
other can be reduced by observing quantities (here the value x 2 of X 2 ) related to X 1 .
Exercise 2.18
[Proof of Theorem 2.3] Expand E [ Var(X 1 |X 2 ) ] and Var ( E[X 1 |X 2 ] ) .
Hence show that Var(X 1 ) = E [ Var(X 1 |X 2 ) ] + Var ( E[X 1 |X 2 ] ) .
‖
Exercise 2.19
Continuing Exercise 2.16, in which Θ ∼ U (0, 1), (X|Θ) ∼ Bin(2, Θ), and E[X |Θ] = 2Θ, find Var ( E[X |Θ] )
and E [ Var(X |Θ) ] . Hence or otherwise show that VarX = 2/3, and comment on the effect on the uncertainty
in X of observing Θ.
‖
23

2.11 Problems
1. Two fair coins are tossed independently. Let A 1 , A 2 and A 3 be the following events:
A 1 = ‘coin 1 comes down heads’
A 2 = ‘coin 2 comes down heads’
A 3 = ‘results of both tosses are the same’.
(a) Show that A 1 , A 2 and A 3 are pairwise independent (i.e. A 1 ⊥ A 2 , A 1 ⊥ A 3 and A 2 ⊥ A 3 ) but
not mutually independent.
(b) Hence or otherwise construct three random variables X 1 , X 2 , X 3 such that E[X 3 |X 1 = x 1 ] and
E[X 3 |X 2 = x 2 ] are constant, but E[X 3 |X 1 = x 1 &X 2 = x 2 ] isn’t.
2. Construct three random variables X 1 , X 2 , X 3 with continuous distributions such that X 1 ⊥ X 2 ,
X 1 ⊥ X 3 and X 2 ⊥ X 3 , but any two X i ’s determine the remaining one.
3. (a) Show that for any random variables X and Y ,
i. E[Y ] = E [ E[Y |X] ] ,
ii. Var[Y ] = E [ Var[Y |X] ] + Var [ E[Y |X] ] .
(b) Suppose that the random variables X i and P i , i = 1, . . . , n, have the following distributions:
X i =
P i
IID
∼
{
1 with probability Pi ,
0 with probability 1 − P i ,
Beta(α, β),
i.e. P i has density
f(p) =
with mean µ and variance σ 2 given by
Γ(α + β)
Γ(α) Γ(β) pα−1 (1 − p) β−1
µ = E[P i ] = α
α + β , αβ
σ2 = Var[P i ] =
(α + β) 2 (α + β + 1) ,
and X i has a Bernoulli(P i ) distribution.
Find
i. E[X 1 |P 1 ],
ii. Var[X 1 |P 1 ],
iii. Var [ E[X 1 |P 1 ] ] , and
iv. E [ Var[X 1 |P 1 ] ] .
Hence find E[Y ] where Y = ∑ n
i=1 X i, and show that Var[Y ] = nαβ/(α + β) 2 .
(c) Express E[Y ] and Var[Y ] in terms of µ and σ 2 , and comment on the result.
From Warwick ST217 exam 1998
4. Suppose that the number N of bye-elections occurring in Government-held seats over a 12-month
period follows a Poisson distribution with mean 10.
Suppose also that, independently for each such bye-election, the probability that the Government
hold onto the seat is 1/4. The number X of seats retained in the N bye-elections therefore follows a
binomial distribution:
[X|N] ∼ Bin(N, 0.25).
(a) What are E[N], Var[N], E[X|N] and Var[X|N]
(b) What are E[X] and Var[X]
(c) What is the distribution of X
[HINT : try using generating functions—see MSA]
24

5. (a) For continuous random variables X 1 , X 2 , X 3 with joint density f X (x), define the following in
terms of f X (x):
i. the marginal densities f X1 (x 1 ) and f X1,X 2
(x 1 , x 2 ),
ii. the conditional density f X2|X 1
(x 2 |x 1 ) of X 2 given X 1 = x 1 , and
iii. the conditional expectation E[h(X 2 )|X 1 ] of a function h(X 2 ) of X 2 given X 1 .
(b) Show that
i. E[h(X 2 )] = E [ E[h(X 2 )|X 1 ] ] ,
ii. Var[X 2 ] = E [ Var[X 2 |X 1 ] ] + Var [ E[X 2 |X 1 ] ] ,
iii. Cov[X 2 , X 3 ] = E [ Cov[X 2 , X 3 |X 1 ] ] + Cov [ E[X 2 |X 1 ], E[X 3 |X 1 ] ] , and
iv. Cov[X 1 , X 2 |X 1 ] = 0.
(c) Suppose that the random variables X and Y have a continuous joint distribution, with PDF
f(x, y), means µ X & µ Y respectively, variances σX 2 & σ2 Y respectively, and correlation ρ. Suppose
also that
E[Y |x] = β 0 + β 1 x
where β 0 and β 1 are constants.
Show that
i. ∫ ∞
−∞ yf(x, y)dy = (β 0 + β 1 x)f X (x),
ii. µ Y = β 0 + β 1 µ X , and
iii. ρ σ X σ Y + µ X µ Y = β 0 µ X + β 1 (σ 2 X + µ2 X ).
(Hint: use the fact that E[XY ] = E[E[XY |X]]).
(d) Hence or otherwise express β 0 and β 1 in terms of µ X , µ Y , σ X , σ Y & ρ.
6. For discrete random variables X and Y , define:
(i) The conditional expectation of Y given X, E[Y |X], and
(ii) The conditional variance of Y given X, Var[Y |X].
Show that
(iii) E[Y ] = E [ E[Y |X] ] , and
(iv) Var[Y ] = E [ Var[Y |X] ] + Var [ E[Y |X] ] .
(v) Show also that if E[Y |X] = β 0 + β 1 X for some constants β 0 and β 1 , then
E[XY ] = β 0 E[X] + β 1 E[X 2 ].
From Warwick ST217 exam 2002
The random variable X denotes the number of leaves on a certain plant at noon on Monday, Y
denotes the number of greenfly on the plant at noon on Tuesday, and Z denotes the number of
ladybirds on the plant at noon on Wednesday.
Suppose that, given X = x, Y has a Poisson distributions with mean µx.
distribution with mean λ, show that
E[Y ] = λµ
and
Var[Y ] = λµ(1 + µ),
(you may assume that for a Poisson distribution the mean and variance are equal).
If X has a Poisson
Suppose further that, given Y = y, Z has a Poisson distributions with mean νy. Find E[Z], Var[Z],
and the correlation between X and Z.
From Warwick ST217 exam 1996
25

7. Using the relationship
E [ E[h(X 1 )|X 2 ] ] = E[h(X 1 )],
where
h(x 1 ) = (x 1 − E[X 1 |x 2 ] + E[X 1 |x 2 ] − EX 1 ) 2 ,
prove that
Var(X 1 ) = E [ Var(X 1 |X 2 ) ] + Var ( E[X 1 |X 2 ] )
for any two random variables X 1 & X 2 .
8. Prove that, for any three RVs X, Y and Z for which the various expectations exist,
(a) X and Y − E(Y |X) are uncorrelated,
(b) Var ( Y − E(Y |X) ) = E ( Var(Y |X) ) ,
(c) if X and Y are uncorrelated then E ( Cov(X, Y |Z) ) = −Cov ( E(X|Z), E(Y |Z) ) ,
(d) Cov ( Z, E(Y |Z) ) = Cov(Z, Y ).
In scientific thought we adopt the simplest theory which will explain all the facts under consideration
and enable us to predict new facts of the same kind. The catch in this criterion lies
in the word ‘simplest’. It is really an aesthetic canon such as we find implicit in our criticisms
of poetry or painting.
J. B. S. Haldane
All models are wrong, some models are useful.
G. E. P. Box
A child of five would understand this. Send somebody to fetch a child of five.
Groucho Marx
26

Chapter 3
The Multivariate Normal
Distribution
3.1 Motivation
A Normally distributed RV
has PDF
where
X ∼ N (µ, σ 2 )
[
f(x; µ, σ 2 ) = constant × exp − 1 (x − µ) 2 ]
2 σ 2
µ is the mean of X,
σ 2 is the variance of X, and
‘constant’ is there to make f integrate to 1.
(3.1)
The Normal distribution is important because, by the CLT, as n → ∞, the CDF of a MLE such as
̂θ = ∑ X i /n or ̂θ = ∑ (X i − ΣX j /n) 2 / n, tends uniformly (under reasonable conditions) to the CDF of a
Normal RV with the appropriate mean and variance.
i.e. the log-likelihood tends to a quadratic in θ.
Similarly it can be shown that, for a model with parameter vector θ = (θ 1 , . . . , θ p ) T , under reasonable
conditions the log-likelihood will tend to a quadratic in (θ 1 , . . . , θ p ).
Therefore, by analogy with Equation 3.1, we will want to define a distribution with PDF
[
f(x; µ, V) = constant × exp − 1 ]
2 (x − µ)T V −1 (x − µ)
(3.2)
where
µ is a (p × 1) matrix or column vector,
V is a (p × p) matrix, and
‘constant’ is again there to make f integrate to 1.
27

IID
As an example of a PDF of this form, if X 1 , X 2 , . . . , X p ∼ N (0, 1), then
f(x) = f 1 (x 1 ) × f 2 (x 2 ) × · · · × f p (x p ) by independence
1
=
(2π) exp ( )
− 1 1
p/2 2 Σx2 i =
(2π) exp ( − 1 p/2 2 xT x ) . (3.3)
Definition 3.1 (Multivariate standard Normal)
The distribution with PDF
f(z) = f(z 1 , z 2 , . . . , z p ) =
is called the multivariate standard Normal distribution.
1
(2π) exp ( − 1 p/2 2 zT z )
The statement ‘Z has a multivariate standard Normal distribution’ is often written
Z ∼ N (0, I), Z ∼ MVN (0, I), Z ∼ N p (0, I), or Z ∼ MVN p (0, I),
and the CDF and PDF of Z are often written Φ(z) and φ(z), or Φ p (z) and φ p (z), respectively.
In the more general case, where the component RVs X 1 , X 2 , . . . , X p in Equation 3.2 aren’t independent,
we need an expression for the constant term.
3.2 Digression: Transforming a Random Vector
Exercise 3.1
Suppose that the RVs Z 1 , Z 2 , . . . , Z n have a continuous joint distribution, with joint PDF f Z (z).
Consider a 1-1 transformation (i.e. a bijection between the corresponding sample spaces) to new RVs
X 1 , X 2 , . . . , X n . What is the PDF f X (x) of the transformed RVs Solution: Because the transformation
is 1-1 we can invert it and write
Z = u(X)
i.e. a given point (z 1 , . . . , z n ) transforms to (x 1 , . . . , x n ), where
z 1 = u 1 (x 1 , . . . , x n ),
z 2 = u 2 (x 1 , . . . , x n ),
.
z n = u n (x 1 , . . . , x n ).
(3.4)
Now assume that each function u i (·) is continuous and differentiable.
Then we can form the following matrix:
⎛
∂u 1 ∂u 1 ∂u 1
. . .
∂x 1 ∂x 2 ∂x n
∂u
∂u 2 ∂u 2 ∂u 2
. . .
=
∂x
∂x
1 ∂x 2 ∂x n
.
⎜ . . .. .
⎝ ∂u n ∂u n ∂u n
. . .
∂x 1 ∂x 2 ∂x n
⎞
⎟
⎠
(3.5)
and its determinant J, which is called the Jacobian of the transformation u
[i.e. of the joint transformation (u 1 , . . . , u n )].
Then it can be shown that
f X (x) = |J| × f Z (z)
at all points in the ‘sample space’ (i.e. set of possible values) of X.
‖
28

z + δ 2 z + δ 1 + δ 2 infinitesimal δ 1 × δ 2 rectangle
z z + δ 1
✻
∴
density = f Z (z)
area = δ 1 δ 2
probability content = δ 1 δ 2 f Z (z)
u
✏ ✏
u −1 (z + δ 1 + δ 2 )
u −1 (z + δ 2 ) ✏✏
✏ ✏✏ ✏ ✁✁
✁ ✁ u −1 (z + δ 1 )
x = u −1 (z)
infinitesimal parallelogram
∴
area = δ 1 δ 2 /|J|
probability content = δ 1 δ 2 f Z (z)
density = |J| × f Z (z)
Figure 3.1: Bivariate Parameter Transformation
3.3 The Bivariate Normal Distribution
Suppose that Z 1 and Z 2 are IID with N (0, 1) distributions, i.e. (as in Equation 3.3):
f Z (z 1 , z 2 ) = 1
2π exp( − 1 2 (z2 1 + z 2 2) ) .
Now let µ 1 , µ 2 ∈ (−∞, ∞), σ 1 , σ 2 ∈ (0, ∞) & ρ ∈ (−1, 1), and define (as in DeGroot §5.12):
X 1 = σ 1 Z 1 + µ 1 ,
X 2 = σ 2
(
ρZ1 + √ 1 − ρ 2 Z 2
)
+ µ2 .
Then the Jacobian of the transformation from Z to X is given by
∣ J =
σ 1 0 ∣∣∣
√
∣
= √ 1 − ρ
ρ σ 2 1 − ρ2 σ 2 σ 1 σ 2 .
2
(3.6)
Therefore the Jacobian of the inverse transformation from X to Z is 1/ (√ 1 − ρ 2 σ 1 σ 2
)
, and the PDF of
X is given by Equations 3.7 & 3.8 below.
Definition 3.2 (Bivariate Normal Distribution)
The continuous bivariate distribution with PDF
where
f X (x) =
1
|J| f Z(z) =
σ 1
(
)
1
√ × 1
1 − ρ2 σ 1 σ 2
2π × exp 1
−
2 ( 1 − ρ 2)Q , (3.7)
( ) 2 ( )( ) ( ) 2 x1 − µ 1 x1 − µ 1 x2 − µ 2 x2 − µ 2
Q =
− 2ρ
+
. (3.8)
σ 1 σ 2 σ 2
is called the bivariate Normal distribution.
29

Exercise 3.2
If the RV X = (X 1 , X 2 ) has PDF given by Equations 3.7 & 3.8, then show by substituting
or otherwise, that X 1 ∼ N (µ 1 , σ 2 1).
v = x 2 − µ 2
followed by w = v − ρ(x 1 − µ 1 )/σ
√ 1
,
σ 2 1 − ρ
2
Hence or otherwise show that the conditional distribution of X 1 given X 2 = x 2 is Normal with mean
µ 1 + (ρσ 1 /σ 2 )(x 2 − µ 2 ) and variance σ 2 1(
1 − ρ
2 ) .
‖
Comments
1. It’s easy to show (problem 3.4.2, page 31) that EX i = µ i , VarX i = σ 2 i and corr(X 1, X 2 ) = ρ. This
suggests that we will be able to write
X = (X 1 , X 2 ) T ∼ MVN (µ, V), where
µ = (µ 1 , µ 2 ) T is the ‘mean vector’ of X, and
( )
σ
2
V =
1 ρ σ 1 σ 2
ρ σ 1 σ 2 σ2
2 is the ‘variance-covariance matrix’ of X.
2. The ‘level curves’ (i.e. contours in 2-d) of the bivariate Normal PDF are given by Q = constant in
formula 3.8; i.e. ellipses provided the discriminant is negative:
( ) 2 ρ
− 1 1
σ 1 σ 2 σ1
2 σ2
2
= ρ2 − 1
σ 2 1 σ2 2
< 0.
This holds as we are only considering ‘nonsingular’ bivariate Normal distributions with ρ ≠ ±1.
3. PLEASE MAKE NO ATTEMPT TO MEMORISE FORMULAE 3.7 & 3.8!!
Exercise 3.3
(
σ
2
Show that the inverse of the variance-covariance matrix V = 1 ρ σ 1 σ 2
ρ σ 1 σ 2 σ2
2
V −1 = 1
1 − ρ 2 ( 1/σ
2
1 −ρ/σ 1 σ 2
−ρ/σ 1 σ 2 1/σ 2 2
)
.
)
is
‖
30

3.4 Problems
1. Suppose that the RVs X 1 , X 2 , . . . , X n have a continuous joint distribution with PDF f X (x), and that
the RVs Y 1 , Y 2 , . . . , Y n are defined by Y = AX, where the (n × n) matrix A is nonsingular. Show
that the joint density of the Y i s is given by
1
f Y (y) =
| det A| f (
X A −1 y ) for y ∈ R n .
Hence or otherwise show carefully that if X 1 and X 2 are independent RVs with PDFs f 1 and f 2
respectively, then the PDF of Y = X 1 + X 2 is given by
or equivalently by
f Y (y) =
f Y (y) =
∫ ∞
−∞
∫ ∞
−∞
f 1 (y − z)f 2 (z)dz
f 1 (z)f 2 (y − z)dz
IID
If X i ∼ Exp(1), i = 1, 2, then what is the distribution of X 1 + X 2
for −∞ < y < ∞
for −∞ < y < ∞
2. Suppose that Z 1 and Z 2 are i.i.d. random variables with standard Normal N(0, 1) distributions.
Define the random vector (X 1 , X 2 ) by:
where σ 1 , σ 2 > 0 and −1 ≤ ρ ≤ 1.
X 1 = µ 1 + σ 1 Z 1 , X 2 = µ 2 + σ 2
[
ρZ 1 + √ 1 − ρ 2 Z 2
]
,
(a) Show that E[X 1 ] = µ 1 , E[X 2 ] = µ 2 , Var[X 1 ] = σ 2 1, Var[X 2 ] = σ 2 2, and corr[X 1 , X 2 ] = ρ.
(b) Find E[X 2 |X 1 ] and Var[X 2 |X 1 ].
(c) Derive the joint PDF f(x 1 , x 2 ).
(d) Find the distribution of [X 2 |X 1 ]. Hence or otherwise show that two r.v.s. with a joint bivariate
Normal distribution are independent if and only if they are uncorrelated.
(e) Now suppose that σ 1 = σ 2 . Show that the RVs Y 1 = X 1 +X 2 and Y 2 = X 1 −X 2 are independent.
3. Suppose that X and Y have the joint density
1
f X,Y (x, y) =
√
2π σ X σ Y 1 − ρ
2
(
[ (x
1 −
× exp −
2 ( µX
1 − ρ 2)
σ X
) 2 ( )( ) ( ) ]) 2 x − µX y − µY y − µY
− 2ρ
+
.
σ X σ Y σ Y
(a) Show by substituting u = (x−µ X )/σ X and v = (y −µ Y )/σ Y followed by w = (u−ρv)/ √ 1 − ρ 2 ,
or otherwise, that f X,Y does indeed integrate to 1.
(b) Show that the ‘joint MGF’ M X,Y (s, t) = E [ exp(sX + tY ) ] is given by
(c) Show that
M X,Y (s, t) = exp [ µ X s + µ Y t + 1 2 (σ2 Xs 2 + 2ρσ X σ Y st + σ 2 Y t 2 ) ] .
&
∂M X,Y
∂s
∣ = µ X ,
s,t=0
∂ 2 M X,Y
∂s 2 ∣
∣∣∣s,t=0
= µ 2 X + σ 2 X,
∂2 M X,Y
∂s∂t
∣ = µ X µ Y + ρσ X σ Y .
s,t=0
(d) Guess the formula for the MGF M X (s) of X, where X ∼ MVN (µ, V).
4. Suppose that (X 1 , X 2 ) have a bivariate Normal distribution. Show that any linear combination
Y = a 0 + a 1 X 1 + a 2 X 2 has a univariate Normal distribution.
31

3.5 The Multivariate Normal Distribution
Definition 3.3 (Multivariate Normal distribution)
Let µ = (µ 1 , µ 2 , . . . , µ p ) be a p-vector, and let V be a symmetric positive-definite (p × p) matrix.
Then the multivariate probability density defined by
f X (x; µ, V) =
1
√
(2π)p |V| exp[ − 1 2 (x − µ)T V −1 (x − µ) ] (3.9)
is called a multivariate Normal PDF with mean vector µ and variance-covariance matrix V.
Comments
1. Expression 3.9 is a natural generalisation of the univariate Normal density, with V taking the rôle of
σ 2 in the exponent, and its determinant |V| taking the rôle of σ 2 in the ‘normalising constant’ that
makes the whole thing integrate to 1. Many of the properties of the MVN distribution are guessable
from properties of the univariate Normal distribution—in particular, it’s helpful to think of 3.9 as
‘exponential of a quadratic’.
2. The statement ‘X = (X 1 , X 2 , . . . , X p ) has a multivariate Normal distribution with mean vector µ
and variance-covariance matrix V’ may be written
X ∼ N (µ, V), X ∼ MVN(µ, V), X ∼ N p (µ, V), or X ∼ MVN p (µ, V).
3. The mean vector µ is sometimes called just the mean, and the variance-covariance matrix V is
sometimes called the dispersion matrix, or simply the variance matrix or covariance matrix.
4. µ = EX, (or equivalently, componentwise, EX i = µ i , i = 1, 2, . . . , p). This fact should be obvious
from the name ‘mean vector’, and can be proved in various ways, e.g. by differentiating a multivariate
generalization of the MGF, or simply by symmetry.
5. V = E ( (X − µ)(X − µ) T ) = E(XX T ) − µµ T , i.e.
⎛
E(XX T ) − µµ T = E ⎜
⎝
X1 2 X 1 X 2 . . . X 1 X p
X 2 X 1 X2 2 . . . X 2 X p
.
. . .. .
⎞ ⎛
⎟
⎠ − ⎜
⎝
µ 2 1 µ 1 µ 2 . . . µ 1 µ p
µ 2 µ 1 µ 2 2 . . . µ 2 µ p
.
. . .. .
⎞
⎟
⎠
X p X 1 X p X 2 . . . X 2 p
µ p µ 1 µ p µ 2 . . . µ 2 p
=
⎛
⎜
⎝
E ( )
X1 2 ⎞
− µ
2
1 E(X 1 X 2 − µ 1 µ 2 . . . E(X 1 X p ) − µ 1 µ p
E(X 2 X 1 ) − µ 2 µ 1 E ( )
X2 2 − µ
2
2 . . . E(X 2 X p ) − µ 2 µ p
.
.
.
..
⎟
.
E(X p X 1 ) − µ p µ 1 E(X p X 2 ) − µ p µ 2 . . . E ( ⎠
)
Xp
2 − µ
2
p
= V =
⎛
⎜
⎝
⎞
v 11 v 12 . . . v 1p
v 21 v 22 . . . v 2p
.
. . ..
⎟
. ⎠ ,
v p1 v p2 . . . v pp
say,
from which it follows that
⎛
V = ⎜
⎝
σ1 2 ρ 12 σ 1 σ 2 . . . ρ 1p σ 1 σ p
ρ 12 σ 1 σ 2 σ2 2 . . . ρ 2p σ 2 σ p
.
. . .. .
⎞
⎟
⎠ , (3.10)
ρ 1p σ 1 σ p ρ 2p σ 2 σ p . . . σ 2 p
32

where σ i is the standard deviation of X i and ρ ij is the correlation between X i and X j . Again these
results can be proved using a multivariate generalization of the MGF.
6. The p-dimensional MVN p (µ, V) distribution can therefore be parametrised by—
NB. —a total of 1 2p(p + 3) parameters.
p means µ i ,
p variances σi 2, and
1
2 p(p − 1) correlations ρ ij
7. Given n random vectors X i = (X i1 , X i2 , . . . , X ip ) IID
∼ MVN (µ, V), i = 1, 2, . . . , n,
a set of minimal sufficient statistics for the unknown parameters is given by:
&
n∑
X ij j = 1, . . . , p,
i=1
n∑
Xij 2 j = 1, . . . , p,
i=1
n∑
X ij X ik j = 2, . . . , p, k = 1, . . . , (j − 1), ⎪⎭
i=1
⎫
⎪⎬
(3.11)
and MLEs for µ and V are given by:
or, in matrix notation,
̂µ j = 1 ∑
X ij , (3.12)
n
i
̂σ j 2 = 1 ∑
(X ij − ̂µ j ) 2 , (3.13)
n
i
∑
1
n i
̂ρ jk =
(X ij − ̂µ j )(X ik − ̂µ k )
, (3.14)
̂σ ĵσ k
̂µ = 1 n
̂V = 1 n
= 1 n
n∑
X i , (3.15)
i=1
n∑
(X i − ̂µ)(X i − ̂µ) T (3.16)
i=1
n∑
X i X T i − ̂µ̂µ T . (3.17)
i=1
8. The fact that V is positive-definite implies various (messy!) constraints on the correlations ρ ij .
9. Surfaces of constant density form concentric (hyper-)ellipsoids (concentric hyper-spheres in the case
of the standard MVN distribution). In particular, the contours of a bivariate Normal density form
concentric ellipses (or concentric circles for the standard bivariate Normal).
10. It can be proved that all conditional and marginal distributions of a MVN are themselves MVN. The
proof of this important fact is quite straightforward, quite tedious, and mercifully omitted from this
course.
33

3.6 Distributions Related to the MVN
Because of the CLT, the MVN distribution is important throughout statistics. For example, the joint
distribution of the MLEs ̂θ 1 , ̂θ 2 , . . . , ̂θ p of unknown parameters θ 1 , θ 2 , . . . , θ p will under reasonable conditions
tend to a MVN as the size of the sample from which ̂θ = (̂θ1 , ̂θ 2 , . . . , ̂θ p ) T was calculated increases.
Therefore various distributions arising from the MVN by transformation are also important.
Throughout this Section we shall usually denote independent standard Normal RVs by Z i , i.e.:
Z i
IID
∼ N (0, 1), i = 1, 2, . . .
i.e. Z = (Z 1 , Z 2 , . . . , Z n ) T ∼ MVN (0, I).
Exercise 3.4
Show that if a is a constant (n × 1) column vector, B is a constant nonsingular (n × n) matrix, and Z =
(Z 1 , Z 2 , . . . , Z n ) T is a random n-vector with a MVN (0, I) distribution, then Y = a+BZ ∼ MVN ( a, BB T ) .
‖
3.6.1 The Chi-squared Distribution
Definition 3.4 (Chi-squared Distribution)
If Z i
IID
∼ N(0, 1) for i = 1, 2, . . . , n, then the distribution of
X = Z 2 1 + Z 2 2 + · · · + Z 2 n
is called a Chi-squared distribution on n degrees of freedom, and we write X ∼ χ 2 n.
Comments
1. In particular, if Z ∼ N (0, 1), then Z 2 ∼ χ 2 1.
2. The above construction of the χ 2 n distribution shows that if X ∼ χ 2 m, Y ∼ χ 2 n, and X ⊥ Y ,
then (X + Y ) ∼ χ 2 m+n.
This summation property accounts for the importance and usefulness of the χ 2 distribution:
essentially a squared length is split into two orthogonal components, as in Pythagoras’ theorem.
3. If X ∼ χ 2 n, then the (unmemorable) density of X can be shown to be
with f X (x) = 0 for x ≤ 0.
f X (x) =
1
2 n/2 Γ(n/2) x(n/2)−1 e −x/2 for x > 0, (3.18)
Comparing this with the definition of a Gamma distribution (MSA) shows that a Chi-squared distribution
on n degrees of freedom is just a Gamma distribution with α = n/2 and β = 1/2 (in the
usual parametrisation).
4. It can be shown that if X ∼ χ 2 n then EX = n and VarX = 2n.
Note that this implies that E[X/n] = 1 and Var[X/n] = 2/n.
5. The χ 2 distributions are positively skewed—for example, χ 2 2 is just an exponential distribution with
mean 2. However, because of the CLT, the χ 2 n distribution tends (slowly!) to Normality as n → ∞.
6. The PDF 3.18 cannot be integrated analytically except for the special case n = 2. Therefore the
CDFs of χ 2 n distributions for various n are given in standard Statistical Tables.
34

Figure 3.2: Chi-squared distributions for 1, 2, 5 & 20 d.f. Vertical lines show the 2.5%, 16%, 50%, 84%
and 97.5% points (which for N(0, 1) are at −2, −1, 0, 1, 2).
3.6.2 Student’s t Distribution
Definition 3.5 (t Distribution)
If Z ∼ N(0, 1), Y ∼ χ 2 n and Y ⊥ Z, then the distribution of
X =
Z √
Y/n
is called a (Student’s) t distribution on n degrees of freedom, and we write X ∼ t n .
Comments
1. The shape of the t distribution is like that of a Normal, but with heavier tails (since there is variability
in the denominator of t as well as in the Normally-distributed numerator Z).
However, as n → ∞, the denominator becomes more and more concentrated around 1, so (loosely
speaking!) ‘t n → N(0, 1) as n → ∞’.
2. The (highly unmemorable) PDF of X ∼ t n can be shown to be
f X (x) = Γ( (n + 1)/2 ) (
√ 1 + x 2 /n ) −(n+1)/2
nπ Γ(n/2)
for −∞ < x < ∞. (3.19)
35

Figure 3.3: t distributions for 1, 2, 5 & 20 d.f. Vertical lines show the 2.5%, 16%, 50%, 84% and 97.5%
points.
3. The t distribution on 1 degree of freedom is also called the Cauchy distribution—note that it arises
as the distribution of Z 1 /Z 2 where Z i
IID
∼ N (0, 1).
The Cauchy distribution is infamous for not having a mean. More generally, only the first n − 1
moments of the t n distribution exist.
IID
4. Note that if X i ∼ N (0, σ 2 ), then the RV
T =
X 1
√∑ n
i=2 X2 i /(n − 1)
has a t n−1 distribution, and is a measure of the length of X 1 compared to the root mean square
length of the other X i s.
i.e. if X has a spherical MVN (0, σ 2 I) distribution, then we would expect T not to be too large. This
is, in effect, how the t distribution usually arises in practice.
5. The PDF 3.19 cannot be integrated analytically in general (exception: n = 1 d.f.). The CDF must
be looked up in Statistical Tables or approximated using a computer.
36

3.6.3 Snedecor’s F Distribution
Definition 3.6 (F Distribution)
If Y ∼ χ 2 m, Z ∼ χ 2 n and Y ⊥ Z, then the distribution of
X = Y/m
Z/n
is called an F distribution on m & n degrees of freedom, and we write X ∼ F m,n .
Figure 3.4: F distributions for selected d.f. Vertical lines show the 2.5%, 16%, 50%, 84% and 97.5% points.
Comments
1. Note that the numerator Y/m and denominator Z/n of X both have mean 1. Therefore, provided
both m and n are large, X will usually take values around 1.
2. If X ∼ F m,n , then the (extraordinarily unmemorable) density of X can be shown to be
with f X (x) = 0 for x ≤ 0.
f X (x) = Γ( (m + n)/2 ) m m/2 n n/2
Γ(m/2) Γ(n/2)
×
x (m/2)−1
(mx + n) (m+n)/2 for x > 0, (3.20)
37

3.7 Problems
1. Let Z ∼ N (0, 1) & Y = Z 2 , and let φ(·) & Φ(·) denote the PDF & CDF respectively of the standard
Normal N (0, 1) distribution.
(a) Show that F Y (y) = Φ( √ y) − Φ(− √ y).
(b) Express f Y (y) in terms of φ( √ y).
(c) Hence show that
(d) Find the MGF of Y .
f Y (y) = 1 √
2π
y −1/2 e −y/2 for y > 0.
2. Using Formula 3.18 for the PDF of the χ 2 distribution, show that if X ∼ χ 2 n then the MGF of X is
M X (t) = (1 − 2t) −n/2 . (3.21)
Deduce that if X ∼ χ 2 m & Y ∼ χ 2 n with X ⊥ Y , then (X + Y ) ∼ χ 2 m+n.
IID
3. Given Z 1 , Z 2 ∼ N (0, 1), what is the probability that the point (Z 1 , Z 2 ) lies
(a) in the square {(z 1 , z 2 ) | −1 < z 1 < 1 & −1 < z 2 < 1},
(b) in the circle {(z 1 , z 2 ) | (z 2 1 + z 2 2) < 1}
4. Let Z 1 , Z 2 , . . . be independent random variables, each with mean 0 and variance 1, and let µ i , σ i and
ρ ij be constants with −1 ≤ ρ ij ≤ 1. Let
and define X i = µ i + σ i Y i , i = 1, 2.
Y 1 = Z 1 ,
Y 2 = ρ 12 Z 1 +
√
1 − ρ 2 12 Z 2,
(a) Show that E[X i ] = µ i , Var[X i ] = σi
2 (i = 1, 2), and that ρ 12 is the correlation between X 1 and
X 2 .
(b) Find constants c 0 , c 1 , c 2 and c 3 such that
Y 3 = c 0 + c 1 Z 1 + c 2 Z 2 + c 3 Z 3
has mean 0, variance 1, and correlations ρ 13 & ρ 23 with Y 1 and Y 2 respectively.
(c) Hence show that the random vector Z = (Z 1 , Z 2 , Z 3 ) T with zero mean vector and identity
variance-covariance matrix can be transformed to give a random vector X = (X 1 , X 2 , X 3 ) T with
specified first and second moments, subject to constraints on the correlations corr[X i , X j ] = ρ ij
including
ρ 2 12 + ρ 2 13 + ρ 2 23 ∈ [0, 1 + 2ρ 12 ρ 13 ρ 23 ].
(d) What can you say about the distribution of X when Z has a standard trivariate Normal distribution
and ρ 2 12 + ρ 2 13 + ρ 2 23 is at one of the extremes of its allowable range (i.e. 0 or 1 + 2ρ 12 ρ 13 ρ 23 )
From Warwick ST217 exam 2001
5. Suppose that the RVs O i have independent Poisson distributions: O i ∼ Poi(np i ), i = 1, . . . , k, where
∑ k
i=1 p i = 1.
(a) Find EO i and Var O i . Hence or otherwise show that E[O i − np i ] = 0 and Var[O i − np i ] = np i .
(b) Define the RV N by N = ∑ k
i=1 O i. What is the distribution of N
(c) Define the RVs E i = Np i , i = 1, . . . , k. Show that EE i = np i and VarE i = np 2 i .
(d) By writing E[O 1 E 1 ] = p 1
(
E[O
2
1 ] + E[O 1
∑ k
i=2 O i] ) , or otherwise, show that Cov(O 1 , E 1 ) = np 2 1.
(e) Deduce that the RV (O i − E i ) has mean 0 and variance np i (1 − p i ) for i = 1, . . . , k.
38

6. Suppose that Y i are independent RVs with Poisson distributions: Y i ∼ Poi(λ i ), i = 1, . . . , k.
(a) Assuming that λ i is large, what is the approximate distribution of Z i = (Y i − λ i )/ √ λ i
(b) Hence or otherwise show that if all the λ i s are large, then the RV X = ∑ k
i=1 (Y i − λ i ) 2 /λ i has
approximately a χ 2 k distribution.
7. Let Z = (Z 1 , Z 2 , . . . , Z m+n ) T ∼ MVN m+n (0, I).
(a) Describe the distribution of Y = Z/
√ ∑m+n
1
Z 2 i .
(b) Show that the RV X = (n ∑ m
1 Y 2
i )/(m ∑ m+n
m+1 Y 2
i ) has an F m,n distribution.
(c) Hence show that if Y = (Y 1 , Y 2 , . . . , Y m+n ) T has any continuous spherically symmetric distribution
centred at the origin, then X = (n ∑ m
1 Y i 2)/(m
∑ m+n
m+1 Y i 2)
has an F m,n distribution.
8. (a) Define a multivariate standard Normal distribution N(0, I), where I denotes the identity matrix.
Given Z = (Z 1 , Z 2 , . . . , Z n ) T ∼ N(0, I), write down functions of Z (i.e. transformed random
variables) having
i. a chi-squared distribution on (n − 1) degrees of freedom, and
ii. a t distribution on (n − 1) degrees of freedom.
(b) Let Z = (Z 1 , Z 2 , . . . , Z n ) T have a multivariate standard Normal distribution, and let Z =
∑ n
i=1 Z i/n.
Also let A = (a ij ) be an n × n orthogonal matrix, i.e. AA T = A T A = I, and define the random
vector Y = (Y 1 , Y 2 , . . . , Y n ) T by Y = AZ.
Quoting any properties of probability distributions that you require, show the following:
i. Show that ∑ n
i=1 Y i
2
= ∑ n
i=1 Z2 i .
ii. Show that Y ∼ N(0, I).
iii. Show that for suitable choices of k i , i = 1, . . . , n (where k i > 0 for all i), the following
matrix A is orthogonal, and find k i :
⎛
⎞
k 1 −k 1 0 . . . 0 0
k 2 k 2 −2k 2 . . . 0 0
.
A =
. . . .. .
.
.
⎜ k n−2 k n−2 k n−2 . . . −(n − 2)k n−2 0
⎟
⎝ k n−1 k n−1 k n−1 . . . k n−1 −(n − 1)k n−1
⎠
k n k n k n . . . k n k n
iv. With the above definition of A, show that ∑ n−1
i=1 Y i
2 = ∑ n
i=1 (Z i − Z) 2 and that Y n = √ n Z.
v. Hence show that the RVs Z and ∑ n
i=1 (Z i − Z) 2 are independent and have N (0, 1/n) and
χ 2 n−1 distributions respectively.
IID
vi. Suppose that X 1 , X 2 , . . . , X n ∼ N (µ, σ 2 ), and define the random variables
X =
S 2 =
n∑
X i /n,
i=1
1
n − 1
n∑
(X i − X) 2 ,
i=1
T = X − µ √
S2 /n .
Show that T has a t distribution on n − 1 degrees of freedom.
From Warwick ST217 exam 2003
39

9. Suppose that X has a χ 2 n distribution with PDF given by Formula 3.18. Find the mean, mode &
variance of X, and an approximate variance-stabilising transformation.
10. Let z(m, n, P ) denote the P % point of the F m,n distribution. Without looking in statistical tables,
what can you say about the relationships between the following values:
(a) z(2, 2, 50) and z(20, 20, 50), (b) z(2, 20, 50) and z(20, 2, 50),
(c) z(2, 20, 16) and z(20, 2, 84), (d) z(20, 20, 2.5) and z(20, 20, 97.5)
11. Let the random variables X and Y have finite first and second moments.
(a) By considering the quadratic (tX − Y ) 2 , or otherwise, prove the Cauchy-Schwarz inequality:
(
E(XY )
) 2
≤ E
(
X
2 ) E ( Y 2) ,
with equality if and only if P(Y = cX) = 1 for some constant c.
(b) Hence or otherwise prove the following
i.
Var(X + Y ) ≤
( √Var(X)
+
√
Var(Y )
) 2
.
Under what conditions does equality hold
ii. If the random variable Z has a finite variance, then E ( |Z| ) is also finite.
Show that the converse is untrue, i.e. give an example of a probability distribution D such
that if Z ∼ D then E ( |Z| ) is finite but the variance of Z does not exist.
From Warwick ST217 exam 2002
IID
12. Suppose that Z i ∼ N (0, 1), i = 1, 2, . . . What is the distribution of the following RVs
(a)
(b)
(c)
(d)
(e)
X 4 =
X 5 =
X 1 = Z 1 + Z 2 − Z 3
X 2 = Z 1 + Z 2
Z 1 − Z 2
X 3 = (Z 1 − Z 2 ) 2
(Z 1 + Z 2 ) 2
(
(Z1 + Z 2 ) 2 + (Z 1 − Z 2 ) 2)
2
2Z 1
√
Z
2
2 + Z 2 3 + Z2 4 + Z2 5
(f)
X 6 = (Z 1 + Z 2 + Z 3 )
√
Z
2
4 + Z 2 5 + Z2 6
40

stating under what conditions σ 2 X , µ Y , µ T and σ 2 T exist. From Warwick ST217 exam 1998
(g)
X 7 =
3(Z 1 + Z 2 + Z 3 + Z 4 ) 2
(Z 1 + Z 2 − Z 3 − Z 4 ) 2 + (Z 1 − Z 2 + Z 3 − Z 4 ) 2 + (Z 1 − Z 2 − Z 3 + Z 4 ) 2
(h)
X 8 = 2Z 2 1 + (Z 2 + Z 3 ) 2
13. For each of the RVs X i defined in the previous question, use Statistical Tables to find c i (i = 1 . . . 8)
such that P(X i > c i ) = 0.95.
14. Show that the PDFs of the t and F distributions (definitions 3.5 & 3.6) are indeed given by formulae
3.19 & 3.20.
15. (a) Define the Standard Multivariate Normal distribution MVN (0, I).
(b) Given Z = (Z 1 , Z 2 , . . . , Z m+n ) T ∼ MVN (0, I), write down transformed random variables X(Z),
T (Z) and Y (Z) with the following distributions:
i. X ∼ χ 2 n,
ii. T ∼ t n ,
iii. Y ∼ F m,n .
(c) Given that the PDF of X ∼ χ 2 n is
f X (x) =
and f X (x) = 0 elsewhere, show that
i. E[X] = n,
ii. E[X 2 ] = n 2 + 2n, and
iii. E[1/X] = 1/(n − 2) (provided n > 2).
(d) Hence or otherwise find
i. the variance σX 2 of X ∼ χ2 n,
ii. the mean µ Y of Y ∼ F m,n and
iii. the mean µ T and variance σT 2 of T ∼ t n,
1
2 n/2 Γ(n/2) x(n/2)−1 e −x/2 for x > 0,
Theory is often just practice with the hard bits left out.
J. M. Robson
Get a bunch of those 3–D glasses and wear them at the same time. Use enough to get it up to
a good, say, 10– or 12–D.
Rod Schmidt
The Normal . . . is the Ordinary made beautiful; it is also the Average made lethal.
Peter Shaffer
Symmetry, as wide or as narrow as you define is meaning, is one idea by which man through
the ages has tried to comprehend and create order, beauty and perfection.
Hermann Weyl
41

Chapter 4
Inference for Multiparameter Models
4.1 Introduction: General Concepts
4.1.1 Modelling
Given a random vector X = (X 1 , X 2 , . . . , X p ), we can describe the joint distribution of the X i s by the
CDF F X (x) or, usually more conveniently, by the PMF or PDF f X (x).
Interrelationships between the X i s can be described using
1. marginals F i (x i ), f i (x i ), F ij (x i , x j ), etc.,
2. conditionals G i (x i |x j , j ≠ i), g i (x i |x j , j ≠ i), G ij (x i , x j |x k , k ≠ i, j), etc.,
3. conditional expectations E[X i |X j ], Var[X i |X j ], etc.
Often F X (x) is assumed to lie in a family of probability distributions:
where Ω Θ is the ‘parameter space’.
F = {F (x|θ) | θ ∈ Ω Θ } (4.1)
The process of formulating, choosing within, & checking the reasonableness of, such families F, is called
statistical modelling (or probability modelling, or just modelling).
Exercise 4.1
The data-set in Table 4.1, plotted in Figure 1.1 (page 2), shows patients’ blood pressures before and after
treatment. Suggest some reasonable models for the data.
‖
4.1.2 Data
In practice, we typically have a set of data in which d variables are measured on each of n ‘cases’ (or
‘individuals’ or ‘units’):
D =
case.1
case.2
.
case.n
var.1 var.2 · · · var.d
⎛
⎞
x 11 x 12 · · · x 1d
x 21 x 22 · · · x 2d
⎜
⎝
.
. . ..
⎟
. ⎠
x n1 x n2 · · · x nd
(4.2)
42

Patient Systolic Diastolic
Number before after change before after change
1 210 201 -9 130 125 -5
2 169 165 -4 122 121 -1
3 187 166 -21 124 121 -3
4 160 157 -3 104 106 2
5 167 147 -20 112 101 -11
6 176 145 -31 101 85 -16
7 185 168 -17 121 98 -23
8 206 180 -26 124 105 -19
9 173 147 -26 115 103 -12
10 146 136 -10 102 98 -4
11 174 151 -23 98 90 -8
12 201 168 -33 119 98 -21
13 198 179 -19 106 110 4
14 148 129 -19 107 103 -4
15 154 131 -23 100 82 -18
Table 4.1: Supine systolic and diastolic blood pressures of 15 patients with moderate
hypertension (high blood pressure), immediately before and 2 hours after taking 25mg
of the drug captopril.
Data from HSDS, set 72
Definition 4.1 (Data Matrix)
A set of data D arranged in the form of 4.2 is called a data matrix or a cases-by-variables array.
The data-set D is assumed to be a representative sample (of size n) from an underlying population of
potential cases. This population may be actual, e.g. the resident population of England & Wales at noon
on June 30th 1993, or purely theoretical/hypothetical, e.g. MVN(µ, V).
Exercise 4.2
Table 4.2 presents data on ten asthmatic subjects, each tested with 4 drugs. Describe various ways that
the data might be set out as a data matrix for analysis by a statistical computing package.
‖
4.1.3 Statistical Inference
Statistical inference is the art/science of using the sample to learn about the population (and hence,
implicitly, about future samples).
Typically we use statistics (properties of the sample)
to learn about parameters (properties of the population).
This activity might be:
1. Part of analysing a formal probability model,
e.g. calculating the MLEs ̂θ of θ, after making an assumption as in Expression 4.1, or
2. Purely to summarise the data as a part of ‘data analysis’ (Section 4.2),
IID
For example, given X 1 , X 2 , . . . , X n ∼ F X (unknown), the statistics
∑
Xi = X,
∑
(Xi − X) 2 ,
∑
(Xi − X) 3
S 1 = 1 n
S 2 = 1 n
S 3 = 1 n
43

Patient number
Drug Time 1 2 3 4 5 6 7 8 9 10
P −5 mins 0.0 2.3 2.4 1.9 1.6 4.8 0.6 2.7 0.9 1.3
+15 mins 3.8 9.2 5.4 3.3 4.2 15.1 1.3 6.7 4.2 3.1
C −5 mins 0.5 1.0 2.0 1.1 2.1 6.8 0.6 3.1 1.5 3.0
+15 mins 2.0 5.3 7.5 6.4 4.1 9.1 0.6 14.8 2.4 2.3
D −5 mins 0.8 2.3 0.8 0.8 1.2 9.6 1.1 9.7 0.8 4.9
+15 mins 2.4 4.8 2.4 1.9 1.2 12.5 1.7 12.5 4.3 8.1
K −5 mins 0.2 1.7 2.2 0.1 1.7 9.2 0.6 12.7 1.1 2.8
+15 mins 0.4 3.4 2.0 1.3 3.4 6.7 1.1 12.5 2.7 5.7
Table 4.2: NCF (Neutrophil Chemotactic Factor) of ten individuals, each tested with
4 drugs: P (Placebo), C (Clemastine), D (DSCG), K (Ketotifen). On a given day,
an individual was administered the chosen drug, and his NCF measured 5 minutes
before, and 15 minutes after, being given a ‘challenge’ of allergen.
Data from Dr. R. Morgan of Bart’s Hospital
provide measures of location, scale and skewness.
Note that here we’re implicitly estimating the corresponding population quantities
µ X = EX, E [ (X − µ X ) 2] , E [ (X − µ X ) 3] ,
and using these as measures of population location, scale and skewness. Without a formal probability
model, it can be hard to judge whether these or some other measures may be most appropriate.
In both cases, the CLT & its generalisations (to higher dimensions and to ‘near-independence’) show that,
under reasonable conditions, the joint distribution of the statistics of interest, such as ̂θ or (S1 , S 2 , S 3 ), is
approximately MVN. This approximation improves if
1. the sample size n → ∞, and/or
2. the joint distribution of the random variables being summed (e.g. the original random vectors
X 1 , X 2 , . . . , X n ) is itself close to MVN.
QUESTIONS: How should we interpret this How should we try to link probability models to reality
4.2 Data Analysis
Data analysis is the art of summarising data while attempting to avoid probability theory.
For example, you can calculate summary statistics such as means, medians, modes, ranges, standard
deviations etc., thus summarising in a few numbers the main features of a possibly huge data-set. For
example, the (0%, 25%, 50%, 75%, 100%) points of the data distribution (i.e. minimum, lower quartile,
median, upper quartile and maximum) form the five-number summary, and the inter-quartile range (IQR =
upper quartile - lower quartile) is a measure of spread, containing the ‘middle 50%’ of the data.
These summaries can be formalised as follows
Definition 4.2 (Order statistics)
Given RVs X 1 , X 2 , . . . , X n , one can order them and denote the smallest of the X i s by X (1) , the second
smallest by X (2) , etc. Then X (k) is called the kth order statistic.
44

Thus X (1) , X (2) , . . . , X (n) are a permutation of X 1 , X 2 , . . . , X n , and x (n) , the observed value of X (n) , denotes
the largest observed value in a sample of size n.
Given ordered data x (1) ≤ x (2) ≤ · · · ≤ x (n) , one can define:
Definition 4.3 (Sample median)
⎧
⎪⎨
x M =
⎪⎩
[
1
2
x ( n
2
x ( )
n+1 if n is odd,
2 ]
) + x ( n
2 + 1)
if n is even.
We can always write x M = x ( n
2 + ) 1 , provided we adopt the following convention:
2
1. If the number in brackets is exactly half-way between two integers, then take the average of the two
corresponding order statistics.
2. Otherwise round the bracketed subscript to the nearest integer, and take the corresponding order
statistic.
Similarly the quartiles etc. can be formally defined as follows:
Definition 4.4 (Sample lower quartile)
Definition 4.5 (Sample upper quartile)
Definition 4.6 (100p th sample percentile)
x L = x ( n
4 + ) 1 ,
2
x U = x ( 3n
4 + ) 1 ,
2
x 100p% = x ( pn
100 + ) 1 ,
2
Definition 4.7 (Five number summary)
(
x(1) , x L , x M , x U , x (n)
)
.
4.3 Classical Inference
4.3.1 Introduction
In ‘classical statistical inference’, the typical procedure is:
1. Choose a family F of models indexed by θ (formula 4.1).
2. Assume temporarily that the true distribution lies in F
i.e. data D ∼ F (d|θ) for some true but unknown parameter vector θ ∈ Ω Θ .
3. Compare possible models according to some criterion of compatibility between the model & the data
(equivalently, between the population & the sample).
4. Assess the chosen model(s), and go back to step (1) or (2) if the model proves inadequate.
45

Comments
1. Step 1 is a compromise between
(a) what we believe is the true underlying mechanism that produced the data, and
(b) what we can do mathematically.
If in doubt, keep it simple.
2. Step 2, by assuming a true θ exists, implicitly interprets probability as a property of Nature
e.g. a ‘fair’ coin is assumed to have an intrinsic property: if you toss it n times, then the proportion
of ‘heads’ tends to 1/2 as n → ∞.
Thus probability represents a ‘long-run relative frequency’.
3. Most statistical computer packages currently use the classical approach, and we’ll mainly be using
classical inference in MSB.
4. There are many possible criteria at step 3. For example, hypothesis-testing and likelihood approaches
are both discussed briefly below.
4.3.2 Point Estimation (Univariate)
Given RVs X = (X 1 , X 2 , . . . , X n ), a point estimator for an unknown parameter Θ ∈ Ω Θ is simply a function
̂Θ(X) taking values in the parameter space Ω Θ . Once data X = x are obtained, one can calculate the
corresponding point estimate ̂θ = ̂Θ(x).
There are many plausible criteria for ̂Θ to be considered a ‘good’ estimator of Θ. For example:
1. Mean Squared Error
One would like the mean squared error (MSE) of ̂Θ to be small whatever the true value θ of Θ, where
MSE(̂Θ) = E [ (̂Θ − θ) 2] . (4.3)
In particular, an estimator ̂Θ has minimum mean squared error if
2. Unbiasedness
Definition 4.8 (Bias)
The bias of an estimator ̂Θ is
MSE(̂Θ) = min
̂Θ ′ MSE(̂Θ ′ ).
Bias(̂Θ) = E[̂Θ − θ|Θ = θ]. (4.4)
Exercise 4.3
Show that MSE(̂Θ) = Var(̂Θ) + (Bias ̂Θ) 2 .
‖
Definition 4.9 (Unbiasedness)
An estimator ̂Θ for a parameter Θ is called unbiased if E[̂Θ|Θ = θ] = θ for all possible true
values θ of Θ.
46

Example
IID
Given a random sample X 1 , X 2 , . . . , X n , i.e. X i ∼ F X (x), where F X is a member of some family F
of probability distributions,
(a) X = ∑ n
i=1 X i/n is an unbiased estimate of the mean µ X = EX of F X .
(b) More generally, any statistic of the form ∑ n
i=1 w iX i , where ∑ n
i=1 w i = 1, is an unbiased estimate
of µ X .
(c) ̂σ 2 1 = ∑ n
i=1 (X i − X) 2 /(n − 1) is an unbiased estimate of the variance σ 2 X of F X, but
(d) ̂σ 2 2 = ∑ n
i=1 (X i − X) 2 /n is NOT an unbiased estimate of the variance σ 2 X of F X.
3. Efficiency & Minimum Variance Unbiased Estimation
Given two unbiased estimators ̂Θ 1 & ̂Θ 2 for a parameter Θ, the efficiency of ̂Θ 1 relative to ̂Θ 2 is
defined by
Eff(̂Θ 1 , ̂Θ 2 ) = Var(̂Θ 1 )
Var(̂Θ 2 ) . (4.5)
Definition 4.10 (MVUE)
The Minimum Variance Unbiased Estimator of a parameter Θ is the unbiased estimator ̂Θ, out
of all possible unbiased estimators, that has minimum variance.
Example
IID
Given X i ∼ F X (x) ∈ F, the family of all probability distributions with finite mean & variance, it can
be shown that
(a) X is the MVUE of the mean µ X = EX of F X , and
(b) ∑ n
i=1 (X i − X) 2 /(n − 1) is the MVUE of the variance σ 2 X of F X.
Note that there are major problems with using MVUE as a criterion for estimation:
(a) The MVUE may not exist (e.g. in general there is no unbiased estimator for the underlying
standard deviation σ X of X).
(b) The MVUE may exist but be nonsensical (see Problems).
(c) Even if the MVUE exists and appears reasonable, other (biased) estimators may be better by
other criteria, for example by having smaller mean squared error, which is much more important
in practice than being unbiased.
4. Consistency
Definition 4.11 (Consistency)
A sequence of estimators ̂Θ 1 , ̂Θ 2 , . . . is consistent for Θ ∈ Ω Θ if, for all ɛ > 0 and for all θ ∈ Ω Θ ,
lim P(|̂Θ n − θ| > ɛ|Θ = θ) = 0.
n→∞
5. Sufficiency
̂Θ(X 1 , . . . X n ) is sufficient for Θ if the conditional distribution of (X 1 , . . . X n ) given ̂Θ = ̂θ & Θ = θ
does not depend on θ. See MSA.
6. Maximum likelihood
See MSA.
7. Invariance
See Casella & Berger, page 300.
47

8. The ‘plug in’ property
If θ is a specified property of the CDF F (x), then ̂θ is the corresponding property of the empirical
CDF
̂F (x) = 1 n × (number of X i ≤ x i ). (4.6)
For example (assuming the named quantities exist):
(a) the sample mean ̂θ = x = ∑ x i /n is the plug-in estimate of the population mean θ = EX,
(b) the sample median is the plug-in estimate of the population median F −1 (0.5).
4.3.3 Hypothesis Testing (Introduction)
In this approach you
1. Choose a statistic T that has a known distribution F 0 (t) if the true parameter value is θ = θ 0 (for
some particular parameter value θ 0 of interest). The statistic T should provide a measure of the
discrepancy of the data D from what would be reasonable if θ = θ 0 .
2. Test the hypothesis ‘θ = θ 0 ’ using the tail probabilities of F 0 .
An example is the ‘Chi-squared’ statistic used in MSA. Hypothesis testing will be covered in more detail
in chapter 5.
Some problems with the standard hypothesis testing approach are:
1. In practice, we don’t really believe that θ = θ 0 is ‘true’ and all other possible values of θ are ‘false’;
instead we just wish to adopt ‘θ = θ 0 ’ as a convenient assumption, because it’s as good as, and
simpler than, other models.
2. If we really do want to make a decision [e.g. to give drug ‘A’ or drug ‘B’ to a particular patient],
then we should weigh up the possible consequences.
3. It’s hard to create appropriate hypothesis tests in complex situations, such as to test whether or not
θ lies in a particular subset Ω 0 of the parameter space Ω.
Unfortunately, real life is a complex situation.
4.3.4 Likelihood Methods
Use the likelihood function
{ P(D|θ) (discrete case)
L(θ; D) =
(constant) × f(D|θ) (continuous case),
(4.7)
or equivalently the log-likelihood or ‘support’ function
l(θ; D) = log L(θ; D) (4.8)
as a measure of the compatibility between data D and parameter θ.
In particular, the MLE corresponds to the particular ∈ F that is most compatible with the data D.
F̂θ
Likelihood underlies the most useful general approaches to statistics:
1. It can handle several parameters simultaneously.
2. The CLT implies that in many cases the log-likelihood will be approximately quadratic in θ (at least
near the MLE).
This makes both theory and numerical computation easier.
48

However, there are difficulties with basing inference solely on likelihood:
1. How should we handle ‘nuisance parameters’ (i.e. components θ i that we’re not interested in)
Note that it makes no sense to integrate over values of θ i to get a ‘marginal likelihood’ for the other
θ j s, since L(θ; d) is NOT a probability density or probability function—we would get a different
marginal likelihood if we reparametrised say by θ i ↦→ log θ i .
2. A more fundamental problem is that likelihood takes no account of how far-fetched the model might
be (‘high likelihood’ does NOT mean ‘likely’!)
This suggests that in practice we may wish to incorporate information not contained in the likelihood:
1. Prior information/Expert opinion: Are there external reasons for doubting some values of θ more
than others
2. For decision-making: How relatively important are the possible consequences of our inferences
[e.g. an innocent person is punished / a murderer walks free].
4.4 Problems
1. How might the mortality data in Tables 1.1 and 1.2 (pages 8 & 9) be set out as a data matrix
2. Suppose that ̂θ(X 1 , . . . X n ) is unbiased. Show that ̂θ is consistent iff lim n→∞
(
Var(̂θ(X1 , . . . X n )) ) = 0.
IID
3. Given X i ∼ F X (x), where F X is a member of some family F of probability distributions, show that
(a) Any statistic of the form ∑ n
i=1 w iX i , where ∑ n
i=1 w i = 1, is an unbiased estimate of µ X = EX,
(b) The mean X = ∑ n
i=1 X i/n is the unique UMVUE of this form,
(c) ̂σ 2 = ∑ n
i=1 (X i − X) 2 /(n − 1) is an unbiased estimate of the variance σ 2 X of F X.
4. The number of mistakes made each lecture by a certain lecturer follow independent Poisson distributions,
each with mean λ > 0.
You decide to attend the Monday lecture, note the number of mistakes X, and use X to estimate
the probability p that there will be no mistakes in the remaining two lectures that week.
(a) Show that p = exp(−2λ).
(b) Show that the only unbiased estimator of p (and hence, trivially, the MVUE), is
{
1 if X is even,
̂p =
−1 if X is odd.
(c) What is the maximum likelihood estimator of p
(d) Discuss (briefly) the relative merits of the MLE and the MVUE in this case.
5. Let T be an unbiased estimator for g(θ), let S be a sufficient statistic for θ, and let φ(S) = E[T |S].
Prove the Rao-Blackwell theorem:
φ(S) is also an unbiased estimator of g(θ), and Var[φ(S)|θ] ≤ Var[T |θ], for all θ,
and interpret this result.
49

6. (a) Explain what is meant by an unbiased estimator for an unknown parameter θ.
(b) Show, using moment generating functions or otherwise, that if X 1 & X 2 have independent
Poisson distributions with means λ 1 & λ 2 respectively, then their sum (X 1 + X 2 ) follows a
Poisson distribution with mean (λ 1 + λ 2 ).
(c) A particular sports game comprises four ‘quarters’, each lasting 15 minutes, and a statistician
attending the game wishes to predict the probability p that no further goals will be scored before
full time.
The statistician assumes that the numbers X k of goals scored in the kth quarter follow independent
Poisson distributions, each with (unknown) mean λ, so that
P(X k = x) = λx
x! e−λ
(k = 1, 2, 3, 4; x = 0, 1, 2, . . .).
Suppose that the statistician makes his prediction halfway through the match (i.e. after observing
X 1 = x 1 & X 2 = x 2 ). Show that an unbiased estimator of p is
{
1 if (x1 + x
T =
2 ) = 0,
0 otherwise.
(d) Suppose the statistician also made a prediction after 15 minutes. Show that in this case the
ONLY unbiased estimator of p given X 1 = x 1 is
{ 2
x 1
if x
T =
1 is even,
if x 1 is odd.
−2 x1
(e) What are the maximum likelihood estimators of p after 15 and after 30 minutes
(f) Briefly compare the advantages of maximum likelihood and unbiased estimation for this situation.
7. (a) Show that for any random variables X and Y ,
i. E[Y ] = E [ E[Y |X] ] ,
ii. Var[Y ] = E [ Var[Y |X] ] + Var [ E[Y |X] ] .
(b) Explain what is meant by a minimum variance unbiased estimator (MVUE).
From Warwick ST217 exam 1997
(c) Let S be a sufficient statistic for a parameter θ, let T be an unbiased estimator for φ(θ), and
define W = E[T |S]. Show that
i. W is an unbiased estimator for φ(θ), and
ii. Var[W ] ≤ Var[T ] for all θ.
Deduce that a MVUE, if one exists, must be a function of a sufficient statistic.
(d) Let X 1 , X 2 , . . . , X n be IID random variables with a geometric distribution, i.e.
P(X i = x) = θ(1 − θ) x−1 x = 1, 2, . . . .
i. Show that S = ∑ n
i=1 X i is a sufficient statistic for θ, and find the maximum likelihood
estimator ̂θ of θ.
ii. Define T by
{ 1 if X1 = 1,
T =
0 otherwise.
What is E[T ]
iii. Find E[T |S], and briefly discuss its advantages & disadvantages compared to the maximum
likelihood estimator ̂θ.
From Warwick ST217 exam 2001
50

8. (a) For random variables X (continuous) and Y (discrete), define
i. the joint cumulative distribution function (CDF) F X,Y (x, y),
ii. the conditional probability mass function (PMF) f Y |X (y|x) of Y given X = x, and
iii. the conditional expectation E[Y |X] of Y given X.
(b) Show that E[g(Y )] = E [ [E[g(Y )|X] ] , for any function g(·) such that the expectations exist.
(c) People arrive randomly at a public phone box at a rate of one per minute, and the length Λ of a
phone call has an exponential distribution with mean µ minutes. Thus the number Y of people
arriving at the phone box during a call has PMF
f Y |Λ (y|λ) = λy e −λ
, (y = 0, 1, 2, . . .),
y!
where Λ has PDF
f Λ (λ) = 1 exp(−λ/µ), (λ > 0).
µ
Show that the conditional distribution of Y given Λ = λ has moment generating function (MGF)
M Y |Λ (t) = E [ e tY |Λ = λ ] given by
M Y |Λ (t) = exp ( λ(e t − 1) ) .
Hence or otherwise find the MGF M Y (t) = E [ e tY ] of Y .
(d) By differentiation of M Y (·), or otherwise, show that Y is an unbiased estimator of µ, and find
its variance.
From Warwick ST217 exam 2000
9. (a) Explain what is meant by an unbiased estimator for an unknown parameter θ.
(b) A statistician working for a CD manufacturer assumes that the number of faults on each CD
follow independent Poisson distributions with mean θ, and hence the number of faults X on a
faulty CD follows a truncated Poisson distribution, i.e.
P(X = x) =
e −θ θ x
(1 − e −θ ) x!
(x = 1, 2, . . .).
Find the mean and variance of this distribution.
(c) The statistician counts the number of faults x on a faulty CD, and is interested in φ = 1 − e −θ ,
the probability that a random CD is fault-free. Show that the only unbiased estimator T (X) of
φ takes the form:
{ 0 if x is odd,
T (x) =
2 if x is even.
(d) Find an equation satisfied by the maximum likelihood estimator of θ given observations x 1 , x 2 , . . . , x n
on n faulty CDs.
(e) Comment briefly on the relative advantages and disadvantages of unbiased estimation and maximum
likelihood estimation for φ.
From Warwick ST217 exam 2002
51

IID
10. Given X i ∼ Poi(θ), compare the following possible estimators for θ in terms of unbiasedness, consistency,
relative efficiency, etc.
̂θ 1 = X = 1 n∑
X k , ̂θ6 = (̂θ 1 +
n
̂θ 5 )/2,
( k=1 )
̂θ 2 = 1 n∑
100 + X k , ̂θ7 = median(X 1 , X 2 , . . . , X n ),
n
k=1
̂θ 3 = 1 2 (X 2 − X 1 ) 2 , ̂θ8 = mode(X 1 , X 2 , . . . , X n ),
̂θ 4 = 1 n∑
(X k − X) 2 2
n∑
, ̂θ9 =
kX k ,
n
n(n + 1)
k=1
k=1
1
n∑
̂θ 5 =
(X k − X) 2 1
n∑
, ̂θ10 =
X k .
n − 1
n − 1
k=1
11. [Light relief] Discuss the following possible defence submission at a murder trial:
‘The supposed DNA match placing the defendant at the scene of the crime would have arisen with
even higher probability if the defendant had a secret identical twin
[the more people with that DNA, the more chances of getting a match at the crime scene].
‘Now assume that my client has been cloned θ times, θ ∈ {0, 1, . . . , n} for some n > 0. Clearly the
larger the value of θ, the higher the probability of obtaining the observed DNA results
[every increase in θ means another clone who might have been at the scene of the crime].
‘Therefore the MLE of θ is n.
‘But then, even assuming somebody with my client’s DNA committed this terrible crime, the probability
that it was my client is only 1/(n + 1) (under reasonable assumptions).
‘Therefore you cannot say that my client is, beyond a reasonable doubt, guilty.
‘The defence rests.’
k=2
4.5 Bayesian Inference
4.5.1 Introduction
Classical inference regards probability as a property of physical objects (e.g. a ‘fair coin’).
An alternative interpretation uses probability to represent an individual’s (lack of) understanding of an
uncertain situation.
Examples
1. ‘I have no reason to suspect that “heads” or “tails” are more likely. Therefore, by symmetry, my
current probability for this particular coin’s coming down “heads” is 1/2.’
2. ‘I doubt the accused has any previously-unknown identical siblings. I’d bet 100,000 to 1 against’
(i.e. if θ is the number of identical siblings, then my probability for θ > 0 is 1/100001).
Different people, with different knowledge, can legitimately have different probabilities for real-world events
(therefore it’s good discipline to say ‘my probability for. . . ’ rather than ‘the probability of. . . ’).
As you learn. your probabilities can be continually updated using Bayes’ theorem, i.e.
P(A|B) =
P(B|A) × P(A)
P(B)
(4.9)
(
assuming P(B) is positive, and using the fact that P(A&B) = P(A|B)P(B) = P(B|A)P(A)
)
.
52

The Bayesian approach to statistical inference treats all uncertainty via probability, as follows:
1. You have a probability model for the data, with PMF p(D|Θ).
2. Your prior PMF for Θ (i.e. your PMF for Θ based on a combination of expert opinion, previous
experience, and your own prejudice), is p(θ).
3. Then Bayes’ theorem says
p(θ|D) =
p(D|θ) p(θ)
p(D)
or, since once the data have been obtained p(D) is a constant,
p(θ|D) ∝ p(D|θ) p(θ)
∝ L(θ; D) p(θ)
i.e. ‘posterior probability ∝ ‘likelihood’ × ‘prior’
(4.10)
Formula 4.10 also applies in the continuous case, in which case p(·) represents a PDF.
Comments
1. Further applications to decision theory are given in the third year course ST301.
2. Note that if θ = (θ 1 , θ 2 , . . . , θ p ), then p(θ|D) is a p-dimensional function, and may prove difficult to
manipulate, summarise or visualise.
3. Treating all uncertainty via probability has the advantage that one-off events (e.g. management
decisions, or the results of horse races) can be handled. However, it’s not at all obvious that all
uncertainty can be treated via probability!
4. As with Classical inference, a Bayesian analysis of a problem should involve checking whether the
assumptions underlying p(D|θ) and p(θ) are reasonable, and rethinking & reanalysing the model if
necessary.
Exercise 4.4
Describe the Bayesian approach to statistical inference, denoting the data by x, the prior by f Θ (θ), and
the likelihood by L(θ; x) = f X|Θ (x|θ).
‖
4.6 Nonparametric Methods
IID
Standard Classical and Bayesian methods make strong assumptions, e.g. X i ∼ F (x|θ) for some θ ∈ Ω.
Assumptions of independence are critical (what aspects of the problem provide information about other
aspects)
Assumptions about the form of probability distributions are often less important, at least provided the
sample size n is large. However, there are exceptions to this:
1. It might be that the probability distribution encountered in practice is fundamentally different from
the form assumed in our model. For example, some probability distributions are so ‘heavy-tailed’
that their means don’t exist ( e.g. the Cauchy distribution with f(x) = 1/π(1 + x 2 ), x ∈ R ) .
2. Some data may be recorded incorrectly, or there may be a few atypically large/small data values
(‘outliers’), etc.
3. In any case, what if n is small and the CLT can’t be invoked
53

‘Nonparametric’ methods don’t assume that the actual probability distribution F (·|θ) lies in a particular
parametric family F; instead they make more general assumptions, for example
1. ‘F (x) is symmetric about some unknown value Θ’.
Note that this may be a reasonable assumption even if EX doesn’t exist.
Θ is the (unknown) median of the population, i.e. P(X < Θ) = P(X > Θ).
Therefore one could estimate Θ by the median of the data (though better methods may exist).
2. ‘F (x, y) is such that if (X i , Y i ) IID
∼ F , (i = 1, 2), then P(Y 1 < Y 2 |X 1 < X 2 ) = 1/2’.
This is a nonparametric version of the statement ‘X & Y are uncorrelated’.
Many statistical methods involve estimating means, as we’ll see in the rest of the course (t-tests, linear
regression, many MLEs etc.)
Corresponding nonparametric methods typically involve medians—or equivalently, various probabilities.
Exercise 4.5
Suppose that X has a continuous distribution. Show that a test of the statement ‘median of X is θ 0 ’ is
equivalent to a test of the statement ‘P(X < θ 0 ) = 1/2’.
If X i are IID, what is the distribution of R = (number of X i < θ T ), where θ T is the true value of θ
‖
Other nonparametric methods involve ranking the data X i : replacing the smallest X i by 1, the next
smallest by 2, etc. Classical statistical methods can then be applied to the ranks. Note that the effect of
outliers will be reduced.
Example
Given data (X i , Y i ), i = 1, . . . , n from a continuous bivariate distribution, ‘Spearman’s rank correlation’
(often written ρ S ) can be calculated as follows:
1. replace the X i values by their ranks R i ,
2. similarly replace the Y i values by their ranks S i ,
3. calculate the usual (‘product-moment’ or ‘Pearson’s’) correlation between the R i s and S i s.
Comments
1. If the distribution of the original RVs is not continuous, then some data values may be repeated (‘tied
ranks’). Repeated X i s are given averaged ranks (for example, if there are two X i with the smallest
value, then they are each given rank 1.5 = (1 + 2)/2).
2. If X ⊥ Y , so the ‘true’ ρ S is zero, then the distribution of the calculated ρ S is easily approximated
(using the standard formulae for ∑ n
i=1 ik ).
3. ‘Easily approximated’ does not necessarily mean ‘well approximated’!
4. Most books give another formula for ρ S , which is equivalent unless there are tied ranks, but which
obscures the relationship with the standard product-moment correlation
∑ (xi − x)(y i − y)
ρ = √∑ (xi − x) ∑ 2 (y i − y) . 2
5. Other, perhaps better, types of nonparametric correlation have been defined (‘Kendall’s τ’).
54

4.7 Graphical Methods
A vital part of data analysis is to plot the data using bar-charts, histograms, scatter diagrams etc. Plotting
the data is important no matter what further formal statistical methods will be used:
1. It enables you to ‘get a feel for’ the data,
2. It helps you look for patterns and anomalies,
3. It helps in checking assumptions (such as independence, linearity or Normality).
Many useful plots can be easily churned out using a computer, though sometimes you have to devise original
plots to display the data in the most appropriate way.
Exercise 4.6
The following table shows 66 measurements on the speed of light, made by S. Newcomb in 1882. Values
are the times in nanoseconds (ns), less 24,800 ns, for light to travel from his laboratory to a mirror and
back. Values are to be read row-by-row, thus the first to observations are 24,828 ns and 24,826 ns.
28 26 33 24 34 -44 27 16 40 -2
29 22 24 21 25 30 23 29 31 19
24 20 36 32 36 28 25 21 28 29
37 25 28 26 30 32 36 26 30 22
36 23 27 27 28 27 31 27 26 33
26 32 32 24 39 28 24 25 32 25
29 27 28 29 16 23
Produce a histogram, a Normal probability plot and a time plot of Newcomb’s data. Decide which (if any)
observations to ignore, and produce a normal probability plot of the remaining reduced data set. Finally
compare the mean of this reduced data set with (i) the mean and (ii) the 10% trimmed mean of the original
data. Solution: Plots are shown in Figure 4.1. There are clearly 2 large outliers, but the time plot also
suggests that the 6th to 10th observations are unusually variable, and that the last two observations are
atypically low (both being lower than the previous 20 observations).
The Normal probability plot is calculated by calculating y (i) (the sorted data) and z i as follows, and
plotting y (i) against z i .
i y (i) x i = (i−0.5)/n z i = Φ −1 (x i )
1 −44 0.0075 −2.434
2 −2 0.0224 −2.007
3 16 0.0373 −1.783
4 16 0.0522 −1.624
. . .
.
65 39 0.9776 2.007
66 40 0.9925 2.434
Omitting the first 10 and the last 2 recorded observations leaves a data-set where the Normality and
independence assumptions are much more reasonable—see plot (d) of Figure 4.1.
Location estimates are (i) 26.2, (ii) 27.4, (iii) 27.9. The trimmed mean is reasonably close to the mean of
observations 11–64.
‖
55

Figure 4.1: Plots of Newcomb’s data: (a) histogram, (b) Normal probability plot, (c) time plot, (d) Normal
probability plot of data after excluding the first 10 and last 2 observations.
4.8 Bootstrapping
‘Bootstrap’ methods have become increasingly used over the past few years.
question:
They address the general
‘What are the properties of the calculated statistics (e.g. MLEs ̂θ) given that the underlying
distributional assumptions may be false (and, in reality, will be false)’
Bootstrapping uses the observed data directly as an estimate of the underlying population, then uses
‘plug-in’ estimation, and typically involves computer simulation.
Several other computer-intensive approaches to statistical inference have also become very popular recently.
56

4.9 Problems
1. [Light relief] Discuss the following quote:
‘As a statistician, I want to use mathematics to help deal with practical uncertainty. The natural
mathematical way to handle uncertainty is via probability.
‘About the simplest practical probability statement I can think of is “The probability that a fair coin,
tossed at random, will come down ‘heads’ is 1/2”.
‘Now try to define “fair coin”, “at random” and “probability 1/2” without using subjective probability
or circular definitions.
‘Summary: if a practical probability statement is not subjective, then it must be tautologous, illdefined,
or useless.
‘Of course, for balance, some of the time I teach subjective methods, and some of the time I teach
useless methods :-).’
Ewart Shaw (Internet posting 13–Aug–1993).
2. (a) Plot the captopril data (Table 4.1), and suggest what sort of models seem reasonable.
(b) Roughly estimate from your graph(s) the effect of captopril (C) on systolic and diastolic blood
pressure (SBP & DBP).
(c) Suggest a single summary measure (SBP, DBP or a combination of the two) to quantify the
effect of treatment.
(d) Do you think a transformation of the data would be appropriate
(e) Comment on the number of parameters in your model(s).
(f) Calculate ρ S and ρ between ∆ S , the change (after-before) in SBP, and ∆ D , the change (afterbefore)
in DBP.
Suggest some advantages and disadvantages in using ρ S and ρ here.
(g) Calculate some further summary statistics such as means, variances, correlations and fivenumber
summaries, and comment on how useful they are as summaries of the data.
(h) Are there any problems in using the data to estimate the effect of captopril
information would be useful
What further
(i) What advantages/disadvantages would there be in using bootstrapping here, i.e. using the discrete
distribution that assigns probability 1/15 to each of the 15 points x 1 = (210, 201, 130, 125),
x 2 = (169, 165, 122, 121), . . . , x 15 = (154, 131, 100, 82) as an estimate of the underlying population,
and working out the properties of ρ S , ρ, etc. based on that assumption
57

Chapter 5
Hypothesis Testing
5.1 Introduction
A hypothesis is a claim about the real world; statisticians will be interested in hypotheses like:
1. ‘The probabilities of a male panda or a female panda being born are equal’,
2. ‘The number of flying bombs falling on a given area of London during World War II follows a Poisson
distribution’,
3. ‘The mean systolic blood pressure of 35-year-old men is no higher than that of 40-year-old women’,
4. ‘The mean value of Y = log(systolic blood pressure) is independent of X = age’
(i.e. E[Y |X = x] = constant).
These hypotheses can be translated into statements about parameters within a probability model:
1. ‘p 1 = p 2 ’,
2. ‘N ∼ Poi(λ) for some λ > 0’, i.e.: p n = P(N = n) = λ n exp(−λ)/n! (within the general probability
model p n ≥ 0 ∀n = 0, 1, . . .; ∑ p n = 1),
3. ‘θ 1 ≤ θ 2 ’ and
4. ‘β 1 = 0’ (assuming the linear model E[Y |x] = β 0 + β 1 x).
Definition 5.1 (Hypothesis test)
A hypothesis test is a procedure for deciding whether to accept a particular hypothesis as a reasonable
simplifying assumption, or to reject it as unreasonable in the light of the data.
Definition 5.2 (Null hypothesis)
The null hypothesis H 0 is the simplifying assumption we are considering making.
Definition 5.3 (Alternative hypothesis)
The alternative hypothesis H 1 is the alternative explanation(s) we are considering for the data.
Definition 5.4 (Type I error)
A type I error is made if H 0 is rejected when H 0 is true.
Definition 5.5 (Type II error)
A type II error is made if H 0 is accepted when H 0 is false.
58

Comments
1. In the first example above (pandas) the null hypothesis is H 0 : p 1 = p 2 .
2. The alternative hypothesis in the first example would usually be H 1 : p 1 ≠ p 2 , though it could also
be (for example)
(a) H 1 : p 1 < p 2 ,
(b) H 1 : p 1 > p 2 , or
(c) H 1 : p 1 − p 2 = δ for some specified δ ≠ 0.
5.2 Simple Hypothesis Tests
The simplest type of hypothesis testing occurs when the probability distribution giving rise to the data is
specified completely under the null and alternative hypotheses.
Definition 5.6 (Simple hypotheses)
A simple hypothesis is of the form H k : θ = θ k ,
i.e. the probability distribution of the data is specified completely.
Definition 5.7 (Composite hypotheses)
A composite hypothesis is of the form H k : θ ∈ Ω k ,
i.e. the parameter θ lies in a specified subset Ω k of the parameter space Ω Θ .
Definition 5.8 (Simple hypothesis test)
A simple hypothesis test tests a simple null hypothesis H 0 : θ = θ 0 against a simple alternative
H 1 : θ = θ 1 , where θ parametrises the distribution of our experimental random variables X =
X 1 , X 2 , . . . X n .
There may be many seemingly sensible approaches to testing a given hypothesis. A reasonable criterion
for choosing between them is to attempt to minimise the chance of making a mistake: incorrectly rejecting
a true null hypothesis, or incorrectly accepting a false null hypothesis.
Definition 5.9 (Size)
A test of size α is one which rejects the null hypothesis H 0 : θ = θ 0 in favour of the alternative
H 1 : θ = θ 1 iff
X ∈ C α where P(X ∈ C α | θ = θ 0 ) = α
for some subset C α of the sample space S of X.
Definition 5.10 (Critical region)
The set C α in Definition 5.9 is called the critical region or rejection region of the test.
Definition 5.11 (Power & power function)
The power function of a test with critical region C α is the function
β(θ) = P(X ∈ C α | θ),
and the power is β = β(θ 1 ), i.e. the probability that we reject H 0 in favour of H 1 when H 1 is true.
A hypothesis test typically uses a test statistic T (X), whose distribution is known under H 0 , and such that
extreme values of T (X) are more compatible with H 1 that H 0 .
Many useful hypothesis tests have the following form:
59

Definition 5.12 (Simple likelihood ratio test)
A simple likelihood ratio test (SLRT) of H 0 : θ = θ 0 against H 1 : θ = θ 1 rejects H 0 iff
X ∈ C ∗ α =
{
x ∣ L(θ 0; x)
}
L(θ 1 ; x) ≤ A α
where L(θ; x) is the likelihood of θ given the data x, and the number A α is chosen so that the size of
the test is α.
Exercise 5.1
IID
Suppose that X 1 , X 2 , . . . , X n ∼ N (θ, 1). Show that the likelihood ratio for testing H 0 : θ = 0 against
H 1 : θ = 1 can be written
λ(x) = exp [ n ( x − 2)] 1 .
Hence show that the corresponding SLRT of size α rejects H 0 when the test statistic T (X) = X satisfies
T > Φ −1 (1 − α)/ √ n.
‖
Comments
1. For a simple hypothesis test, both H 0 and H 1 are ‘point hypotheses’, each specifying a particular
value for the parameter θ rather than a region of the parameter space.
2. The size α is the probability of rejecting H 0 when H 0 is in fact true; clearly we want α to be small
(α = 0.05, say).
3. Clearly for a fixed size α of test, the larger the power β of a test the better.
However, there is an inevitable trade-off between small size and high power (as in a jury trial: the
more careful one is not to convict an innocent defendant, the more likely one is to free a guilty one
by mistake).
4. In practice, no hypothesis will be precisely true, so the whole foundation of classical hypothesis testing
seems suspect!
5. Regarding likelihood as a measure of compatibility between data and model, an SLRT compares the
compatibility of θ 0 and θ 1 with the observed data x, and accepts H 0 iff the ratio is sufficiently large.
6. One reason for the importance of likelihood ratio tests is the following theorem, which shows that
out of all tests of a given size, an SLRT (if one exists) is ‘best’ in a certain sense.
Theorem 5.1 (The Neyman-Pearson lemma)
Given random variables X 1 , X 2 , . . . , X n , with joint density f(x|θ), the simple likelihood ratio test of
a fixed size α for testing H 0 : θ = θ 0 against H 1 : θ = θ 1 is at least as powerful as any other test of
the same size.
Exercise 5.2
[Proof of Theorem 5.1] Prove the Neyman-Pearson lemma. Solution: Fix the size of the test to be α.
Let A be a positive constant and C 0 a subset of the sample space satisfying
(a) P(X ∈ C 0 | θ = θ 0 ) = α,
(b) X ∈ C 0 ⇐⇒ L(θ 0; x)
L(θ 1 ; x) = f(x|θ 0)
f(x|θ 1 ) ≤ A.
Suppose that there exists another test of size α, defined by the critical region C 1 , i.e.
60

C 0
C 1
Reject H 0 iff x ∈ C 1 , where P(x ∈ C 1 |θ = θ 0 ) = α.
Let B 1 = C 0 ∩ C 1 , B 2 = C 0 ∩ C c 1, B 3 = C c 0 ∩ C 1 .
B 2
B 1
B 3
Ω X
Figure 5.1: Proof of Neyman-Pearson lemma
Note that B 1 ∪ B 2 = C 0 , B 1 ∪ B 3 = C 1 , and B 1 , B 2 & B 3 are disjoint.
Let the power of the likelihood ratio test be I 0 = P(X ∈ C 0 | θ = θ 1 ),
and the power of the other test be I 1 = P(X ∈ C 1 | θ = θ 1 ).
We want to show that I 0 − I 1 ≥ 0.
But
I 0 − I 1 = ∫ C 0
f(x|θ 1 )dx − ∫ C 1
f(x|θ 1 )dx
= ∫ B 1∪B 2
f(x|θ 1 )dx − ∫ B 1∪B 3
f(x|θ 1 )dx
= ∫ B 2
f(x|θ 1 )dx − ∫ B 3
f(x|θ 1 )dx.
Also B 2 ⊆ C 0 , so f(x|θ 1 ) ≥ A −1 f(x|θ 0 ) for x ∈ B 2 ,
similarly B 3 ⊆ C c 0, so f(x|θ 1 ) ≤ A −1 f(x|θ 0 ) for x ∈ B 3 ,
Therefore
as required.
[ ∫
I 0 − I 1 ≥ A −1 B 2
f(x|θ 0 )dx − ∫ ]
B 3
f(x|θ 0 )dx
[ ∫
= A −1 C 0
f(x|θ 0 )dx − ∫ ]
C 1
f(x|θ 0 )dx
= A −1 [α − α] = 0
‖
5.3 Simple Null, Composite Alternative
Suppose that we wish to test the simple null hypothesis H 0 : θ = θ 0 against the composite alternative
hypothesis H 1 : θ ∈ Ω 1 .
The easiest way to investigate this is to imagine the collection of simple hypothesis tests with null hypothesis
H 0 : θ = θ 0 and alternative H 1 : θ = θ 1 , where θ 1 ∈ Ω 1 . Then, for any given θ 1 , an SLRT is the most
powerful test for a given size α. The only problem would be if different values of θ 1 result in different
SLRTs.
61

Definition 5.13 (UMP Tests)
A hypothesis test is called a uniformly most powerful test of H 0 : θ = θ 0 against H 1 : θ = θ 1 , θ 1 ∈ Ω 1 ,
if
1. There exists a critical region C α corresponding to a test of size α not depending on θ 1 ,
2. For all values of θ 1 ∈ Ω 1 , the critical region C α defines a most powerful test of H 0 : θ = θ 0
against H 1 : θ = θ 1 .
Exercise 5.3
Suppose that X 1 , X 2 , . . . , X n
IID
∼ N(0, σ 2 ).
(a) Find the UMP test of H 0 : σ 2 = 1 against H 1 : σ 2 > 1.
(b) Find the UMP test of H 0 : σ 2 = 1 against H 1 : σ 2 < 1.
(c) Show that no UMP test of H 0 : σ 2 = 1 against H 1 : σ 2 ≠ 1 exists.
‖
Comments
1. If a UMP test exists, then it is clearly the appropriate test to use.
2. Often UMP tests don’t exist!
3. A UMP test involves the data only via a likelihood ratio, so is a function of the sufficient statistics.
4. The critical region C α therefore often has a simple form, and is usually easily found once the distribution
of the sufficient statistics have been determined
(hence the importance of the χ 2 , t and F distributions).
5. The above three examples illustrate how important is the form of alternative hypothesis being considered.
The first two are one-sided alternatives whereas H 1 : σ 2 ≠ 1 is a two-sided alternative
hypothesis, since σ 2 could lie on either side of 1.
5.4 Composite Hypothesis Tests
The most general situation we’ll consider is where the parameter space Ω is divided into two subsets:
Ω = Ω 0 ∪ Ω 1 , where Ω 0 ∩ Ω 1 = ∅, and the hypotheses are H 0 : θ ∈ Ω 0 , H 1 : θ ∈ Ω 1 .
For example, one may want to test the null hypothesis that the data come from an exponential distribution
against the alternative that the data come from a more general gamma distribution. Note that here, as in
many other cases, dim(Ω 0 ) < dim(Ω 1 ) = dim(Ω).
One possible approach to this situation is to regard the maximum possible likelihood over θ ∈ Ω i as a
measure of compatibility between the data and the hypothesis H i (i = 0, 1). It’s therefore convenient to
define the following:
̂θ is the MLE of θ over the whole parameter space Ω,
̂θ 0 is the MLE of θ over Ω 0 , i.e. under the null hypothesis H 0 , and
̂θ 1 is the MLE of θ over Ω 1 , i.e. under the alternative hypothesis H 1 .
62

Note that ̂θ must therefore be the same as either ̂θ0 or ̂θ1 , since Ω = Ω 0 ∪ Ω 1 .
One might consider using the likelihood ratio criterion L(̂θ1 ; x)/L(̂θ0 ; x), by direct analogy with the SLRT.
However, it’s generally easier to use the equivalent ratio L(̂θ; x)/L(̂θ0 ; x):
Definition 5.14 (Likelihood Ratio Test (LRT))
A likelihood ratio test rejects H 0 : ̂θ ∈ Ω0 in favour of the alternative H 1 : ̂θ ∈ Ω1 = Ω \ Ω 0 iff
λ(x) = L(̂θ; x)
≥ λ, (5.1)
L(̂θ0 ; x)
where ̂θ is the MLE of θ over the whole parameter space Ω, ̂θ0 is the MLE of θ over Ω 0 , and the
value λ is fixed so that
sup
θ∈Ω 0
P(λ(X) ≥ λ|θ) = α
where α, the size of the test, is some chosen value.
Equivalently, the test criterion uses the log LRT statistic:
r(x) = l(̂θ; x) − l(̂θ0 ; x) ≥ λ ′ , (5.2)
where l(θ; x) = log L(θ; x), and λ ′ is chosen to give chosen size α = sup θ∈Ω0 P(r(X) ≥ λ ′ |θ).
Comments
1. The size α is typically chosen by convention to be 0.05 or 0.01.
2. Note that high values of the test statistic λ(x), or equivalently of r(x), are taken as evidence against
the null hypothesis H 0 .
3. The test given in Definition 5.14 is sometimes referred to as a generalized likelihood ratio test, and
Equation 5.1 a generalized likelihood ratio test statistic.
4. Equation 5.2 is often easier to work with than Equation 5.1—see the exercises and problems.
Exercise 5.4
IID
[Paired t-test] Suppose that X 1 , X 2 , . . . , X n ∼ N (µ, σ 2 ), and let X = ∑ X i /n, S 2 = ∑ (X i − X) 2 /(n − 1).
What is the distribution of T = X/(S/ √ n)
Is the test based on rejecting H 0 : µ = 0 for large T a likelihood ratio test
Assuming that the observed differences in diastolic blood pressure (after–before) are IID and Normally
distributed with mean δ D , use the captopril data (4.1) to test the null hypothesis H 0 : δ D = 0 against the
alternative hypothesis H 1 : δ D ≠ 0.
Comment: this procedure is called the paired t test
Exercise 5.5
[Two sample t-test] Suppose X 1 , X 2 , . . . , X m
IID
∼ N (µ X , σ 2 ) and Y 1 , Y 2 , . . . , Y n
IID
∼ N (µ Y , σ 2 ).
(a) Derive the LRT for testing H 0 : µ X = µ Y versus H 1 : µ X ≠ µ Y .
(b) Show that the LRT can be based on the test statistic
‖
where
T =
X √
− Y
1
S p
m + 1 n
. (5.3)
∑ m
Sp 2 i=1
=
(X i − X) 2 + ∑ n
i=1 (Y i − Y ) 2
. (5.4)
m + n − 2
63

(c) Show that, under H 0 , T ∼ t m+n−2 .
(d) Two groups of female rats were placed on diets with high and low protein content, and the gain
in weight (grammes) between the 28th and 84th days of age was measured for each rat, with the
following results:
High protein diet
134 146 104 119 124 161 107 83 113 129 97 123
Low protein diet
70 118 101 85 107 132 94
Using the test statistic T above, test the null hypothesis that the mean weight gain is the same under
both diets.
Comment: this is called the two sample t-test, and S 2 p is the pooled estimate of variance.
‖
Exercise 5.6
IID
[F -test] Suppose X 1 , X 2 , . . . , X m ∼ N (µ X , σX 2 ) and Y IID
1, Y 2 , . . . , Y n ∼ N (µ Y , σY 2 ), where µ X, µ Y , σ X and σ Y
are all unknown.
Suppose we wish to test the hypothesis H 0 : σX 2 = σ2 Y against the alternative H 1 : σX 2 ≠ σ2 Y .
(a) Let S 2 X = ∑ m
i=1 (X i − X) 2 and S 2 Y = ∑ n
i=1 (Y i − Y ) 2 .
What are the distributions of S 2 X /σ2 X and S2 Y /σ2 Y
(b) Under H 0 , what is the distribution of the statistic
V = S2 X /(m − 1)
/(n − 1)
S 2 Y
(c) Taking values of V much larger or smaller than 1 as evidence against H 0 , and given data with m = 16,
n = 16, ∑ x i = 84, ∑ y i = 18, ∑ x 2 i = 563, ∑ y 2 i = 72, test the null hypothesis H 0.
Comment: with the alternative hypothesis H 1 : σX 2 > σ2 Y , the above procedure is called an F test.
‖
Even in simple cases like this, the null distribution of the log likelihood ratio test statistic r(x) (5.2) can be
difficult or impossible to find analytically. Fortunately, there is a very powerful and very general theorem
that gives the approximate distribution of r(x):
Theorem 5.2 (Wald’s Theorem)
Let X 1 , X 2 , . . . , X n
IID
∼ f(x|θ) where θ ∈ Ω, and let r(x) denote the log likelihood ratio test statistic
r(x) = l(̂θ; x) − l(̂θ0 ; x),
where ̂θ is the MLE of θ over Ω and ̂θ0 is the MLE of θ over Ω 0 ⊂ Ω.
Then under reasonable conditions on the PDF (or PMF) f(·|·), the distribution of 2r(x) converges
to a χ 2 distribution on dim(Ω) − dim(Ω 0 ) degrees of freedom as n → ∞.
64

Comments
1. A proof is beyond the scope of this course, but may be found in e.g. Kendall & Stuart, ‘The Advanced
Theory of Statistics’, Vol. II.
2. Wald’s theorem implies that, provided the sample size is large, you only need tables of the χ 2
distribution to find the critical regions for a wide range of hypothesis tests.
Another important theorem, see Problem 3.7.8, page 39, is the following:
IID
Theorem 5.3 (Sample Mean and Variance of X i ∼ N (µ, σ 2 ))
IID
Let X 1 , X 2 , . . . , X n ∼ N(µ, σ 2 ). Then
1. X = ∑ X i /n and Y = ∑ (X i − X) 2 are independent RVs,
2. X has a N(µ, σ 2 /n) distribution,
3. Y/σ 2 has a χ 2 n−1 distribution.
Exercise 5.7
Suppose X 1 , X 2 , . . . , X n
IID
∼ N (θ, 1), with hypotheses H 0 : θ = 0 and H 1 : θ arbitrary.
Show that 2r(x) = nx 2 , and hence that Wald’s theorem holds exactly in this case.
‖
Exercise 5.8
Suppose now that X i ∼ N (θ i , 1), i = 1, . . . , n are independent, with null hypothesis H 0 : θ i = θ ∀i and
alternative hypothesis H 1 : θ i arbitrary.
Show that 2r(x) = ∑ n
i=1 (x i − x) 2 . and hence (quoting any other theorems you need) that Wald’s theorem
again holds exactly.
‖
65

5.5 Problems
1. Suppose that X ∼ Bin(n, p). Under the null hypothesis H 0 : p = p 0 , what are EX and VarX
Show that if n is large and p 0 is not too close to 0 or 1, then
X/n − p
√ 0
∼ N (0, 1)
p0 (1 − p 0 )/n approximately.
Out of 1000 tosses of a given coin, 560 were heads and 440 were tails. Is it reasonable to assume that
the coin is fair Justify your answer.
2. Out of 370 new-born babies at a Hospital, 197 were male and 173 female.
Test the null hypothesis H 0 : p < 1/2 versus H 1 : p ≥ 1/2, where p denotes the probability that a
baby born at the Hospital will be male.
Discuss any assumptions you make.
3. X is a single observation whose density is given by
{
(1 + θ)x
θ
if 0 < x < 1,
f(x) =
0 otherwise.
Find the most powerful size α test of H 0 : θ = 0 against H 1 : θ = 1.
Is there a U.M.P. test of H 0 : θ ≤ 0 against H 1 : θ > 0 If so, what is it
IID
4. Suppose X 1 , X 2 , . . . , X n ∼ N (µ, σ 2 ) with null hypothesis H 0 : σ 2 = 1 and alternative H 1 : σ 2 is
arbitrary. Show that the LRT will reject H 0 for large values of the test statistic r(x) = n(̂v−1−log ̂v),
where ̂v = ∑ n
i=1 (x i − x) 2 /n.
5. Let X 1 , . . . , X n be independent each with density
{
λx
f(x) =
−2 e −λ/x if x > 0,
0 otherwise,
where λ is an unknown parameter.
(a) Show that the UMP test of H 0 : λ = 1 2 against H 1 : λ > 1 2
is of the form:
‘reject H 0 if ∑ n
i=1 X−1 i ≤ A ∗ ’, where A ∗ is chosen to fix the size of the test.
(b) Find the distribution of ∑ n
i=1 X−1 i under the null & alternative hypotheses.
(c) You observe values 0.59, 0.36, 0.71, 0.86, 0.13, 0.01, 3.17, 1.18, 3.28, 0.49 for X 1 , . . . , X 10 .
Test H 0 against H 1 , & comment on the test in the light of any assumptions made.
6. (a) Define the size and power of a hypothesis test of a simple null hypothesis H 0 : θ = θ 0 against a
simple alternative hypothesis H 1 : θ = θ 1 .
(b) State and prove the Neyman-Pearson Lemma for continuous random variables X 1 , . . . , X n when
testing the null hypothesis H 0 : θ = θ 0 against the alternative H 1 : θ = θ 1 .
(c) Assume that a particular bus service runs at regular intervals of θ minutes, but that you do
not know θ. Assume also that the times you find you have to wait for a bus on n occasions,
X 1 , . . . , X n , are independent and identically distributed with density
{ θ
−1
if 0 ≤ x ≤ θ,
f(x|θ) =
0 otherwise.
i. Discuss briefly when the above assumptions would be reasonable in practice.
ii. Find the likelihood L(θ; x) for θ given the data (X 1 , . . . , X n ) = x = (x 1 , . . . , x n ).
iii. Find the most powerful test of size α of the hypothesis H 0 : θ = θ 0 = 20 against the
alternative H 1 : θ = θ 1 > 20.
66

From Warwick ST217 exam 1997
7. The following problem is quoted verbatim from Osborn (1979), ‘Statistical Exercises in Medical
Research’ :
A study of immunoglobulin levels in mycetoma patients in the Sudan involved 22 patients to be
compared to 22 normal individuals. The levels of IgG recorded for the 22 mycetoma patients are
shown below. The mean level for the normal individuals was calculated to be 1,477 mg/100ml before
the data for this group was lost overboard from a punt on the river Nile. Use the data below to
estimate the within group variance and hence perform a ‘t’ test to investigate the significance of the
difference between the mean levels of IgG in mycetoma patients and normals.
IgG levels (mg/100ml) in 22 mycetoma patients
1,047 1,135 1,350 1,122 1,345
1,377 1,375 804 1,062 1,204
1,210 1,067 1,032 1,002 1,053
1,103 907 960 960 936
1,270 1,230
IID
8. Let X 1 , X 2 , . . . , X n ∼ Exp(θ), i.e. f(x|θ) = θe −θx for θ ∈ (0, ∞).
Show that a likelihood ratio test for H 0 : θ ≤ θ 0 versus H 1 : θ > θ 0 has the form:
‘Reject H 0 iff θ 0 x < k, where k is given by α =
∫ nk
0
1
Γ(n) zn−1 e −z dz’.
Show that a test of this form is UMP for testing H 0 : θ = θ 0 versus H 1 : θ > θ 0 .
9. (a) Define the size and power function of a hypothesis test procedure.
Osborn (1979) 4.6.16
(b) State and prove the Neyman-Pearson lemma in the case of a test statistic that has a continuous
distribution.
IID
(c) Let X 1 , X 2 , . . . , X n ∼ N (µ, σ 2 ), where σ 2 is known. Find the likelihood ratio
f X (x|µ 1 )/f X (x|µ 0 )
and hence show that the most powerful test of size α for testing the null hypothesis H 0 : µ = µ 0
against the alternative H 1 : µ = µ 1 , for some µ 1 < µ 0 , has the form:
‘Reject H 0 if X < µ 0 + σ Φ −1 (α)/ √ n ’,
where X = ∑ n
i=1 X i/n is the sample mean, and Φ −1 (α) is the 100 α% point of the standard
Normal N (0, 1) distribution.
(d) Define a uniformly most powerful (UMP) test, and show that the above test is UMP for testing
H 0 : µ = µ 0 against H 1 : µ < µ 0 .
(e) What is the UMP test of H 0 : µ = µ 0 against H 1 : µ > µ 0
(f) Deduce that no UMP test of size α exists for testing H 0 : µ = µ 0 against H 1 : µ ≠ µ 0 .
(g) What test would you choose to test H 0 : µ = µ 0 against H 1 : µ ≠ µ 0 , and why
From Warwick ST217 exam 1999
10. A group of clinicians wish to study survival after heart attack, by classifying new heart attack patients
according to
(a) whether they survive at least 7 days after admission, and
(b) whether they currently smoke 10 or more cigarettes per day.
67

From previous experience, the clinicians predict that after N days the observed counts
Survive
Die
Smoker R 1 R 2
Non-smoker R 3 R 4
will follow independent Poisson distributions with means
Survive
Die
Smoker Nr 1 Nr 2
Non-smoker Nr 3 Nr 4
The clinicians intend to estimate the population log-odds ratio l = log(r 1 r 4 /r 2 r 3 ) by the sample
value L = log(R 1 R 4 /R 2 R 3 ), and they wish to choose N to give a probability 1 − β of being able to
reject the hypothesis H 0 : l = 0 at the 100α% significance level, when the true value of l is l 0 > 0.
Using the formula Var ( f(X) ) ≈ ( f ′ (EX) ) 2
Var(X), show that L has approximate variance
1
Nr 1
+ 1
Nr 2
+ 1
Nr 3
+ 1
Nr 4
,
and hence, assuming a Normal approximation to the distribution of L, that the required number of
days is roughly
N = 1 l 2 0
{ 1
r 1
+ 1 r 2
+ 1 r 3
+ 1 r 4
} {Φ −1 (α/2) + Φ −1 (β) } 2
,
where Φ is the standard Normal cumulative distribution function.
Comment critically on the clinicians’ method for choosing N.
From Warwick ST332 exam 1988
11. (a) Define the size and power of a hypothesis test, and explain what is meant by a simple likelihood
ratio test and by a uniformly most powerful test.
(b) Let X 1 , X 2 , . . . , X n be independent random variables, each having a Poisson distribution with
mean λ. Find the likelihood ratio test for testing H 0 : λ = λ 0 against H 1 : λ = λ 1 , where
λ 1 > λ 0 .
Show also that this test is uniformly most powerful.
(c) Twenty-five leaves were selected at random from each of six similar apple trees. The number of
adult female European red mites on each was counted, with the following results:
No. of mites 0 1 2 3 4 5 6 7
Frequency 70 38 17 10 9 3 2 1
Assuming that the number of mites per leaf follow IID Poisson distributions, and using a Normal
approximation to the Poisson distribution, carry out a test of size 0.05 of the null hypothesis
H 0 that the mean number of mites per leaf is 1.0, against the alternative H 1 that it is greater
than 1.0.
Discuss briefly whether the assumptions you have made in testing H 0 appear reasonable here.
From Warwick ST217 exam 2000
12. Hypothesis test procedures can be inverted to produce confidence intervals or more generally confidence
regions. Thus, given a size α test of the null hypothesis H 0 : θ = θ 0 , the set of all values θ 0
that would NOT be rejected forms a ‘100(1 − α)% confidence interval for θ’.
An amateur statistician argues as follows:
68

Suppose something starts at time t 0 and ends at time t 1 . Then at time t ∈ (t 0 , t 1 ), the
ratio r of its remaining lifetime (t 1 − t) to its current age (t − t 0 ), i.e.
r(t) = t 1 − t
t − t 0
,
is clearly a monotonic decreasing function of t. Also it is easy to check that r = 39 after
(1/40)th of the total lifetime, and that r = 1/39 after (39/40)th of the total lifetime.
Therefore, for 95% of something’s existence, its remaining lifetime lies in the interval
(
(t − t0 )/39, 39(t − t 0 ) ) ,
where t is the time under consideration, and t 0 is the time the thing came into existence.
The statistician is also an amateur theologian, and firmly believes that the World came into existence
6006 year ago. Using his pet procedure outlined above, he says he is ‘95% confident that the World
will end sometime between 154 years hence, and 234234 years hence’.
His friend, also an amateur statistician, says she has an even more general procedure to produce
confidence intervals:
In any situation I simply roll an icosahedral (fair 20-sided) die. If the die shows ‘13’ then I
quote the empty set ∅ as a 95% confidence interval, otherwise I quote the whole real line R.
She rolls the die, which comes up 13. She therefore says she is ‘95% confident that the World ended
before it even began (although presumably no-one has noticed yet).’
Discuss.
5.6 The Multinomial Distribution and χ 2 Tests
5.6.1 Multinomial Data
Definition 5.15 (Multinomial Distribution)
The multinomial distribution Mn(n, θ) is a probability distribution on points y = (y 1 , y 2 , . . . , y k ),
where y i ∈ {0, 1, 2, . . .}, i = 1, 2, . . . , k, and ∑ k
i=1 y i = n, with PMF
f(y 1 , y 2 , . . . , y k ) =
where θ i > 0 for i = 1, . . . , k, and ∑ k
i=1 θ i = 1.
n!
y 1 !y 2 ! · · · y k !
k∏
i=1
θ yi
i (5.5)
Comments
1. The multinomial distribution arises when one has n independent observations, each classified in one
of k ways (e.g. ‘eye colour’ classified as ‘Brown’, ‘Blue’ or ‘Other’; here k = 3).
Let θ i denote the probability that any given observation lies in category number i, and let Y i denote
the number of observations falling in category i. Then the random vector Y = (Y 1 , Y 2 , . . . , Y k ) has a
Mn(n, θ) distribution.
2. A binomial distribution is the special case k = 2, and is usually parametrised by p = θ 1 (so θ 2 = 1−p).
69

Exercise 5.9
By partial differentiation of the likelihood function, show that the MLEs ̂θ i of the parameters θ i of the
Mn(n, θ) satisfy the equations
y i
̂θ i
−
and hence that ̂θ i = y i /n for i = 1, . . . , k.
y k
1 − ∑ k−1
j=1 ̂θ j
= 0, (i = 1, . . . , k − 1)
‖
5.6.2 Chi-Squared Tests
Suppose one wishes to test the null hypothesis H 0 that, in the multinomial distribution 5.5, θ is some
function θ(φ) of another parameter φ. The alternative hypothesis H 1 is that θ is arbitrary.
Exercise 5.10
IID
Suppose H 0 is that X 1 , X 2 , . . . X n ∼ Bin(3, φ). Let Y i (for i = 1, 2, 3, 4) denote the number of observations
X j taking value i − 1. What is the null distribution of Y = (Y 1 , Y 2 , Y 3 , Y 4 )
‖
The log likelihood ratio test statistic r(X) is given by
k∑
r(X) = Y i log ̂θ i −
where ̂θ i = y i /n for i = 1, . . . , k.
i=1
k∑
Y i log θ i (̂φ) (5.6)
By Walds theorem, under H 0 , 2r(X) has approximately a χ 2 distribution:
k∑
2 Y i [log ̂θ i − log θ i (̂φ)] ∼ χ 2 k 1−k 0
(5.7)
where
̂θ i = Y i /n,
i=1
k 0 is the dimension of the parameter φ, and
k 1 = k − 1 is the dimension of θ under the constraint ∑ k
i=1 θ i = 1.
Comments
1. in Example 5.10, k = 4, k 0 = 1, k 1 = 3 and ̂φ = X = ∑ 4
i=1 Y i.
We would reject H 0 , that the sample comes from a Bin(3, φ) distribution for some φ, if 2r(x) is
greater than the 95% point of the χ 2 2 distribution, where r(x) is given in Formula 5.6.
2. It is straightforward to check, using a Taylor series expansion of the log function, that provided EY i
is large ∀ i,
k∑
2 Y i [log ̂θ
k∑ (Y i − µ i ) 2
i − log θ i (̂φ)] ≏
, (5.8)
µ i
i=1
where µ i = nθ i (̂φ) is the expected number of individuals (under H 0 ) in the ith category.
i=1
i=1
Definition 5.16 (Chi-squared Goodness of Fit Statistic)
X 2 =
k∑ (o i − e i ) 2
i=1
e i
, (5.9)
where o i is the observed count in the ith category and e i is the corresponding expected count under
the null hypothesis, is called the χ 2 goodness-of-fit statistic.
70

Comments
1. Under H 0 , X 2 has approximately a χ 2 distribution with number of degrees of freedom being (number
of categories) - 1 - (number of parameters estimated under H 0 ).
This approximation works well provided all the expected counts are reasonably large (say all are at
least 5).
2. This χ 2 test was suggested by Karl Pearson before the theory of hypothesis testing was fully developed.
5.7 Problems
1. In a genetic experiment, peas were classified according to their shape (‘round’ or ‘angular’) and
colour (‘yellow’ or ‘green’). Out of 556 peas, 315 were round+yellow, 108 were round+green, 101
were angular+yellow and 32 were angular+green.
Test the null hypothesis that the probabilities of these four types are 9/16, 3/16, 3/16 and 1/16
respectively.
2. A sample of 300 people was selected from a population, and classified into blood type (O/A/B/AB,
and Rhesus positive/negative), as shown in the following table:
O A B AB
Rh positive 82 89 54 19
Rh negative 13 27 7 9
The null hypothesis H 0 is that being Rhesus negative is independent of whether an individual’s blood
group is O, A, B or AB. Estimate the probabilities under H 0 of falling into each of the 8 categories,
and hence test the hypothesis H 0 .
3. The random variables X 1 , X 2 , . . . , X n are IID with P(X i = j) = p j for j = 1, 2, 3, 4, where ∑ p j = 1
and p j > 0 for each j = 1, 2, 3, 4.
Interest centres on the hypothesis H 0 that p 1 = p 2 and simultaneously p 3 = p 4 .
(a) Define the following terms
i. a hypothesis test,
ii. simple and composite hypotheses, and
iii. a likelihood ratio test.
(b) Letting θ = (p 1 , p 2 , p 3 , p 4 ), X = (X 1 , . . . , X n ) T with observed values x = (x 1 , . . . , x n ) T , and
letting y j denote the number of x 1 , x 2 , . . . , x n equal to j, what is the likelihood L(θ|x)
(c) Assume the usual regularity conditions, i.e. that the distribution of −2 log L(θ|x) tends to χ 2 ν
as the sample size n → ∞. What are the dimension of the parameter space Ω θ and the number
of degrees of freedom ν of the asymptotic chi-squared distribution
(d) By partial differentiation of the log-likelihood, or otherwise, show that the maximum likelihood
estimator of p j is y j /n.
(e) Hence show that the asymptotic test statistic of H 0 : p 1 = p 2 and p 3 = p 4 is
−2 log L(x) = 2
4∑
y j log(y j /m j ),
j=1
where m 1 = m 2 = (y 1 + y 2 )/2 and m 3 = m 4 = (y 3 + y 4 )/2.
(f) In a hospital casualty unit, the numbers of limb fractures seen over a certain period of time are:
71

Side
Left Right
Arm 46 49
Leg 22 32
Using the test developed above, test the hypothesis that limb fractures are equally likely to
occur on the right side as on the left side.
Discuss briefly whether the assumptions underlying the test appear reasonable here.
From Warwick ST217 exam 1998
Prudens quaestio dimidium scientiae.
Half of science is asking the right questions.
Roger Bacon
We all learn by experience, and your lesson this time is that you should never lose sight of the
alternative.
Sir Arthur Conan Doyle
One forms provisional theories and then waits for time or fuller knowledge to explode them.
Sir Arthur Conan Doyle
What used to be called prejudice is now called a null hypothesis.
A. W. F. Edwards
The conventional view serves to protect us from the painful job of thinking.
John Kenneth Galbraith
Science must begin with myths, and with the criticism of myths.
Sir Karl Raimund Popper
72

Chapter 6
Linear Statistical Models
6.1 Introduction
Definition 6.1 (Response Variable)
a response variable is a random variable Y whose value we wish to predict.
Definition 6.2 (Explanatory Variable)
An explanatory variable is a random variable X whose values can be used to predict Y .
Definition 6.3 (Linear Model)
A linear model is a prediction function for Y in terms of the values x 1 , x 2 , . . . , x k of X 1 , X 2 , . . . , X k
of the form
E[Y |x 1 , x 2 , . . . , x k ] = β 0 + β 1 x 1 + β 2 x 2 + · · · + β k x k (6.1)
Thus if Y 1 , Y 2 , . . . , Y n are the responses for cases 1, 2, . . . , n, and x ij is the value of X j (j = 1, . . . , k) for
case i, then
E[Y|X] = Xβ (6.2)
where
is the vector of responses,
is the matrix of explanatory variables, and
is the (unknown) parameter vector.
⎛
Y = ⎜
⎝
Y 1
Y 2
.
Y n
⎞
⎟
⎠
X = (x ij ) where x i0 = 1 for i = 1, . . . n,
⎛
β = ⎜
⎝
β 0
β 1
.
β k
⎞
⎟
⎠
73

Examples
Consider the captopril data (page 43), and let
X 1 = Diastolic BP before treatment, X 2 = Systolic BP before treatment,
X 3 = Diastolic BP after treatment, X 4 = Systolic BP after treatment,
Z 1 = 2X 1 + X 2 , Z 2 = 2X 3 + X 4 .
Some possible linear models of interest are:
1. Response Y = X 4 ,
(a) explanatory variable X 2
variable),
(this is a ‘simple linear regression model’, with just 1 explanatory
(b) explanatory variable X 3
(c) explanatory variables X 1 and X 2 (a ‘multiple regression model’).
2. Response Y = Z 2 ,
(a) explanatory variable Z 1
(b) explanatory variables Z 1 and Z 2 1 (a ‘quadratic regression model’).
Note how new explanatory variables may be obtained by transforming and/or combining old ones.
3. Looking just at the interrelationship between SBP and DBP at a given time:
Comments
(a) response Y = X 2 , explanatory variable X 1 ,
(b) response Y = X 1 , explanatory variable X 2 ,
(c) response Y = X 4 , explanatory variable X 3 , etc.
1. A linear relationship is the simplest possible relationship between response variables and explanatory
variables, so linear models are easy to understand, interpret and also to check for plausibility.
2. One can (in theory) approximate an arbitrarily complicated relationship by a linear model, for example
quadratic regression can obviously be extended to ‘polynomial regression’
3. Linear models have nice links with
• geometry,
• linear algebra,
• conditional expectations and variances,
• the Normal distribution.
E[Y |x] = β 0 + β 1 x + β 2 x 2 + · · · + β m x m .
4. Distributional assumptions (if any!) will typically be made ONLY about the response variable Y ,
NOT about the explanatory variables.
Therefore the model makes sense even if the X i s are chosen nonrandomly (‘designed experiments’).
5. The response variable Y is sometimes called the ‘dependent variable’, and the explanatory variables
are sometimes called ‘predictor variables’, ‘regressor variables’, or (very misleadingly) ‘independent
variables’.
74

6.2 Simple Linear Regression
Definition 6.4
A simple linear regression model is a linear model with one response variable Y and one explanatory
variable X, i.e. a model of the form
E[Y |x 1 ] = β 0 + β 1 x 1 . (6.3)
Typically in practice we have n data points (x i , y i ) for i = 1, . . . , n, and we want to predict a future
response Y from the corresponding observed value x of X.
Often there’s a natural candidate for which variable should be treated as the response:
1. X may precede Y in time, for example
(a) X is BP before treatment and Y is BP after treatment, or
(b) X is number of hours revision and Y is exam mark;
2. X may be in some way more fundamental, for example
(a) X is age and Y is height or
(b) X is height and Y is weight;
3. X may be easier or cheaper to observe, so we hope in future to estimate Y without measuring it.
In simple linear regression we don’t know β 0 or β 1 , but need to estimate them in order to predict Y by
Ŷ = ̂β 0 + ̂β 1 x.
To make accurate predictions we require the prediction error
to be small.
Y − Ŷ = Y − ̂β 0 + ̂β 1 x
This suggests that, given data (x i , y i ) for i = 1, . . . , n, we should fit ̂β 0 and ̂β 1 by simultaneously making
all the vertical deviations of the observed data points from the fitted line y = ̂β 0 + ̂β 1 x small.
The easiest way to do this is to minimise the sum of squared deviations ∑ (y i − ŷ i ) 2 , i.e. to use the ‘least
squares’ criterion.
6.3 Method of Least Squares
For simple linear regression,
ŷ i = β 0 + β 1 x i (i = 1, . . . , n) (6.4)
Therefore to estimate β 0 and β 1 by least squares, we need to minimise
Q =
n∑
[y i − (β 0 + β 1 x i )] 2 . (6.5)
i=1
Exercise 6.1
Show that Q in equation 6.5 is minimised at values β 0 and β 1 satisfying the simultaneous equations
β 0 n + β 1
∑
xi = ∑ y i ,
β 0
∑
xi + β 1
∑ x
2
i = ∑ x i y i ,
(6.6)
75

and hence that
̂β 1 =
∑
xi y i − n x y
∑ x
2
i − nx 2 , (6.7)
̂β 0 = y − ̂β 1 x. (6.8)
‖
Comments
1. Forming ∂ 2 Q/∂β 2 0, ∂ 2 Q/∂β 2 1 and ∂ 2 Q/∂β 0 β 1 verifies that Q is minimised at β = ̂β.
2. Equations 6.6 are called the ‘normal equations’ for β 0 and β 1 (‘normal’ as in ‘perpendicular’ rather
than as in ‘standard’ or as in ‘Normal distribution’).
3. y = ̂β 0 + ̂β 1 x is called the ‘least squares fit’ to the data.
4. From equations 6.7 and 6.8, the least squares fitted line passes through (x, y), the centroid of the
data points.
5. Concentrate on understanding and remembering the method for finding ̂β, rather than on memorising
the formulae 6.7 and 6.8 for ̂β 1 and ̂β 1 .
6. Geometrical interpretation
We have a vector y = (y 1 , y 2 , . . . , y n ) T of observed responses, i.e. a point in n-dimensional space,
together with a surface S representing possible joint predicted values under the model (for simple
linear regression, it’s the 2-dimensional surface β 0 + β 1 x for real values of β 0 and β 1 ).
Minimising ∑ (y i − ŷ i ) 2 is equivalent to dropping a perpendicular from the point y to the surface S;
the perpendicular hits the surface at ŷ. Thus we are literally finding the model closest to the data.
6.4 Problems
1. Show that the expression ∑ x i y i − n x y occurring in the formula for ̂β 1 could also be written as
∑ (xi − x)(y i − y), ∑ (x i − x)y i , or ∑ x i (y i − y).
2. Show that the ‘residual sum of squares’, ∑ n
i=1 (y i − ŷ i ) 2 , satisfies the following identity:
n∑
(y i − ŷ i ) 2 =
i=1
n∑
(y i − ̂β 0 − ̂β 1 x i ) 2 =
i=1
3. For the captopril data, find the least squares lines
(a) to predict SBP before captopril from DBP before captopril,
(b) to predict SBP after captopril from DBP after captopril,
(c) to predict DBP before captopril from SBP before captopril.
Compare these three lines.
n∑
(y i − y) 2 − ̂β
∑ n
1 (x i − x)(y i − y).
Discuss whether it is sensible to combine the before and after measurements in order to obtain a
better prediction of SBP at a given time from DBP measured at that time.
4. Illustrate the geometrical interpretation of least squares (see above comments) in the following two
cases
(a) model E[Y |x] = β 0 + β 1 x with 3 data points (x 1 , y 1 ), (x 2 , y 2 ) and (x 3 , y 3 ),
(b) model E[Y |x] = βx with 2 data points (x 1 , y 1 ) and (x 2 , y 2 ).
i=1
What does Pythagoras’ theorem tell us in the second case
i=1
76

5. Suppose that the random variables X and Y have the joint density
f X,Y (x, y) =
1
√ ×
2π σ X σ Y 1 − ρ
2
( [ (x−µX ) 2 ( )(
1
exp −
2 ( x−µX y−µY
1−ρ 2) − 2ρ
σ X
σ X
σ Y
)
+
( y−µY
σ Y
) 2
])
.
(a) show by substituting
or otherwise, that Y ∼ N (µ Y , σ 2 Y ).
v = x − µ X
followed by w = v − ρ(y − µ Y )/σ
√ Y
,
σ X 1 − ρ
2
(b) Hence or otherwise show that the conditional distribution of Y given X = x is Normal with
mean µ Y + (ρσ Y /σ X )(x − µ X ) and variance σ 2 Y
(
1 − ρ
2 ) .
(c) Find E[XY ], and show that the symbols µ X , µ Y , σ X , σ Y and ρ have their usual interpretations
of means, standard deviations, and correlation.
(d) Discuss briefly the similarities and differences between the above model for (X, Y ), and the
Simple Normal Linear Regression Model for Y given X.
From Warwick ST217 exam 2003
6.5 The Normal Linear Model (NLM)
6.5.1 Introduction
Definition 6.5 (NLM)
Given n response RVs Y i (i = 1, 2, . . . , n), with corresponding values of explanatory variables x T i , the
NLM makes the following assumptions:
1. (Conditional) Independence
The Y i are mutually independent given the x T i .
2. Linearity
The expected value of the response variable is linearly related to the unknown parameters β:
3. Normality
EY i = x T i β.
The random variation Y i |x i is Normally distributed.
4. Homoscedasticity (Equal Variances)
i.e. Y i |x i ∼ N(x T i β, σ2 ).
6.5.2 Matrix Formulation of NLM
The NLM for responses y = (y 1 , y 2 , . . . , y n ) T can be recast as follows
1. E[Y] = Xβ for some parameter vector β = (β 1 , β 2 , . . . , β p ) T ,
2. ɛ = Y − E[Y] ∼ MVN(0, σ 2 I), where I is the (n × n) identity matrix.
77

It can be shown that the least squares estimates of β are given by solving the simultaneous linear equations
(the normal equations), with solution (assuming that X T X is nonsingular)
X T y = X T Xβ (6.9)
̂β = (X T X) −1 X T y, (6.10)
Comments
1. Note that, by formula 6.10, each estimator ̂β j is a linear combination of the Y i s.
Therefore under the NLM, ̂β has a MVN distribution.
2. Even if the Normality assumption doesn’t hold, the CLT implies that, provided the number n of
cases is large, the distribution of the estimator ̂β will still be approximately MVN.
3. The most important assumption is independence, since it’s relatively easy to modify the standard
NLM to account for
• nonlinearity: transform the data, or include e.g. x 2 ij as an explanatory variable,
• unequal variances (‘heteroscedasticity’): e.g. transform from y i − ŷ i to z i = (y i − ŷ i )/̂σ i .
• non-Normality: transform, or simply get more data!
4. In the general formulation the constant term β 0 is omitted, though in practice the first column of the
matrix X will often contain 1’s and the corresponding parameter β 1 will be the ‘constant term’.
5. The corresponding fitted values are ŷ = X T ̂β, and the vector of residuals is r = y − ŷ,
i.e. r i = y i − ŷ i , where ŷ i = x T i ̂β = ∑ p
j=1 x ij ̂β j .
Definition 6.6 (RSS)
The residual sum of squares (RSS) in the fitted NLM is
s 2 = ∑ n
i=1 (y i − ŷ i ) 2
= (y − X̂β) T (y − X̂β)
(6.11)
Important Fact about the RSS
Considering the RSS s 2 to be the observed value of a corresponding RV S 2 , it can be shown that
• S 2 /σ 2 ∼ χ 2 (n−p) ,
• S 2 is independent of ̂β.
Exercise 6.2
(a) Show that the log-likelihood function for the NLM is
(constant) − n 2 log(σ2 ) − 1
2σ 2 (y − Xβ)T (y − Xβ). (6.12)
(b) Show that the maximum likelihood estimate of β is identical to the least squares estimate.
What is the distribution of ̂β
78

(c) Show that the MLE ̂σ 2 of σ 2 is
What are the mean and variance of ̂σ 2
̂σ 2 = s2
n . (6.13)
(d) Show that an unbiased estimator of σ 2 is given by the formula
Residual Sum of Squares
Residual Degrees of Freedom
‖
6.5.3 Examples of the NLM
1. Simple Linear Regression (again)
Y i = β 0 + β 1 x i + ɛ i , (6.14)
IID
where ɛ i ∼ N (0, σ 2 ).
2. Two-sample t-test
⎛
y =
⎜
⎝
x 1
x 2
.
x m
y 1
.
y n
⎞
⎛
1 0
1 0
⎞
. .
, X =
1 0
, β =
0 1
⎟ ⎜ ⎟
⎠ ⎝ . . ⎠
0 1
(
β0
β 1
)
, (6.15)
and we’re interested in the hypothesis H 0 : (β 0 − β 1 ) = 0.
3. Paired t-test
Some quantity Y is measured on each of n individuals under 2 different conditions (e.g. drugs A
and B), and we want to test whether the mean of Y can be assumed equal in both circumstances.
⎛
y =
⎜
⎝
y 11
y 21
.
y n1
y 12
y 22
.
.
y n2
⎞ ⎛
⎞
1 0 · · · 0 0
0 1 · · · 0 0
⎛
. . . .. . .
, X =
0 0 · · · 1 0
1 0 · · · 0 1
, β =
⎜
0 1 · · · 0 1
⎝
⎟ ⎜
⎠ ⎝
. ⎟
. . .. . . ⎠
0 0 · · · 1 1
α 1
α 2
.
α n
δ
⎞
, (6.16)
⎟
⎠
where δ is the difference between the expected responses under the two conditions, and the α i are
‘nuisance parameters’ representing the overall level of response for the ith individual.
The null hypothesis is H 0 : δ = 0.
4. Multiple Regression (example thereof)
79

Y = SBP after captopril, x 1 = SBP before captopril, x 2 = DBP before captopril,
⎛ ⎞ ⎛
⎞
201
1 210 130
165
1 169 122
166
1 187 124
157
1 160 104
147
1 167 112
145
1 176 101
⎛
168
1 185 121
y =
180
, X =
1 206 124
, β = ⎝ β ⎞
0
β 1
⎠ , (6.17)
147
1 173 115
β 2
136
1 146 102
151
1 174 98
168
1 201 119
⎜ 179
⎟ ⎜ 1 198 106
⎟
⎝ 129 ⎠ ⎝ 1 148 107 ⎠
131
1 154 100
where (roughly speaking) β 1 represents the increase in EY per unit increase in SBP before captopril
(x 1 ), allowing for the fact that EY also depends partly on DBP before captopril (x 2 ), and β 2 has a
similar interpretation in terms of the effect of x 2 allowing for x 1 .
In all the above examples, it’s straightforward to calculate ̂β = (X T X) −1 X T y, and also (for example) to
calculate the sampling distribution of ̂β i under the null hypothesis H 0 : β i = 0.
Exercise 6.3
Verify the following calculations from the data given in 6.17 above:
⎛
X T X = ⎝
⎛
(X T X) −1 = ⎝
15 2654 1685
2654 475502 300137
1685 300137 190817
⎞
⎛
⎠ , X T y = ⎝
8.563 −0.009165 −0.06120
−0.009165 0.0003026 −0.0003951
−0.06120 −0.0003951 0.001167
⎞
⎠ ,
2370
424523
268373
̂β =
⎛
⎝
⎞
⎠ ,
−20.7
0.724
0.450
⎞
⎠ .
‖
6.6 Checking Assumptions of the NLM
Clearly it’s very important in practice to check that your assumptions seem reasonable; there are various
ways to do this
6.6.1 Formal hypothesis testing
χ 2 tests are not very powerful, but are simple and general: count the number of data points satisfying
various (exhaustive & mutually exclusive) conditions, and compare with the expected counts under your
assumptions.
Other tests, for example to test for Normality, have been devised. However, a general problem with
statistical tests is that they don’t usually suggest what to do if your null hypothesis is rejected.
Exercise 6.4
How might you use a χ 2 test to check whether SBP after captopril is independent of SBP before captopril
‖
80

Exercise 6.5
A possible test for linearity in the simple Normal linear regression model (i.e. the NLM with just one
explanatory variable x) is to fit the quadratic NLM
and test the null hypothesis H 0 : β 2 = 0.
EY = β 0 + β 1 x + β 2 x 2 (6.18)
Suppose that Y is SBP and x is dose of drug, and that you have rejected the above null hypothesis.
Comment on the advisability of using Formula 6.18 for predicting Y given x.
‖
6.6.2 Graphical Methods and Residuals
If all the assumptions of the NLM are valid, then the residuals
r i = y i − ŷ i
= y i − x T ̂β i
(6.19)
should resemble observations on IID Normal random variables.
Therefore plots of r i against ANYTHING should be patternless
SEE LECTURE
Comments
1. Before fitting a formal statistical model (including e.g. performing a t-test), you should plot the data,
particularly the response variable against each explanatory variable.
2. After fitting a model, produce several residual plots. The computer is your friend!
3. Note that it’s the residual plots that are most informative. For example, the NLM DOESN’T assume
that the Y i are Normally distributed about µ Y , but DOES assume that each Y i is Normally distributed
about EY i |x i .
i.e. it’s the conditional distributions, not the marginal distributions, that are important.
81

6.7 Problems
1. Show that the following is an equivalent formulation of the two-sample t-test to that given above in
Formulae 6.15
⎛
x 1
⎞ ⎛
1
⎞
0
x 2
⎟ ⎜ 1 0 ⎟
Y =
⎜
⎝
.
x m
y 1
.
y n
with null hypothesis H 0 : β 1 = 0.
. .
, X =
1 0
, β =
1 1
⎟ ⎜ ⎟
⎠ ⎝ . . ⎠
1 1
(
β0
β 1
)
, (6.20)
2. Independent samples of 10 U.S. men aged 25–34 years, and 15 U.S. men aged 45–54 years were taken.
Their heights (in inches) were as follows:
(a) Age 25–34
73.3 64.8 72.1 68.9 68.7 70.4 66.8 70.7 74.4 71.8
(b) Age 45–54
73.2 68.5 62.4 65.5 71.3 69.5 74.5 70.6 69.3 67.1 64.7 73.0 66.7 68.1 64.3
Use a two-sample t-test to test the hypothesis that the population means of the two age-groups are
equal (the 90%, 95%, 97.5%, and 99% points of the t 23 distribution are 1.319, 1.714, 2.069 and 2.500
respectively).
Comment on whether the underlying assumptions of the two-sample t-test appear reasonable for this
set of data.
Comment also on whether the data can be used to suggest that the population of the U.S. has (or
hasn’t) tended to get taller over the last 20 years.
3. Verify that the least squares estimates in simple linear regression
∑
xi y i − n x y
̂β 1 = ∑ x
2
i − nx 2 , ̂β0 = y − ̂β 1 x,
are a special case of the general formula ̂β = (X T X) −1 X T y.
4. The following data-set shows average January minimum temperature in degrees Fahrenheit (y), together
with Latitude (x 1 ) and Longitude (x 2 ) for 28 US cities. Plot y against x 1 , and comment on
what this plot suggests about the reasonableness of the various assumptions underlying the NLM for
predicting y from x 1 and x 2 .
y x 1 x 2 y x 1 x 2 y x 1 x 2
44 31.2 88.5 38 32.9 86.8 35 33.6 112.5
31 35.4 92.8 47 34.3 118.7 42 38.4 123.0
15 40.7 105.3 22 41.7 73.4 26 40.5 76.3
30 39.7 77.5 45 31.0 82.3 65 25.0 82.0
58 26.3 80.7 37 33.9 85.0 22 43.7 117.1
19 42.3 88.0 21 39.8 86.9 11 41.8 93.6
22 38.1 97.6 27 39.0 86.5 45 30.8 90.2
12 44.2 70.5 25 39.7 77.3 23 42.7 71.4
21 43.1 83.9 2 45.9 93.9 24 39.3 90.5
8 47.1 112.4
Data from HSDS, set 262
82

5. (a) Assuming the model
E[Y |x] = β 0 + β 1 x,
Var[Y |x] = σ 2 independently of x,
derive formulae for the least squares estimates ̂β 0 and ̂β 1 from data (x i , y i ), i = 1, . . . , n.
What advantages are gained if the corresponding random variables Y i |x i can be assumed to be
independently Normally distributed
(b) The following table shows the tensile strength (y) of different batches of cement after being
‘cured’ (dried) for various lengths of time x: 3 batches were cured for 1 day, 3 for 2 days, 5 for
3 days, etc. The batch means and standard deviations (s.d.) are also given.
Curing time
Tensile strength
(days) x (kg/cm 2 ) y mean s.d.
1 13.0 13.3 11.8 12.7 0.8
2 21.9 24.5 24.7 23.7 1.6
3 29.8 28.0 24.1 24.1 26.2 26.5 2.5
7 32.4 30.4 34.5 33.1 35.7 33.2 2.0
28 41.8 42.6 40.3 35.7 37.3 40.0 3.0
Plot y against x and discuss briefly how reasonable seem each of the following assumptions:
(i) linearity: E[Y i |x i ] = β 0 + β 1 x i for some constants β 0 and β 1 .
(ii) independence: the Y i are mutually independent given the x i .
If conditional independence (ii) is assumed true, then how reasonable here are the further assumptions:
(iii) homoscedasticity: Var[Y i |x i ] = σ 2 for all i = 1, . . . , n,
(iv) Normality: the random variables Y i are each Normally distributed.
Say briefly whether you consider any of the above assumptions (i)–(iv) would be more plausible
following
(A) transforming from y to y ′ = log e (y), and/or
(B) transforming x in an appropriate way.
NOTE: you do not need to carry out numerical calculations such as finding the
least-squares fit explicitly.
From Warwick ST217 exam 2000
83

6. To monitor an industrial process for converting ammonia to nitric acid, the percentage of ammonia
lost (y) was measured on each of 21 consecutive days, together with explanatory variables representing
air flow (x 1 ), cooling water temperature (x 2 ) and acid concentration (x 3 ). The data, together with
the residuals after fitting the model ŷ = 3.614 + 0.072 x 1 + 0.130 x 2 − 0.152 x 3 , are given in the
following table:
Air Water Acid
Flow Temp. Conc.
Day y (x 1) (x 2) (x 3) Resid.
1 4.2 80 27 58.9 0.323
2 3.7 80 27 58.8 −0.192
3 3.7 75 25 59.0 0.456
4 2.8 62 24 58.7 0.570
5 1.8 62 22 58.7 −0.171
6 1.8 62 23 58.7 −0.301
7 1.9 62 24 59.3 −0.239
8 2.0 62 24 59.3 −0.139
9 1.5 58 23 58.7 −0.314
10 1.4 58 18 58.0 0.127
11 1.4 58 18 58.9 0.264
12 1.3 58 17 58.8 0.278
13 1.1 58 18 58.2 −0.143
14 1.2 58 19 59.3 −0.005
15 0.8 50 18 58.9 0.236
16 0.7 50 18 58.6 0.091
17 0.8 50 19 57.2 −0.152
18 0.8 50 19 57.9 −0.046
19 0.9 50 20 58.0 −0.060
20 1.5 56 20 58.2 0.141
21 1.5 70 20 59.1 −0.724
Some residual plots are shown on the next page (Fig. 6.1).
(a) Discuss whether the pattern of residuals casts doubt on any of the assumptions underlying the
Normal Linear Model (NLM).
Describe any further plots or calculations that you think would help you assess whether the
fitted NLM is appropriate here.
Continued. . .
84

(b) Various suggestions could be made for improving the model, such as
i. transforming the response (e.g. to log y or to y/x 1 ),
ii. transforming some or all of the explanatory variables,
iii. deleting outliers,
iv. including quadratic or even higher-order terms (e.g. x 2 2),
v. including interaction terms (e.g. x 1 x 3 ),
vi. carrying out a nonparametric analysis of the data,
vii. applying a bootstrap procedure,
viii. fitting a nonlinear model.
Outline the merits and disadvantages of each of these suggestions here. What would be your
next step in analysing this data-set
Figure 6.1: Residual plots
From Warwick ST217 exam 1999
85

7. Table 6.1, originally from Narula & Wellington (1977), shows data on selling prices of 28 houses
in Erie, Pennsylvania, together with explanatory variables that could be used to predict the selling
price. The variables are:
X 1 = current taxes (local, school and county) ÷ 100,
X 2 = number of bathrooms,
X 3 = lot size ÷ 1000 (square feet),
X 4 = living space ÷ 1000 (square feet),
X 5 = number of garage spaces,
X 6 = number of rooms,
X 7 = number of bedrooms,
X 8 = age of house (years),
X 9 = number of fireplaces,
Y = actual sale price ÷ 1000 (dollars).
Find a function of X 1 –X 9 that predicts Y reasonably accurately (such functions are used to fix
property taxes, which should be based on the current market value of each property).
X 1 X 2 X 3 X 4 X 5 X 6 X 7 X 8 X 9 Y
4.9176 1.0 3.4720 0.9980 1.0 7 4 42 0 25.9
5.0208 1.0 3.5310 1.5000 2.0 7 4 62 0 29.5
4.5429 1.0 2.2750 1.1750 1.0 6 3 40 0 27.9
4.5573 1.0 4.0500 1.2320 1.0 6 3 54 0 25.9
5.0597 1.0 4.4550 1.1210 1.0 6 3 42 0 29.9
3.8910 1.0 4.4550 0.9880 1.0 6 3 56 0 29.9
5.8980 1.0 5.8500 1.2400 1.0 7 3 51 1 30.9
5.6039 1.0 9.5200 1.5010 0.0 6 3 32 0 28.9
15.4202 2.5 9.8000 3.4200 2.0 10 5 42 1 84.9
14.4598 2.5 12.8000 3.0000 2.0 9 5 14 1 82.9
5.8282 1.0 6.4350 1.2250 2.0 6 3 32 0 35.9
5.3003 1.0 4.9883 1.5520 1.0 6 3 30 0 31.5
6.2712 1.0 5.5200 0.9750 1.0 5 2 30 0 31.0
5.9592 1.0 6.6660 1.1210 2.0 6 3 32 0 30.9
5.0500 1.0 5.0000 1.0200 0.0 5 2 46 1 30.0
8.2464 1.5 5.1500 1.6640 2.0 8 4 50 0 36.9
6.6969 1.5 6.9020 1.4880 1.5 7 3 22 1 41.9
7.7841 1.5 7.1020 1.3760 1.0 6 3 17 0 40.5
9.0384 1.0 7.8000 1.5000 1.5 7 3 23 0 43.9
5.9894 1.0 5.5200 1.2560 2.0 6 3 40 1 37.5
7.5422 1.5 4.0000 1.6900 1.0 6 3 22 0 37.9
8.7951 1.5 9.8900 1.8200 2.0 8 4 50 1 44.5
6.0931 1.5 6.7265 1.6520 1.0 6 3 44 0 37.9
8.3607 1.5 9.1500 1.7770 2.0 8 4 48 1 38.9
8.1400 1.0 8.0000 1.5040 2.0 7 3 3 0 36.9
9.1416 1.5 7.3262 1.8310 1.5 8 4 31 0 45.8
12.0000 1.5 5.0000 1.2000 2.0 6 3 30 1 41.0
Table 6.1: House price data
Weisberg (1980)
86

8. The number of ‘hits’ recorded on J.E.H.Shaw’s WWW homepage in late 1999 are given below. ‘Local’
means the homepage was accessed from within Warwick University, ‘Remote’ means it was accessed
from outside. Data for the week beginning 7–Nov–1999 were unavailable. Note that there was an
exam on Wednesday 8–Dec–1999 for the course ST104, taught by J.E.H.Shaw.
Week Number of Hits
Beginning Local Remote Total
26 Sept 0 182 182
3 Oct 35 253 288
10 Oct 901 315 1216
17 Oct 641 443 1084
24 Oct 1549 525 2074
31 Oct 823 344 1167
7 Nov — — —
14 Nov 1136 383 1519
21 Nov 2114 584 2698
28 Nov 2097 536 2633
5 Dec 3732 461 4193
12 Dec 5 352 357
19 Dec 0 296 296
(a) Fit a linear least-squares regression line to predict the number of remote hits (Y ) in a week from
the observed number x of local hits.
(b) Calculate the residuals and plot them against date. Does the plot give any evidence that the
interrelationship between X and Y changes over time
(c) Using both general considerations and residual plots, comment on how reasonable here are the
assumptions underlying the simple Normal linear regression model, and suggest possible ways
to improve the prediction of Y .
9. The following table shows the assets x (billions of dollars) and net income y (millions of dollars) for
the 20 largest US banks in 1973.
Bank x y Bank x y Bank x y Bank x y
1 49.0 218.8 6 14.2 63.6 11 11.6 42.9 16 6.7 42.7
2 42.3 265.6 7 13.5 96.9 12 9.5 32.4 17 6.0 28.9
3 36.3 170.9 8 13.4 60.9 13 9.4 68.3 18 4.6 40.7
4 16.4 85.9 9 13.2 144.2 14 7.5 48.6 19 3.8 13.8
5 14.9 88.1 10 11.8 53.6 15 7.2 32.2 20 3.4 22.2
(a) Plot income (y) against assets (x), and also log(income) against log(assets).
(b) Verify that the least squares fit regression lines are
fit 1: y = 4.987 x + 7.57,
fit 2: log(y) = 0.963 log(x) + 1.782 (Note: logs to base e),
and show the fitted lines on your plots.
(c) Produce Normal probability plots of the residuals from each fit.
(d) Which (if either) of these models would you use to describe the relationship between total assets
and net income Why
(e) Bank number 19 (the Franklin National Bank) failed in 1974, and was the largest ever US bank to
fail. Identify the point representing this bank on each of your plots, and discuss briefly whether,
from the data presented, one might have expected beforehand that the Franklin National Bank
was in trouble.
87

10. The following data show the blood alcohol levels (mg/100ml) at post mortem for traffic accident
victims. Blood samples in each case were taken from the leg (A) and from the heart (B). Do these
results indicate that blood alcohol levels differ systematically between samples from the leg and the
heart
Case A B Case A B
1 44 44 11 265 277
2 265 269 12 27 39
3 250 256 13 68 84
4 153 154 14 230 228
5 88 83 15 180 187
6 180 185 16 149 155
7 35 36 17 286 290
8 494 502 18 72 80
9 249 249 19 39 50
10 204 208 20 272 290
11. (a) Assume the linear model
E[Y|X] = Xβ,
Var[Y|X] = σ 2 I n ,
Osborn (1979) 4.6.5
where I n denotes the n × n identity matrix, and X T X is nonsingular. By writing Y − Xβ =
(Y − X̂β) + X(̂β − β), or otherwise, show that for this model, the residual sum of squares
is minimised at β = ̂β = (X T X) −1 X T Y.
(Y − Xβ) T (Y − Xβ)
(b) Show that E[̂β] = β and that Var[̂β] = σ 2 (X T X) −1 .
(c) Let A = X(X T X) −1 X T . Show that A and I n − A are both idempotent, i.e. AA = A and
(I n − A)(I n − A) = I n − A.
(d) For the particular case of a Normal linear model, find the joint distribution of the fitted values
Ŷ = X̂β, and show that Y − Ŷ is independent of Ŷ. Quote carefully any properties of the
Normal distribution you use.
(e) For the simple linear regression model (EY i = β 0 + β 1 x i ), write down the corresponding matrix
X and vector Y, find (X T X) −1 , and hence find the least squares estimates ̂β 0 and ̂β 1 and their
variances.
From Warwick ST217 exam 2001
88

6.8 The Analysis of Variance (ANOVA)
6.8.1 One-Way Analysis of Variance: Introduction
This is a generalization of the two-sample t-test to p > 2 groups.
Suppose there are observations y ij (j = 1, 2, . . . , n i ) in the ith group (i = 1, 2, . . . , p),
and let n = n 1 + n 2 + · · · + n p denote the total number of observations.
Denote the corresponding RVs by Y ij , and assume that Y ij ∼ N (β i , σ 2 ) independently.
Traditionally the main aim has been to test the null hypothesis
H 0 : β 1 = β 2 = . . . = β p
i.e. : β = β 0 = (β 0 , β 0 , . . . , β 0 )
The idea is to fit MLEs ̂β and ̂β 0 and apply a likelihood ratio test, i.e. test whether the ratio
change in RSS
RSS
= squared distance from ŷ to ŷ 0
squared distance from y to ŷ
(where ŷ and ŷ 0 are the corresponding fitted values) is larger than would be expected by chance.
A useful notation for group means etc. uses overbars and ‘+’ suffices as follows:
(
)
y i+ = 1 ∑n i
y ij , y
n ++ = 1 p∑ ∑n i
y ij = 1 p∑
n i y
i
n
n i+ , etc.
j=1
i=1 j=1
The underlying models fit naturally in the NLM framework:
i=1
Definition 6.7 (One-Way ANOVA)
The one-way ANOVA model is a NLM of the form
Y ∼ MVN(Xβ, σ 2 I),
where
⎛
⎞
1 0 0 · · · 0
1 0 0 · · · 0
. . . . . . . .. .
1 0 0 · · · 0
0 1 0 · · · 0
⎛
⎞
⎛ ⎞
Y 1
. . . . .. .
1 0 0 · · · 0
⎛ ⎞
Y 2
Y = ⎜ ⎟
⎝ . ⎠ , X = 0 1 0 · · · 0
0 1 0 · · · 0
β 1
0 0 1 · · · 0
=
0 0 1 · · · 0
β 2
, β = ⎜ . ⎟
⎜
⎝
.
Y n
. . . . .
. . . ..
⎟ ⎝
. ⎠
. ⎠ ,
β 0 0 1 · · · 0
0 0 0 · · · 1
p
. . . . .
0 0 0 · · · 1
⎜
⎝
. ⎟
. . . .. . ⎠
0 0 0 · · · 1
(6.21)
where X has n 1 rows of the first type, . . . n p rows of the last type, and n 1 + n 2 + · · · + n p = n.
Exercise 6.6
Show that for one-way ANOVA, X T X = diag(n 1 , n 2 , . . . , n p ), and hence ̂β = (Y 1+ , Y 2+ , . . . , Y p+ ) T .
‖
89

6.8.2 One-Way Analysis of Variance: ANOVA Table
Let
β 0 = E[Y ++ ] = 1 n
p∑ ∑n i
EY ij = 1 n
i=1 j=1
α i = β i − β 0 (i = 1, 2, . . . , p).
p∑
n i β i ,
Typically the p groups correspond to p different treatments, and α i is then called the ith treatment effect.
We’re interested in the hypotheses
H 0 : α i = 0 (i = 1, 2, . . . , p),
H 1 : the α i are arbitrary.
i=1
Note that
1. Y ++ is the MLE of β = β 0 under H 0 ,
2. Y i+ is the MLE of β + α i , i.e. the mean response given the i treatment.
Hence the fitted values under H 0 and H 1 are given by Y ++ and Y i+ respectively.
If we also include the ‘null model’ that all the β i are zero, then the possible models of interest are:
Model # params DF RSS
∑
β i = 0 ∀ i i.e. ŷ ij = 0 0 n
i,j y2 ij (1)
∑
β i = β 0 ∀ i i.e. ŷ ij = y ++ 1 n − 1
i,j (y ij − y ++ ) 2 (2)
∑
β i arbitrary, i.e. ŷ ij = y i+ p n − p
i,j (y ij − y i+ ) 2 (3)
The calculations needed to test H 0 , involving the RSS formulae given above, can be conveniently presented
in an ‘ANOVA table’:
Source of Degrees of Sum of squares Mean square
variation freedom (DF) (SS) (MS) = SS/DF
Overall mean 1 (1)–(2) = ny 2 ++ ny 2 ++
Treatment p − 1 (2)–(3) = ∑ i n i(y i+ − y ++ ) 2 ∑i n i(y i+ − y ++ ) / 2 (p − 1)
Residual n − p (3) =
∑i,j (y ij − y i+ ) 2 ∑
i,j (y ij − y i+ ) / 2 (n − p)
Total n (1) =
∑
i,j y2 ij
Finally, calculate the ‘F ratio’
F =
Treatment MS
Residual MS
=
Treatment SS/(p − 1)
Residual SS/(n − p)
(6.22)
which, under H 0 , has an F distribution on (p − 1) and (n − p) d.f.
Large values of F are evidence against H 0 .
Note: DON’T try too hard to remember formulae for sums of squares in an ANOVA table.
Instead THINK OF THE MODELS BEING FITTED. The ‘lack of fit’ of each model is given by the
corresponding RSS, & the formulae for the differences in RSS simplify.
90

6.9 Problems
1. Show that the formulae for sums of squares in one-way ANOVA simplify:
p∑
n i (Y i+ − Y ++ ) 2 =
i=1
p∑ ∑n i
(Y ij − Y i+ ) 2 =
i=1 j=1
p∑
n i Y 2 i+ − nY 2 ++,
i=1
p∑ ∑n i
Yij 2 −
i=1 j=1
p∑
n i Y 2 i+.
i=1
2. (a) Define the Normal Linear Model, and describe briefly how each of its assumptions may be
informally checked by plotting residuals.
(b) The following data summarise the number of days survived by mice inoculated with three strains
of typhoid (31 mice with ‘9D’, 60 mice with ‘11C’ and 133 mice with ‘DSCI’).
Days Numbers of Mice
to Inoculated with. . .
Death 9D 11C DSCI Total
2 6 1 3 10
3 4 3 5 12
4 9 3 5 17
5 8 6 8 22
6 3 6 19 28
7 1 14 23 38
8 11 22 33
9 4 14 18
10 6 14 20
11 2 7 9
12 3 8 11
13 1 4 5
14 1 1
Total 31 60 133 224
∑
Xi 125 442 1037 1604
∑ X
2
i 561 3602 8961 13124
(X i is the survival time of the ith mouse in the given group).
Without carrying out any calculations, discuss briefly how reasonable seem the assumptions
underlying a one-way ANOVA on the data, and whether a transformation of the data may be
appropriate.
(c) Carry out a one-way ANOVA on the untransformed data. What do you conclude about the
responses to the three strains of typhoid
From Warwick ST217 exam 1997
3. The amount of nitrogen-bound bovine serum albumin produced by three groups of mice was measured.
The groups were: normal mice treated with a placebo (i.e. an inert substance), alloxan-diabetic mice
treated with a placebo, and alloxan-diabetic mice treated with insulin. The resulting data are shown
in the following table:
91

Normal Alloxan-diabetic Alloxan-diabetic
+ placebo + placebo + insulin
156 391 82
282 46 100
197 469 98
297 86 150
116 174 243
127 133 68
119 13 228
29 499 131
253 168 73
122 62 18
349 127 20
110 276 100
143 176 72
64 146 133
26 108 465
86 276 40
122 50 46
455 73 34
655 44
14
(a) Produce appropriate graphical display(s) and numerical summaries of these data, and comment
on what can be learnt from these.
(b) Carry out a one-way analysis of variance on the three groups. You may feel it necessary to
transform the data first.
Data from HSDS, set 304
4. The following table shows measurements of the steady-state haemoglobin levels for patients with
different types of sickle-cell anaemia (‘HB SS’, ‘HB S/-thalassaemia’ and ‘HB SC’). Construct an
ANOVA table and hence test whether the steady-state haemoglobin levels differ between the three
types.
HB SS HB S/-thalassaemia HB SC
7.2 8.1 10.7
7.7 9.2 11.3
8.0 10.0 11.5
8.1 10.4 11.6
8.3 10.6 11.7
8.4 10.9 11.8
8.4 11.1 12.0
8.5 11.9 12.1
8.6 12.0 12.3
8.7 12.1 12.6
9.1 12.6
9.1 13.3
9.1 13.3
9.8 13.8
10.1 13.9
10.3
Data from HSDS, set 310
92

5. The data in Table 6.2, collected by Brian Everitt, are described in HSDS as being the ‘weights, in
kg, of young girls receiving three different treatments for anorexia over a fixed period of time with
the control group receiving the standard treatment’.
(a) Using a one-way ANOVA on the weight gains, compare the three methods of treatment.
(b) Plot the data so as to clarify the effects of the three treatments, and discuss whether the above
formal analysis was appropriate.
Cognitive
behavioural
Family
treatment Control therapy
Weight Weight Weight
before after before after before after
80.5 82.2 80.7 80.2 83.8 95.2
84.9 85.6 89.4 80.1 83.3 94.3
81.5 81.4 91.8 86.4 86.0 91.5
82.6 81.9 74.0 86.3 82.5 91.9
79.9 76.4 78.1 76.1 86.7 100.3
88.7 103.6 88.3 78.1 79.6 76.7
94.9 98.4 87.3 75.1 76.9 76.8
76.3 93.4 75.1 86.7 94.2 101.6
81.0 73.4 80.6 73.5 73.4 94.9
80.5 82.1 78.4 84.6 80.5 75.2
85.0 96.7 77.6 77.4 81.6 77.8
89.2 95.3 88.7 79.5 82.1 95.5
81.3 82.4 81.3 89.6 77.6 90.7
81.3 82.4 81.3 89.6 77.6 90.7
76.5 72.5 78.1 81.4 83.5 92.5
70.0 90.9 70.5 81.8 89.9 93.8
80.4 71.3 77.3 77.3 86.0 91.7
83.3 85.4 85.2 84.2 87.3 98.0
83.0 81.6 86.0 75.4
87.7 89.1 84.1 79.5
84.2 83.9 79.7 73.0
86.4 82.7 85.5 88.3
76.5 75.7 84.4 84.7
80.2 82.6 79.6 81.4
87.8 100.4 77.5 81.2
83.3 85.2 72.3 88.2
79.7 83.6 89.0 78.8
84.5 84.6
80.8 96.2
87.4 86.7
Table 6.2: Anorexia data
Data from HSDS, set 285
93

6. The following data come from a study of pollution in inland waterways. In each of seven localities,
five pike were caught and the log concentration of copper in their livers measured.
Locality
Log concentration of copper (ppm)
1. Windermere 0.187 0.836 0.704 0.938 0.124
2. Grassmere 0.449 0.769 0.301 0.045 0.846
3. River Stour 0.628 0.193 0.810 0.000 0.855
4. Wimbourne St Giles 0.412 0.286 0.497 0.417 0.337
5. River Avon 0.243 0.258 -0.276 -0.538 0.041
6. River Leam 0.134 0.281 0.529 0.305 0.459
7. River Kennett 0.471 0.371 0.297 0.691 0.535
(a) The data are plotted in Figure 6.2. Discuss briefly what the plot suggests about the relative
copper pollution in the various localities.
Figure 6.2: Concentration of copper in pike livers
(b) Carry out a one-way analysis of variance to test for differences between the data between localities.
Do the results of the formal analysis agree with your subjective impressions from
Figure 6.2
94

7. (a) Define the Normal Linear Model, and show how the ‘One-Way Analysis of Variance’ may be
written in matrix form as a Normal Linear Model.
(b) The following table gives the concentrations (nanograms per litre) of two pollutants (aldrin
and HCB) found in ten separate samples at each of three different depths of the Wolf River in
Tennessee. (Data from P. R. Jaffe, F. L. Parker & D. J. Wilson (1982), ‘Distribution of toxic
substances in rivers’, Journal of the Environmental Engineering Division 108:639–649.)
Depth: Surface Middepth Bottom
Pollutant: Aldrin HCB Aldrin HCB Aldrin HCB
3.08 3.74 5.17 6.03 4.81 5.44
3.58 4.61 6.17 6.55 5.71 6.88
3.81 4.00 6.26 3.55 4.90 5.37
4.31 4.67 4.26 4.59 5.35 5.44
4.35 4.87 3.17 3.77 5.26 5.03
4.40 5.12 3.76 4.81 6.26 6.48
3.67 4.52 4.76 5.85 3.76 3.89
5.17 5.29 4.90 5.74 8.07 5.85
5.17 5.74 6.57 6.77 8.79 6.85
4.35 5.48 5.17 5.64 7.30 7.16
Column sum: 41.9 48.0 50.2 53.3 60.2 58.4
Sum of squares: 179.5 234.4 262.9 295.1 385.0 350.2
Carry out a one-way ANOVA on the aldrin data. What do you conclude about the relative
concentration at different depths
(c) Plot the tabulated data in what you consider an appropriate way. Discuss briefly whether your
plot casts any doubt on the assumptions underlying the One-Way ANOVA, and how you might
continue your analysis (but do NOT perform any further numerical calculations!)
From Warwick ST217 exam 2002
6.10 Two-Way Analysis of Variance
Here there are two factors (e.g. two treatments, or patient number and treatment given) that can be varied
independently.
Factor A has I ‘levels’ 1, 2, . . . , I, and factor B has J ‘levels’ 1, 2, . . . , J. For example:
(a) A is patient number 1, 2, . . . , I, every patient receiving each treatment j = 1, 2, . . . , J in turn,
(b) A is treatment number 1, 2, . . . , I, and B is one of J possible supplementary treatments.
Data can be conveniently tabulated:
Factor B
1 2 . . . J
1 Y 11 Y 12 . . . Y 1J
2 Y 21 Y 22 . . . Y 2J
Factor A 3 Y 31 Y 32 . . . Y 3J
. . .
. .. .
I Y I1 Y I2 . . . Y IJ
i.e. there is precisely one observation Y ij at each (i, j) combination of factor levels.
95

Again assume the NLM with
E[Y ij ] = θ i + φ j for i = 1 . . . I and j = 1 . . . J.
i.e.
Y ij ∼ N (θ i + φ j , σ 2 ) independently. (6.23)
A problem here is that one could transform θ i ↦→ θ i + c and φ j ↦→ φ j − c for each i and j, where c is
arbitrary. Therefore for identifiability one needs to impose some (arbitrary) constraints.
The simplest and most symmetrical reformulation for the two-way ANOVA model is
Y ij ∼ N (µ + α i + β j , σ 2 ), where
∑ I
i=1 α i = 0,
(6.24)
∑ J
j=1 β j = 0.
Exercise 6.7
What is the matrix formulation of the model 6.24
‖
Particular models of interest within the framework of Formulae 6.24 are:
(1) Y ij ∼ N (0, σ 2 ),
RSS = ∑ i,j
Y 2
ij,
DF = n = IJ.
(2) Y ij ∼ N (µ, σ 2 ),
RSS = ∑ i,j
(Y ij − Y ++ ) 2 , DF = n − 1 = IJ − 1.
(3) Y ij ∼ N (µ + α i , σ 2 ),
Ŷ ij = ̂µ + ̂α i = Y i+ .
Therefore RSS = ∑ (Y ij − Y i+ ) 2 , DF = n − I = I(J − 1).
i,j
(4) Y ij ∼ N (µ + β j , σ 2 ),
Ŷ ij = ̂µ + ̂β j = Y +j .
Therefore RSS = ∑ (Y ij − Y +j ) 2 , DF = n − J = (I − 1)J.
i,j
(5) Y ij ∼ N (µ + α i + β j , σ 2 ),
Ŷ ij = ̂µ + ̂α i + ̂β j = Y i+ + Y +j − Y ++ .
Therefore RSS = ∑ (Y ij − Y i+ − Y +j + Y ++ ) 2 , DF = n − I − J + 1 = (I−1)(J−1).
i,j
Again, we can form an ANOVA table summarising the independent ‘sources of variation’.
The degrees of freedom are the differences between the DFs associated with the various models.
The sums of squares are the differences between the SSs associated with the various models.
96

Source of Degrees of Sum of squares Mean square
variation freedom (DF) (SS) (MS)
Overall mean 1 (1)−(2) ( ) /
Effect of Factor A I−1 (2)−(3) ( (2)−(3) ) / (I−1)
Effect of Factor B J−1 (2)−(4) (2)−(4) (J−1)
Residuals (I−1)(J−1) (5) (5) / ( (I−1)(J−1) )
Total IJ = n (1)
Table 6.3: Two-way ANOVA table
Comments
1. DeGroot gives a more general version.
2. As with one-way ANOVA, one can test H 0 : α i = 0, i = 1 . . . I, by comparing
with the 95% point of F (I−1),([I−1][J−1]) .
(SS due to A)/(I − 1)
(Residual SS)/([I − 1][J − 1])
3. Similarly one can test H 0 : β j = 0, j = 1 . . . J, by comparing
with the 95% point of F (J−1),([I−1][J−1]) .
(SS due to B)/(J − 1)
(Residual SS)/([I − 1][J − 1])
4. The above two F tests are using completely separate aspects of the data (row sums of the Y ij table,
column sums of the Y ij table).
5. The case J = 2 is equivalent to the paired t-test (Exercise 5.4).
6. As for one-way ANOVA, the formulae for sums of squares simplify:
‘sum over each observation the squared difference between the fitted values under the two models
being considered’.
The residual SS is then most easily obtained by subtraction.
See problem 6.11.1
97

6.11 Problems
1. For the two-way analysis of variance (Table 6.3, page 97), find simplified formulae for the sums of
squares analogous to those found for the one-way ANOVA (exercise 6.9.1).
2. Three pertussis vaccines were tested on each of ten days. The following table shows estimates of the
log doses of vaccine (in millions of organisms) required to protect 50% of mice against a subsequent
infection with pertussis organisms.
Vaccine
Day A B C Total
1 2.64 2.93 2.93 8.50
2 2.00 2.52 2.56 7.08
3 3.04 3.05 3.35 9.44
4 2.07 2.97 2.55 7.59
5 2.54 2.44 2.45 7.43
6 2.76 3.18 3.25 9.19
7 2.03 2.30 2.17 6.50
8 2.20 2.56 2.18 6.94
9 2.38 2.99 2.74 8.11
10 2.42 3.20 3.14 8.76
Total 24.08 28.14 27.32 79.54
Test the statistical significance of the differences between days and between vaccines.
Osborn (1979) 8.1.2
3. (a) Explain what is meant by the Normal Linear Model (NLM), and show how the two-way analysis
of variance may be formulated in this way.
(b) The following table gives the average UK cereal yield (tonnes per hectare) from 1994 to 1998,
together with the row, column, and overall totals.
1994 1995 1996 1997 1998 Total
Wheat 7.35 7.70 8.15 7.38 7.56 38.14
Barley 5.37 5.73 6.14 5.76 5.29 28.29
Oats 5.50 5.52 6.14 5.78 6.00 28.94
Other cereal 5.65 5.52 5.86 5.52 5.04 27.59
Total 23.87 24.47 26.29 24.44 23.89 122.96
Calculate the fitted yields and residuals for Wheat in each of the five years
i. under the NLM assuming no column effect, and
ii. under the NLM assuming that row & column effects are additive.
(c) Describe briefly how to test the null hypothesis that there is no column effect (i.e. no consistent
change in yield from year to year). You do not need to carry out the numerical calculations.
(d) A nonparametric test of the above hypothesis may be carried out as follows: rank the data for
each row from lowest to highest (thus for Wheat the values 7.35, 7.70, 8.15, 7.38 and 7.56 are
replaced by 1, 4, 5, 2 and 3 respectively), then sum the four ranks for each year, and finally
carry out a one-way analysis of variance on the five sums of ranks.
Comment on the advantages and disadvantages of applying this procedure, rather than the
standard two-way ANOVA, to the above data.
From Warwick ST217 exam 2001
98

4. The following table gives the estimated hospital waiting lists (000s) by month & region, throughout
the years 2000 & 2001.
Month Year NY T E L SE SW WM NW
1 2000 137.6 105.5 121.6 173.3 192.6 111.0 99.4 177.6
2 2000 132.0 103.2 118.7 167.9 190.4 107.7 95.8 172.2
3 2000 125.1 98.7 111.9 162.3 184.0 100.7 89.6 164.8
4 2000 129.5 99.5 114.3 163.5 186.9 101.5 92.1 166.5
5 2000 129.9 99.5 114.6 163.2 186.4 100.6 92.4 166.2
6 2000 128.9 99.9 114.4 163.3 183.7 100.2 91.9 165.5
7 2000 127.9 99.0 113.6 160.8 183.9 99.6 90.7 164.9
8 2000 126.7 98.9 113.2 159.6 183.4 100.1 90.2 165.6
9 2000 124.6 98.2 111.9 158.1 183.8 99.6 91.0 164.6
10 2000 123.4 97.1 112.0 156.0 183.4 98.8 91.1 163.1
11 2000 121.0 97.2 111.7 155.7 183.4 98.9 92.1 161.2
12 2000 121.3 99.0 112.7 158.0 188.2 99.4 92.8 162.9
1 2001 122.4 97.7 113.3 159.5 188.8 99.8 93.4 164.0
2 2001 121.6 96.9 113.0 159.2 186.7 100.1 92.1 163.3
3 2001 119.6 95.3 109.7 156.3 181.1 97.1 87.2 160.5
4 2001 122.6 96.5 110.7 158.9 184.1 99.3 88.8 162.7
5 2001 124.0 97.2 111.0 160.0 185.9 100.3 90.1 164.4
6 2001 124.2 98.0 111.8 160.6 187.2 99.7 90.8 165.6
7 2001 123.2 98.9 111.4 160.9 187.4 100.2 91.0 165.5
8 2001 123.2 99.6 111.9 161.4 185.7 100.0 91.6 166.2
9 2001 123.0 99.4 111.6 159.1 185.2 100.2 91.0 165.8
10 2001 124.1 99.0 111.8 156.7 184.6 101.1 90.7 165.5
11 2001 123.0 99.6 113.2 155.4 183.7 102.0 89.8 164.7
12 2001 124.3 100.7 115.8 159.1 186.7 103.6 92.1 168.0
Key
NY Northern & Yorkshire T Trent
E Eastern L London SE South East
SW South West WM West Midlands NW North West
[Data extracted from archived Press Releases at http://tap.ccta.gov.uk/doh/intpress.nsf]
Fit a two-way ANOVA model, possibly after transforming the data, and address (briefly) the following
questions:
(a) Does the pattern of change in waiting lists differ across the regions
(b) Is there a simple (but not misleading) description of the overall change in waiting lists over the
two years
(c) Predict the values for the eight regions in March 2002 (to the nearest 100, as in the Table).
(d) The set of figures for March 2001 were the latest available at the time of the General Election
in May 2001. A cynical acquaintance suggests to you that the March 2001 waiting lists were
‘unusually good’. What do you think
99

5. Table 4.2, page 44, presented data on the preventive effect of four different drugs on allergic response
in ten patients.
A simple way to analyse the data is via a two-way ANOVA on a suitable measure of patient response,
such as the increase in √ NCF, which is tabulated below (for example, 1.95 = √ 3.8 − √ 0.0 and
1.52 = √ 9.2 − √ 2.3).
Patient number
Drug 1 2 3 4 5 6 7 8 9 10
P 1.95 1.52 0.77 0.44 0.78 1.69 0.37 0.95 1.10 0.62
C 0.71 1.30 1.32 1.48 0.58 0.41 0.00 2.09 0.32 −0.22
D 0.65 0.67 0.65 0.48 0.00 0.44 0.26 0.42 1.18 0.63
K 0.19 0.54 −0.07 0.82 0.54 −0.44 0.27 −0.03 0.59 0.71
(a) Test the statistical significance of the differences between drugs and between patients.
(b) Plot the original data (Table 4.2) in a way that would help you assess whether the assumptions
underlying the above two-way ANOVA are reasonable.
(c) Comment on the analysis you have made suggesting possible improvements where appropriate.
You do NOT need to carry out any further complicated calculations.
6. Table 6.4 shows purported IQ scores of identical twins, one raised in a foster home (Y ), and the
other raised by natural parents (X). The data are also categorised according to the social class of
the natural parents (upper, middle, low). The data come from Burt (1966), and are also available in
Weisberg (1980).
upper class middle class lower class
Case Y X Case Y X Case Y X
1 82 82 8 71 78 14 63 68
2 80 90 9 75 79 15 77 73
3 88 91 10 93 82 16 86 81
4 108 115 11 95 97 17 83 85
5 116 115 12 88 100 18 93 87
6 117 129 13 111 107 19 97 87
7 132 131 20 87 93
21 94 94
22 96 95
23 112 97
24 113 97
25 106 103
26 107 106
27 98 111
Table 6.4: Burt’s twin IQ data
(a) Plot the data.
(b) Fit simple linear regression models to predict Y from X within each social class.
(c) Fit parallel lines predicting Y from X within each social class (i.e. fit regression models with
the same slope in each of the three classes, but possibly different intercepts).
(d) Produce an ANOVA table and an F -test to test whether the parallelism assumption is reasonable.
Comment on the calculated F ratio.
100

7. (a) Give a matrix formulation of the Normal Linear Model, and show how the two-sample t-test and
the two-way analysis of variance may both be considered to be examples of the Normal Linear
Model.
(b) The following table shows the effect of oestrogen on weight gain in sheep in four separate ranches.
Treatments are a combination of sex of sheep (M or F) and level of oestrogen (S0 or S3). Thus
the value 52 in the table denotes the weight gain of a female sheep in Ranch II, given level S0
of oestrogen.
Ranch
Treatment I II III IV
F-S0 47 52 62 51
M-S0 50 54 67 57
F-S3 57 53 69 57
M-S3 54 65 74 59
data from an experiment reported by Little & Hills (1978)
Produce a two-way analysis of variance table, and test whether there are statistically significant
differences between the treatments and/or the ranches, in terms of weight gain.
(c) Discuss briefly how you would check the assumptions underlying your analysis (you do NOT
need to produce any plots or further calculations).
From Warwick ST217 exam 2003
For we know in part, and we prophesy in part.
But when that which is perfect is come, then that which is in part shall be done away.
1 Corinthians 13:9–10
Everything should be made as simple as possible, but not simpler.
Albert Einstein
A theory is a good theory if it satisfies two requirements: it must accurately describe a large
class of observations on the basis of a model that contains only a few arbitrary elements, and
it must make definite predictions about the results of future observations.
Stephen William Hawking
The purpose of models is not to fit the data but to sharpen the question.
Samuel Karlin
Science may be described as the art of systematic oversimplification.
Sir Karl Raimund Popper
101

Chapter 7
Further Topics
7.1 Generalisations of the Linear Model
You can generalise the systematic part of the linear model, i.e. the formula for E[Y |x]
and/or the random part, i.e. the distribution of Y − E[Y |x].
7.1.1 Nonlinear Models
These are models of the form
E[Y |x] = g(x, β) (7.1)
where Y is the response, x is a vector of explanatory variables, β = (β 1 . . . β p ) T is a parameter vector, and
the function g is nonlinear in the β i s.
Examples
1. Asymptotic regression:
Y i = α − βγ xi + ɛ i (i = 1, 2, . . . , n),
ɛ i
IID
∼ N (0, σ 2 ).
There are four parameters to be estimated: β = (α, β, γ, σ 2 ) T .
Assuming that 0 < γ < 1, we have:
(a) E[Y |x] is monotonic increasing in x,
(b) E[Y |x = 0] = α − β,
(c) as x → ∞, E[Y |x] → α.
This ‘asymptotic regression’ model might be appropriate, for example, if
(a) x = age of an animal,
y = height or weight, or
(b) x = time spent training,
y = height jumped
(for n people of similar build).
2. The ‘Michaelis-Menten’ equation in enzyme kinetics
E[Y |x] =
β 1x
β 2 + x
with various possible distributional assumptions, the simplest of which is
[Y |x] ∼ N (β 1 x/(β 2 +x), σ 2 ).
102

Comments
1. Nonlinear models can be fitted, in principle, by maximum likelihood.
2. In practice one needs computers and iteration.
3. Even if the random variation is assumed to be Normal, the likelihood may have a very non-Normal
shape.
7.1.2 Generalised Linear Models
Definition 7.1 (GLM)
A generalized linear model (GLM) has a random part and a systematic part:
Random Part
1. The ith response Y i has a probability distribution with mean µ i .
2. The distributions are all of the same form (e.g. all Normal with variance σ 2 , or all Poisson, etc.)
3. The Y i s are independent.
Systematic Part
g(µ i ) = x T i β
p∑
= β j x ij ,
j=1
where
1. x i = (x i1 . . . x ip ) T is a vector of explanatory variables,
2. β = (β 1 . . . β p ) T is a parameter vector, and
3. g(·) is a monotonic function called the link function.
Comments
1. If Y i ∼ N (µ i , σ 2 ) and g(·) is the identity function, then we have the NLM.
2. Other GLMs typically must have their parameters estimated by maximising the likelihood numerically
(iteratively in a computer).
3. The principles behind fitting GLMs are similar to those for fitting NLMs
Example: ‘logistic regression’
1. Random part: binary response
e.g. Y i |x i =
{ 1 if individual i survived
0 if individual i died
(and all Y i s are conditionally independent given the corresponding x i s).
Note that µ i = E[Y i |x i ] is here the probability of surviving given explanatory variables x i , and is
usually written p i or π i .
2. Systematic part:
( )
πi
g(π i ) = log .
1 − π i
103

Exercise 7.1
Show that under the logistic regression model, if n patients have identical explanatory variables x say, then
(a) Each of these n patients has probability of survival given by
π = exp(xT β)
1 + exp(x T β) ,
(b) The number R surviving out of n has expected values nπ and variance nπ(1 − π).
‖
7.2 Simpson’s Paradox
Simpson’s paradox occurs when there are three RVs X, Y and Z, such that the conditional distributions
[X, Y |Z] show a relationship between [X|Z] and [Y |Z], but the marginal distribution [X, Y ] apparently
shows a very different relationship between X and Y . For example,
1. X(Y )=male(female) death rate, Z=age,
2. X(Y )=male(female) admission rate to University, Z=admission rate for student’s chosen course.
7.3 Problems
1. (a) Explain what is meant by
i. the Normal linear model,
ii. simple linear regression, and
iii. nonlinear regression.
(b) For simple linear regression applied to data (x i , y i ), i = 1, . . . , n, show that the maximum likelihood
estimators ̂β 0 and ̂β 1 of the intercept β 0 and slope β 1 satisfy the simultaneous equations
and
Hence find ̂β 0 and ̂β 1 .
̂β 0
̂β 0 n + ̂β 1
n
∑
i=1
x i + ̂β 1
n
∑
i=1
n
∑
i=1
x i =
x 2 i =
n∑
i=1
y i
n∑
x i y i .
(c) The following table shows Y , the survival time (weeks) of leukaemia patients and x, the corresponding
log of initial white blood cell count.
i=1
x Y x Y x Y
3.36 65 4.00 121 4.54 22
2.88 156 4.23 4 5.00 1
3.63 100 3.73 39 5.00 1
3.41 134 3.85 143 4.72 5
3.78 16 3.97 56 5.00 65
4.02 108 4.51 26
Plot the data and, without carrying out any calculations, discuss how reasonable are the assumptions
underlying simple linear regression in this case.
From Warwick ST217 exam 1998
104

2. Because of concerns about sex discrimination, a study was carried out by the Graduate Division at
the University of California, Berkeley. In fall 1973, there were 8,442 male applications and 4,321
female applications to graduate school. It was found that about 44% of the men and 35% of the
women were admitted.
When the data were investigated further, it was found that just 6 of the more than 100 majors
accounted for over one-third of the total number of applicants. The data for these six majors (which
Berkeley forbids identifying by name) are summarized in the table below.
Men
Women
Number of Percent Number of Percent
Major applicants admitted applicants admitted
A 825 62 108 82
B 560 63 25 68
C 325 37 593 34
D 417 33 375 35
E 191 28 393 24
F 373 6 341 7
Discuss the possibility of sex discrimination in admission, with particular reference to explanatory
variables, conditional probability, independence and Simpson’s paradox.
Data from Freedman et al. (1991), page 17
3. (a) At a party, the POTAS 1 of your dreams approaches you, and says by way of introduction:
Hi—I’m working on a study of human pheromones, and need some statistical help. Can
you explain to me what’s meant by ‘logistic regression’, and why the idea’s important
Give a brief verbal explanation of logistic regression, without (i) using any formulae, (ii) saying
anything that’s technically incorrect, (iii) boring the other person senseless and ruining a
potentially beautiful friendship, (iv) otherwise embarrassing yourself.
(b) Repeat the exercise, replacing logistic regression successively with:
Bayesian inference, conditional expectation, likelihood,
a multinomial distribution, multiple regression, the Neyman-Pearson lemma,
nuisance parameters, one-way ANOVA, order statistics,
the Poisson distribution, a linear model, size & power,
statistical independence, a t-test.
(c) Suddenly, a somewhat inebriated student (SIS) appears and interrupts your rather impressive
explanation with the following exchange:
SIS: Think of a number from 1 to 10.
POTASOYD: Erm—seven
SIS: Wrong. Get your clothes off.
You then watch aghast while he starts introducing himself in the same way to everyone in the
room. As a statistician, you of course note down the numbers x i he is given, namely
7, 2, 3, 1, 5, 2, 10, 10, 7, 3, 9, 1, 2, 2, 7, 10, 5, 8, 5, 7, 3, 10, 6, 1, 5, 3, 2, 7, 8, 5, 7.
His response y i is ‘Wrong’ in each case, and you formulate the hypotheses
H 0 : y i = ‘Wrong’ irrespective of x i
{ ‘Right’ if xi = x
H 1 : y i =
0 , for some x 0 ∈ {1, 2, 3, 4, 5, 6, 7, 8, 9, 10},
‘Wrong’ if x i ≠ x 0 .
How might you test the null hypothesis H 0 against the alternative H 1
1 Person Of The Appropriate Sex
105

4. (a) Explain what is meant by:
i. a generalised linear model,
ii. a nonlinear model.
(b) Discuss the models you would most likely consider for the following data sets:
i. Data on the age, sex, and weight of 100 people who suffered a heart attack (for the first
time), and whether or not they were still alive two years later.
ii. Data on the age, sex and weight of 100 salmon in a fish farm.
From Warwick ST217 exam 1996
I have yet to see any problem, however complicated, which, when you looked at it the right
way, did not become still more complicated.
Poul Anderson
The manipulation of statistical formulas is no substitute for knowing what one is doing.
Hubert M. Blalock, Jr.
A judicious man uses statistics, not to get knowledge, but to save himself from having ignorance
foisted upon him.
Thomas Carlyle
The best material model of a cat is another, or preferably the same, cat.
A. Rosenblueth & Norbert Wiener
A little inaccuracy sometimes saves tons of explanation.
Saki (Hector Hugh Munro)
karma police arrest this man he talks in maths he buzzesLikeAfridge hes like a detuned radio.
Thom Yorke
Better is the end of a thing than the beginning thereof.
Ecclesiastes 7:8
106

Bibliography
Barnett, V. (1982). Comparative Statistical Inference (Second ed.). New York: John Wiley and Sons.
Burt, C. (1966). The genetic determination of differences in intelligence: A study of monozygotic twins
reared together and apart. Brit. J. Psych. 57: 137–153.
Casella, G. and R. L. Berger (1990). Statistical Inference. Pacific Grove, CA: Wadsworth &
Brooks/Cole.
Casella, G. and R. L. Berger (2001). Statistical Inference (second ed.). Pacific Grove, CA: Duxbury
Press.
DeGroot, M. H. (1989). Probability and Statistics (Second ed.). Reading, Mass.: Addison-Wesley.
DeGroot, M. H. and M. J. Schervish (2002). Probability and Statistics (Third ed.). Reading, Mass.:
Addison-Wesley.
Efron, B. and R. J. Tibshirani (1993). An Introduction to the Bootstrap, Volume 57 of Monographs
on Statistics and Applied Probability. New York: Chapman and Hall.
Freedman, D., R. Pisani, R. Purves, and A. Adhikari (1991). Statistics (Second ed.). New York:
W. W. Norton.
Gilks, W. R., S. Richardson, and D. J. Spiegelhalter (eds.) (1996). Markov Chain Monte Carlo
in Practice. London: Chapman and Hall.
Hand, D. J., F. Daly, A. D. Lunn, K. J. McConway, and E. Ostrowski (eds.) (1994). A
Handbook of Small Data Sets. London: Chapman and Hall.
Hogg, R. V. and A. T. Craig (1970). Introduction to Mathematical Statistics. New York: MacMillan.
Lindgren, B. W. (1994). Statistical Theory (Fourth ed.). London: Chapman and Hall.
Mood, A. M., F. A. Graybill, and D. C. Boes (1974). Introduction to the Theory of Statistics
(Third ed.). New York: McGraw-Hill.
Moore, D. S. and G. S. McCabe (1998). Introduction to the Practice of Statistics (Third ed.).
Oxford, UK: W. H. Freeman & Company Limited.
Moore, D. S. and G. S. McCabe (2002). Introduction to the Practice of Statistics (Fourth ed.).
Oxford, UK: W. H. Freeman & Company Limited.
Narula, S. C. and J. F. Wellington (1977). Prediction, linear regression and minimum sum of
relative errors. Technometrics 19: 185–190.
O.P.C.S. (1995). 1993 Mortality Statistics, Volume 20 of DH2. London: Her Majesty’s Stationery Office.
Osborn, J. F. (1979). Statistical Exercises in Medical Research. Oxford, UK: Blackwell Scientific Publications.
Rice, J. A. (1995). Mathematical Statistics and Data Analysis (Second ed.). Pacific Grove, CA:
Wadsworth.
Sprent, P. (1998). Data Driven Statistical Methods. London: Chapman and Hall.
Weisberg, S. (1980). Applied Linear Regression. New York: John Wiley and Sons.
107