Theory of Statistics - George Mason University

36 1 Probability Theory
distribution used in an expectation, we may use notation for the expectation
operator similar to that we use on the individual distribution, as described
on page 23. Given the random variables X1 and X2, we use the notation EX1
to indicate an expectation taken with respect to the marginal distribution of
X1.
We often denote the expectation taken with respect to the joint distribution
as simply E, but for emphasis, we may use the notation EX1,X2.
We also use notation of the form EP, where P denotes the relevant probability
distribution of whatever form, or Eθ in a parametric family of probability
distributions.
Expectations of PDFs and of Likelihoods
If the marginal PDFs of the random variables X1 and X2 are fX1 and fX2,
we have the equalities
fX2(X1) fX1(X2)
EX1 = EX2 = 1. (1.61)
fX1(X1) fX2(X2)
On the other hand,
EX1(−log(fX1(X1))) ≤ EX1(−log(fX2(X1))), (1.62)
with equality only if fX1(x) = fX2(x) a.e. (see page 41).
When the distributions are in the same parametric family, we may write
fθ with different values of θ instead of fX1 and fX2. In that case, it is more
natural to think of the functions as likelihoods since the parameter is the
variable. From equation (1.61), for example, we have for the likelihood ratio,
L(θ2; X)
Eθ1 = 1. (1.63)
L(θ1; X)
Covariance and Correlation
Expectations are also used to define relationships among random variables.
We will first consider expectations of scalar random variables, and then discuss
expectations of vector and matrix random variables.
For two scalar random variables, X and Y , useful measures of a linear
relationship between them are the covariance and correlation. The covariance
of X and X, if it exists, is denoted by Cov(X, Y ), and is defined as
Cov(X, Y ) = E((X − E(X))(Y − E(Y ))) (1.64)
From the Cauchy-Schwarz inequality (B.21) (see page 845), we see that
(Cov(X, Y )) 2 ≤ V(X)V(Y ). (1.65)
Theory of Statistics c○2000–2013 James E. Gentle