WEEK 8 - Of the Clux

WEEK 8
Expectation
Let (Ω, F, P) be a probability space. Let X be a random variable on this probability
space and µ X its distribution. If its cummulative distribution function (c.d.f.)
F X (y) = µ X ((−∞, y)) is differentiable, then we define the probability density function
(p.d.f.) f X as the derivative of F X . In this case, we know that the expectation
of the random variable is defined as
∫
E(X) = xf X (x)dx.
R
This is well defined provided that f X is measurable. However, if the random variable
takes only finitely (or counably) many values {a 1 , . . . , a p }, then the c.d.f is a step
function that is not differentiable everywhere (in particular, it is not differentiable at
{a 1 , . . . , a p }). Then, we have seen in previous years that the expectation is defined
as the sum:
p∑
E(X) = a i P(X = a i ).
i=1
However, it is possible to define a random variable that does not fit in any of the above
categories (consider, for example, the random variable with a uniform distribution on
the Cantor set, as defined in homework 6). How do we define its expectation Can
we find a way to define the expectation for all random variables, without using their
p.d.f. or c.d.f. but only their distribution µ X It turns out that this is possible by
using the Lebesgue construction of the integral, only now it is not with respect to the
Lebesgue measure but the measure µ X :
Definition 0.1. Let X be a random variable defined on a probability space (Ω, F, P).
We define the expectation of X with respect to the probability measure P,
E(X) as follows:
(i) if X = 1 A is a indicator random variable, i.e. a random variable that is 1 on a
set A ∈ F and zero otherwise, then
E (1 A ) = P(A).
(ii) if X = ∑ n
i=1 a i1 Ai for some A i ∈ F, i = 1, . . . , n is a simple random variable,
i.e. a random variable that only takes finitely many values, then
E
( n∑
i=1
a i 1 Ai
)
=
n∑
a i P(A i ).
i=1
(iii) if X is a non-negative random variable, then
E(X) = lim
n→∞
E(X n ),
1

for (X n ) n≥1 a sequence of simple random variables defined as:
⎧
i−1
⎨ , if i−1 ≤ X(ω) < i and for i = 1, . . . , n2 n
2 n 2 n 2 n
X n (ω) = n, X(ω) ≥ n
⎩
0, otherwise
. (1)
So,
( n2 n
∑
E(X) = lim
n→∞
i=1
( n2 n
∑
n→∞
= lim
i=1
i − 1
2 n P(i − 1
2 n ≤ X < i
2 n ) + nP(X ≥ n) )
i − 1
2 n µ X
(
[ i − 1
2 n , i
2 n ) )
+ nµ X ([n, +∞))
)
. (2)
(iv) if X is any random variable, we can always write it as a difference between two
non-negative random variables
X = X + − X − ,
where X + = X1 X≥0 and X − = −X1 X

We now state the Radon-Nikodym theorem:
Theorem 0.5. A measure µ on B(R) is absolutely continuous with respect to the
Lebesgue measure if and only if there exists a measurable function f : R → R such
that
∫
∫
∀A ∈ B(R), µ(A) = 1 A (x)f(x)dx = f(x)dx.
R
We call f the Radon-Nikodym derivative of measure µ with respect to the Lebesgue
measure and we often write f = dµ
dx .
We will show that if X has a p.d.f. f X , then
∫
E(X) = xf X (x)dx (3)
as expected.
(i) We define
R
∫
˜µ X (A) =
A
f X (x)dx.
It is easy to check that this is a probability measure. Its c.d.f. will be
˜F X (y) =
∫ y
−∞
f X (x)dx =
∫ y
−∞
A
F ′ X(x)dx = F X (y)
since f X is the derivative of F X and F X (−∞) = 0. So, the measures µ X and
˜µ X have the same c.d.f. and by Caratheodory’s extension theorem, the two
measures should be the same on B(R):
∀A ∈ B(R), µ X (A) = ˜µ X (A).
We conclude that µ X is absolutely continuous w.r.t. the Lebesgue measure and
its Radon-Nikodym derivative is dµ X
dx = f X.
(ii) Suppose X ≥ 0. Then,
( n2 n
∑
E(X) = lim
n→∞
i=1
( n2 n
∑
n→∞
= lim
= lim
n→∞
∫R
i=1
( n2 n
(
i − 1
2 µ n X [ i − 1 )
2 , i
n 2 ) + nµ n X ([n, +∞))
i − 1
1
2
∫R
n [
i−1
2 n , i
2 n ) (x)f X(x)dx + n
∑
i=1
i − 1
2 n 1 [ i−1
2 n , i
2 n ) (x) + n1 [n,+∞)(x)
∫
R
)
)
=
1 [n,+∞) (x)f X (x)dx
f X (x)dx
It is easy to see that the limit of the function inside the parenthesis is x. Using
the Monotone Convergence Theorem which we state below, we can take the
limit inside the integral and we get
∫ ∞
0
3
xf X (x)dx
)
=

as promised. It is now easy to check that (3) is also true for any random variable
(not necessarily non-negative).
We define a discrete random variable in the usual way:
Definition 0.6. X is called a discrete random variable if there exists a finite or
countantable set A ∈ B(R) (|A| ≤ | N |) such that µ X (A) = 1.
Clearly, the distribution of a discrete random variable is not absolutely continuous
with respect to the Lebesgue measure. In fact, they are mutually singular: we say
that a measures µ on B(R) and the Lebesgue measure are mutually singular if there
exists an A ∈ B(R) such that Leb(A) = 0 and µ(A c ) = 0.
We said that if X is a discrete random variable, its distribution and the Lebesgue
measure are mutually singular. The opposite is not true: for example, the randon
variable with uniform distribution on the Cantor set (see homework 6) is not discrete
but its distribution and the Lebesgue measure are mutually singular: the uniform
distribution on the Cantor set has measure 1 on the Cantor set while the Lebesgue
measure of the Cantor set is 0.
It is often convenient to write the distribution of a discrete random variable as a
linear combination of delta measures: A delta measure on some x ∈ R, denoted by
δ x is defined as
{ 1 , if x ∈ A
∀A ∈ B(R), δ x (A) =
0 , otherwise
If the distribution of a random variable X is δ x for some x ∈ R, then we know that
whatever the outcome of the random experiment, X = x with probability one. Thus,
the expectation of any measurable function g of X will be Eg(X) = g(x).
If X is a discrete random variable taking values in some set (a i ) p i=1 (with the
possibility that p = ∞), i.e.
X =
p∑
a i 1 Ai , where A i = X −1 ({a i })
i=1
then its distribution µ X can be written as
µ X =
p∑
p i δ ai where p i = P(A i ).
i=1
The expectation of X will then be
E(X) =
p∑
a i p i
i=1
which is consistent with the definition we used before.
4

Monotone and Dominated Convergence Theorems
In order to define the expectation of a non-negative random variable X, we have used
a particular sequence of partitions of R + corresponding to the sequence of simple
functions (X n ) n≥1 defined in (1). However, any other sequence of partitions would
have given the same result. This is a consequence of the Monotone Convergence Theorem.
Before stating this theorem, we have to define almost sure (a.s.) convergence
with respect to a measure µ:
Definition 0.7. Let X n and X be random variables defined on (Ω, F, P). We say
that a sequence of random variables (X n ) n≥1 converges almost surely (a.s.) to X with
respect to P if the set of ω’s for which it does not converge to X(ω) has measure 0:
P ({ω : X n (ω) does not converge toX(ω) }) = 0
or, equivalently,
P ({ω : X n (ω) → X(ω)}) = 1.
Note that a.s. convergence with respect to any measure is weaker than pointwise
convergence: if X n (ω) → X(ω) as n → ∞ for all ω, then the set of all ω for which
X n (ω) does not converge to X(ω) will be the empty set and, consequently, will have
measure zero.
Theorem 0.8 (Monotone Convergence Theorem). Let X n be an increasing
sequence of non-negative random variables defined on a probability space (Ω, F, P).
If the sequence X n converges almost surely to a random variable X, then
lim E(X n) = E(X).
n→∞
This theorem has the following consequence: suppose X is a non-negative measurable
random variable defined on (Ω, F, P). Let (y (n)
i ) Nn
i=0 be any sequence of partitions
of [0, ∞), i.e.
0 = y (n)
0 < y (n)
1 < y (n)
2 < · · · < y (n)
N < n−1 y(n) N n
and y (n)
N n
→ ∞, such that
lim
n→∞
max (y i − y i−1 ) = 0 and {y (n)
i
i=1,...,N n
For every n ≥ 1, we define X y n as follows:
∑N n
Xn(x) y = y (n)
i−1 1 )
.
X
([y −1 (n)
i−1 ,y(n) i )
i=1
} Nn
i=1 ⊆ {y(n+1) i } N n+1
i=1
Then, one can show using the same arguments as before that (X y n) n≥1 is an increasing
sequence of non-negative random variables that converges pointwise to X. Thus, by
the monotone convergence theorem,
lim
n→∞ EXy n = EX,
5

i.e. the limit of the expectation of the simple functions Xn y corresponding to a partition
(y (n)
i ) Nn
i=0 that approximate X is independent of the choice of partitions, as promised!
Another corollary of the monotone convergence theorem is the following:
Corollary 0.9. Suppose that the Riemann-Stieltjes integral of function f exists.
Then, its Lebesgue integral will be equal to the Riemann-Stieltjes integral.
Proof. Suppose f ≥ 0. Since the Riemann-Stieltjes integral exists, it will be equal to
∫
(
Nn
)
∑
( )
f(x)dx = lim
inf f(z
R
n→∞
∫R
(n)
i ) 1 (n) (x
dx
i−1 ,x(n) i ]
i=1
z (n)
i
∈(x (n)
i−1 ,x(n) i ]
for (x (n)
i ) Nn
i=0 a sequence of partitions of satisfying
lim
n→∞
max (x i − x i−1 ) = 0 and {x (n)
i
i=1,...,N n
} Nn
i=1 ⊆ {x(n+1) i } N n+1
i=1 .
Remember that since the Riemann-Stieltjes integral exists, it will be independent of
the choice of partition. The functions
f n (x) =
∑N n
i=1
z (n)
i
inf
∈(x (n)
i−1 ,x(n) i ]
( )
f(z (n)
i ) 1 (n) (x i−1 ,x(n) i ]
will be non-negative, increasing and will converge to f almost surely – remember that
for the Riemann-Stieltjes integral to exist, f has to be continuous or have up to many
discontinuities. f n might not converge to f on the discontinuities, but since there are
only finitely many of them, their measure is 0. Thus, by the monotone convergence
theorem, the Lebesgue integral will also be equal to the limit of the integrals of f n
and thus equal to the Riemann-Stieltjes integral.
The monotone convergence theorem allows us to interchange limits and expectations.
Another theorem that allows us to do that is the Dominated Convergence
Theorem:
Theorem 0.10 (Dominated Convergence Theorem). Let X n be a sequence of
random variables defined on a probability space (Ω, F, P). If the sequence X n is
bounded almost surely by a random variable Y , i.e. P(|X n | ≤ Y ) = 1 and EY < ∞,
and the X n converge almost surely to a random variable X, then
lim E(X n) = E(X).
n→∞
Example 1. We are not always allowed to interchange limits and probabilities. For
example, let f n (x) = n (0,
1
)(x). Then,
n
while
∀x ∈ [0, 1],
∀n ≥ 1,
∫ 1
0
lim f n (x) = 0
n→∞
f n (x)dx = n · 1
n .
6

Consequently
0 =
∫ 1
0
lim f n(x)dx ≠ lim
n→∞
n→∞
∫ 1
0
f n (x)dx = 1.
Note that the sequence (f n (x)) n≥1 is neither bounded nor increasing and thus, neither
the monotone convergence theorem not the dominated convergence theorem apply.
7