From Algorithms to Z-Scores - matloff - University of California, Davis

52 CHAPTER 3. DISCRETE RANDOM VARIABLES
constants, we have
so the minimizing c is c = EX!
In other words, the minimum value of E[(X − c) 2 ] occurs at c = EX.
0 = −2EX + 2c (3.64)
Moreover: Plugging c = EX into (3.63) shows that the minimum value of g(c) is E(X − EX) 2 ] ,
which is Var(X)!
3.9 Covariance
This is a topic we’ll cover fully in Chapter 8, but at least introduce here.
A measure of the degree to which U and V vary together is their covariance,
Cov(U, V ) = E[(U − EU)(V − EV )] (3.65)
Except for a divisor, this is essentially correlation. If U is usually large (relative to its expectation)
at the same time V is small (relative to its expectation), for instance, then you can see that the
covariance between them will be negative. On the other hand, if they are usually large together or
small together, the covariance will be positive.
Again, one can use the properties of E() to show that
Also
Cov(U, V ) = E(UV ) − EU · EV (3.66)
V ar(U + V ) = V ar(U) + V ar(V ) + 2Cov(U, V ) (3.67)
Suppose U and V are independent. Then (3.17) and (3.66) imply that Cov(U,V) = 0. In that case,
V ar(U + V ) = V ar(U) + V ar(V ) (3.68)
By the way, (3.68) is actually the Pythagorean Theorem in a certain esoteric, infinite-dimesional
vector space (related to a similar remark made earlier). This is pursued in Section 19.7 for the
mathematically inclined.