Student Notes To Accompany MS4214: STATISTICAL INFERENCE

Definition 2.8 (Score function). For the (possibly vector valued) observation X = x
to be informative about θ, the density must vary with θ. If f(x|θ) is smooth and
differentiable, this change is quantified to first order by the score function
S(θ) = ∂
∂θ ln f(x|θ) ≡ f ′ (x|θ)
f(x|θ) .
Under suitable regularity conditions (differentiation wrt θ and integration wrt x can
be interchanged), we have
E{S(θ)} =
=
� ′ f (x|θ)
f(x|θ)dx =
f(x|θ)
�
∂
�� �
f(x|θ)dx =
∂θ
∂
∂θ
f ′ (x|θ)dx ,
1 = 0.
Thus the score function has expectation zero. �
True frequentism evaluates the properties of estimators based on their “long-run”
behaviour. The value of x will vary from sample to sample so we have treated the score
function as a random variable and looked at its average across all possible samples.
Lemma 2.7 (Fisher information). The variance of S(θ) is the expected Fisher
information about θ
Proof. Using the chain rule
I(θ) = E{S(θ) 2 } ≡ E
∂2 ∂
ln f =
∂θ2 ∂θ
� �
1 ∂f
f ∂θ
= − 1
f 2
�
∂f
∂θ
�
∂ ln f
= −
∂f
�� � �
2
∂
ln f(x|θ)
∂θ
� 2
� 2
+ 1 ∂
f
2f ∂θ2 + 1 ∂
f
2f ∂θ2 If integration and differentiation can be interchanged
�
1 ∂
E
f
2f ∂θ2 � �
=
∂2f ∂2
dx =
∂θ2 ∂θ2 �
dx = ∂2
1 = 0,
∂θ2 thus
X
X
� � �� � �
2
∂
∂
−E ln f(x|θ) = E ln f(x|θ) = I(θ). (2.1)
∂θ2 ∂θ
Variance measures lack of knowledge. Reasonable that the reciprocal of the variance
should be defined as the amount of information carried by the (possibly vector valued)
observation x about θ.
15