30) Review of Asymptotic Theory

Econ 311
PART I: REVIEW OF ASYMPTOTIC
THEORY
Elements of Asymptotic Theory
Inequalities for Random Variables
EX 2 / 2
Chebyshev’s Inequality
If EX 2 < ∞, thenPr [|X| >] ≤
Schwarz Inequality
E |XY | 2 ≤ E |X| 2 E |Y | 2
Convergence concepts for random
variables
1

Almost sure convergence
X T (ω)
as
−→ X(ω) iff
2

Pr [lim T →∞ |X T (ω) − X(ω)| ≤ ] =
1 ∀ >0
Strong Law of Large Numbers
Let X T = 1 TX
Y i and X = lim T →∞ E(X T )
T
be finite. If X T (ω)
i=1
as
−→ X, then{Y i } obeys
the SLLN. Sufficient conditions for {Y i } to
obey the SLLN:
{Y i } independent and
(Kolmogorov)
∞X
i=1
1
i 2V (Y i) < ∞
{Y i } iid and E(Y i ) exists (also a necessary
condition)
3

Convergence in rth Moment
X T (ω)
rth moment
−→
X if E(X r T ) and
E(X r ) exist and lim T →∞ E [|X T − X| r ]=0
Convergence in Probability
X T (ω)
p
−→ X(ω) iff
lim T →∞ Pr [|X T (ω) − X(ω)| >]=0
∀ >0
In other words, ∃ T 0 such that
T > T 0 =⇒ Pr [|X T − X| >]=0. If
X T
p
−→ k, a constant, we write plimX T = k.
4

Weak Law of Large Numbers
Let X T = 1 TX
Y i and X = lim T →∞ E(X T )
T
i=1
be finite. If X T (ω)
p
−→ X, then{Y i }
obeys the WLLN. A sufficient condition
for {Y i } to obey the WLLN is that
1
TX
lim T −→∞ V (X
T 2
T )=0.
T =1
Slutsky Theorem
X T
p
−→ X and g(X) continuous =⇒
g(X T )
p
−→ g(X).
If X is a constant, we write plim g(X T )=
g(plim X T ). So if g(X T ) is an estimate
5

of some parameter and we don’t know the
distribution of g(X T ), then approximate it
with the distribution of g(X).
Convergence in Distribution
X T
d
−→ X (rv or constant) iff F T → F
pointwise, where F is the cdf of X.
Preliminary Results
d
1. g(·) continuous and X T −→ X =⇒
d
g(X T ) −→ g(X)
p
d
d
2. X T −→ Y T and Y T −→ Y =⇒ X T −→
p
Y where X T −→ Y T means
lim
T −→∞ Pr[|X T(ω) − Y T (ω)| >ε]=0.
Man and Wald Theorem
6

d
X T −→ X and Y T
d
1. X T + Y T −→ X + c
p
−→ c =⇒
2. X T Y T
d
−→ Xc
3. The limit of the joint distribution of
(X T ,Y T ) exists and equals the joint
distribution of (X, c).
Delta Method
If {α T } is a sequence of nonstochastic
numbers tending to ∞, α T (Z T − θ)
d
−→ X
where θ is a constant, and g 00 (·) exists, then
α T [g(Z T ) − g(θ)]
d
−→ g 0 (θ)X. If g 00 (·)
exists and √ T (Z T − θ)
d
−→ N(0,σ 2 θ ),then
√
T [g(ZT ) − g(θ)]
d
−→ N ¡ 0,g 0 (θ) 2 σ 2 θ¢
.
7

Comparing Convergence Concepts
X T
X T
d
−→ X
as
−→ X =⇒ X T
p
−→
⇑ X =⇒
X T
rth moment
−→
X
If X is a constant, then
X T
d
−→ X =⇒ X T
p
−→ X.
8

Central Limit Theorems
Lindberg-Levy CLT
{Y t } iid, Ȳ = 1 T
TX
t=1
E(Y t )=µ, V (Y t )=
Liapounov CLT
Y t
σ 2 =⇒ √ T (Y − µ)
d
−→ N(0,σ 2 ) .
{Y T } independent, E(Y t )=µ t ,V(Y t )=
σ 2 t ,andE(Y 3
t ) exist =⇒ √ T (Y − µ)
d
−→
N(0, σ 2 ),where
σ 2 = lim 1 T
TX
σ 2 t.
t=1
9

[1] Amemiya, Advanced Econometrics,
1985, chapter 3.
[2] Greenberg and Webster, Advanced
Econometrics, 1991, chapter 1.
10

PART II: ANALOGY PRINCIPLE AND
EXTREMUM ESTIMATORS
METHODS OF ESTIMATION
Consider Four Methods
(1) MLE
(2) [Nonlinear] Least Squares
(3) Methods of Moments
(4) GMM
11

Modern Methods in Econometrics
These methods are not always distinct in a
particular problem:
e.g.
classical normal linear regression
model
Y t = X t β + U t U t ∼ N(0,σ 2 u)
U t
iid
X t
Fixed constant
P
Xt X 0 t
T
Full rank .
In this case, OLS is all four rolled into one.
12

One basic principle underlies these methods:
Analogy Principle: (Large Sample version).
This is due to Karl Pearson or Goldberger.
Suppose ∃ true model in population
M(Y,X,θ 0 )=0, an implicit equation. θ 0
is a true parameter vector. Defined for other
values of θ (at least some other). Construct
some “criterion” or function of model and
data:
Q(Y,X,θ)
which has property P in the population when
13

θ = θ 0 .
In the sample, construct something which
converges to Q for each θ
14

Q T (Y,X,θ) almost surely, uniformly
−−−−−−−−−−−−−−−−→ Q
for some θ ∈ Θ
(T isthesamplesize)
and mimics in sample properties of Q in
population.
Let ˆθ T be the estimator of θ 0 in sample
T formed from the analog in sample
(which causes sample Q T (
) to have the
Property).
Then, we have under “regularity conditions”
∧
θ T p −→
θ 0
15

Regularity conditions:
(Basic ideas)
(a) Identification: only one value of θ
satisfies P in population (at least in nbd of
θ 0 ). One can define this using a Local vs. a
Global definition.
(b) Need Q T ( ∧ θ T ) to converge “nicely”: in
particular
Q T (θ) → Q(θ)
all θ ∈ Θ.
Need convergence to be uniform:
i.e. if ∧ θ T → C, Q T ( ∧ θ T ) → Q(C) .
This is the intuition underlying the method:
Consider the property that
Q is maximized at θ = θ 0
(in population).
16

Select criterion so Q T (θ) → Q
for
all θ ∈ θ
=
.
Select b θ T maximize Q T (θ) for each T .
Then if convergence is “O.K” we get that
under regularity, we have convergence in the
sample to maximizer in the population.
Now suppose θ = {1, 2, 3} a finite set.
..
·
θ ∧ assumes only one of 3 values.
by (a) Q is maximized at one of these.
each θ.
by hypothesis Q T (θ)
p
−→ Q(θ)
for
17

..
·
the rule picks
∧
θ T
= argmax Q T (θ).
This must work for T “big enough,”
because Q T
p −→
Q for each θ. Why
As T gets “big” we get Q T (θ) arbitrarily
close to Q(θ)
..
·
evaluate Q T (θ) at θ =1, 2, 3
∧
θ is the value that maximizes Q(θ), but
maximizing value is θ 0 .
..
·
∧
θ T p −→
θ.
Otherwise one gets a contradiction Q T ( ∧ θ)
18

Q(θ 0 ). This can’t happen as Q T → Q since
θ 0 is defined to be the maximizing value.
Here Principle P is extremum principle:
Examples
max : Maximum Likelihood
min : Nonlinear Least Squares.
Some Definitions
Convergence of sequences of Functions:
(Assume Real)
For each x ∈ S form a sequence {F n (x)} .
Let B be the set of points for which F n (x)
19

converges.
F (x) = lim
n→∞
F n (x) if x ∈ B. Then we
call F (x) a limit function. {F n (x)} converges
pointwise on B.
Basic Fact:
Pointwise convergence is not enough to
guarantee that if F n has a property (continuity,
differentiability, etc.) that the property is
inherited by F .
For this notion we need uniform convergence.
(Sufficient Condition)
20

e.g. does lim
x→x0
F n (x) =F n (x 0 ) for each n
⇒ lim
x→x0
F (x) =F (x 0 )
(where F is the limit function)
e.g. does lim lim F n (x) = lim lim F n (x)
x→x0 n→∞ n→∞ x→x0
No, not in general.
Example:
F n (x) =n 2 (x)(1 − x) n ; 0 ≤ x ≤ 1
lim n→∞ F n (x) =0
for each fixed x
but set x = 1
µ n +1
1
lim n→∞ F n
n +1
∞
µ µ n 1 n
= n 2 →
n +1 n +1
A sequence of functions {F n } converges
21

uniformly to F on B if for every ε>0 , ∃ an
N (depending only on ε and not on x) such
that for all
n>N⇒ |F n (x) − F (x)|

(y, x) related to θ in population.
Q 0 = Q
(Population moments (y, x))
In the sample
∧
θ T = θ (Sample moments)
Now two cases:
Case A: Q T
= Q (OLS) Slutsky’s
Theorem:
ˆθ T = θ (Sample Moments)
Sample moments → population moments
Given (a)
∧
θ T
p
−→ θ 0
(both solve same equation – we don’t need B.)
23

Case B: Q T
p
−→ Q We need uniform
convergence in order for Q T (sample moments)
→ Q
for (sample moments) and
ˆθ → θ for (population moments)
Example:
OLS
Y t = X t β 0 + U t E(U t )=0 Var(U t )=σ 2 u
E(X 0 tU t )=0
E(X 0 tX t )= X xx positive definite E(X0 tY t )= X xy
Assume (Y t X t ) iid sequence.
In population: E(X 0 tU t )=0
24

or
Q :
Xty 0 t = XtX 0 t β 0 + XtU 0 t
X
= X β 0 +0
xy xx
β 0 = X −1 X
xx xy
in sample:
⎛X
∧
β T = ⎝
T
X 0 tX t
⎞
⎠
X
⎝
−1 ⎛
T
X t Y t
⎞
⎠
..
·
rhs →
∧
β → β 0
³X
´−1 X
xx xy
Another Analogy:
Define U as Û
E(X 0 tU t )=0 b U = y − x b β mimics
U !
Pick ∧ β as
25

(1)
1
T
1
T
(Leads to Different Analogy)
then
X ∧
Xt (Y t − X tβ)=0
⎛X
⎞
X X t X t
Xt Y t = ⎝ ⎠ β
∧
ObservethatwehavethatwhenX t =1
T
∧
β =
X
Yt
T
:themean.
Extremum Principle which underlies Nonlinear
Least Squares (and MLE) is also illustrated
in linear regression:
Y t = X t β +[X t (β 0 − β)+U t ]
(Y t − X t β) 2 =(β 0 − β) 0 XtX 0 t (β − β 0 )
26

+2U t X t (β − β 0 )+U 2 t .
Let Q = E[Y t − X t β] 2
= (β −
β 0 ) 0 X xx (β − β 0)+σ 2 u .SinceU t and X t
independently E(U t )=0.
min Q wrt β at min we have β = β 0 .
β = β 0 .
Now in population: min wrt β occurs at
..
·
Form Sample Analog:
Q T = 1 T
TX
(Y t − X t β) 2
t=1
Analog Principle:
27

plim Q T = plim 1 T
TX
(Y t − Xtβ) 0 2
t=1
=(β −β 0 ) 0 X xx (β −β 0)+σ 2 U
pick ∧ β to min . Can’tmakeitanysmallerthan
β 0 in limit. Get contradiction unless ∧ β → β 0 .
28

Extremum Principle:
In the population
Y t = g(X t ; θ)+[g(X t ; θ 0 ) − g(X t ; θ)] + U t
assume (U t ,X t ,Y t ) iid
U t ⊥X t
..
·
U t ⊥g(X t ; θ) all θ
Q = E(Y t −g(X t ; θ)) 2 = E [g(X t ; θ 0 ) − g(X t ; θ)] 2 +σ 2 U
at θ = θ 0 ,Qminimized in population.
If θ = θ 0 is the only such value, model is
29

identified
(wrt Q criterion).
Analogy in Sample: Pick
Q T (θ) = 1 TX
(Y t − g(X t ; θ)) 2
T
t=1
∧
θ T = argmin Q T (θ)
min Q T (θ) for all T : Q T (ˆθ T )

U t ⊥g(X t ; θ 0 ) ⇒ E(U t g(X t ; θ)) = 0
(only one implication of ⊥).
Now we may write
Y t − g(X t ; θ) =U t
..
·
E [(Y t − g(X t ; θ)] = 0
just by analogy.
Sample b Q = 1 T
TX
(Y t − g(X t ; θ))g(X t ; θ).
t=1
Find θ which set = ˆQ (or is close to zero
as possible). Now we can make analogy
complete.
Maximum Likelihood:
31

Suppose the joint density of data
f(y t ,x t ; θ) =f(y t | x t ; θ)f(x t )
assume that x t is “exogenous” i.e. density of
x t is uninformative about θ.
Assume random sampling. Then, we
have that
Q
$ = T f(y t ,x t ; θ)
t=1
or taking (y t ,x t ) as data, is a function of θ
TX
ln £ = ln f(y t, x t ; θ) =
TX
ln f(x t )
t=1
t=1
TX
ln f(y t | x t ,θ)+
t=1
32

Principle of Maximum Likelihood: Pick
θ thatmaximizes£. Thisisanextremum
version of analogy principle.
In the population define
33

Q = E θ0 (ln f(y t ,x t ; θ))
Z
= [ln f(y t ,x t ; θ)] f(y t ,x t ; θ 0 )dy t dx t
(Assume exists)
This is always at max when θ = θ 0 .
Why
because
· ¸ f(yt ,x t ; θ)
E θ0
f(y t ,x t ; θ 0 )
=1
=
Z f(yt ,x t ; θ)
f(y t ,x t ; θ 0 ) f(y t,x t ; θ 0 )dy t dx t =1.
Now apply Jensen’s inequality
E(ln(x)) ≤ ln E(x)
(ln is concave function)
· ¸¸
f(yt ,x
..E
·
t ; θ)
θ0
·ln
≤ 0
f(y t ,x t ; θ 0 )
34

..E
·
θ0 [ln f(y t ,x t ,θ)] ≤ E θ0 [ln f(y t ,x t ,θ 0 )] ∀θ
Identified globally in the population if the
inequality is strict for all θ 6= θ 0 .
..
·
Construct sample Q b :
$ = 1 TX
ln f(y t ,x t | θ)
T
t=1
b θ =argmax£ (assume exists)
..
·
if Q T → Q all θ ∈ Θ
and if convergence is uniform, consistency
follows immediately.
Local form of the principle
Z
Q(θ) = ln f(y; θ)f(y; θ 0 )dy = E θ0 (ln f(y; θ))
35

Max θ
FOC:
SOC:
Z ¸
·∂ ln f(y, θ)
f(y; θ 0 )dy =0
∂θ
Z ∂ 2 ln f(y; θ)
∂θ∂θ 0 f(y; θ 0 ) negative definite
Requirement:
1
TX
T
t=1
1
TX
T
t=1
∂ 2 ln f
∂θ∂θ 0
(Local conditions)
∂ ln f(y t ; θ)
∂θ
=0
negative definite
Either way, same basic idea. For each T
pick ˆθ T such that:
Q T
³
bθT´
>Q T (θ).
Now if Q T
³
bθT´
→ Q(lim b θ T ) (uniform
36

convergence) we get Q( b θ T ) > Q(θ θ ),
assumed to be a maximum value. Thus we get
a contradiction unless plim b θ T = θ 0 . Q.E.D.
To be more precise: we need definitions.
Random Function: Let (Ω, A,P) be probability
space and let Θ ⊂ R k . A real function
F (θ) =F (θ, w) on Θ× Ω is called a random
function on Θ if for every t ∈ R 1 and θ ∈ Θ
{w ∈ Ω : F (θ, w)

Proposition:
If ϕ(θ, x) is continuous real
function on Θ × R n
where Θ is compact,
sup θ∈Θ ϕ(θ, x) and inf ϕ(θ, x)
θ∈Θ
are continuous functions.
Proof: See any Real Analysis text (see, e.g.
Royden).
Proposition: If for almost all values of x in
X, g(x,θ) is continous wrt θ at point θ 0 ,and
if for all θ in nbd of θ 0 we have
|g(x, θ)|

Z
lim
θ→θ 0
g(x, θ)dF (x) =
Z
g(x, θ 0 )dF (x)
i.e. lim θ→θ0 E(g(x, θ)) = E(g(x, θ 0 ))
Proposition:
If for almost all values of x in
X and for a fixed value of θ
(a)
(b)
∂g(x, θ)
exists (in nbd of θ)
∂θ
g(x, θ + h) − g(x, θ)
¯ h ¯

function to another. Let F (θ) and F n (θ) be
random functions on Θ ⊂ R k for each θ ∈ Θ.
1
F n (θ) almost surely converges to F (θ) as
n →∞if
P {w | lim n→∞ |F n (θ, w) − F (θ, w)| n 0 (θ, ε) i.e. the set S θ such that
|F n (θ, w) − F (θ, w)| ≥ ε if n ≥ n 0 (θ, ε)
has no probability. Now: S = ∪ S θ may
θ∈θ
=
have non-negligible probability even though
any one set has negligible probability. We
avoid this by the following definition.
40

Definition: Almost sure uniform convergence:
(2) F n (θ) → F (θ) almost surely uniformly in
Θ
if sup |F n (θ) − F (θ)| → 0 almost surely as
θ∈Θ
n →∞
i.e. for every ε>0 and w ∈ Ω.
P {w : lim n→∞ sup θ∈Θ |F n (θ, w) − F (θ, w)| ≤ ε} =
1.
In this case, the negligible set not indexed by
θ.
41

Definition:
Convergence in Probability. Let
F n (θ) and F (θ) be random functions on Θ.
Then F n (θ) → F (θ) in probability
uniformly in θ on Θ if
lim n→∞ Prob{sup θ∈Θ |F n (θ) − F (θ)| >ε} =
0
converges i.e. Q T (θ) → Q(θ) for all θ.
42

Theorem 1 Let {x n } be a sequence of random
k×1 vectors. Suppose iid. Let F (x, θ) be
a continuous real function on R k . Θ is compact
(Closed and Bounded so it has a finite
subcover).
43

Define ψ(a) = sup sup |F (x, θ)| .
kx

1
sup
T
T
TX
E(ψ | x j |) 1+σ < ∞.
j=1
Need a bound in either case.
45

PART III: ANALOG PRINCIPLE
1. Analog Principle
A. Intuition
Suppose we know some properties that
are satisfied for the “true parameter” in the
population. If we can find a parameter value
in the sample that causes the sample to mimic
the properties of the population, we might
use this parameter value to estimate the true
parameter.
B. Definition
46

Suppose there exists a true model of the
population:
M(y, x, θ) =0.
Create based on this model, a “criterion
function” for the population,
Q(y, x, θ),
which, at θ = θ 0 , achieves a property P .Now
in the sample, construct an “analog” to Q,
Q T (y, x, θ), a normalized (by T ) likelihood or
least squares criterion which has the following
two properties
pr. a.s
(a) Q T (y, x, θ) −→ Q(y, x, θ) ∀θ ∈ Θ,
and
47

(b) Q T (y, x, θ) mimics, in the sample, the
properties of Q(y, x, θ), in the population.
Suppose ˆθ T is the value for θ that causes
Q T to assume the property P in the sample.
By the analogy principle, ˆθ T should be our
sample estimator for θ 0 .
C. Consistency
In general, in order to prove that ˆθ T is a
consistent estimator of θ 0 ,i.e.ˆθ T
pr. a.s.
−→ θ 0 ,we
need to make the following two assumption,
whichareoftenreferredtoas“regularity
conditions”:
48

(a) Only θ 0 causes Q to possess property P ,at
least in some neighborhood, i.e. θ 0 must be
at least locally identified; and
(b) Q T → Q uniformly ∀θ ∈ Θ.
The first condition insures that θ 0 can be
identified. If there is more than one value of
θ that causes Q to have property P ,thenwe
cannot be sure that only one value of θ will
cause Q T to assume property P in the sample,
in which case we may not be able to determine
what ˆθ T estimates. The second condition is
a technical one that insures that if Q T has a
property such as continuity or differentiability,
49

this property is inherited by Q. This property
is uniform convergence.
Specific proofs of consistency depend
on which property we suppose Q has when
θ = θ 0 .
50

D. Applications
1. Moment principle
According to the moment principle,
let the property P be some moments of
the population. Set Q equal to a vector of
moments and θ 0 equal to the value of those
moments in the population,
Q (population moments [y, x]) = θ 0 .
Then, select Q T and ˆθ T such that
Q T ([sample moments (y, x)]) = ˆθ T ,
where either Q T = Q or Q T
P
−→ Q. Note
51

that in the case where Q T = Q, we don’t need
uniform convergence to prove consistency,
while in the case where Q T
P
−→ Q, we do.
Example: OLS
The OLS estimator can be interpreted
as an application of the moment principle
and is therefore also a GMM estimator. OLS
models the population as follows: y = xβ
+ u. Q and θ 0 are selected such that the
population moment to be matched in the
sample is E(x 0 u)=0.Fromthisitiseasyto
52

analytically derive the OLS estimator:
53

E[x 0 (y − xβ)] = E[x 0 y]-E[x 0 x]β
=0=⇒ β = E[x 0 x] −1 E[x 0 y].
OLS can also be interpreted (a fact which will
be proven later in the notes) as a nonlinear
least squares estimator, which in turn is an
application of the extremum principle, which
we will now describe.
2. Extremum principle
According to the extremum principle, let
the property P be the maximum or the
minimum so that at θ = θ 0 , Q is maximized
54

or minimized in the population. Two common
applications of the extremum principle are
the maximum likelihood estimator (MLE)
and the nonlinear least squares (NLLS)
estimator. As their names suggest, the MLE
chooses P to be the maximum and the NLLS
estimator chooses P to be the minimum.
The following two sections will review
the MLE and NLLS estimators. Proofs of
consistency and asymptotic normality for
the two estimators will be presented in a
55

somewhat specialized, but easily generalized,
form in the next section.
The following theorems lay out the conditions
under which extremum estimators are
consistent and asymptotically normal. They
each talk about estimators using estimators
using the maximum principle, but can trivally
be extended to minimum principle estimators
by placing a negative sign in front of Q, i.e.
min Q = max [−Q].
a. Consistency of extremum estimators
56

The first theorem proves consistency
when the criterion function has a globally
unique maximum or minimum, respectively in
the population. Thus Θ is uniquely identified.
No differentiability is required. The second
theorem states the additional assumptions you
have to make if Q is only locally identified,
i.e. there are multiple solutions to max Q but
only one is in the neighborhood N(θ 0 )ofθ 0 .
It assumes differentiability.
Theorem 1 (Global): Assume that
57

(a) Parameter space Θ is a compact subset of
< k ,
(b) Q T (y, θ) is continuous in θ, θ ∈ Θ, ∀y and
is a measurable function of y ∀θ ∈ Θ;
(c) Q T (y, θ) P → Q(θ), a nonstochastic function,
in probability uniformly as T →∞;
and
(d) θ 0 =argmax θ∈Θ Q 0 (y, θ) is globally
identified. (Achieves global maximum
at θ 0 ).
If we let ˆθ T =argmax θ∈Θ Q T (y, θ), then
ˆθ T
P
−→ θ 0 .
Observe that continuity of Q(θ) follows
from the fact that limits of uniformly continuous
functions are continuous and continuity of
Q in θ and compactness of Θ implies uniform
58

continuity.
Proof: Let N(θ 0 ) be an open neighborhood
in < k containing θ 0 . Then N c (θ 0 ), the
complement of N(θ 0 ), is closed, so N c (θ 0 )∩Θ
is compact. Therefore
max Q(θ)
θ∈{N c (θ 0 )∩Θ}
exists. Denote ε = Q(θ 0 )− max Q(θ) >
θ∈{N c (θ 0 )∩Θ}
0.
Let A T be the event
A T = {| Q T (θ) − Q(θ) |< ε/2} =
{−ε/2

due to uniformity of convergence. Then A T
implies
(a) Q(ˆθ T ) >Q T (ˆθ T ) − ε/2,
(b) Q T (θ 0 ) >Q(θ 0 ) − ε/2.
But, because Q T (ˆθ T ) >Q T (θ 0 )bythe
definition of ˆθ T ,wehave
Q(ˆθ T ) >Q T (θ 0 )-ε/2 >Q(θ 0 )-ε,
using (a). This fact and (b) imply Q(ˆθ T ) >
Q(θ 0 ) − ε. Rearranging, these facts imply
that A T implies ε>Q(θ 0 ) − Q(ˆθ T ),which,
since we have a strict inequality, implies
ˆθ T ∈ N(θ 0 ),forT sufficiently big. Since
60

A T =⇒ {ˆθ T ∈ N(θ 0 )},itmustbethat
Pr{A T } ≤ Pr{ˆθ T ∈ N(θ 0 )}.
But, by assumption (c),
lim T →∞ Pr{A T } → 1,soˆθ T
P
−→ θ 0
because ε is arbitrary.
Theorem 2 (Local): Assume that
(a) parameter space Θ is an open subset of < k
that contains θ 0 ;
(b) Q T (y, θ) is a measurable function of y ∀
θ ∈ Θ;
(c) ∂Q T
exists and is continuous in an open
∂θ
neighborhood N 1 (θ 0 ) of θ 0 (this implies
Q T is continuous ∀ θ ∈ N 1 (θ 0 ));
(d) there exists an open neighborhood N 2 (θ 0 )
of θ 0 such that Q T (y, θ) → Q(θ), a
nonstochastic function, in probability
uniformly ∀ θ ∈ N 2 (θ 0 ) as T →∞;and
61

(e) θ 0 = argmax θ∈N2 (θ 0 )Q(θ) is locally identified.
∂Q T
∂θ
If we let ˆΘ T denote the set of roots of
=0corresponding to the local maxima;
then, for any ε>0,
½
lim Pr inf
T →∞
θ∈ ˆΘ T
¾
| θ − θ 0 |>ε
=0.
Proof: See Amemiya, chapter 4, or the next
set of notes.
b. Asymptotic normality of extremum
estimators
Now we will show that under certain
conditions on the first and second derivatives
62

of Q, the criterion function for an estimator
which uses the extremum principle and
choose Θ T as the maximum, the asymptotic
distribution of ˆθ T is normal.
Theorem 3 (Cramer): Assume the conditions
of Theorem 2 and, in addition,
(a) parameter space Θ is an open subset of < k ;
(b) ∂2 Q T
0 exists and is continuous in an open
∂θ∂θ
neighborhood of θ 0 ;
(c) there exists an open neighborhood N(θ 0 )
of θ 0 such that Q T (y, θ) → Q(θ), a
nonstochastic function, in probability
uniformly ∀θ ∈ N(θ 0 ) as T →∞;
(d) θ 0 = argmax θ∈N(θ0 ) Q(θ) is locally
identified;
63

(e) ∂2 Q T (θ) p ¯¯θ∗
∂θ∂θ 0 T
−→ A(θ 0 ) if θ ∗ T → θ 0 in
µ ∂ 2
Q T (θ)
probability where A(θ 0 )=plim
∂θ∂θ 0 | θ0
is nonsingular; and
(f) √ µ ∂QT (θ)
T |θ 0 ∼ N(0,B(θ 0 ))
∂θ ·∂Q(θ)
where B(θ 0 )=E · ∂Q(θ)0 |θ 0¸
.
∂θ ∂θ
If we let ˆθ T denote the root of ∂Q T
∂θ =0
then
√
T (ˆθT −θ 0 ) ∼ N(0,A(θ 0 ) −1 B(θ 0 )A(θ 0 ) −1 ).
64

Proof: By assumption,
∂Q T ¯
=0. ¯ˆθT
∂θ
Taking a Taylor expansion of the l.h.s.
around θ 0 ,wehave
∂Q T ¯
= ∂Q T
¯ˆθT
∂θ ∂θ | θ 0
+ ∂2 Q T
∂θ∂θ 0 | θ ∗ (ˆθ T − θ 0 ),
where θ ∗ lies between ˆθ T and θ 0 . Multiplying
by √ T ,
√
T
∂Q T
∂θ
| θ 0
+ ∂2 Q T
∂θ∂θ 0 | θ ∗ √
T (ˆθT − θ 0 )=0.
Rearranging, we get
√
T (ˆθT − θ 0 )=
µ ∂ 2 −1
Q T
−
∂θ∂θ 0 | √ ∂Q T
θ ∗ T
∂θ | θ 0
.
Since ˆθ T
p
−→ θ 0 =⇒ θ ∗
p
−→ θ 0 , we see
65

the first object on the r.h.s. becomes
∂ 2 Q T
∂θ∂θ 0 | θ ∗
p
−→ ∂2 Q T
∂θ∂θ 0 | θ 0
= A(θ 0 ),
where A(θ 0 ) is a constant. As for second
object on the r.h.s., by assumption,
µ
√ ∂QT (θ) T |θ 0 ∼ N(0,B(θ 0 )).
∂θ
Putting this all together we have, by
Slutsky’s Theorem,
√
T (ˆθT − θ 0 ) ∼
N(0,A(θ 0 ) −1 B(θ 0 )A(θ 0 ) −1 ).¤
66

PART IV: HYPOTHESIS TESTING:
LARGE SAMPLE
3Tests:
(1)L.R.Test
(2) Wald
(3) Rao, Score or L. M.
Consider a Simple Null Hypothesis:
H 0 : θ = θ 0
H A : θ 6= θ 0 .
Natural test uses the fact that
√
T (ˆθ − θ0 ) ∼ N(0,I −1
θ 0
)
I θ = − 1 T
∂ 2 n$
∂θ∂θ 0 ¯¯¯¯θ0
67

√
T (ˆθ − θ0 ) 0 I θ0
√
T (ˆθ − θ0 ) ∼ χ 2 (k).
Let k = Numbers of Parameters.
This is Wald Test: Like F test
From uniform convergence of ˆθ to θ 0 we
obtain
plim = I Iˆθ0 θ0
Second Test (Rao Test): More intuitive.
Observes that if θ = θ 0 (a known value), θ 0
should be a root of the likelihood equation
∂n$
TX
∂θ | ∂nf(y t ,θ)
θ 0
=
¯ =0.
∂θ
¯θ=θ0
Now
t=1
68

1 ∂n$
√ ¯ = 1 TX
√
T ∂θ
¯θ=θ0 T
t=1
N(0,I θ0 )
∂nf(y t ,θ)
∂θ
¯
¯
¯θ0
∼
Test Score =0.
µ 0 µ
1 ∂n$
1
√ I −1 ∂n$
θ T ∂θ
0
√ χ 2 (k)
θ=θ 0 T ∂θ
θ=θ 0
again.
tests:
Look at the relationships between the two
X ∂nf
∂θ
k
0
by construction
X ∂nf
¯ =
µ
∂θ
¯θ=ˆθ X
∂nf
∂θ∂θ 0
¯
¯θ=θ0
+
(ˆθ − θ 0 )
θ ∗
θ ∗ is an intermediate
69

value
kθ 0 k ≤ kθ ∗ °
k ≤ °ˆθ °
Duality relationships between score and
parameter vectors.
µ (ˆθ − θ 0 )=
X ∂ 2
nf
−
∂θ∂θ 0
X ∂nf
θ=θ
∂θ
∗
¯
¯θ=θ0
µ −1
T
√
T (ˆθ − θ0 )=
µ X ∂ 2 −1
nf 1 X ∂nf
∂θ∂θ 0 √ ¯
θ=θ ∗ T ∂θ
¯θ=θ0
Substitution into Wald: we get
µ 0 µ
1 ∂n$
1
√ Iθ −1
T ∂θ
0
I θ0 Iθ −1 ∂n$
0
√ =
T ∂θ
Rao.
This is sometimes called the Score Test.
70

∂n$
∂θ
is the score or L. M. test from the
solution to the following problem
n$ − λ 0 (θ − θ 0 )
∂n$
− λ =0
∂θ
at null λ = ∂n$ =0
∂θ
(can test on λ or on the score).
Likelihood Ratio Test: Expand L. F. around
Random Pt.
n$(θ 0 )=n$(ˆθ)+ ∂n$
¯
∂θ
¯θ=ˆθ(θ 0 − ˆθ)
+(ˆθ − θ 0 ) 0∂2 n$
∂θ∂θ
¯¯¯¯θ∗
0 (ˆθ − θ 0 )
kθ 0 k < kθ ∗ °
k < °ˆθ °
71

2[n $(ˆθ) − n $(θ 0 )]
θ 0 )
" #
$(ˆθ)
2 n
$(θ 0 )
= √ µ −1
T (ˆθ−θ 0 ) 0 ∂ 2
n$ √T
(ˆθ−
T ∂θ∂θ 0
d
−→
χ 2 (k) ⇐⇒ Rao Test
√
T (ˆθ − θ0 ) 0 I θ
√
T (ˆθ − θ0 )
high values - reject // low values - don’t reject.
θ 0 = True Parameter
θ H = Hypothesized Value
Consider the L. R. Derivation
L. R. Test: Expand around a random point
n$(θ 0 )=n$(ˆθ)
72

θ 0 )
+ ∂n$
¯
∂θ
¯θ=ˆθ
(θ 0 −ˆθ)+ 1 2 (ˆθ−θ 0 ) 0∂2 n$
∂θ∂θ
¯¯¯¯θ∗
0 (ˆθ−
θ ∗ is an intermediate value.
Tests: 2[n £(ˆθ)− n $(θ 0 )]
·
=2 − 1 √
T (ˆθ − θ0 ) 0 1 ∂ 2 n$
2
T ∂θ∂θ
¯¯¯¯θ∗
0
√ √
T (ˆθ − θ0 ) 0 I θ0 T (ˆθ − θ0 )
d
=
χ 2 (k) ⇐⇒ Wald
¸
√
T (ˆθ − θ0 )
73

Composite Hypothesis
Let θ =(θ 1 ,θ 2 )
unconstrained
H 0 : θ 1 = 0,θ 2
H A : θ 1 ,θ 2
unrestricted
$(ˆθ 1 ,
L. R. Test =2n
ˆθ 2 )
$(θ 1 =0, ˆθ 2 ) =
−2n $(θ 1 =0, ˆθ 2 )
$(ˆθ 1 , ˆθ 2 )
˜θ 2 is unrestricted value when θ 1 =0imposed.
But observe that in testing the null, we have
θ 1 =0isassumedtobetrueso(¯θ 1 =0, ¯θ 2 ) is
the true value θ 0 .
74

L. R. = −2[n £(θ 1 =0, ˜θ 2 ) − n
£(¯θ 1 , ¯θ 2 )]
+2[n £(ˆθ 1 , ˆθ 2 ) − n £(¯θ 1 , ¯θ 2 )].
By previous logic, we have
√
T (ˆθ1 − ¯θ 1 , ˆθ 2 − ¯θ 2 ) 0 I(¯θ 1 , ¯θ 2 )( √ T (ˆθ 1 −
¯θ 1 , ˆθ 2 − ¯θ 2 )
¯θ 2 ).
− √ T (0, ˜θ 2 − ¯θ 2 ) 0 I(¯θ 1 , ¯θ 2 ) √ T (0, ˜θ 2 −
Now consider the relationship between a
restricted and an unrestricted system.
Note: Use θ 0 and θ ∗ interchangeably
75

(neglecting o p (1) terms)
76

Unrestricted MLE
(***)
1 ∂n$
√
T ∂θ 1
¯¯¯¯θ=ˆθ
¯θ 1 )
= √ 1
·
∂n$ 1
+
T ∂θ 1
¯¯¯¯(θ0 ) T
+
· 1
T
∂ 2 ¸
ln $ √ ∧
T (
∂θ 1 ∂θ 0 θ2 − ¯θ 2 )
2 (θ 0 )
∂ 2 ¸
ln $ √T
∂θ 1 ∂θ 0 (ˆθ1 −
(θ 0 )
(*)
1 ∂ ln $
√ =
T
∧
∂θ 2
¯¯¯¯θ= 1 ∂ ln $
√
θ T ∂θ 2
¯¯¯¯θ0
· 1 ∂ 2 ln $ √ ³
+
T bθ2 − θ 2´
¸θ 0
+
T
· 1
T
∂θ 2 ∂θ 0 2
∂ 2 ln $ √ ³
T bθ1
∂θ 2 ∂θ 0 − θ 1´
1
¸θ 0
77

µ
Iθ1 θ
I θθ = 1
I θ1 θ 2
I θ2 θ 1
I θ2 θ2
Replace with Info Matrix
0= √ 1 ∂ ln $
T ∂θ 1
¯¯¯¯θ0
√ √
−I θ1 θ 1 T ( b θ1 − θ 1 ) − I θ1 θ 2 T ( b θ2 − θ 2 )
(*) 0= 1 √
T
∂ ln $
∂θ 2
¯¯¯¯θ0
− I θ2 θ 2
√
T ( b θ2 − θ 2 )
−I θ2 θ 1
√
T ( b θ1 − θ 1 ).
Recall that θ 0 =(¯θ 1 , ¯θ 2 ).
78

Collect Terms
Ã√ !
T ( b θ1 − θ 1 )
√
T ( b θ2 − θ 2 )
Next: Consider e θ 2 :
= I −1
θθ
⎛
⎜
⎝
1 ∂ ln $
√
T ∂θ 1
1 ∂ ln $
√
T ∂θ 2
MLE of θ 2 given
⎞
¯¯¯¯θ0
⎟
⎠ . ¯¯¯¯θ0
θ 1 = θ 1 . Expand around Root of Likelihood
¯θ 1 =0(assume true point).
Equation:
1 ∂ ln $
√ =
T ∂θ 2
¯¯¯¯θ1 1 ∂ ln $
√
, e θ 2 T ∂θ 2
¯¯¯¯θ0
+ 1 µ ∂ 2
ln $ √
T
T ∂θ 2 ∂θ 0 (˜θ2 − θ 2 ).
2 θ 0
Now the first term on the right hand side is in
common with the term on rhs of (*) above.
79

Therefore we have that
I θ2 θ 2
√
T ( e θ2 − θ 2 )=
I θ2 θ 2
√
T ( b θ2 − θ 2 )+I θ2 θ 1
√
T ( b θ1 − θ 1 )
or
(*) √ T ( e θ 2 − θ 2 ) = √ T ( b θ 2 − θ 2 )+
I −1
θ 2 θ 2
I θ2 θ 1
√
T ( b θ1 − θ 1 ).
Observe similarity to OLS Specification Bias
Formula.
Y = X 1 β 1 + X 2 β 2 + U
Delete X 1
Y = X 2 Π + {U +
X 1 β 1 }
80

µ
plim
Π 2 = β 2 +
X
0 −1 µ
plim 2 X 2 X
0
plim 2 X 1
T
T
Now let ¯£=(n f 1 ,nf 2 ,...,nf T )
β 1
81

⎡⎛
I θθ = E ⎢⎜
⎣⎝
µ
Iθ1 θ 1
I θ1 θ 2
I θ2 θ 1
I θ2 θ 2
∂ ln ¯$
∂θ 1
∂ ln ¯$
∂θ 2
⎞ ⎛
⎟
⎠
⎜
⎝
∂ ln ¯$
∂θ 1
∂ ln ¯$
∂θ 2
⎞0⎤
⎟
⎠
⎥
⎦ =
Score vector like X 0 U vector
∂ ln ¯$ = X
∂θ
1U
0
1
∂ ln ¯$ = X
∂θ
2U 0 etc.
2
only if scores uncorrelated
"
∂ ln
E
¯$ µ 0
#
∂ ln ¯$
=0
∂θ 2 ∂θ 1
i.e. if I θ2 θ 1
=0we will have that
√
T (˜θ2 − ¯θ 2 )= √ T (ˆθ 2 − ¯θ 2 ).
Discrepancy between ˜θ 2 and ˆθ 2 depends
82

on the correlation between score wrt θ 1 and
score wrt θ 2 .
Equivalence between Likelihood Ratio
and Wald Test: Substitute in formula in
bottomofpage5:˜θ 2 − ¯θ 2
LR = √ T (ˆθ 1 − ¯θ 1 , ˆθ 2 − ¯θ 2 ) 0 I¯θ1 ,¯θ 2
√
T (ˆθ1 −
¯θ 1 , ˆθ 2 − ¯θ 2 )
− √ h
T (ˆθ 2 − ¯θ
√
2 )+(I θ2 θ 2
) −1 I θ2 θ 1 T (ˆθ1 − ¯θ
0
1 )i
Iθ2 θ 2
h √T
(ˆθ2 − ¯θ
√
2 )+(I θ2 θ 2
) −1 I θ2 θ 1 T (ˆθ1 − ¯θ 1 )i
.
Call ˆθ 1 − ¯θ 1 = a
ˆθ 2 − ¯θ 2 = b
83

= √ Ta 0 I θ1 θ 1
√
Ta+
√
Tb 0 I θ2 θ 1
√
T · a +
√
Ta 0 I θ1 θ 2
b √ T
a
+ √ Tb 0 I θ2 θ 2
√
Tb
– √ Ta 0 I θ1 θ 2
(I θ2 θ 2
) −1 I θ2 θ 2
I θ2 θ 1
√
T · a
− √ Tb 0 I θ2 θ 2
√
T ·b 0 − √ Tb 0 I θ2 θ 2
I −1
θ 2 θ 2
I θ2 θ 1
√
T ·
= √ Ta 0 [I θ1 θ 1
− I θ2 θ 2
(I θ2 θ 2
) −1 I θ2 θ 1
] √ T · a
= √ T (ˆθ 1 −¯θ 1 )(I θ 1θ 1
) −1√ T ·a Q.E.D.
..χ
. 2 (k) k = rank (I θ1 θ 1
).
84

fact that
Obviously ⇔ Wald which exploits the
√
T ( b θ1 − θ 1 ) ∼
N(0,I θ 1θ 1
)
where I θ 1θ 1
is the upper left diagonal
block of
I −1
µ −1
θθ = Iθ1 θ 1
I θ1 θ 2
I θ2 θ 2
I θ2 θ 1
Score Test Version: Compute Derivative
wrt θ 1 using b θ 2 − value (estimate of θ 2 when
θ 1 constraint is imposed).
85

1 ∂ ln $
√ =
T ∂θ 1
¯¯¯¯θ1,ˆθ2 1
+
· 1
T
∂ ln $
√
T ∂θ 1
¯¯¯¯θ0
√
T
³˜θ2 − θ 2´
¸θ 0
∂ 2 ln $
∂θ 1 ∂θ 0 2
Substitute for ˜θ 2 − θ 2 from above (page 6).
' √ 1 ∂ ln $
− I
T ∂θ
θ1 θ 2
Iθ 1
¯¯¯¯θ0 −1 1 ∂ ln $
2 θ 2
√
T ∂θ 2
¯¯¯¯θ0
since
√T ³˜θ2 − θ 2´
= Iθ −1 1 ∂ ln $
2 θ 2
√
T ∂θ 2
¯¯¯¯θ0
Ã
!
1 ∂ ln $
Var √ = I θ1 θ
T ∂θ
1
+
1
¯¯¯¯θ1 ,ˆθ 2
I θ1 θ 2
I −1
θ 2 θ 2
I θ2 θ 2
I −1
θ 2 θ 2
I θ2 θ 1
(I θ 1θ 1
) −1
−2 I θ1 θ 2
I −1
θ 2 θ 2
I θ2 θ 1
=
..
·
We have the Rao test:
86

Ã
! Ã
!
1 ∂ ln $
√ I
T ∂θ 1
¯¯¯¯θ1 θ 1θ 1
1 ∂ ln $
√
, e θ 2 T ∂θ 1
¯¯¯¯θ1 ,˜θ
"
2
· ¸0
1 ∂ ln $
1
√ − I θ1 θ T ∂θ
2
Iθ 1
¯¯¯¯θ0 −1 ∂ ln $
2 θ 2
√
T ∂θ 2
=
"
1
√
T
∂ ln $
∂θ 1
¯¯¯¯θ0
− I θ1 θ 2
I −1
θ 2 θ 2
· 1
¸
∂ ln $
√
T ∂θ 2
θ 0
#
θ 0
# 0
I θ 1θ 1·
Observe that
·
√
T ( b θ1 − θ 1 )=I θ 1θ 1
1 ∂ ln $
√ + I
T ∂θ 1
¯¯¯¯θ0 θ 1θ 2 1 ∂n$ ¯¯θ0¸
√
T ∂θ
"
2
#
= I θ 1θ 1
1 ∂ ln $
√ − I θ1 θ 2
Iθ T ∂θ 1
¯¯¯¯θ0 −1 1 ∂ ln $
2 θ 2
√
T ∂θ 2
¯¯¯¯θ0
Substitute into expressions for L.R. or Wald to
get ⇔ with score
"
# 0 "
1 ∂ ln $
√ − I θ1 θ
T ∂θ
2
Iθ 1
¯¯¯¯θ0 −1 1 ∂ ln $
2 θ 2
√ I
T ∂θ 2
¯¯¯¯θ0 θ 1θ 1
.
1 ∂ ln $
√ − I θ1
T ∂θ 1
¯¯¯¯θ0
⇔ to Rao
Q.E.D.
87

Now Key Result in Regression is
Theorem of Partitioned Inverse:
Partition X =(X 1 X 2 )
X 0 X =
µ
X
0
1 X 1 X1X 0 2
X 1 X2 0 X2X 0 2
Let M 1 = I − X 1 (X 0 1X 1 ) −1 X 0 1
D = X 0 2X 2 − X 0 2X 1 (X 0 1X 1 ) −1 X 0 1X 2 =
X 0 2M 1 X 2
(X 0 X) −1 =
³ ´
(X
0
1 X 1 ) −1 )(I+X1X 0 2 D −1 X2D 0 −1 X2X 0 1 (X1X 0 1 ) −1 −(X1X 0 1 ) −1 X1X 0 2 D −1 )
−D −1 X2 0X 1(X1 0X 1) −1 D −1
=
³ ´
(X
0
1 X 1 ) −1 +(X1X 0 1 ) −1 X1X 0 2 (D −1 )X2X 0 1 (X1X 0 1 ) −1 −(X1X 0 1 ) −1 X1X 0 2 D −1
−D −1 X2 0X 1(X1 0X 1) −1 D −1
88

Thus we may always write
à ! bβ1
µ X
=(X
bβ 0 X) −1 0
1 Y
2
X2 0Y bβ 1 =(X 0 1X 1 ) −1 (X 0 1Y )+(X 0 1X 1 ) −1·
X 0 1X 2 D −1 X 0 2X 1 (X 0 1X 1 ) −1 X 0 1Y −
(X 0 1X 1 ) −1 X 0 1X 2 D −1 X 0 2Y
=(X 0 1X 1 ) −1 X 0 1Y −(X 0 1X 1 ) −1 (X 0 1X 2 )D −1
2 (M)Y
bβ 2 = −D −1 X2X 0 1 (X1X 0 1 ) −1 X1Y 0 +
D −1 X2Y
0 = D −1 X2(I 0 − X 1 (X1X 0 1 ) −1 X 1 )Y =
D −1 X2M 0 1 Y
89

Review Tests - Regression
LR k Wald k RAO
Analogy for MLE in neighborhood of
optimum we have that
√
T
³
bθ1 − θ 1´
⎛
= plim ⎜
⎝
∂ ln ¯$
∂θ
µ ∂ ln ¯$
T
∂θ
0
⎞
⎟
⎠
−1
1
√
T
∂n ¯£
∂θ ι0 T
or
90

⎛
T −1 ⎜
⎝
∂ ln ¯$ µ 0 ∂ ln ¯$ ∂ ln ¯$ µ 0
⎞
∂ ln ¯$
∂θ 1 ∂θ 1 ∂θ 1 ∂θ 2
∂ ln ¯$
µ 0 µ 0 ⎟
∂ ln ¯$ ∂ ln f ∂ ln ¯$ ⎠
∂θ 2 ∂θ 1 ∂θ 2 ∂θ 2
−1
⎛
1
√ ⎜
T
⎝
∂ ln ¯$
∂θ 1
∂ ln ¯$
∂θ 2
⎞
⎟
⎠ ι0 T
where ι T is a 1 × T vector of 1 s . Regress
scores on vector of 1 s .
91

√ ³
T bθ2 − θ 0,2´
=
⎡
⎢
"
⎣
I-
⎛
⎝I −
Analogy to
µ ∂ ln ¯$
∂θ 2
0
µ
∂ ln ¯$ ∂ ln ¯$0 ∂ ln ¯$ −1
∂ ln ¯$
#
0
∂θ 1 ∂θ 1 ∂θ 1 ∂θ 1
∂ ln ¯$
∂θ 1
" µ∂ ln ¯$
∂θ 1
0
∂ ln ¯$
∂ ln ¯$
∂θ
⎞ 2
# −1
∂ ln ¯$0
⎠ .
∂θ 1 ∂θ 1
⎤
⎥
⎦
−1
= [−D −1 (X2X 0 1 ) 1
(X1X 0 1 ) −1 X1 0 +
D −1 X2]
0
=(D −1 X2[I 0 − X 1 (X1X 0 1 ) −1 X1])
0
i.e.
92

⎡
⎢
"
⎣
I −
⎡
⎢
"
⎣
I −
µ 0 ∂ ln ¯$
∂θ
µ 2
∂ ln ¯$ ∂ ln ¯$0 ∂ ln ¯$ −1
∂ ln ¯$
#
0
∂θ 1 ∂θ 1 ∂θ 1 ∂θ 1
µ 0 ∂ ln ¯$
∂θ
µ 2
∂ ln ¯$ ∂ ln ¯$0 ∂ ln ¯$ −1
∂ ln ¯$
#
0
∂θ 1 ∂θ 1 ∂θ 1 ∂θ 1
purged score vectorsk
What is going on Simultaneous orthogonalization
Y = X 1 β 1 + X 2 β 2 + U
∂ ln ¯$
∂θ 2
⎤
⎥
⎦
⎤
⎥
⎦
−1
Regress X 2 on X 1 clean out X 1
Regress Y on X 1 clean out X 1
bβ 2 is regression of cleaned out Y on cleaned
93

out X 2
Cleaned out
Y = £ I − X 1 (X 0 1X 1 ) −1 X 0 1¤
Y =
YM 1 = Y − X 1 β 1
Cleaned out X 2 = £ I − X 1 (X1X 0 1 ) −1 X1¤ 0 X2
Regression β b 2 = ¡ X2M 0 1 X 2 ) −1 X2M 0 1 M 1 Y ¢ =
D −1 X2M 0 1 Y
Observe: That we really don’t have to clean
out Y ,justX since M 1 is idempotent. Also
observe that
bY 0 Y b =
94

Y 0 X 1 (X 0 1X 1 ) −1 X 0 1Y 1
Part due to X 1
+ Y 0 M 1 X 2 D −1 X 2 M 1 Y
Part due to orthogonalized X 2
Obviously unless regressors are orthogonal,
there are no unique contributions.
95

Specification Error
Y = X 1 β 1 + X 2 β 2 + U
b b1 =(X 0 1X 1 ) −1 X 0 1Y
plim b 1 =(X1X 0 1 ) −1 X1(X 0 1 β 1 + X 2 β 2 + U)
µ X
0 −1 µ
= β 1 + 1 X 1 X
0
1 X 2
β
T T 2 .
F test in classical linear regression model is
H o : β 2 =0
Y 0 M 1 X 2 D −1 X2M 0 1 Y/k
µ Y 0
MY
∼ F (k 2 ,T − k).
T − k
↑ unrestricted
Consider MLE in classical linear regression
model.
96

Y t = X t β+U t
U t ∼ N ¡ 0,σ 2 U
¢
iid
f(Y t | X t β)=f(Y t − X t β | X t β)
Now
f(U t )=
f(Y t | X t β)=
iid sampling
1
√
2πσ
2
U
exp − 1 2U 2 t
σ 2 U
1
√ exp − 1
2πσ
2
U
2σ 2 (Y t −X t β) 2
U
97

ln $ = − T 2 ln σ2 u − 1
2σ 2 u
Solution:
TX
(Y t − X t β) 2
t=1
bβ =
³X
X
0
t X t´−1 X
X
0
t Y t
∂ ln $
∂β
= X U t X t
bσ 2 u = 1 T
TX
(Y t − X tβ) b 2
t=1
98

1
T
1 ∂ 2 ln $
T ∂β∂β 0 = − (X0 X)
T
∂ 2 ln $
∂(σ 2 u)∂β = ∂ ·
− T ∂β 2
=
1
σ 2
1
σ 2 u
+ 1 X
¸
2(σ 2 u) 2 (Yt − X tβ) b 2
1
(2σ 2 u) 2 X
(Yt − X t
b β) 0 X t =0
(unless we have that the model is such that
there’s a functional relationship between σ 2 u
and β. This we exclude.)
1 ∂ 2 ln $
T ∂(σ 2 u) = − T
2 2bσ 2 u
I θ0 θ 0
=
⎛
⎜
⎝
1
σ 2 u
β and σ u Block Diagonal.
0
X 0 X 0
99
1
2σ 4 u
⎞
⎟
⎠

(By previous result we may ignore
parameter estimation error between β b and bσ 2 ).
Now consider L.R. test
Partition: β =(β 1 ,β 2 )
H 0 : β 2 =0
L.R.
⎛X ⎞
ln $(θ 0 )=− T 2 ln ⎜
³Y t − X 1tβ1´2
e
⎟
⎝
⎠
T
³X X
eβ 1 = X
0
1t X 1t´−1
X
0
1t Y t
⎛X ⎞
ln $( b θ)=− T (Yt
2 ln − X 1tβ1 b − X 2tβ2 b ) 2
⎝
⎠
T
"
$(
2ln
b # Ã
θ)
$( b = T £ Y
0
θ 0 ) 2 ln I − X 1 (X1X 0 1 ) −1¤ !
Y
.
Y 0 [I − X(X 0 X) −1 X 0 ] Y
100

Wald Test (Use partitioned inverse theorem on
information matrix above.)
√
T ( b β2 − β 0 2) ∼
N ¡ 0,σ 2 (X 0 2[I − X 1 (X 0 1X) −1 X 0 ]X 2 ) −1¢
.. √ ³
·
T bβ2 -β H 2
´0
"
X
0
2
¡
I-X1 (X 0 1X 1 ) −1 Y 0
β H 2
σ 2 u
T
is hypothesized
1¢
X2
# ³bβ2
-β H 2
´
Rao:
Compute the derivative of the Log
Likelihood at the point where we have that b β 1
is imposed. Use (*) rewrite (*) above as
X ³Y 1t − X 1tβ1 b − X 2t β 2´
2¯¯¯¯¯¯¯β2 =0
ln $(θ 0 )=− T 2 ln bσ2 U−
1
T
∂ ln $(θ 0 )
= −
∂β 2
¯¯¯¯β2=0 1 T
101
2bσ 2 U
X ³Y 1t − X 1tβ1´
b
2bσ 2 U
X 2t
.

Now what do we have Look at orthogonality
between bε t = Y 1t − X 1t
b β1 and X 2t . If the
residual is orthogonal, we are okay.
..
·
We have that the proposal is to construct
residual ⎛ test:
X
⎝ 1 T
⎛X
⎝
bε t X 2t
2bσ 2 u
bε t X 2t
T 2bσ 2 u
⎞0 ¡I ¢
⎠ β 2 β 2
1
T
⎞
⎠
0
X
bε t X 2t
2σ 2 u
σ 2 u
¡
X
0
2
¡
I − X1 (X 0 1X 1 ) −1 X 0 1¢
X2
¢ −1
·
1
T
X
bε t X 2t
2σ 2 u
Proof that Wald > LRT use fact that (b − 1) ≥
ln(b) for b>1
.
102

ESS(R) − ESS(U)
Wald = T
ESS(U)
¸
LRT = T ln
b = ESS(R)
ESS(U)
·ESS(R)
ESS(U)
Now b − 1 ≥ ln b for b ≥ 1
LRT = T ln b
Wald = T (b − 1)
Rao = T
T [1 − c]
¸
·ESS(R) − ESS(U)
ESS(R)
=
− ln c ≥ 1 − c
let c = ESS(U)
ESS(R)
103
LRT = −T ln c

·.. LRT ≥ Rao.
Observe a feature of Rao test. Is it
equivalent to regressing M 1 Y 1 on X 2
and testing for statistical significance No!
Regression residuals from regression of ε 2 on
x 1 and x 2
M 1 Y 1 = X 2 Π + V
bΠ =(X 0 2X 2 ) −1 X 0 2M 1 Y
OLS test of Π =0
104

T b Π 0 (X 0 2X 2 ) b Π
=
Y 0 M 1 X 0 2(X 0 2X 2 ) −1 (X 0 2X 2 )(X 0 2X 2 ) −1 X 0 2M 1 Y
((Y −X 2 (X 0 2 X 2) −1 X 0 2 M 1Y ) 0 Y −X 2 (X 0 2 X 2) −1 X 0 2 M 1Y )
= T
= T
Rao is:
Y 0 M 1 X 0 2(X 0 2X 2 ) −1 X 0 2M 1 Y
Y 0 [I−M 1 X 2 (X 0 2 X 2) −1 X 0 2 ][I−X 2(X 0 2 X 2) −1 X 0 2 M 1]Y
Y 0 M 1 X 0 2(X 0 2X 2 ) −1 X 0 2M 1 Y
Y 0 [I+M 1 X 2 (X 0 2 X 2) −1 X 2 M 1 −M 1 X 2 (X 0 2 X 2) −1 X 0 2 −X 2(X 0 2 X 2) −1 X 2 M 1 ]Y
T
k
Y 0 M 1 X2(X 0 2X 0 2 ) −1 X2M 0 1 Y
Y 0 M 1 M 2 M 1 Y
= T Y 0 M 1 X 2 (X2M 0 1 X 2 ) −1 X2M 0 1 Y
k Y 0 M 1 Y
only if M 1 X 2 = X 2 (so X 1 ⊥X 2 )arethese
equal but note.
If we regress MY 1 on X 1 and X 2 ,the
105

need to orthogonalize X 2 test is valid. Just
regress on X 2 purged of X 1 .
MY 1 = X 1 Π 1 + X 2 Π 2 + b V
OLS: test Π 2 =0
(verify as a homework exercise)
bΠ 2 =(X 0 2M 1 X 2 ) −1 X 0 2M 1 Y
OLS test of Π 2 =0
To set Rao test, replace denominator
with Y 0
1M 1 Y
T
(OLS constrained residuals -
obtained from constrained regression).
Nonlinear Least Squares:
Analogies.
Consider the following nonlinear least squares
106

model
Y t = f(X t | θ 0 )+U t .
Assume
107

1. U t iid E(U t )=0 E(Ut 2 )=σ 2 u
2. θ 0 is a k vector of unknown parameters
3. Assume X t ⊥U 0 t for all t, t 0
4.
∂f t
∂β exists and is continuous in nbd of β 0
5. f t (β) is continuous in β uniformly in t
(i.e. for ε>0 there exists δ>0 such that
|f t (β 1 ) − f t (β 2 )|

Proof: Amemiya p. 129. An application of
Extremum Analogy Theorem.
b θT =argminQ
Gallant (1975): Notice important analogy b θ T
is a root of equation
X
(Yt − f(X t | θ)) ∂f t
∂θ =0
Now expand f in nbd of θ ∗ in Taylor series
Y t = f(x t | θ ∗ )+ ∂f(x t | θ)
∂θ
¯¯¯¯θ=θ 0 (θ−θ ∗ )+U t .
∗
Rewrite the equation as
z t = y t − f(x t | θ ∗ )+ ∂f(x t | θ)
∂θ 0
= ∂f
∂θ (x t | θ)θ + U t + remainder.
¯¯¯¯θ=θ
∗
Now by analogy with classical linear regres-
109
θ ∗

sion model
y t − f(x t | θ ∗ )+ ∂f(x t | θ)
∂θ 0
is a dependent variable.
X role played by ∂f(x t | θ)
.
∂θ
¯¯¯¯θ=θ
∗
Then in nbd of true parameter (θ ∗ = θ 0 ) we
have that the remainder term is negligible.
NLLS estimator is
" TX (*) b ∂f(x t | θ)
¯
θ=
∂θ
t=1
X ∂f(xt | θ) ¯
∂θ
¯
¯θ=θ
0
¯
¯θ=θ
0
z t
µ ∂f(xt | θ)
∂θ 0 ¯¯¯¯θ=θ
0
# −1
110

X 0 X
X 0 Y
replaced by
replaced by
Ã
∂ ˜f
∂θ
! 0 Ã
∂ ˜f
∂θ
µ ∂f
∂θ
z
where ˜f =(f 1 , ..., f T ),Z =(Z 1 , ..., Z T ).
Then analogy with OLS goes through exactly.
Do hypothesis testing, etc. using derivatives
in nbd of optimum.
!
To justify large sample normality, we
need additional conditions on the model. The
required conditions for asymptotic normality.
111

1
TX
∂f t
(1) lim T →∞
¯
∂f t
→ C
T ∂θ ∂θ
t=1
¯θ0 0¯¯¯¯θ0
a positive definite matrix
(2) 1 TX ∂f t ∂f t
T ∂θ ∂θ 0
t=1
converges uniformly to a finite matrix in an
open nbd of θ 0 .
(3)
∂ 2 f t
∂θ∂θ 0
is continuous in θ in an open nbd of θ 0
uniformly (.. · need uniform continuity of first
and second partials).
(4) lim 1 TX
· ∂2 f 0¸ t
=0
T 2 ∂θ∂θ ∼
t=1
for all θ in open nbd of θ 0 .
TX
(5) 1 f t (θ 1 ) ∂2 f t
T ∂θ∂θ 0¯¯¯¯θ2
t=1
converges to a finite matrix uniformly for
all θ 1 ,θ 2 in open nbd of θ 0 .
112

Then
√
T ( b θT − θ 0 ) ∼ N(0,σ 2 uC −1 ).
The intuition for this is exactly as in Cramer
Theorem.
Look at first order condition:
∂Q
TX
∂θ = −2 (Y t − f t (θ)) ∂f t
∂θ
t=1
1 ∂Q
√ ¯ = − 1 X
√ (Ut ) ∂f t
¯
T ∂θ
¯θ=θ0 T ∂θ
¯θ=θ0
.
This is asymptotically normal (iid r.v.) by
Lindeberg-Levy Central Limit Theorem.
(Look at b θ).Nowuse(*)onpage17toobtain
113

"
b
1 θ−θ0 =
T
1
TX
√
T
t=1
TX
t=1
∂f(x t | θ)
∂θ
∂f(x t | θ)
(U t ).
∂θ
Asymptotically normal in nbd of θ 0 if
µ # −1
∂f(xt | θ)
∂θ 0
converges uniformly C by hypothesis.
Evaluate expressions at
θ = θ 0 .
.. √ ·
T ( b θ − θ 0 ) ∼ N(0,σ 2 C −1 )
Analogy with MLE proof, complete. (See
Amemiya for a rigorous derivation.) Also, see
the result in Gallant.
b θ
114

..
·
Like OLS and MLS.
Analogy with MLE complete if we
assume u t is normal. Then, $
ln $ = − T 2 ln σ2 u − 1 X
(Yt − f(X
2σ 2 t | θ)) 2
u
and FOC and asy. theory as before. General
Result: any nonlinear regresion model ⇐⇒
MLE if we have that U t normal. Nonlinear
regression is picking another criterion.
..
·
Efficient in normal case.
Now, consider estimation. How to obtain b θ
Two methods:
Newton-Raphson, Gauss-
115

Newton
Newton-Raphson:. (Expand criterion Q(θ) in
nbd of value b θ 1
Q(θ) ' Q( b θ 1 )+ ∂Q
0
∂θ ¯ (θ − b θ 1 )
b θ1
+ 1 2 (θ − b θ 1 ) 0 ∂ 2 Q
∂θ∂θ 0(θ − b θ 1 )
Quadratic Problem has a solution if Hessian
matrix ∂2 Q
0 is a definite matrix (pos. def.
∂θ∂θ
for min).
min Q(θ) wrt θ to get algorithm:
·
b θ2 = b θ 1 −
∂2 Q
0¸−1
∂Q
¯
∂θ∂θ ∂θ
b θ1
¯bθ1
.
Continue iteration until convergence occurs.
116

Method assumes that we can approximate
with a quadratic. There is a problem if
Hessian singular: method fails.
Correction:
·
Use λ such that
∂2 Q
λI¸
∂θ∂θ 0 − neg. def.
λ is scalar (obviously can pick vectors).
Fiddle with this to get out of nbd of local
singularity.
Notice in nonlinear model:
∂ 2 Q
TX
∂θ∂θ 0 = ∂f t ∂f t
∂θ ∂θ 0
t=1
in MLE use idea due to T.W. Anderson on
117

eading list and note that asymptotically (need
X 0 X)
µ 1
E
T
∂ 2
ln f
∂θ∂θ 0
= E
µ
− 1 T
∂ ln f
∂θ
∂ ln f
∂θ 0
sometimes called BHHH but method due to
Anderson.
Another problem: May overshoot. Then
scale step back by ϕ:
·
b θ2 = b θ 1 − ϕ
∂2 Q
0¸−1
∂θ∂θ
b θ1
0

Z t = Y t − f(y t | b θ 1 )+ ∂f
∂θ (y t | b θ 1 ) b θ 1
= ∂f
∂θ (y t | b θ 1 ) b θ 2 + U 2 .
These are data once one knows b θ.
Then do OLS:
"
b
1
TX
θ2 =
T
t=1
∂f t
∂θ
# −1
∂f t 1
∂θ 0 T
b θ1
∂f t
∂θ z t
Revise — update , start all over again.
Some problems as in Newton-Raphson.
(A)
(B)
To solve for optimum use
·∂f ∂f
λI¸
∂θ ∂θ 0 + λ scalar
To avoid overshouting, use Hartley
modification.
119

Form
∆ 1 = − 1 2
" TX
t=1
∂f t
∂θ
¯
# −1 ∂Q(θ)
¯
∂θ 0¯¯¯¯bθ1 ∂θ
¯bθ1
∂f t
¯bθ1
(Q(θ) = X (Y t − f t (X t | θ)) 2 choose λ
to minimize.
S( b θ 1 + λD 1 ) 0 ≤ λ ≤ 1
Then algorithim converges to a root of the
equation. General Global convergence is a
mess, unresolved.
120

Theorem: One Newton-Raphson Step
toward an optimum is fully efficient if you
start from an initial consistent estimator.
Suggests strategy. Get cheap – low cost
estimator. Consistent – not efficient. Then
iterate once – avoids computational cost.
(True also for Gauss-Newton).
Must use unmodified Hessians.
Proof:
Suppose b θ 1
P
−→ θ 0 and √ T ( b θ 1 − θ 0 )
d
−→
N(0, X ). It is consistent but not necessarily
θ
121

efficient. Now expand root of likehood
equation in nbd of b θ 1
∂ ln $
¯ = ∂ ln $
¯ + ∂2 ln $
∂θ
¯bθ1 ∂θ ∂θ∂θ
¯θ0
¯¯¯¯θ1 0 ( b θ 1 −
θ 0 )
b θ1 doesn’t, necessarily set left hand side
to zero. If it did, we would have an efficient
estimator. As before θ ∗ 1
is intermediate value.
Now look at Newton-Raphson criterion.
·∂2
¸−1 ¸
b θ2 −θ 0 =( b ln $
·∂ ln $
θ 1 −θ 0 )−
∂θ∂θ 0 b θ1
∂θ
·∂2 b
"
θ1
¸−1
=( b ln $ ∂ ln $
θ 1 −θ 0 )−
¯
∂θ∂θ 0 + ∂2 ln $
b θ1
∂θ ∂θ∂θ
¯θ0
¯¯¯¯θ∗
0
( b θ 1 − θ 0 )
#
Multiply by √ T ; collect terms.
122

√ ³
T bθ2 − θ 0´
= −
" · 1
+ I −
T
√
T
³
bθ1 − θ 0´
· 1
T
∂ 2 ¸−1
ln $ 1
∂θ∂θ 0 b θ1
T
∂ 2 ¸−1
ln $ 1
∂θ 2 ∂θ 0 2 b θ1
P
−→ √ 1 I −1 ∂ ln $
θ T
0
∂ ln $
√ ¯
T ∂θ
#
∂ 2 ln $
∂θ∂θ
¯¯¯¯θ∗ 0 1
∂θ
¯θ0
h £I ¤ √ ³
+ − I
−1
θ 0
I θ0 T bθ1 − θ 0´i
term vanishing if √ T (ˆθ 1 − θ 0 ) is 0 p (1).
..
·
One Newton-Raphson step satisfies
likelihood equation at θ 0 .
Asymptotically Efficient:
Same results obviously holds for Gauss-
Newton. One G-N step for a consistent
¯
¯θ0
123

estimator is fully efficient (or at least as
efficient as NLLS). Method saves computer
time. Avoids nonlinear optimatric. Avoids
local optimization problem.
“Durbin Problem”
Use an initial consistent estimator for
θ 2 . (Treat likelihood as if θ 2 known but it is
estimated by ˜θ 2 ).Letθ 0 =(¯θ 1 , ¯θ 2 ).Theseare
“true values”.
∂ ln $
¯ =0= 1 ∂ ln $
√
∂θ
¯bθ1 e θ2 T ∂θ 1
¯¯¯¯θ0
+ 1 ∂ 2 ln $
√ ³
¯ T bθ1
T ∂θ 1 ∂θ 0 − θ 1´
1
¯θ∗
124

+ √ 1
∂ 2 ln $
¯
T ∂θ 1 ∂θ 0 2
¯θ∗
³˜θ2 − θ 2´
↑
“Durbin Problem”.
125

√
T
³
bθ1 − θ 1´
= Iθ −1 1 ∂ ln $
1 θ 1
√
T ∂θ 1
¯¯¯¯θ0
√
−Iθ −1
1 θ 1
I θ1 θ 2 T
³˜θ2 − θ 2´
= I −1
θ 1 θ 1
1
√
T
∂ ln $
∂θ 1
| θ0
−I −1
θ 1 θ 1
I θ1 θ 2
I −1
θ 2 θ 2
1
√
T
∂n˜$
∂θ 2
¯¯¯θ2
where ˜£ is from the likelihood with sample
size used to produce ˜θ 2 . Need covariance
between the two score vectors.
126

à ! −1 Ã
√
T (˜θ2 − ¯θ 1 ∂ 2 ˜$ ∂
2 )=−
˜$
!
T ∂θ 2 ∂θ 0 2 ∂θ 2
√
T (ˆθ1 − ¯θ 1 )=−I −1 1 ∂n$
θ 1 ,θ 1
√ | θ0
T ∂θ 1
à ! −1
− Iθ −1 1 ∂ 2 ˜$ 1 ∂
1 θ 1
I θ1 θ 2
T ∂θ 2 ∂θ 0 √ ˜$ .
2 T ∂θ 2
Compute covariance to get the right
standard errors. Just form a new covariance
matrix
(*) Var(ˆθ 1 −¯θ 1 )=I −1
θ 1 θ 1
+ I −1
θ 1 θ 1
I θ1 θ 2
Ĩ −1
θ 2 θ 2
I θ2 θ 1
I −1
θ 1 θ 1
- I −1
θ 1 θ 1
I θ1 θ 2
Ĩ θ2 θ 2
Cov(Ŝ1, ˜S 2 )I −1
θ 1 θ 1
- Iθ −1
1 θ 1
Cov(Ŝ1, ˜S 2 )Ĩθ 2 θ 2
I θ2 θ 1
Iθ −1
1 θ 1
where S 1 = √ 1 ∂n £
T
¯¯¯θ2 =˜θ
∂θ 2 ,θ 1 =ˆθ 1
1
127
0

T is sample size
S 2 = 1 √ ˜T
∂n ˜£
∂θ 2
| θ2
˜T is sample size. Independent sample case:
Last two terms in (*) vanish.
Concentrated Likelihood Problem:
n £(α, β)
Solve ∂n$ (α, β) =0to get β(α). Then
∂β
optimize with respect to β(α). Same as joint
maximization of β,α.
128

Envelope Theorem:
dn £(α, β(α)) ∂n $(α, β(α))
=
dα
∂(α)
d 2 n £(α, β(α))
= ∂2 n$(α, β(α))
dαdα
∂α∂α 0
+ ∂β ∂ 2 n$(α, β(α))
.
∂(α) ∂α∂β
Now
¸
∂
·∂n £(α, β)
∂α ∂β
+ ∂β
∂(α)
∂β
∂α = n$
−∂2 ∂β∂α
∂ 2 n$
∂β∂β 0 =0
= ∂2 n£
∂β∂α
µ ∂ 2 n£
∂β∂β 0 −1
.
Substitute into previous expression
plim 1 T
I αβ I −1
ββ I βα)
d 2 n$(α, β(α))
dαdα
=(I αα −
129

Same asymptotic dist. for α as if we
estimate jointly.
130