Lecture3 Slide - The Department of Statistics and Applied Probability ...

1
ST5207 Nonparametric Regression, Lecture 3
Lijian Yang
Department of Statistics & Probability
Michigan State University
East Lansing, MI 48824
and
Department of Statistics & Applied Probability
National University of Singapore
Singapore 117546
ST5207 Nonparametric Regression, 27th January 2005

2
Step one, continued
• E |ξ in | 2 [
= n −2 E |K h (X i − x) − EK h (X i − x)| 2
= n −2 E {K h (X i − x)} 2 − {EK h (X i − x)} 2]
ST5207 Nonparametric Regression, 27th January 2005

3
Step one, continued
• E |ξ in | 2 [
= n −2 E |K h (X i − x) − EK h (X i − x)| 2
= n −2 E {K h (X i − x)} 2 − {EK h (X i − x)} 2]
•
E {K h (X i − x)} 2 =
∫
K 2 h (u − x) f (u) du = 1 h 2 ∫
K 2 ( u − x
h
)
f (u) du
ST5207 Nonparametric Regression, 27th January 2005

4
Step one, continued
• E |ξ in | 2 [
= n −2 E |K h (X i − x) − EK h (X i − x)| 2
= n −2 E {K h (X i − x)} 2 − {EK h (X i − x)} 2]
•
E {K h (X i − x)} 2 =
= 1 h 2 ∫
∫
K 2 h (u − x) f (u) du = 1 h 2 ∫
K 2 (v) f (x + hv) hdv = 1 h
∫
K 2 ( u − x
h
K 2 (v) f (x + hv) dv
)
f (u) du
ST5207 Nonparametric Regression, 27th January 2005

5
Step one, continued
• E |ξ in | 2 [
= n −2 E |K h (X i − x) − EK h (X i − x)| 2
= n −2 E {K h (X i − x)} 2 − {EK h (X i − x)} 2]
•
E {K h (X i − x)} 2 =
= 1 h 2 ∫
= f (x)
h
∫
K 2 h (u − x) f (u) du = 1 h 2 ∫
K 2 (v) f (x + hv) hdv = 1 h
∫
K 2 (v) dv + 1 f (x)
u (h) =
h h
∫
K 2 ( u − x
h
K 2 (v) f (x + hv) dv
∫
K 2 (v) dv + u (1)
)
f (u) du
ST5207 Nonparametric Regression, 27th January 2005

6
Step one, continued
• E |ξ in | 2 [
= n −2 E |K h (X i − x) − EK h (X i − x)| 2
= n −2 E {K h (X i − x)} 2 − {EK h (X i − x)} 2]
•
E {K h (X i − x)} 2 =
•
= 1 h 2 ∫
= f (x)
h
σ 2 in = f (x)
n 2 h
∫
K 2 h (u − x) f (u) du = 1 h 2 ∫
K 2 (v) f (x + hv) hdv = 1 h
∫
∫
K 2 (v) dv + 1 f (x)
u (h) =
h h
K 2 (v) dv {1 + u (1)} , V 2 n = f (x)
nh
∫
K 2 ( u − x
h
K 2 (v) f (x + hv) dv
∫
K 2 (v) dv + u (1)
∫
)
f (u) du
K 2 (v) dv {1 + u (1)}
ST5207 Nonparametric Regression, 27th January 2005

7
Step one, continued
• E |ξ in | 2+δ = n −(2+δ) E |K h (X i − x) − EK h (X i − x)| 2+δ
ST5207 Nonparametric Regression, 27th January 2005

8
Step one, continued
• E |ξ in | 2+δ = n −(2+δ) E |K h (X i − x) − EK h (X i − x)| 2+δ
• E |ξ in | 2+δ ≤ n −(2+δ) E {|K h (X i − x)| + |EK h (X i − x)|} 2+δ
ST5207 Nonparametric Regression, 27th January 2005

9
Step one, continued
• E |ξ in | 2+δ = n −(2+δ) E |K h (X i − x) − EK h (X i − x)| 2+δ
• E |ξ in | 2+δ ≤ n −(2+δ) E {|K h (X i − x)| + |EK h (X i − x)|} 2+δ
• Using (a + b) p ≤ 2 p−1 (a p + b p ) , a, b > 0, p ≥ 1
ST5207 Nonparametric Regression, 27th January 2005

10
Step one, continued
• E |ξ in | 2+δ = n −(2+δ) E |K h (X i − x) − EK h (X i − x)| 2+δ
• E |ξ in | 2+δ ≤ n −(2+δ) E {|K h (X i − x)| + |EK h (X i − x)|} 2+δ
• Using (a + b) p ≤ 2 p−1 (a p + b p ) , a, b > 0, p ≥ 1
E |ξ in | 2+δ ≤ n −(2+δ) 2 1+δ E
{|K h (X i − x)| 2+δ + |EK h (X i − x)| 2+δ}
ST5207 Nonparametric Regression, 27th January 2005

11
Step one, continued
• E |ξ in | 2+δ = n −(2+δ) E |K h (X i − x) − EK h (X i − x)| 2+δ
• E |ξ in | 2+δ ≤ n −(2+δ) E {|K h (X i − x)| + |EK h (X i − x)|} 2+δ
• Using (a + b) p ≤ 2 p−1 (a p + b p ) , a, b > 0, p ≥ 1
E |ξ in | 2+δ ≤ n −(2+δ) 2 1+δ E
{|K h (X i − x)| 2+δ + |EK h (X i − x)| 2+δ}
•
E |K h (X i − x)| 2+δ =
∫
K 2+δ
h
(u − x) f (u) du
ST5207 Nonparametric Regression, 27th January 2005

12
Step one, continued
• E |ξ in | 2+δ = n −(2+δ) E |K h (X i − x) − EK h (X i − x)| 2+δ
• E |ξ in | 2+δ ≤ n −(2+δ) E {|K h (X i − x)| + |EK h (X i − x)|} 2+δ
• Using (a + b) p ≤ 2 p−1 (a p + b p ) , a, b > 0, p ≥ 1
E |ξ in | 2+δ ≤ n −(2+δ) 2 1+δ E
{|K h (X i − x)| 2+δ + |EK h (X i − x)| 2+δ}
•
E |K h (X i − x)| 2+δ =
= 1
h 2+δ ∫
∫
K 2+δ
h
K 2+δ ( u − x
h
(u − x) f (u) du
)
f (u) du
ST5207 Nonparametric Regression, 27th January 2005

13
Step one, continued
• E |ξ in | 2+δ = n −(2+δ) E |K h (X i − x) − EK h (X i − x)| 2+δ
• E |ξ in | 2+δ ≤ n −(2+δ) E {|K h (X i − x)| + |EK h (X i − x)|} 2+δ
• Using (a + b) p ≤ 2 p−1 (a p + b p ) , a, b > 0, p ≥ 1
E |ξ in | 2+δ ≤ n −(2+δ) 2 1+δ E
{|K h (X i − x)| 2+δ + |EK h (X i − x)| 2+δ}
•
E |K h (X i − x)| 2+δ =
= 1
h 2+δ ∫
= 1
h 2+δ ∫
∫
K 2+δ
h
K 2+δ ( u − x
h
(u − x) f (u) du
)
f (u) du
K 2+δ (v) f (x + hv) hdv
ST5207 Nonparametric Regression, 27th January 2005

14
Step one, continued
• E |ξ in | 2+δ = n −(2+δ) E |K h (X i − x) − EK h (X i − x)| 2+δ
• E |ξ in | 2+δ ≤ n −(2+δ) E {|K h (X i − x)| + |EK h (X i − x)|} 2+δ
• Using (a + b) p ≤ 2 p−1 (a p + b p ) , a, b > 0, p ≥ 1
E |ξ in | 2+δ ≤ n −(2+δ) 2 1+δ E
{|K h (X i − x)| 2+δ + |EK h (X i − x)| 2+δ}
•
= 1
h 2+δ ∫
E |K h (X i − x)| 2+δ =
= 1
h 2+δ ∫
∫
K 2+δ
h
K 2+δ ( u − x
h
K 2+δ (v) f (x + hv) hdv = 1
h 1+δ ∫
(u − x) f (u) du
)
f (u) du
K 2+δ (v) f (x + hv) dv
ST5207 Nonparametric Regression, 27th January 2005

15
Step one, continued
• E |K h (X i − x)| 2+δ ≤ 1
h 1+δ
max f (t) ∫ K 2+δ (v) dv
t∈[a,b]
ST5207 Nonparametric Regression, 27th January 2005

16
Step one, continued
• E |K h (X i − x)| 2+δ ≤ 1
h 1+δ
|EK h (X i − x)| 2+δ =
max f (t) ∫ K 2+δ (v) dv
t∈[a,b]
{ }
∣
∣Bias ˆf (x)
+ f (x) ∣ 2+δ = f (x) 2+δ +u ( h 2)
ST5207 Nonparametric Regression, 27th January 2005

17
Step one, continued
• E |K h (X i − x)| 2+δ ≤ 1
h 1+δ
|EK h (X i − x)| 2+δ =
max f (t) ∫ K 2+δ (v) dv
t∈[a,b]
{ }
∣
∣Bias ˆf (x)
+ f (x) ∣ 2+δ = f (x) 2+δ +u ( h 2)
E |ξ in | 2+δ ≤ n −(2+δ) 2 1+δ 1
h 1+δ
∫
max f (t)
t∈[a,b]
K 2+δ (v) dv {1 + u (1)}
ST5207 Nonparametric Regression, 27th January 2005

18
Step one, continued
• E |K h (X i − x)| 2+δ ≤ 1
h 1+δ
|EK h (X i − x)| 2+δ =
max f (t) ∫ K 2+δ (v) dv
t∈[a,b]
{ }
∣
∣Bias ˆf (x)
+ f (x) ∣ 2+δ = f (x) 2+δ +u ( h 2)
E |ξ in | 2+δ ≤ n −(2+δ) 2 1+δ 1
h 1+δ
∫
max f (t)
t∈[a,b]
K 2+δ (v) dv {1 + u (1)}
•
V −(2+δ)
n
n∑
i=1
E |ξ in | 2+δ = V −(2+δ)
n nE |ξ in | 2+δ
ST5207 Nonparametric Regression, 27th January 2005

19
Step one, continued
• E |K h (X i − x)| 2+δ ≤ 1
h 1+δ
|EK h (X i − x)| 2+δ =
max f (t) ∫ K 2+δ (v) dv
t∈[a,b]
{ }
∣
∣Bias ˆf (x)
+ f (x) ∣ 2+δ = f (x) 2+δ +u ( h 2)
E |ξ in | 2+δ ≤ n −(2+δ) 2 1+δ 1
h 1+δ
∫
max f (t)
t∈[a,b]
K 2+δ (v) dv {1 + u (1)}
•
=
V −(2+δ)
n
[ f (x)
nh
∫
n∑
i=1
E |ξ in | 2+δ = V −(2+δ)
n nE |ξ in | 2+δ
] −(2+δ)/2
K 2 (v) dv {1 + u (1)} nE |ξ in | 2+δ
ST5207 Nonparametric Regression, 27th January 2005

20
Step one, continued
•
≤
[ f (x)
nh
∫
K 2 (v) dv {1 + u (1)}] −(2+δ)/2
× n×
ST5207 Nonparametric Regression, 27th January 2005

21
Step one, continued
•
≤
[ f (x)
nh
∫
n −(2+δ) 2 1+δ 1
h 1+δ
K 2 (v) dv {1 + u (1)}] −(2+δ)/2
× n×
∫
max f (t)
t∈[a,b]
K 2+δ (v) dv {1 + u (1)}
ST5207 Nonparametric Regression, 27th January 2005

22
Step one, continued
•
≤
[ f (x)
nh
∫
n −(2+δ) 2 1+δ 1
h 1+δ
K 2 (v) dv {1 + u (1)}] −(2+δ)/2
× n×
∫
max f (t)
t∈[a,b]
K 2+δ (v) dv {1 + u (1)}
≤ Cn (2+δ)/2+1−(2+δ) h (2+δ)/2−(1+δ) = Cn −δ/2 h −δ/2 = C (nh) −δ/2
ST5207 Nonparametric Regression, 27th January 2005

23
Step one, continued
•
≤
[ f (x)
nh
∫
n −(2+δ) 2 1+δ 1
h 1+δ
K 2 (v) dv {1 + u (1)}] −(2+δ)/2
× n×
∫
max f (t)
t∈[a,b]
K 2+δ (v) dv {1 + u (1)}
≤ Cn (2+δ)/2+1−(2+δ) h (2+δ)/2−(1+δ) = Cn −δ/2 h −δ/2 = C (nh) −δ/2
• Lyapunov CLT shows
V −1
n
n∑
i=1
ξ in
D
→ N (0, 1) ,
{ f (x)
nh
∫
} −1/2 ∑ n
K 2 (v) dv
i=1
ξ in
D
→ N (0, 1)
ST5207 Nonparametric Regression, 27th January 2005

24
Step one, continued
•
≤
[ f (x)
nh
∫
n −(2+δ) 2 1+δ 1
h 1+δ
K 2 (v) dv {1 + u (1)}] −(2+δ)/2
× n×
∫
max f (t)
t∈[a,b]
K 2+δ (v) dv {1 + u (1)}
≤ Cn (2+δ)/2+1−(2+δ) h (2+δ)/2−(1+δ) = Cn −δ/2 h −δ/2 = C (nh) −δ/2
• Lyapunov CLT shows
V −1
n
n∑
i=1
{ f (x)
nh
∫
ξ in
D
→ N (0, 1) ,
{ f (x)
nh
∫
} −1/2 ∑ n
K 2 (v) dv
i=1
ξ in
D
→ N (0, 1)
K 2 (v) dv} −1/2 [
ˆf (x) − f (x) − Bias
{
ˆf (x)
}] D→ N (0, 1)
ST5207 Nonparametric Regression, 27th January 2005

25
Digression to density estimation
• [
ˆf (x) − f (x) −
f (2) (x)
2!
h ∫ ]
2 v 2 K (v) dv
D
∫ } 1/2 → N (0, 1)
K2 (v) dv
because
{ ∫ f (x)
nh
{
f(x)
nh
K 2 (v) dv} −1/2
u ( h 2) = u
(
h 2 n 1/2 h 1/2) (
= u n 1/2 h 5/2) = u (1)
ST5207 Nonparametric Regression, 27th January 2005

26
Digression to density estimation
• [
ˆf (x) − f (x) −
f (2) (x)
2!
h ∫ ]
2 v 2 K (v) dv
D
∫ } 1/2 → N (0, 1)
K2 (v) dv
because
{ ∫ f (x)
nh
{
f(x)
nh
K 2 (v) dv} −1/2
u ( h 2) = u
(
h 2 n 1/2 h 1/2) (
= u n 1/2 h 5/2) = u (1)
• Confidence interval for probability density function f (x) is
constructed as
ˆf (x) − f ′′ ∫
{ ∫
1/2
(x)
f (x)
h 2 v 2 K (v) dv ± K 2 (v) dv}
z 1−α/2
2
nh
ST5207 Nonparametric Regression, 27th January 2005

27
Digression to density estimation
• [
ˆf (x) − f (x) −
f (2) (x)
2!
h ∫ ]
2 v 2 K (v) dv
D
∫ } 1/2 → N (0, 1)
K2 (v) dv
because
{ ∫ f (x)
nh
{
f(x)
nh
K 2 (v) dv} −1/2
u ( h 2) = u
(
h 2 n 1/2 h 1/2) (
= u n 1/2 h 5/2) = u (1)
• Confidence interval for probability density function f (x) is
constructed as
ˆf (x) − f ′′ ∫
{ ∫
1/2
(x)
f (x)
h 2 v 2 K (v) dv ± K 2 (v) dv}
z 1−α/2
2
nh
• XploRe quantlets are denxci, denxest, we now explore their use
ST5207 Nonparametric Regression, 27th January 2005

28
Digression to density estimation
From ˆf (x) − f (x) = n ∑
i=1
ξ in + Bias
{
ˆf (x)
}
, one gets
E
{
{ } 2 n
} 2
∑
{ }
ˆf (x) − f (x) = E ξ in +Bias 2 ˆf (x)
i=1
= V 2 n +Bias 2 { ˆf (x)
}
ST5207 Nonparametric Regression, 27th January 2005

29
Digression to density estimation
From ˆf (x) − f (x) = n ∑
i=1
ξ in + Bias
{
ˆf (x)
}
, one gets
E
{
{ } 2 n
} 2
∑
{ }
ˆf (x) − f (x) = E ξ in +Bias 2 ˆf (x)
i=1
= V 2 n +Bias 2 { ˆf (x)
}
E
{ } { 2 f (x)
f ˆf (2) (x) − f (x) =
nh c (x)
K {1 + u (1)}+ h 2 d K + u ( h 2)} 2
2!
ST5207 Nonparametric Regression, 27th January 2005

30
Digression to density estimation
From ˆf (x) − f (x) = n ∑
i=1
ξ in + Bias
{
ˆf (x)
}
, one gets
E
{
{ } 2 n
} 2
∑
{ }
ˆf (x) − f (x) = E ξ in +Bias 2 ˆf (x)
i=1
= V 2 n +Bias 2 { ˆf (x)
}
E
{ } { 2 f (x)
f ˆf (2) (x) − f (x) =
nh c (x)
K {1 + u (1)}+ h 2 d K + u ( h 2)} 2
2!
E
{ } 2 f (x)
ˆf (x) − f (x) =
nh c K + f (2) (x) 2
h 4 d 2 K + u
4
( 1
nh + h4 )
ST5207 Nonparametric Regression, 27th January 2005

31
Digression to density estimation
From ˆf (x) − f (x) = n ∑
i=1
ξ in + Bias
{
ˆf (x)
}
, one gets
E
{
{ } 2 n
} 2
∑
{ }
ˆf (x) − f (x) = E ξ in +Bias 2 ˆf (x)
i=1
= V 2 n +Bias 2 { ˆf (x)
}
E
∫
{ } { 2 f (x)
f ˆf (2) (x) − f (x) =
nh c (x)
K {1 + u (1)}+ h 2 d K + u ( h 2)} 2
2!
( 1
nh + h4 )
{ } 2 f (x)
E ˆf (x) − f (x) =
nh c K + f (2) (x) 2
h 4 d 2 K + u
4
{ } ∫ ∫
2 f (x) dx f
E ˆf ′′ (x) 2 dx
(x) − f (x) dx = c K +
h 4 d 2
nh
4
K+o
( 1
nh + h4 )
ST5207 Nonparametric Regression, 27th January 2005

Digression to density estimation
From ˆf (x) − f (x) = n ∑
i=1
ξ in + Bias
i=1
{
ˆf (x)
}
, one gets
{
{ } 2 n
} 2
∑
{ }
E ˆf (x) − f (x) = E ξ in +Bias 2 ˆf { }
(x) = Vn 2 +Bias 2 ˆf (x)
E
∫
{ } { 2 f (x)
f ˆf (2) (x) − f (x) =
nh c (x)
K {1 + u (1)}+ h 2 d K + u ( h 2)} 2
2!
{ } 2 f (x)
E ˆf (x) − f (x) =
nh c K + f (2) (x) 2 ( ) 1
h 4 d 2 K + u
4
nh + h4
{ } ∫ ∫
2 f (x) dx f
E ˆf ′′ (x) 2 ( )
dx
1
(x) − f (x) dx = c K +
h 4 d 2
nh
4
K+o
nh + h4
{ } { } ( ) 1
MISE ˆf (x) ; h = AMISE ˆf (x) ; h + o
nh + h4
32
ST5207 Nonparametric Regression, 27th January 2005

33
Digression to density estimation
•
{ }
AMISE ˆf (x) ; h
= 1
nh c K +
∫
f ′′ (x) 2 dx
h 4 d 2 K
4
ST5207 Nonparametric Regression, 27th January 2005

34
Digression to density estimation
•
{ }
AMISE ˆf (x) ; h
= 1
nh c K +
∫
f ′′ (x) 2 dx
h 4 d 2 K
4
• To find the optimal h, solve the following equation
d
{ }
dh AMISE ˆf (x) ; h = − 1 ∫
nh 2 c K + f ′′ (x) 2 dxh 3 d 2 K = 0
ST5207 Nonparametric Regression, 27th January 2005

35
Digression to density estimation
•
{ }
AMISE ˆf (x) ; h
= 1
nh c K +
∫
f ′′ (x) 2 dx
h 4 d 2 K
4
• To find the optimal h, solve the following equation
d
{ }
dh AMISE ˆf (x) ; h = − 1 ∫
nh 2 c K + f ′′ (x) 2 dxh 3 d 2 K = 0
• The optimal bandwidth is
h opt =
{
c K
d 2 K
} 1/5
1
∫
f
′′
(x) 2 n −1/5
dx
ST5207 Nonparametric Regression, 27th January 2005

36
Digression to density estimation
•
{ }
AMISE ˆf (x) ; h = 1
∫
f ′′
nh c (x) 2 dx
K +
h 4 d 2 K
4
• To find the optimal h, solve the following equation
d
{ }
dh AMISE ˆf (x) ; h = − 1 ∫
nh 2 c K + f ′′ (x) 2 dxh 3 d 2 K = 0
• The optimal bandwidth is
{
h opt =
c K
d 2 K
} 1/5
1
∫
f
′′
(x) 2 n −1/5
dx
• The optimal bandwidth depends on curvature of the unknown
density f, kernel K used and sample size n available. XploRe
quantlets are denrot and denbwsel, we now explore their use
ST5207 Nonparametric Regression, 27th January 2005