Machine Learning 3. Nearest Neighbor and Kernel Methods - ISMLL

Machine Learning / 3. Parzen Windows
k-Nearest Neighbor is locally constant
In k-nearest neighbor the size of the window varies from
point to point: it depends on the density of the data:
in dense parts
the effective window size is small,
in sparse parts
the effective window size is large.
Alternatively, it is also possible to set the size of the
windows to a constant λ, e.g.,
{ 1, if |x − x0 | ≤ λ
K λ (x, x 0 ) :=
0, otherwise
Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), Institute BW/WI & Institute for Computer Science, University of Hildesheim
Course on Machine Learning, winter term 2007 36/48
Machine Learning / 3. Parzen Windows
Kernel Regression
Instead of discrete windows, one typically uses
continuous windows, i.e., continuous weights
K(x, x 0 )
that reflect the distance of a training point x to a
prediction point x 0 , called kernel or Parzen window,
e.g.,
{
1 −
|x−x 0 |
K(x, x 0 ) :=
λ
, if |x − x 0 | ≤ λ
0, otherwise
Instead of a binary neighbor/not-neighbor decision, a
continuous kernel captures a “degree of neighborship”.
Kernels can be used for prediction via kernel
regression, esp. Nadaraya-Watson kernel-weighted
average:
∑
(x,y)∈X
ŷ(x 0 ) :=
K(x, x 0)y
∑
(x,y)∈X K(x, x 0)
Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), Institute BW/WI & Institute for Computer Science, University of Hildesheim
Course on Machine Learning, winter term 2007 37/48