Machine Learning 3. Nearest Neighbor and Kernel Methods - ISMLL

More documents

Recommendations

Info

Machine Learning / 2. k-Nearest Neighbor Method Neighborhoods Let d be a distance measure. For a dataset and x ∈ X let D ⊆ X × Y D = {(x 1 , y 1 ), (x 2 , y 2 ), . . . , (x n , y n )} be an enumeration with increasing distance to x, i.e., d(x, x i ) ≤ d(x, x i+1 ) (ties broken arbitrarily). The first k ∈ N points of such an enumeration, i.e., N k (x) := {(x 1 , y 1 ), (x 2 , y 2 ), . . . (x k , y k )} are called a k-neighborhood of x (in D). Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), Institute BW/WI & Institute for Computer Science, University of Hildesheim Course on Machine Learning, winter term 2007 14/48 Machine Learning / 2. k-Nearest Neighbor Method Nearest Neighbor Regression The k-nearest neighbor regressor Ŷ (x) := 1 ∑ k (x ′ ,y ′ )∈N k (x) y ′ The k-nearest neighbor classifier ˆp(Y = y | x) := 1 ∑ k (x ′ ,y ′ )∈N k (x) I(y = y ′ ) and then predict the class with maximal predicted probability Ŷ (x) := argmax y∈Y ˆp(Y = y | x) i.e., the majority class w.r.t. the classes of the neighbors. Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), Institute BW/WI & Institute for Computer Science, University of Hildesheim Course on Machine Learning, winter term 2007 15/48
Machine Learning / 2. k-Nearest Neighbor Method Decision Boundaries For 1-nearest neighbor, the predictor space is partitioned in regions of points that are closest to a given data point: with region D (x 1 ), region D (x 2 ), . . . , region D (x n ) region D (x) := {x ′ ∈ X | d(x ′ , x) ≤ d(x ′ , x ′′ ) ∀(x ′′ , y ′′ ) ∈ D} These regions often are called cells, the whole partition a Voronoi tesselation. Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), Institute BW/WI & Institute for Computer Science, University of Hildesheim Course on Machine Learning, winter term 2007 16/48 Machine Learning / 2. k-Nearest Neighbor Method Decision Boundaries x2 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 x1 Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), Institute BW/WI & Institute for Computer Science, University of Hildesheim Course on Machine Learning, winter term 2007 17/48
Page 1 and 2: Machine Learning Machine Learning 3
Page 3 and 4: Machine Learning / 1. Distance Meas
Page 9: Machine Learning / 1. Distance Meas
Page 13 and 14: Machine Learning / 2. k-Nearest Nei
Page 21 and 22: Machine Learning 1. Distance Measur
Page 23 and 24: Machine Learning / 3. Parzen Window
Page 29: Machine Learning / 3. Parzen Window

Machine Learning 3. Nearest Neighbor and Kernel Methods - ISMLL

Create successful ePaper yourself

Delete template?

Save as template?