Machine Learning 3. Nearest Neighbor and Kernel Methods - ISMLL

More documents

Recommendations

Info

Machine Learning / 2. k-Nearest Neighbor Method Error rate for nearest-neighbor rule (Cover and Hart 1967) If we have unlimited data, the error rate of the nearest neighbor classifier is bound as follows: p ∗ (error) ≤ lim p n (error) ≤ p ∗ (error)(2 − k n→∞ k − 1 p∗ (error)) where p n (error) denotes the error rate for the nearest neighbor classifier in a sample of n points. Roughly spoken “at least half of the classification information in an infite data set resides in the nearest neighbor” (Duda et al. 2001). Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), Institute BW/WI & Institute for Computer Science, University of Hildesheim Course on Machine Learning, winter term 2007 21/48 Machine Learning / 2. k-Nearest Neighbor Method Error rate for nearest-neighbor rule / proof A strict proof of the error bounds is not so easy. A more informal argument is as follows (cf. Duda et al. 2001, p. 179–182): For x 0 denote by x ′ n the nearest neighbor of x 0 in a sample of n points. p n (error|x 0 , x ′ n) = 1 − ∑ y p(y 0 = y, y ′ n = y|x 0 , x ′ n) = 1 − ∑ y p(y 0 = y|x 0 )p(y ′ n = y|x ′ n) ∫ lim p n(error|x 0 ) = lim n→∞ n→∞ n→∞ ∫ p n (error|x 0 , x ′ n)p(x ′ n|x 0 )dx ′ n = lim (1 − ∑ p(y 0 = y|x 0 )p(y n ′ = y|x ′ n))p(x ′ n|x 0 )dx ′ n y ∫ = (1 − ∑ p(y 0 = y|x 0 )p(y n ′ = y|x ′ n))δ(x ′ n − x 0 )dx ′ n y =1 − ∑ y p(y 0 = y|x 0 ) 2 Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), Institute BW/WI & Institute for Computer Science, University of Hildesheim Course on Machine Learning, winter term 2007 22/48
Machine Learning / 2. k-Nearest Neighbor Method Error rate for nearest-neighbor rule / proof Now let y ∗ (x) := argmax y p(y|x) the Bayes classifier: ∑ p(y 0 = y|x 0 ) 2 =p(y 0 = y ∗ (x 0 )|x 0 ) 2 + y ∑ y≠y ∗ (x 0 ) p(y 0 = y|x 0 ) 2 ≥(1 − p ∗ (error|x 0 )) 2 + 1 k − 1 p∗ (error|x 0 ) 2 =1 − 2p ∗ (error|x 0 ) + k k − 1 p∗ (error|x 0 ) 2 because the sum is minimal if all p(y 0 = y|x 0 ) are equal, and thus p(y 0 = y|x 0 ) = 1 k − 1 (1 − p(y 0 = y ∗ (x 0 )|x 0 )) = 1 k − 1 p∗ (error|x 0 ) Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), Institute BW/WI & Institute for Computer Science, University of Hildesheim Course on Machine Learning, winter term 2007 23/48 Machine Learning / 2. k-Nearest Neighbor Method Error rate for nearest-neighbor rule / proof Then we continue lim n→∞ p n(error|x 0 ) = 1− ∑ y p(y 0 = y|x 0 ) 2 ≤ 2p ∗ (error|x 0 )− k k − 1 p∗ (error|x 0 ) 2 Now lim p n(error) = lim p n (error|x 0 )p(x 0 )dx 0 n→∞ ∫ ≤ (2p ∗ (error|x 0 ) − k k − 1 p∗ (error|x 0 ) 2 )p(x 0 )dx 0 =2p ∗ (error) − k ∫ p ∗ (error|x 0 ) 2 p(x 0 )dx 0 k − 1 n→∞ ∫ Lars Schmidt-Thieme, Information Systems and Machine Learning Lab (ISMLL), Institute BW/WI & Institute for Computer Science, University of Hildesheim Course on Machine Learning, winter term 2007 24/48
Page 1 and 2: Machine Learning Machine Learning 3
Page 3 and 4: Machine Learning / 1. Distance Meas
Page 11 and 12: Machine Learning / 2. k-Nearest Nei
Page 13: Machine Learning / 2. k-Nearest Nei
Page 21 and 22: Machine Learning 1. Distance Measur
Page 23 and 24: Machine Learning / 3. Parzen Window
Page 29: Machine Learning / 3. Parzen Window

Machine Learning 3. Nearest Neighbor and Kernel Methods - ISMLL

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?