Learning Data Mining with Python

Chapter 2
Nearest neighbors can be used for nearly any dataset-however, it can be very
computationally expensive to compute the distance between all pairs of samples.
For example if there are 10 samples in the dataset, there are 45 unique distances
to compute. However, if there are 1000 samples, there are nearly 500,000! Various
methods exist for improving this speed dramatically; some of which are covered in
the later chapters of this book.
It can also do poorly in categorical-based datasets, and another algorithm should be
used for these instead.
Distance metrics
A key underlying concept in data mining is that of distance. If we have two samples,
we need to know how close they are to each other. Further more, we need to answer
questions such as are these two samples more similar than the other two?
Answering questions like these is important to the outcome of the case.
The most common distance metric that the people are aware of is Euclidean
distance, which is the real-world distance. If you were to plot the points on a graph
and measure the distance with a straight ruler, the result would be the Euclidean
distance. A little more formally, it is the square root of the sum of the squared
distances for each feature.
Euclidean distance is intuitive, but provides poor accuracy if some features have
larger values than others. It also gives poor results when lots of features have a
value of 0, known as a sparse matrix. There are other distance metrics in use; two
commonly employed ones are the Manhattan and Cosine distance.
The Manhattan distance is the sum of the absolute differences in each feature (with
no use of square distances). Intuitively, it can be thought of as the number of moves
a rook piece (or castle) in chess would take to move between the points, if it were
limited to moving one square at a time. While the Manhattan distance does suffer if
some features have larger values than others, the effect is not as dramatic as in the
case of Euclidean.
[ 27 ]
www.allitebooks.com