Building Machine Learning Systems with Python - Richert, Coelho

Learning How to Classify with Real-world Examples
Nearest neighbor classification
With this dataset, even if we just try to separate two classes using the previous
method, we do not get very good results. Let me introduce , therefore, a new
classifier: the nearest neighbor classifier.
If we consider that each example is represented by its features (in mathematical
terms, as a point in N-dimensional space), we can compute the distance between
examples. We can choose different ways of computing the distance, for example:
def distance(p0, p1):
'Computes squared euclidean distance'
return np.sum( (p0-p1)**2)
Now when classifying, we adopt a simple rule: given a new example, we look at
the dataset for the point that is closest to it (its nearest neighbor) and look at its label:
def nn_classify(training_set, training_labels, new_example):
dists = np.array([distance(t, new_example)
for t in training_set])
nearest = dists.argmin()
return training_labels[nearest]
In this case, our model involves saving all of the training data and labels and
computing everything at classification time. A better implementation would
be to actually index these at learning time to speed up classification, but this
implementation is a complex algorithm.
Now, note that this model performs perfectly on its training data! For each point,
its closest neighbor is itself, and so its label matches perfectly (unless two examples
have exactly the same features but different labels, which can happen). Therefore, it
is essential to test using a cross-validation protocol.
Using ten folds for cross-validation for this dataset with this algorithm, we obtain
88 percent accuracy. As we discussed in the earlier section, the cross-validation
accuracy is lower than the training accuracy, but this is a more credible estimate
of the performance of the model.
We will now examine the decision boundary. For this, we will be forced to simplify
and look at only two dimensions (just so that we can plot it on paper).
[ 44 ]