Building Machine Learning Systems with Python - Richert, Coelho

Classification – Detecting Poor Answers
Looking behind accuracy – precision
and recall
Let us step back and think again what we are trying to achieve here. Actually, we do
not need a classifier that perfectly predicts good and bad answers, as we measured
it until now using accuracy. If we can tune the classifier to be particularly good in
predicting one class, we could adapt the feedback to the user accordingly. If we had
a classifier, for example, that was always right when it predicted an answer to be
bad, we would give no feedback until the classifier detected the answer to be bad.
Contrariwise, if the classifier succeeded in predicting answers to be always good,
we could show helpful comments to the user at the beginning and remove them
when the classifier said that the answer is a good one.
To find out which situation we are in here, we have to understand how to measure
precision and recall. To understand this, we have to look into the four distinct
classification results as they are described in the following table:
In
reality
it is
Classified as
Positive
Negative
Positive True positive (TP) False negative (FN)
Negative False positive (FP) True negative (TN)
For instance, if the classifier predicts an instance to be positive and the instance
indeed is positive in reality, this is a true positive instance. If on the other hand,
the classifier misclassified that instance saying that it is negative while in reality
it is positive, that instance is said to be a false negative.
What we want is to have a high success rate when we are predicting a post as either
good or bad, but not necessarily both. That is, we want as many true positives as
possible. This is what precision captures:
[ 110 ]