Building Machine Learning Systems with Python - Richert, Coelho

Chapter 6
Y = np.zeros(Y.shape[0])
Y[pos] = 1
Y = Y.astype(int)
return Y
Note that we are talking about two different positives now. The sentiment of a tweet
can be positive, which is to be distinguished from the class of the training data. If, for
example, we want to find out how good we can separate the tweets having sentiment
from neutral ones, we could do this as follows:
>>> Y = tweak_labels(Y, ["positive", "negative"])
In Y we now have a 1 (positive class) for all tweets that are either positive or negative
and a 0 (negative class) for neutral and irrelevant ones.
>>> train_model(create_ngram_model, X, Y, plot=True)
0.767 0.014 0.670 0.022
As expected, the P/R AUC drops considerably, being only 67 percent now.
The accuracy is still high, but that is only due to the fact that we have a highly
imbalanced dataset. Out of 3,642 total tweets, only 1,017 are either positive or
negative, which is about 28 percent. This means that if we created a classifier
that always classified a tweet as not containing any sentiments, we would already
have an accuracy of 72 percent. This is another example of why you should always
look at precision and recall if the training and test data is unbalanced.
[ 131 ]