Learning Data Mining with Python

Authorship Attribution
If it doesn't exist, quotequail couldn't find it. It is possible it found other text in the
e-mail. If that exists, we return only that text. The code is as follows:
elif 'text' in r:
return r['text']
Finally, if we couldn't get a result, we just return the e-mail contents, hoping they
offer some benefit to our data analysis:
return email_contents
We can now preprocess all of our documents by running this function on each
of them:
documents = [remove_replies(document) for document in documents]
Our preceding e-mail sample is greatly clarified now and contains only the e-mail
written by Mark Greenberg:
I am disappointed on the timing but I understand. Thanks. Mark
Putting it all together
We can use the existing parameter space and classifier from our previous
experiments—all we need to do is refit it on our new data. By default, training
in scikit-learn is done from scratch—subsequent calls to fit() will discard any
previous information.
There is a class of algorithms called online learning that update the
training with new samples and don't restart their training each time.
We will see online learning in action later in this book, including the
next chapter, Chapter 10, Clustering News Articles.
As before, we can compute our scores by using cross_val_score and print the
results. The code is as follows:
scores = cross_val_score(pipeline, documents, classes, scoring='f1')
print("Score: {:.3f}".format(np.mean(scores)))
The result is 0.523, which is a reasonable result for such a messy dataset. Adding
more data (such as increasing max_docs_author in the dataset loading) can improve
these results.
[ 206 ]