Learning Data Mining with Python

Chapter 9
This document contains another e-mail attached to the bottom as a reply, a common
e-mail pattern. The first part of the e-mail is from Mark Haedicke, while the second is
a previous e-mail written to Mark Haedicke by Mark Greenberg. Only the preceding
text (the first instance of -----Original Message-----) could be attributed to the author,
and this is the only bit we are actually worried about.
Extracting this information generally is not easy. E-mail is a notoriously badly used
format. Different e-mail clients add their own headers, define replies in different
ways, and just do things however they want. It is really surprising that e-mail
works at all in the current environment.
There are some commonly used patterns that we can look for. The quotequail
package looks for these and can find the new part of the e-mail, discarding replies
and other information.
You can install quotequail using pip: pip3 install quotequail.
We are going to write a simple function to wrap the quotequail functionality,
allowing us to easily call it on all of our documents. First we import quotequail
and set up the function definition:
import quotequail
def remove_replies(email_contents):
Next, we use quotequail to unwrap the e-mail, which returns a dictionary
containing the different parts of the e-mail. The code is as follows:
r = quotequail.unwrap(email_contents)
In some cases, r can be none. This happens if the e-mail couldn't be parsed. In this
case, we just return the full e-mail contents. This kind of messy solution is often
necessary when working with real world datasets. The code is as follows:
if r is None:
return email_contents
The actual part of the e-mail we are interested in is called (by quotequail) the
text_top. If this exists, we return this as our interesting part of the e-mail.
The code is as follows:
if 'text_top' in r:
return r['text_top']
[ 205 ]