Learning Data Mining with Python

Authorship Attribution
With our data loading function, we are going to have a lot of options. Most of these
ensure that our dataset is relatively balanced. Some authors will have thousands
of e-mails in their sent mail, while others will have only a few dozen. We limit our
search to only authors with at least 10 e-mails using min_docs_author and take a
maximum of 100 e-mails from each author using the max_docs_author parameter.
We also specify how many authors we want to get—10 by default using the num_
authors parameter. The code is as follows:
def get_enron_corpus(num_authors=10, data_folder=data_folder,
min_docs_author=10, max_docs_author=100,
random_state=None):
random_state = check_random_state(random_state)
Next, we list all of the folders in the data folder, which are separate e-mail addresses
of Enron employees. We when randomly shuffle them, allowing us to choose a new
set every time the code is run. Remember that setting the random state will allow us
to replicate this result:
email_addresses = sorted(os.listdir(data_folder))
random_state.shuffle(email_addresses)
It may seem odd that we sort the e-mail addresses, only to shuffle
them around. The os.listdir function doesn't always return the
same results, so we sort it first to get some stability. We then shuffle
using a random state, which means our shuffling can reproduce a
past result if needed.
We then set up our documents and class lists. We also create an author_num, which
will tell us which class to use for each new author. We won't use the enumerate trick
we used earlier, as it is possible that we won't choose some authors. For example, if
an author doesn't have 10 sent e-mails, we will not use it. The code is as follows:
documents = []
classes = []
author_num = 0
We are also going to record which authors we used and which class number we
assigned to them. This isn't for the data mining, but will be used in the visualization
so we can identify the authors more easily. The dictionary will simply map e-mail
usernames to class values. The code is as follows:
authors = {}
[ 202 ]