Learning Data Mining with Python

Authorship Attribution
After taking a look at these files, you will see that many of them are quite messy—at
least from a data analysis point of view. There is a large project Gutenberg disclaimer
at the start of the files. This needs to be removed before we do our analysis.
We could alter the individual files on disk to remove this stuff. However, what
happens if we were to lose our data? We would lose our changes and potentially be
unable to replicate the study. For that reason, we will perform the preprocessing as
we load the files—this allows us to be sure our results will be replicable (as long as
the data source stays the same). The code is as follows:
def clean_book(document):
We first split the document into lines, as we can identify the start and end of the
disclaimer by the starting and ending lines:
lines = document.split("\n")
We are going to iterate through each line. We look for the line that indicates the start
of the book, and the line that indicates the end of the book. We will then take the text
in between as the book itself. The code is as follows:
start = 0
end = len(lines)
for i in range(len(lines)):
line = lines[i]
if line.startswith("*** START OF THIS PROJECT GUTENBERG"):
start = i + 1
elif line.startswith("*** END OF THIS PROJECT GUTENBERG"):
end = i - 1
Finally, we join those lines together with a newline character to recreate the book
without the disclaimers:
return "\n".join(lines[start:end])
From here, we can now create a function that loads all of the books, performs
the preprocessing, and returns them along with a class number for each author.
The code is as follows:
import numpy as np
By default, our function signature takes the parent folder containing each of the
subfolders that contain the actual books. The code is as follows:
def load_books_data(folder=data_folder):
[ 190 ]