Learning Data Mining with Python

Chapter 9
We create lists for storing the documents themselves and the author classes:
documents = []
authors = []
We then create a list of each of the subfolders in the parent directly, as the script
creates a subfolder for each author. The code is as follows:
subfolders = [subfolder for subfolder in os.listdir(folder)
if os.path.isdir(os.path.join(folder,
subfolder))]
Next we iterate over these subfolders, assigning each subfolder a number using
enumerate:
for author_number, subfolder in enumerate(subfolders):
We then create the full subfolder path and look for all documents within
that subfolder:
full_subfolder_path = os.path.join(folder, subfolder)
for document_name in os.listdir(full_subfolder_path):
For each of those files, we open it, read the contents, preprocess those contents,
and append it to our documents list. The code is as follows:
with open(os.path.join(full_subfolder_path,
document_name)) as inf:
documents.append(clean_book(inf.read()))
We also append the number we assigned to this author to our authors list,
which will form our classes:
authors.append(author_number)
We then return the documents and classes (which we transform into a NumPy
array for each indexing later on):
return documents, np.array(authors, dtype='int')
We can now get our documents and classes using the following function call:
documents, classes = load_books_data(data_folder)
[ 191 ]