Learning Data Mining with Python

Working with Big Data
This gives us the information we needed for our Naive Bayes implementation:
def compare_words_reducer(self, word, values):
per_gender = {}
for value in values:
gender, s = value
per_gender[gender] = s
yield word, per_gender
Finally, we set the code to run this model when the file is run as a script;
if __name__ == '__main__':
NaiveBayesTrainer.run()
We can then run this script. The input to this script is the output of the previous
post-extractor script (we can actually have them as different steps in the same
MapReduce job if you are so inclined);
python nb_train.py /blogposts/
--output-dir=/models/
--no-output
The output directory is a folder that will store a file containing the output from
this MapReduce job, which will be the probabilities we need to run our Naive
Bayes classifier.
Putting it all together
We can now actually run the Naive Bayes classifier using these probabilities.
We will do this in an IPython Notebook, and can go back to using Python 3 (phew!).
First, take a look at the models folder that was specified in the last MapReduce job.
If the output was more than one file, we can merge the files by just appending them
to each other using a command line function from within the models directory:
cat * > model.txt
If you do this, you'll need to update the following code with model.txt as the
model filename.
Back to our Notebook, we first import some standard imports we need for our script:
import os
import re
import numpy as np
from collections import defaultdict
from operator import itemgetter
[ 288 ]