Learning Data Mining with Python

Working with Big Data
The first value is the word and the second is a dictionary mapping the genders to the
frequency of that word in that gender's writings.
Open a new file in your Python IDE or text editor. We will again need the os and re
libraries, as well as NumPy and MRJob from mrjob. We also need itemgetter, as we will
be sorting a dictionary:
import os
import re
import numpy as np
from mrjob.job import MRJob
from operator import itemgetter
We will also need MRStep, which outlines a step in a MapReduce job. Our previous
job only had a single step, which is defined as a mapping function and then as a
reducing function. This job will have three steps where we Map, Reduce, and then
Map and Reduce again. The intuition is the same as the pipelines we used in earlier
chapters, where the output of one step is the input to the next step:
from mrjob.step import MRStep
We then create our word search regular expression and compile it, allowing us to
find word boundaries. This type of regular expression is much more powerful than
the simple split we used in some previous chapters, but if you are looking for a more
accurate word splitter, I recommend using NLTK as we did in Chapter 6, Social Media
Insight using Naive Bayes:
word_search_re = re.compile(r"[\w']+")
We define a new class for our training:
class NaiveBayesTrainer(MRJob):
We define the steps of our MapReduce job. There are two steps. The first step will
extract the word occurrence probabilities. The second step will compare the two
genders and output the probabilities for each to our output file. In each MRStep,
we define the mapper and reducer functions, which are class functions in this
NaiveBayesTrainer class (we will write those functions next):
def steps(self):
return [
MRStep(mapper=self.extract_words_mapping,
reducer=self.reducer_count_words),
MRStep(reducer=self.compare_words_reducer),
]
[ 286 ]