Learning Data Mining with Python

We again redefine our word search regular expression—if you were doing this in
a real application, I recommend centralizing this. It is important that words are
extracted in the same way for training and testing:
word_search_re = re.compile(r"[\w']+")
Next, we create the function that loads our model from a given filename:
def load_model(model_filename):
Chapter 12
The model parameters will take the form of a dictionary of dictionaries, where the
first key is a word, and the inner dictionary maps each gender to a probability. We
use defaultdicts, which will return zero if a value isn't present;
model = defaultdict(lambda: defaultdict(float))
We then open the model and parse each line;
with open(model_filename) as inf:
for line in inf:
The line is split into two sections, separated by whitespace. The first is the word itself
and the second is a dictionary of probabilities. For each, we run eval on them to get
the actual value, which was stored using repr in the previous code:
word, values = line.split(maxsplit=1)
word = eval(word)
values = eval(values)
We then track the values to the word in our model:
model[word] = values
return model
Next, we load our actual model. You may need to change the model filename—it will
be in the output dir of the last MapReduce job;
model_filename = os.path.join(os.path.expanduser("~"), "models",
"part-00000")
model = load_model(model_filename)
As an example, we can see the difference in usage of the word i (all words are turned
into lowercase in the MapReduce jobs) between males and females:
model["i"]["male"], model["i"]["female"]
[ 289 ]