Learning Data Mining with Python

Chapter 12
The first parameter, /blogs/51* (just remember to change
to the full path to your data folder), obtains a sample of the
data (all files starting with 51, which is only 11 documents). We then set the output
directory to a new folder, which we put in the data folder, and specify not to output
the streamed data. Without the last option, the output data is shown to the command
line when we run it—which isn't very helpful to us and slows down the computer
quite a lot.
Run the script, and quite quickly each of the blog posts will be extracted and stored
in our output folder. This script only ran on a single thread on the local computer so
we didn't get a speedup at all, but we know the code runs.
We can now look in the output folder for the results. A bunch of files are created and
each file contains each blog post on a separate line, preceded by the gender of the
author of the blog.
Training Naive Bayes
Now that we have extracted the blog posts, we can train our Naive Bayes model
on them. The intuition is that we record the probability of a word being written by
a particular gender. To classify a new sample, we would multiply the probabilities
and find the most likely gender.
The aim of this code is to output a file that lists each word in the corpus, along
with the frequencies of that word for each gender. The output file will look
something like this:
"'ailleurs" {"female": 0.003205128205128205}
"'air" {"female": 0.003205128205128205}
"'an" {"male": 0.0030581039755351682, "female": 0.004273504273504274}
"'angoisse" {"female": 0.003205128205128205}
"'apprendra" {"male": 0.0013047113868622459, "female":
0.0014172668603481887}
"'attendent" {"female": 0.00641025641025641}
"'autistic" {"male": 0.002150537634408602}
"'auto" {"female": 0.003205128205128205}
"'avais" {"female": 0.00641025641025641}
"'avait" {"female": 0.004273504273504274}
"'behind" {"male": 0.0024390243902439024}
"'bout" {"female": 0.002034152292059272}
[ 285 ]