Learning Data Mining with Python

Working with Big Data
Before we start collecting blog posts, we need to get the gender of the author
of the blog. While we don't normally use the filename as part of MapReduce jobs,
there is a strong need for it (as in this case) so the functionality is available. The
current file is stored as an environment variable, which we can obtain using the
following line of code:
filename = os.environ["map_input_file"]
We then split the filename to get the gender (which is the second token);
gender = filename.split(".")[1]
We remove whitespace from the start and end of the line (there is a lot of whitespace
in these documents) and then do our post-based tracking as before;
line = line.strip()
if line == "":
self.post_start = True
elif line == "":
self.post_start = False
Rather than storing the posts in a list, as we did earlier, we yield them. This allows
mrjob to track the output. We yield both the gender and the post so that we can keep
a record of which gender each record matches. The rest of this function is defined in
the same way as our loop above:
yield gender, repr("\n".join(self.post))
self.post = []
elif self.post_start:
self.post.append(line)
Finally, outside the function and class, we set the script to run this MapReduce job
when it is called from the command line:
if __name__ == '__main__':
ExtractPosts.run()
Now, we can run this MapReduce job using the following shell command. Note that
we are using Python 2, and not Python 3 to run this;
python extract_posts.py /blogs/51* --outputdir=/blogposts
–no-output
[ 284 ]