Learning Data Mining with Python

Chapter 12
MapReduce originates from Google, where it was developed with distributed
computing in mind. It also introduces fault tolerance and scalability improvements.
The "original" research for MapReduce was published in 2004, and since then there
have been thousands of projects, implementations, and applications using it.
While the concept is similar to many previous concepts, MapReduce has become a
staple in big data analytics.
Intuition
MapReduce has two main steps: the Map step and the Reduce step. These are built
on the functional programming concepts of mapping a function to a list and reducing
the result. To explain the concept, we will develop code that will iterate over a list of
lists and produce the sum of all numbers in those lists.
There are also shuffle and combine steps in the MapReduce paradigm, which we
will see later.
To start with, the Map step takes a function and applies it to each element in a
list. The returned result is a list of the same size, with the results of the function
applied to each element.
To open a new IPython Notebook, start by creating a list of lists with numbers in
each sublist:
a = [[1,2,1], [3,2], [4,9,1,0,2]]
Next, we can perform a map, using the sum function. This step will apply the sum
function to each element of a:
sums = map(sum, a)
While sums is a generator (the actual value isn't computed until we ask for it),
the above step is approximately equal to the following code:
sums = []
for sublist in a:
results = sum(sublist)
sums.append(results)
The reduce step is a little more complicated. It involves applying a function to each
element of the returned result, to some starting value. We start with an initial value,
and then apply a given function to that initial value and the first value. We then
apply the given function to the result and the next value, and so on.
[ 275 ]