Algorithms and Data Structures for External Memory

144 External Memory Programming Environments
size to the block size, and running a copy of the pipeline on each node
of a cluster.
Google’s MapReduce [130] is a framework-oriented system that supports
a simple functional style of batched programming. The input data
are assumed to be in the form of a list of key-value pairs. The programmer
specifies a Map function and a Reduce function. The system
applies Map to each key-value pair, which produces a set of intermediate
key-value pairs. For each k, the system groups together all the
intermediate key-value pairs that have the same key k and passes them
to the Reduce function; Reduce merges together those key-value pairs
and forms a possibly smaller set of values for key k. The system handles
the details of data routing, parallel scheduling, and buffer management.
This framework is useful for a variety of massive data applications on
computer clusters, such as pattern matching, counting access frequencies
of web pages, constructing inverted indexes, and distribution sort.