Magellan Final Report - Office of Science - U.S. Department of Energy

Chapter 10
MapReduce Programming Model
The MapReduce programming [13] was developed by Google to address its large scale data processing
requirements. Just reading terabytes of data can be overwhelming. Thus, the MapReduce programming
model is based on the idea of achieving linear scalability by using clusters of standard commodity computers.
Nutch, an open source search project, that was facing similar scalability challenges implemented the ideas
described in the MapReduce [13] and the Google File System papers [33]. The larger applicability of using
MapReduce and the distributed file systems for other projects was recognized and Hadoop was split off as a
separate project from Nutch in 2006.
MapReduce and Hadoop have gained significant traction in the last few years as a means of processing
large volumes of data. The MapReduce programming model consists of two functions familiar to functional
programming, map and reduce, that are each applied in parallel to a data block. The inherent support
for parallelism built into the programming model enables horizontal scaling and fault-tolerance. The opensource
implementation Hadoop is now widely used by global internet companies such as Facebook, LinkedIn,
Netflix, etc. Teams have developed components on top of Hadoop resulting in a rich ecosystem for web
applications and data-intensive computing.
A number of scientific applications have characteristics in common with MapReduce/Hadoop jobs. A
class of scientific applications which employ a high degree of parallelism or need to operate on large volumes
of data might benefit from MapReduce and Hadoop. However, it is not apparent that the MapReduce and
Hadoop ecosystem is sufficiently flexible to be used for scientific applications without significant development
overheads.
Recently, there have been a number of implementations of MapReduce and similar data processing
tools [21, 25, 36, 45, 59, 73, 87]. Apache Hadoop was the most popular implementation of MapReduce at
the start of the Magellan project and it continuous to gain traction in various communities. It has evolved
rapidly as the leading platform and has spawned an entire ecosystem of supporting products. Thus we
use Hadoop, as a representative MapReduce implementation for core Magellan efforts in this area. We
have also collaborated with other groups to compare different MapReduce implementations (Section 10.6.2).
Additionally, Magellan staff were also involved with an alternative implementation of MapReduce (described
in Section 10.6.3).
In Sections 10.1, 10.2 and 10.3 we provide an overview of MapReduce, Hadoop and Hadoop Ecosystem.
In Section 10.4 we describe our experiences with porting existing applications to Hadoop using the Streaming
model. In Section 10.5 we describe our benchmarking effort with Hadoop and Section 10.6 describes other
collaborative efforts. We present a discussion of the use of Hadoop for scientific applications in Section 10.7.
Additional case studies using Hadoop are described in Chapter 11.
83