Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
10.4.2 Application Examples
Magellan staff worked with some key scientific groups to port their applications to the streaming model. We
describe some of these in this section. Additional case studies are presented in Chapter 11.
BLAST. JGI’s IMG pipeline has been tested in the Hadoop framework to manage a set of parallel BLAST
computations. The performance across an HPC machine, virtual machines, and a Hadoop cluster was found
to be comparable (within 10%), making this a feasible application for cloud environments. The experience
with running the IMG pipeline on virtual machines is described in greater detail in Chapter 11. Here we
describe the challenges with running this in the Hadoop framework. BLAST takes a multi-line sequence
input for each run. By default, MapReduce assumes that inputs are line-oriented and each line can be
processed independently. One could define and write Java classes to handle other input formats. However
there are difficulties with managing API versions and jar files that can make the process cumbersome. We
experimented with a wrapper script that formatted the input to be line oriented when passed to the Hadoop
framework and the input was reformatted in the streaming mapper script to be compatible with application
needs. This highlights the need for more custom plugins in Hadoop for science applications.
Tropical storm detection code. Another application that we plugged into Hadoop was TSTORMS. It is
a single-threaded analysis program to find tropical storms in climate data. TSTORMS input is a NetCDF
file that has a binary format. While Hadoop and HDFS can handle binary files, current versions of Hadoop
streaming can’t handle binary input and output. NetCDF libraries provide utilities to manage an ASCII
version of NetCDF files and to convert between the ASCII and binary versions. While not the most efficient
way, we explored pre-converting to an ASCII format and loading the converted files into HDFS. The streaming
mapper function then converts it back to a binary format before running the application. In addition,
TSTORMS does not take inputs on standard input and thus the file needs to be streamed to disk and the
input file path is then passed to the binary. The application enjoyed the benefits of scalability and faulttolerance
in the Hadoop framework but it highlights the difficulties of working with scientific data formats
in this environment.
Atmospheric River Detection. The atmospheric river detection code, similar to TSTORMS works on
NetCDF input files. The challenges presented for TSTORMS also holds for this application. In addition,
this application takes a parameter that defines the day to look at in the NetCDF file. Thus, each map should
be differentiated based on a combination of file and a parameter. This can’t be handled easily in the current
Hadoop implementation where a mapper is differentiated by the file or data it operates on. Additionally, we
experimented with a wrapper script, that sequentially process all days in a file. Additional parallelism could
be achieved for such applications if Hadoop would allow maps to be differentiated by not just a file but by
additional parameters.
10.5 Benchmarking
In addition to understanding the programming effort required for applications to use the Hadoop framework,
we conducted some benchmarking to understand the effects of various Hadoop parameters, underlying file
systems, network interface etc.
10.5.1 Standard Hadoop Benchmarks
First, we used the standard Hadoop benchmark tests on the NERSC Magellan Hadoop cluster (described in
Chapter 5). The benchmarking data enables us to understand the effect of various parameters and system
configurations.
86