Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
TeraSort is a standard map/reduce application for Hadoop that was used in the terabyte sort competition.
TeraGen generates the data, and TeraSort then samples the input data and uses map/reduce to sort the
data in total order. Figures 10.3 and 10.4 show the comparison of using HDFS and GPFS as the underlying
file system for TeraGen with varying number of maps. HDFS and GPFS have been designed for largely
different usage scenarios and the goal of our comparison is not a quantitative performance comparison of
the two system. Each of these file systems has its own strengths for certain workloads. Our goal here is
to understand if scientific applications can benefit from Hadoop’s job management framework while using
POSIX compliant file systems available in HPC centers.
Figure 10.3 shows the time for TeraGen to generate 1 TB of data in Hadoop on both file systems. We
see that the performance of GPFS shows a slight decrease in performance as the number of concurrent
maps is increased. On the other hand, HDFS’s performance significantly improves as number of maps
increases as HDFS is able to leverage the additional bandwidth available from disks in every compute node.
Figure 10.4 shows the effective bandwidth for both systems and we can see that HDFS’s effective BW is
steadily increasing. Our GPFS system is used in production use and the variability seen here is from other
production workloads using the system. Hadoop and HDFS have been designed to handle high-levels of
parallelism for data-parallel applications. These results show that for small to medium scale parallelism,
applications can use Hadoop with GPFS without any loss in performance.
Time (minutes)
12
10
8
6
4
HDFS
GPFS
Linear(HDFS)
Expon.(HDFS)
Linear(GPFS)
Expon.(GPFS)
2
0
0 500 1000 1500 2000 2500 3000
Number of maps
Figure 10.3: HDFS and GPFS Comparison (Time)
7000
6000
HDFS
GPFS
Bandwidth (MB/s)
5000
4000
3000
2000
1000
0
400 600 800 1000
Number of maps
Figure 10.4: HDFS and GPFS Comparison (Bandwidth)
10.5.2 Data Intensive Benchmarks
The following work was funded through a project evaluating Hadoop specifically for data intensive scientific
applications. Magellan staff supervised the student who conducted these experiments and all experiments
were run on Magellan Hadoop testbed. A paper describing the results in detail is in preparation. Here we
88