Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
35000
35000
30000
30000
25000
25000
Time
(ms)
20000
15000
10000
5000
0
20
 40
 60
 80
 100
Maps
20
100
60
 Reduces
Time
(ms)
20000
15000
10000
5000
0
20
 40
 60
 80
 100
Maps
20
100
60
 Reduces
(a) 100 lines
(b) 1000 lines
Figure 10.1: Hadoop MRBench.
MRBench evaluates the performance of MapReduce systems while varying key parameters such as data
size and the number of Map/Reduce tasks. We varied the number of lines of data written from 100 to 1000
and varied the number of maps and reduces. Figure 10.1 shows the time with varying maps and reduces
for a) 100 and b) 1000 lines. As the number of maps and reduces increases the time increases; however
the difference is less than 10 seconds. The number of lines written at the orders of magnitude we measure
shows no perceptible effect. MRBench can be provided with custom mapper and reducer implementations to
measure specific system or application behavior. This could be used to develop benchmarks that emphasize
the nature of scientific workloads.
TestDFSIO measures the I/O performance of HDFS. Figure 10.2 shows the throughput for small and large
file sizes with varying concurrent writers/files. For small file sizes, the throughput remains fairly constant
with varying number of concurrent writers. However, the throughput decreases rapidly as the number of
concurrent files/writers increases. This seems to be dependent on the HDFS block size and will require
tuning and additional benchmarking to understand the configuration necessary for scientific workloads at a
specific site.
70
60
10MB
10GB
Throughput MB/s
50
40
30
20
10
0
0 10 20 30 40 50 60
Number of concurrent writerss
Figure 10.2: HDFS throughput for different file sizes with varying number of concurrent writers.
87