Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
Table 12.3: Annual Cost of the DOE NERSC HPC Center in the cloud
NERSC
ALCF
Cost in Cloud per Year
Cost in Cloud per Year
· Hopper (152,496 cores) $148,993,548 · Intrepid (163,840 cores) $160,077,004
· Franklin (38,320 cores) $37,439,885 · Surveyor (4096 cores) $4,001,925
· Carver (3,200 cores) $3,126,504 · Challenger (4096 cores) $4,001,925
· HPSS (17 PB, 9 PB transferred)
$8,189,952 · HPSS (16 PB, 9PB transferred) $1,062,297
· File Systems (2 PB Total) $2,642,412 · File Systems (8 PB Total) $10,066,329
Total Cost in Cloud $200,392,301 Total Cost in Cloud $179,209,482
NERSC Total Annual Budget $55,000,000 ALCF Total Annual Budget $41,000,000
(including support and other
(including support and other
services)
services)
NERSC Annual Budget $33,000,000 ALCF Annual Budget $30,750,000
for systems and related costs
only
for systems and related costs
only
estimate. In addition, the budgets for both centers include additional services and support which would not
be captured in the cloud cost. Our users have indicated the need for these services for cloud solutions as
well (Chapter 11). These additional services account for approximately 40% of NERSC’s annual budget and
25% of ALCFs budget (calculated based on the staff at the centers who provide these additional services
and support). Adjusting for these costs results in a comparison of approximately $200M for the commercial
cloud versus $33M for the actual costs of NERSC to DOE, and a comparison of $180M for the commercial
cloud versus $31M for the actual costs of ALCF to DOE.
12.2.4 Application Workload Cost and HPL Analysis
Another approach is to consider the cost of running a given workload in a commercial cloud and multiplying
that out for the anticipated amount that the workload would be performed over a period of time. Chapter 9
showed the costs for bioinformatics workloads and STAR workloads. This approach works best when the
workload can be well encapsulated and doesn’t have extraneous dependencies on large data sets. For example,
performing functional analysis on genomic data sets works well since the input data can be easily quantified
and the run time for a given dataset can be easily measured.
Here we use the Linpack benchmark as a stand-in for the various applications that run at NERSC
and ALCF. One reason for choosing this benchmark is it is a recognized benchmark that is published
for many systems and removes many of the complexities of making a comparison across different system
architectures. Also, it was chosen because it gives some favoritism to the Cloud systems since Linpack has
high computational intensity and makes modest use of the interconnect compared with many real-world
scientific applications using MPI (Section 9.1.5). For the comparison, we calculate the cost of a Teraflop
Year. This represents the cost to deliver a full teraflop for an entire year. Table 12.4 shows a comparison of
the cost of a Teraflop Year (based on Linpack) from Amazon, NERSC, and ALCF. All of the HPL results
come from the published values listed on the TOP500 [80] website from the June 2011 list. The Amazon
costs are based on 1-year reserved Cluster Compute Instances that are assumed to be fully utilized during
the year (as described earlier). The NERSC and ALCF costs are based on each center’s total annual budget
which includes significantly more than just computation as noted in the section above. The comparison
illustrates that for tightly coupled applications (even ones with high computational intensity like Linpack),
the cost of a commercial cloud is historically much higher (over a factor of 4). This factor is lower compared
with some of the other approaches, but, again these calculations use the entire center budgets which include
122