Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
Table 12.2: Hourly Cost of a DOE HPC system
Hopper
Total Acquisition Cost $52,000,000
Annual Acquisition Cost (divided over 4 years) $13,000,000
Maintenance & Support $5,200,000
Power & Cooling $2,600,000
FTEs $300,000
Total Annual Costs $21,100,000
Cores 153,408
Utilization 85%
Annual Utilized Core Hours 1,142,275,968
Cost per Utilized Core Hour (includes storage subsystem) $0.018
HPC systems. Currently, there is no streamlined method to provide this type of capability in commercial
IaaS cloud systems like Amazon. However, even using the best methods available in the existing IaaS system,
creating a file system of this scale would add additional costs for storage and for additional instances to act
as servers for the file system.
12.2.3 Cost of a DOE Center in the Cloud
Another cost analysis approach is to consider the total annual budget of a center and the resources it provides
to the users and translate those into commercial cloud offerings. In Table 12.3, we show a summary of the
approximate cost to operate NERSC and ALCF in the Amazon cloud. The computational systems are
calculated using Amazon’s Cluster Compute offering which provides the closest performance to traditional
HPC systems. Based on the performance studies described in Chapter 9, using less expensive, less capable
offerings would result in significantly lower performance, which would actually increase costs. In addition, the
estimates use a 1-year reserved instance with the generous assumption that the cores are used for an entire
year (see Section 12.2.1 for a more detailed description of the assumptions used in the pricing calculations).
This results in effective core hour cost of $0.13 per core hour. The total core hours delivered by each system in
a year is multiplied by this rate assuming 85% overall utilization (NERSC typically exceeds this value). The
storage costs are calculated using $0.037 per GB Month and $0.05 per GB for transfers out of S3 (transfers
into S3 are currently free). This is based on the marginal cost of single copy storage in Amazon S3. The file
system costs are based on Amazon’s Elastic Block Storage cost of $0.10 per GB Month. The data amounts
are based on the current volume of data stored on disk and tape, not the available capacity. They do not
account for the observed growth in data storage. For example, the annual growth in HPSS (tape) is around
70%. The calculated total annual cost to operate NERSC in the cloud is just over $200M, and to operate
ALCF in the cloud is just under $180M. NERSC has a current annual budget of between $50M-$55M. ALCF
is a relatively new facility and its budget has grown since it started. The budget shown in the table is an
average over a 4 year period, using ALCF’s actual budgets for 2008 through 2011. These budgets include
all system acquisition costs, support and maintenance, power and cooling cost, system administration and
management, and software licensing costs, as well as data analytics and visualization resources that are
provided to the facility users.
While facilities costs are not directly included in NERSC’s budget, and some are not directly included in
ALCF’s budget (data center electricity is included), they are incorporated in overhead charges and burdens
and therefore indirectly included in the budget. However, the facilities costs are negligible. For example,
if we use an industry average cost of $1,200 per square foot for data center space for a system the size of
Hopper, we arrive at $3.6M. Assuming a 10 year life span for the data center, which is low, we arrive at $360k
per year. This translates into approximately 3/100th of a cent per core hour or around a 2% increase in the
121