Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
Cost per TF to operate
for 1 year
Table 12.4: Comparison of cost of a Teraflop Year between DOE Center and Amazon
Amazon (1-year reserved
NERSC
ALCF
instances)
HPL Peak 42TF 1361 TF (Hopper 1054 TF,
Franklin 266 TF, Carver
41TF)
481 TF (Intrepid 459TF,
Surveyor 11TF, Challenger
11TF)
Cost $7.5M/year for 42TF $55M/year (entire annual $41M/year (average an-
center budget)
nual center budget)
$179K/year per TF $40K/year per TF $85K/year per TF
services, storage, network, and other benefits. For example, if we isolate just the Hopper system (1054 TF)
and use the annual costs from Section 12.2.2 ($21.1M per year), we arrive at $20k per TF-Year. The ALCF
numbers demonstrate the impact of older equipment on cost analysis, as the ALCF systems are coming up
on their fifth year of operations. These systems will soon be replaced with new, next generation IBM Blue
Gene systems. A discussion of this impact is provided in the section on historical trends in pricing below.
There are scientific applications where the performance penalty of running in a virtualized environment
including Cloud systems is less significant. For example, many high-throughput applications such as those
used in bioinformatics run relatively efficiently in cloud systems since they are computationally intensive and
can run independent of other tasks. One way of understanding the potential improvements for applications
that run more efficiently in Clouds is to recalculate the Linpack result assuming the same efficiency across
systems. Amazon’s Linpack result achieved 50% of peak compared with 82% of peak for the NERSC and
ALCF systems. If Amazon were able to achieve the same efficiency as the DOE HPC systems, then their
cost would drop to $109K/year per TF which is still higher than the DOE systems.
12.3 Other Cost Factors
In addition to the analysis above, we briefly discuss other factors that can impact costs and should be considered
when performing a cost/benefit analysis for moving to a cloud model.
Utilization. Consolidation has been cited as one of the primary motivators for using cloud resources. Utilization
of IT hardware assets has been reported at between 5% and 20% [5]. This is not a typical utilization
rate at HPC Centers, as they already consolidate demand across a large user base. For example, DOE HPC
Centers provide resources to hundreds of projects and thousands of users. Consequently, many DOE HPC
Centers have very high utilization rates. In the above analysis, we have assumed a utilization of 85% which
is relatively conservative.
Security. Security is another area where outsourcing to the cloud is often expected to reap cost savings,
since the responsibility for security can potentially be shifted to the commercial cloud provider. This may be
more accurate for particular cloud models than others. For example, when outsourcing services like e-mail,
the service provider can centrally manage applying updates, protecting against common spam and phishing
attacks, etc. This can likely be done more efficiently and effectively than typical in-house operations
can achieve. However, for infrastructure services the answer is more complicated. In an IaaS model, the
commercial cloud provider is responsible for certain aspects of security such as physical security, network
security, and maintaining security domains. However, the end user must still insure that the operations of
the virtual resources under their control are compliant with relevant security requirements. Many of the
security mechanisms are implemented and enforced at the host level, such as maintaining the security of
the operating system, configuration management, and securing services. As illustrated in the case studies
(Chapter 11), most scientific users are not experts in these areas and thus will require user support services
123