Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
that will require additional personnel and possibly training. In the end, overall security costs in clouds are
unlikely to go down but will more likely just shift from one area to another.
Power Efficiency. Power efficiency is often noted as a reason commercial cloud providers are cost effective.
Most of the power efficiency comes from consolidation which was discussed elsewhere. This consolidation
results in higher utilization and less wasted power. In addition, warehouse scale data centers can easily justify
the additional design and operations efforts to optimize the facilities and hardware to achieve improved
power efficiency. Consequently, best-in-class commercial cloud providers boast of power usage effectiveness
(PUE) of below 1.2 [35]. The PUE is the ratio of the total power used to operate a system to the amount of
power used directly by the IT equipment. Higher PUEs are typically due to inefficiencies in cooling, power
supplies, air handlers, etc. So a PUE below 1.2 is an example of a very efficient design. In comparison, a
2007 report cited an estimated average PUE of 2.0. More recent reports estimate a range between 1.83 and
1.92, but these surveys likely include results from commercial cloud data centers [54]. DOE HPC Centers
have raised concerns about power requirements for large systems for nearly a decade. This was triggered
by an observation of the growing costs of electricity to operate large HPC systems. Consequently, many
of the large DOE HPC data centers are working towards improving their PUE and the overall efficiency of
the systems. While DOE facilities may not quite match the best commercial warehouse scale data centers,
recent deployments are approaching PUE values between 1.2-1.3. These efficiencies are achieved through
novel cooling design, architectural choices, and power management. For example, the Magellan system at
NERSC utilized rear door heat exchangers which used return water used to cool other systems. Another
example is the cooling for the ALCF data centers which has been designed to maximize free cooling capabilities
when weather conditions are favorable. With high efficiency chillers installed to allow for partial free
cooling and centrifugal chiller cooling operations simultaneously, up to 17,820 KW-hr can be saved per day.
Furthermore, new planned facilities, such as the LBNL Computational Research and Theory Building, will
incorporate free-air cooling to further improve the energy efficiency. Looking towards the future, a major
thrust of the DOE Exascale vision is to research new ways to deliver more computing capability per watt
for DOE applications. This will likely be achieved through a combination of energy efficient system designs
and novel packaging and cooling techniques.
Personnel. A large fraction of the staff effort at DOE HPC Centers does not go directly towards performing
hardware support and basic system support but is, instead, focused on developing and maintaining the
environment, applications, and tools to support the scientists. Consequently, many of these functions would
still need to be carried out in a cloud model as well. Moving towards a cloud model where individual research
teams managed their environment and cloud instances could actually increase the effort since many of those
functions would go from being centralized at DOE centers or institutional clusters to being decentralized
and spread across individual research groups. This could actually reverse the trend in DOE labs towards
consolidating support staff for IT systems.
Storage. While most of the cost analysis has focused on the cost of computation in the Cloud, storage cost
is also an important consideration. Calculating the TCO for a recent disk storage procurement at NERSC
yields a cost of approximately $85 per TB-Year or $0.007 per GB-Month. This is over 14x less expensive
than the current storage cost for Amazon’s EBS of $0.10 per GB-Month. It is even 5x less expensive than
Amazon’s least expensive incremental rate for S3 with reduced redundancy ($0.037 per GB-Month). In addition,
Amazon imposes additional charges for IO transactions and out-bound data transfers. For example,
at Amazon’s lowest published rates, transferring 10 TB of data out of Amazon would cost around $500. For
some data intensive science applications, this could add considerable costs. Other cloud providers have similar
costs. For example, Google’s Cloud Storage offerings are basically the same ($0.105 per GB-Month for
its lowest published rate). These higher costs do not factor in many of the performance differences discussed
in Section 9.3 nor does the need for high-performance parallel file systems by many HPC applications. The
cost, performance, and access models for storage are a significant barrier to adopting Cloud based solutions
for scientific applications.
124