Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
compute instance type. Linpack is able to take full advantage of these increases. However, Linpack is not
representative of all scientific applications. Thus, as shown in our benchmarking (Chapter 9), scientific
applications do not typically realize the same benefits. The increase in FLOPs per cycle often occurs every
other generation of processor from Intel, since it is typically a byproduct of a change in core architecture.
This is the “Tock” of the “Tick-Tock” development cycle in Intel parlance. AMD typically follows a similar
roadmap for their processor architecture. Consequently, these types of improvements typically occur every
3-4 years and will be seen in DOE centers in the next few years.
Amazon announced new pricing in November 2011. This lowered the price for some instance types and
introduced new tiers of reserved instances. Some instances did not change in cost, e.g., the m1.small remains
at $0.085 per hour. However, some of the higher-end instances did drop. For example, the on-demand
price of a cc1.4xlarge instance (the type used in much of the analysis above) dropped by 19%. Amazon
also expanded the reserved instances to three different tiers. The new tiers allow customers to match the
reserved instance to the level of utilization. These different tiers essentially trade higher upfront costs for
lower per-usage costs. The tier with the highest upfront costs, “Heavy Utilization Reserved Instances”,
provides the lowest effective rate if the instance is fully utilized over the reservation period.
Applying the earlier analysis to the new instance type yields some interesting results. The per-core hour
cost of a 1-year reserved instance of the new type results in $0.058 cents per core hour versus $0.13 per core
hour before. This is partly due to a heavily discounted reserved instance cost for this instance type. For
example, fully utilizing a 1-year reserved instance for most types provides a discount around 40% over the
on-demand option. However, for the new instance type this discount is 63%. More impressive, the cost of a
Teraflop Year (Section 12.2.4) drops to approximately $36k per TF-Year. This is a 5x improvement over the
previous calculation. Roughly half of this improvement comes from the switch to Intel Sandy Bridge and the
resulting doubling in FLOPs per cycle. Since Amazon essentially scales its pricing on the number of available
cores in the instance, not floating point performance, there is no premium charge for the doubling of the
FLOPs per cycle for the new instance type. Another fraction of the increase comes from the improvement
in efficiency of the Linpack execution (68% versus 50%). The remainder comes from the drop in pricing and
especially the heavily discounted reserved instance pricing.
The DOE systems used in the previous analysis were deployed over a year ago. ALCF’s Intrepid was
deployed almost 4 years ago. So, much of the improvement in the Amazon’s values come from the deployment
of a very new architecture. Ultimately, the DOE Labs and the commercial cloud vendors are relying on the
same technology trends to deliver improved performance over time. As DOE Centers deploy new technologies
like Intel Sandy Bridge and AMD Interlagos, their costs will drop in a similar manner. DOE deployments
are typically timed about three years apart at each center, with these deployments staggered across centers.
In addition, centers will often perform mid-life upgrades to systems to remain closer to the technology edge.
This combination of strategies enables DOE to deliver a portfolio of systems that closely tracks the technology
improvements. For example, the Mira system is projected to achieve a cost less than $8k per TF-Year when
it is deployed next year. This is approximately 4x better than Amazon’s improved result, and represents an
improvement of around 10x over the previous ALCF system, Intrepid, which was deployed around 4 years
ago. Eventually, DOE HPC centers and commercial cloud providers like Amazon are likely to track each
other, aiming for cost-effectiveness. However, the pricing changes in the commercial cloud will not address
the various other challenges in moving scientific computing and HPC workloads to cloud offerings, such as
workflow and data management challenges, high-performance parallel file systems, and access to legacy data
sets (see Sections 6.4 and 11.4).
The recent announcement by Amazon highlights several lessons. One, it shows that Amazon is responding
to feedback from the HPC community on improving their offerings in this space. This process started with
the introduction of the Cluster Compute and GPU offerings over a year ago and continues with the recent
announcement of the new instance type and associated pricing. Secondly, it demonstrates the importance of
tracking the changes in the cloud space and routinely updating cost analysis to ensure that the appropriate
choices are being made.
127