Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
commercial clouds. The second essentially compares the costs of an entire center. The final approach takes
an application centric approach.
12.2.1 Assumptions and Inputs
In all of the cost analyses, we have attempted to use the most cost effective option available. For example,
based on our benchmarking analysis, the Cluster Compute offering is the most cost effective option from
Amazon for tightly coupled MPI applications and even most CPU intensive applications, since the instances
are dedicated resulting in less interference from other applications. Furthermore, most of the instance
pricing works out to a roughly constant cost per core hour. For example, a Cluster Compute Instance is
approximately 16x more capable than a regular small instance and the cost is approximately 16x more. So
using smaller, less expensive instances isn’t more cost effective if the application can effectively utilize all of
the cores, which is true of most CPU intensive scientific applications. In contrast, many web applications
under utilize the CPU, making small instances more cost effective for those use cases. For compute instances,
we use a one year reserved instance and assume the nodes are fully utilized over the entire year to compute
an effective core hour cost. With reserved instances, you pay a fixed up-front cost in order to pay a lower per
hour cost. If the instance is used for a majority of the reserved period (one year in our analysis), this results
in a lower effective rate. For example, an on-demand Cluster Compute instance costs $1.60 per hour, but a
reserved instance that is used during the entire one year period results in an effective rate of $1.05 (a 30%
reduction). We further divide this by the number of cores in the instance to arrive at an effective core-hour
cost, which simplifies comparisons with other systems. Table 12.1 summarizes this calculation. It is worth
noting that the lowest spot instance pricing is approximately 50% of this effective core hour cost. We do not
use this offering as a basis for the cost analysis, since the runtime for a spot instance is unpredictable and
application programmers need to design their applications to handle pre-emption, which would not match
the requirements for our applications. However, spot pricing does provide an estimate of the absolute lowest
bound for pricing, since it essentially reflects the price threshold at which Amazon is unwilling to offer a
service.
For file system costs, we use elastic block storage to compute the storage costs for file systems. This
most likely underestimates the costs since it omits the costs for I/O requests and the costs for instances that
would be required to act as file system servers. S3 is used to compute the costs for archival storage. S3
uses a tiered cost system where the incremental storage costs decline as more data is stored in the system.
For example, the monthly cost to store the first terabyte of data using reduced redundancy is $0.093 per
gigabyte, while the monthly cost to store data between 1 TB and 49 TB is $0.083 per GB. For simplicity,
we compute all S3 costs at the lowest rate. For example, since the NERSC archival system has 19 PB of
data stored, we use Amazon’s rate of $0.037 per GB (for a month) for data stored above 5 PB with reduced
redundancy. The cost for transactions is also omitted for simplicity, but would further increase the cost of
using the commercial offering. The pricing data was collected from the Amazon website on September 30,
2011.
12.2.2 Computed Hourly Cost of an HPC System
One of the more direct methods to compare the cost of DOE HPC System with cloud offerings is to compute
the effective hourly cost per core hour. This makes it relatively straight forward to compare it with similar
commercial cloud systems. However, determining this cost is problematic, since many of the costs used for the
calculation are indirect or business sensitive. However, for the sake of comparison we will use Hopper, a Cray
XE-6 system recently deployed at NERSC. This system was selected for comparison because it is a relatively
recent deployment, is large enough to capture economy of scale, and is tuned for scientific applications
relevant to the DOE-SC community. In lieu of providing detailed costs that may be business sensitive, we
will use conservative values for the cost which are higher than actual costs. The Hopper contract has been
valued at approximately $52M. We use the peak power of 3 MW of power (it typically uses around 2 MW)
which translates into an power cost of $2.6M per year assuming a cost of $0.10 per KWHour. In general, $0.10
119