Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
data pre- and post-processing steps, such as visualization, are often performed on the scientist’s desktop.
The increased scale of digital data due to low-cost sensors and other technologies has resulted in the need
for these applications to scale [58]. These applications are often penalized by scheduling policies used
at supercomputing centers. The requirements of such applications are similar to those of the Internet
applications that currently dominate the cloud computing space, but with far greater data storage and
throughput requirements. These workloads may also benefit from the MapReduce programming model by
simplifying the programming and execution of this class of applications.
4.2 Usage Scenarios
It is critical to understand the needs of scientific applications and users and to analyze these requirements
with respect to existing cloud computing platforms and solutions. In addition to the traditional DOE
HPC center users, we identified three categories of scientific community users that might benefit from cloud
computing resources at DOE HPC centers:
4.2.1 On-Demand Customized Environments
The Infrastructure as a Service (IaaS) facility commonly provided by commercial cloud computing addresses a
key shortcoming of large-scale grid and HPC systems, that is, the relative lack of application portability. This
issue is considered one of the major challenges of grid systems, since significant effort is required to deploy
and maintain software stacks across systems distributed across geographic locales as well as organizational
boundaries [30]. A key design goal of these unified software stacks is providing the best software for the
widest range of applications. Unfortunately, scientific applications frequently require specific versions of
infrastructure libraries; when these libraries aren’t available, applications may run poorly or not at all. For
example, the Supernova Factory project is building tools to measure the expansion of the universe and Dark
Energy. This project has a large number of custom modules [1]. The complexity of the pipeline makes
it important to have specific library and OS versions which makes it difficult to take advantage of many
large resources due to conflicts or incompatibilities. User-customized operating system images provided by
application groups and tuned for a particular application help address this issue.
4.2.2 Virtual Clusters
Some scientific users prefer to run their own private clusters for a number of reasons. They often don’t need
the concurrency levels achievable at supercomputing centers, but do require guaranteed access to resources
for specific periods of time. They also often need a shared environment between collaborators, since setting
up the software environment under each user space can be complicated and time consuming. Clouds may
be a viable platform to satisfy these needs.
4.2.3 Science Gateways
Users of well-defined computational workflows often prefer to have simple web-based interfaces to their
application workflow and data archives. Web interfaces enable easier access to resources by non-experts,
and enable wider availability of scientific data for communities of users in a common domain (e.g., virtual
organizations). Cloud computing provides a number of technologies that might facilitate such a usage
scenario.
4.3 Magellan User Survey
Cloud computing introduces a new usage or business model and additional new technologies that have
previously not been applied at a large scale in scientific applications. The virtualization technology that
16