Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
configurations. User expertise plays a role here, as it does on HPC systems; some users are comfortable
building new configurations and designing system infrastructure while many others are not.
This difference in capabilities demonstrates an important set of trade-offs between cloud and HPC system
support models; specific purpose built APIs (like those provided by HPC centers) can provide deep support
for a relatively fixed set of services, while general purpose APIs (like those provided by IaaS cloud systems)
can only provide high level support efficiently. The crux of the user support issue on cloud systems is to
determine the level of support for various activities ranging from complete to partial support.
7.2 Magellan User Support Model and Experience
Both Magellan sites leveraged their extensive experience supporting diverse user communities to support
the Magellan user groups. In addition, the Argonne Magellan user support model benefited from extensive
experience supporting users of experimental hardware testbeds. These testbeds are similar to clouds in that
users often need administrative access to resources. This experience enabled us to understand the tradeoff
between depth of support and range of user activities required for private cloud environments. Both sites
also heavily leveraged existing systems to manage mailing lists, track tickets, and manage user accounts. Our
user support model consisted of five major areas: monitoring system services, direct support of the high level
tools and APIs provided by the system, construction and support of a set of baseline VM configurations,
training sessions, and building a community support base for more complex problems. We will discuss each
of these areas.
Monitoring. Monitoring is a critical component for providing any user service; even if the services are being
offered as a test bed. While cloud APIs provide access to more generic functionality than the traditional
HPC system software stack, basic testing is quite straightforward. An initial step to monitor system operational
status was to extend the existing monitoring infrastructure to cover the cloud systems. At ALCF, an
existing test harness was extended in order to run system correctness tests on Magellan easily and often. The
test harness ran a variety of tests, from allocating VM instances to running performance benchmarks. These
tests confirmed that the system was performing reliably and consistently over time. A similar approach
was used at NERSC, where tests were run on a routine basis to ensure that instances could be spawned,
networking could be configured, and the storage system was functioning correctly. These tests enabled the
centers to proactively address the most routine problems.
Cloud setup support. The second component of the support model was helping users make use of the
basic cloud services provided by Magellan. These activities included generating users credentials for access
to the cloud services and providing documentation to address routine questions about how to setup simple
VM instance deployments, storage, networks, and security policies. This approach provided enough help for
users to get started, and is analogous in an HPC setting to ensuring that users could login to a system and
submit jobs.
Image support. Our initial assessment suggested that supporting more complex user activities would be
prohibitively expensive, so we opted to take a different approach. Instead of direct support, we provided a
number of pre-configured baseline configurations that users could build on, without needing to start from
scratch. We provided a number of starting VM instance configurations that were verified to work, as well as
several recipes for building common network configurations and storage setups. This approach was relatively
effective; users could easily build on the basic images and documented examples. NERSC also provided tools
to automate building virtual clusters with pre-configured services such as a batch system and an NFS file
system.
Documentation and Tutorials. Both sites also provided tutorials and online documentation to introduce
users to cloud models and address the most common questions and issues. Tutorials were organized for both
37