Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
Hadoop and Eucalyptus users. These sessions primarily provided users with an overview of the basic steps
to using these technologies and provided an opportunity to discuss some of the details of how to port applications
to these environments. For example, ALCF held a ”Welcome to Magellan Day” workshop that ran
a full day and included 4 hours of intensive hands-on training in using Eucalyptus, building and managing
images, and debugging images. This workshop demonstrated the difficulties in training scientific users to
become, in essence, system administrators.
Community Support Model. For advanced user activities, we relied on a community support model.
In order for community support to work, it needs a critical mass of experienced users willing to share their
expertise. Building a community of experienced users requires significant outreach and training, as well
as an extended period of time to develop this community support base. In addition, it needs to be clear
when issues are system problems versus a user issue. Early in the project, the cloud software stacks were
sufficiently immature that this process was difficult. Aside from these issues, the community support model
worked reasonably well. Successful groups often had a primary user who was either already experienced
or gained the expertise to build the infrastructure for the rest of the group to use. While this approach is
not always feasible, it worked particular well in many cases. For example, one Argonne scientific team was
able to build a complete cluster inside of the ALCF Magellan public cloud space, including the Sun Grid
Enging (SGE) batch queueing system and storage, which also had secure access to kerberized NFS resources
outside the system. They were able to route jobs into the system for execution. The team was able to
successfully achieve this complicated configuration because one member of their team had extensive system
administration expertise.
7.3 Discussion
Many of the problems users faced on both the virtual and hardware-as-a-service clusters involved Linux
administrative tasks. Most HPC users have little systems administration experience, so even relatively
minor issues frequently required support from Magellan staff. Often, this could be accomplished by sharing
advice or links to online documentation via email, but at other times it required launching rescue images to
diagnose and repair a damaged system. Installing software and debugging system errors were often necessary
to kick-start a user’s efforts. This process was complicated further when the users chose Linux distributions
outside of the baseline images which the Magellan staff had pre-configured.
Magellan staff also handle challenges related to access to commercial software or other common tools
or infrastructure. For example, some users were accustomed to using proprietary software which could not
always be bundled into a virtual machine without violating license agreements. If a user had the appropriate
license, they had the freedom to install and customized the images to include the software, but this often
led to issues with supporting modified images.
Security related challenges were also commonplace. Many of these issues revolved around user credentials.
We discuss additional security challenges in Chapter 8. On the virtual cluster, there were typically problems
with the users’ client-side tools installation or SSL libraries. For example, only certain versions of the clientside
ec2 tools work with Eucalyptus. At ALCF, almost all of the users had some initial difficulty using the
cryptographic tokens which were required to access the bare-metal provisioning service. While these were
often easily resolved, they constituted a significant portion of the total support requests for this service.
Another common issue was with externally-routable IPs and firewall settings on the instances, which by
default block all incoming connections. Again these were easily resolved, but increased the support load.
The final challenges that users often encountered were related to managing data and workflows (discussed
in Chapter 11). The solution to these challenges are often application specific requiring a detailed
understanding of the data and workflow. Furthermore, to fully leverage the advantages of the cloud model,
such as elasticity, often requires the user to fundamentally rethink the execution model. All of this makes it
difficult to offer a cost-effective support model.
38