Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
by the networking service, and are then ’attached’ to the requested virtual machine via additional network
address translation rules.
As all networking is handled by one service, this introduces a performance bottleneck for communicationintensive
applications. Furthermore, the complexity of the networking setup is quite large, and requires a
significant amount of effort to understand. Some inflexibility with regard to support of physical network
architecture were encountered in the design as well.
Scalability. Overall, OpenStack scales well, and performance on the 420 node cloud at ALCF was reliable
up to per-project instance counts of more than 200. Instance counts of 300 or greater have been successfully
accomplished, though at that scale the single API node becomes overloaded and instance launch times
increase dramatically.
6.3 Nimbus
Nimbus is an open-source IaaS cloud-computing framework, created by a team of developers at Argonne
National Labs and the University of Chicago. It has been successfully deployed on a number of sites around
the country, one of which is an 82 node cloud running on Magellan project hardware at Argonne.
Internals. Nimbus supports the creation of virtual machines using both Xen and KVM, the latter being
used on the Magellan cloud. The core services include the Workspace Service, which handles all virtual machine
management and user interaction via grid-credential or Amazon EC2 API. The configuration clients
run on individual virtual machines, and control setup of services, user and virtual cluster credentials, and
image metadata acquisition. The cloudinit.d script is one of these clients, so user images can easily be
modified for compatibility with Amazon EC2, similar to Eucalyptus and OpenStack.
Running Nimbus on Magellan. The two biggest challenges encountered running Nimbus on Magellan
hardware involved issues with networking and software dependencies. The network issues revolved primarily
around the security measures present on site. Difficulties with ingress firewall settings caused user ssh
sessions to timeout prematurely, and complicated the application of public IP addresses to virtual machines;
this prompted one user to eschew the use of externally-routable IPs altogether. Taken together, these issues
made management of the cloud difficult; and even simple administrative tasks required exorbitant effort.
In order to alleviate the aforementioned networking difficulties, an image propagation protocol called
LANTorrent was developed. LANTorrent is a multicast system that allows researchers to quickly distribute
VM images across the cloud at one time; tests on Magellan resulted in 1000 simultaneous boots. LANTorrent
forms a store and forward chain of recipients that allows the image to be sent at a rate much faster than can
be accomplished with http or ssh transfers. This significantly lessened the time a user’s cluster required to
boot.
6.4 Discussion
In the previous section, we detailed current cloud offerings. Here we examine some of the gaps and challenges
in using existing cloud offerings directly for scientific computing.
Resources. Cloud computing carries with it the assumption of an unlimited supply of resources available
on demand. While this was true in early days of cloud computing when demand for resources was still
ramping up, more recently users have noticed that their requests have not been satisfied on providers such
as Amazon EC2 due to insufficient capacity. This situation is similar to current day supercomputing and
grid resources that are often over-subscribed and have long wait queues. Thus for the user, scientific cloud
computing as an unlimited supply of cycles tends to be less promising. There is a need for differentiated
levels of service similar to Amazon’s current offerings but with advanced resource request interfaces with
34