Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
Routing IP packets from the virtual machine (VM) to the hypervisor (HV) and then using IP over
InfiniBand is the most readily available approach today. In this approach, the VM communicates through
the HV using a virtual network interface. Then standard IP routing is used to route the packets over the
InfiniBand interface. This approach provides the lowest performance of the various options. Since the packet
must traverse multiple TCP stacks (in the VM and through the HV), the overhead is high. Also there are
numerous integration challenges in integrating this with cloud management software like Eucalyptus and
OpenStack.
Utilizing Ethernet over InfiniBand (EoIB) is another approach. In EoIB, first proposed by Mellanox, the
Ethernet frames are encapsulated in an InfiniBand packet. The InfiniBand stack presents a virtual Ethernet
NIC. This virtual NIC would be presented on the HV and then bridged to the VM. This approach has several
advantages over the other approaches. VLANs should be supported in this model, which would provide some
support for isolating networks. Bridging of the virtual Ethernet interface is similar to how Ethernet NICS
are commonly bridged for a virtual machine. This means that packages like Eucalyptus could more easily
support it. Since the communication would be TCP-based, some performance would be lost compared with
remote memory access offered by native InfiniBand. However, the packets would only go through a single
TCP stack (on the VM) on each side. So the performance should exceed the IP over InfiniBand method.
While this approach looks promising, the current EoIB implementation does not support Linux bridging.
Another disadvantage is that only Mellanox currently plans to support EoIB, and it requires a special bridge
device connected to the InfiniBand network. However, the devices are relatively low cost.
Full virtualization of the InfiniBand adapter is just starting to emerge. This method would provide the
best performance, since it would support full RDMA. However, it would be difficult to isolate networks
and protect the network from actions by the VM. Furthermore, images would have to include and configure
the InfiniBand stack. This would dramatically complicate the process of creating and maintaining
images. Finally, only the latest adapters from Mellanox are expected to support full virtualization. The
version of adapters in Magellan will likely not support this method. We are exploring options for evaluating
virtualization of the NIC and looking at alternative methods.
High-performance networks like InfiniBand can dramatically impact the performance, but integrating
them into virtual machine-based systems requires further research and development. Alternatively, it may
be more attractive to look at how some of the benefits of virtualization can be provided to the users without
using true virtualization. For example, our early experiments with MOAB/xCAT show that dynamic imaging
of nodes or software overlay methods allow the user to tailor the environment for their application without
incurring the cost of virtualization.
9.7.2 I/O on Virtual Machines
The I/O performance results clearly highlight that I/O can be one of the causes of performance bottlenecks
on virtualized cloud environments. Performance in VMs is lower than on physical machines, which may be
attributed to an additional level of abstraction between the VM and the hardware. Also, it is evident from
the results that local disks perform better than block store volumes. Block store volumes can be mounted
on multiple instances and used as a shared file system to run MPI programs, but the performance degrades
significantly. Thus these may not be suitable for all the MPI applications that compose the HPC workload.
Our large-scale tests also capture the variability associated with virtual machine performance on Amazon.
HPC systems typically provide a high-performance scratch file system that is used for reading initial
conditions, saving simulation output, or writing checkpoint files. This capability is typically provided by
parallel file systems such as GPFS, Lustre, or Panasas. Providing this capability for virtual machine-based
systems presents several challenges including security, scalability, and stability.
The security challenges focus around the level of control a user would have within the virtual machine.
Most parallel file systems use a host-based trust model, i.e., a server trusts that a client will enforce access
rules and permissions. Since the user would likely have root privileges on the VM, they would be capable of
viewing and modifying other users’ data on the file system.
Accessing shared file systems from user controlled virtual machines can also create scalability and stability
81