Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
challenges. Since each node could potentially run many virtual machine instances, the file system could see a
multiplier of file system clients. If each node was running eight small instances (one on each core), then the file
system would have to deal with eight times more clients. Furthermore, virtual machines are often terminated
instead of performing a clean shutdown. This could lead to clients frequently joining and leaving the file
system cluster. File systems like GPFS and Lustre employ timeout and heartbeat methods that assume a
relatively stable client pool. Clients randomly disappearing could lead to long hangs while outstanding locks
held by terminated instances were timed out.
There are several potential methods to address these issues. One would be to project only the portions
of the file system that the user owns into the virtual machine. This could be done using protocols like NFS
and re-exporting the parallel file system. This option was explored by ALCF Magellan staff, but ran into
challenges with Linux kernel and I/O libraries. Another approach would be to forward the file I/O operations
to a proxy server that would have the file systems mounted. The operations would then be performed on
the proxy server as the user who owned the virtual machine instance. The standard file system client would
enforce the access controls.
There are existing file system modules that use the Linux FUSE file system interface to forward I/O over
connections like SSH. The performance over SSH would be poor, but would at least provide access to data.
This method was used by the ALCF/LCRC project that built an extension of the Argonne LCRC Fusion
within the OpenStack cloud and worked reasonably well. Alternatively, the I/O Forwarding Scalability Layer
(IOFSL) project [2] is developing a high-performance I/O forwarding layer that could potentially help. While
IOFSL is focused on developing an I/O forwarding system to support ultra-scale HPC systems, the same
mechanisms can be used for virtual machines. These solutions would also help mitigate the scaling and
stability challenges, since they would reduce the number of real clients handled by the file system and would
act as a firewall between the virtual machines and the file system.
9.8 Summary
Applications with minimal communication and I/O perform well in cloud environments. Our evaluation
shows that there is a virtualization overhead that is likely to get better with ongoing research. But the
primary performance impact for HPC applications comes from the absence of high-bandwidth, low-latency
interconnects in current virtual machines. Thus a majority of current DOE HPC applications will not
run efficiently in today’s cloud environments. Our results show that the difference between interconnects
is pronounced even at mid-range concurrencies. Similarly there is an I/O performance issue on virtual
machines, and the absence of high-performance file systems further impacts the productivity of running
scientific applications within the cloud environment.
82