Magellan Final Report - Office of Science - U.S. Department of Energy

More documents

Recommendations

Info

Magellan Final Report 3. Application developers should consider the potential for variability and failures in their design and implementation. While this is a good practice in general, it is even more critical for applications running in the cloud, since they experience significant failures and performance variations. 4. Cloud technologies allow user groups to manage their own machine images, enabling groups with complex software dependencies to achieve portability. This flexibility comes with the challenge for these groups to ensure they have addressed security concerns. Science groups should attempt to use standardized secure images to prevent security and other configuration problems with their images. Science groups will also need to have an action plan on how to secure the images and keep them up to date with security patches. 5. Cloud technologies such as message queues, tabular storage, and blob or object store provide a number of key features that enable applications to scale without the need to use synchronization where it may not be necessary. These technologies fundamentally change the application execution model. Scientific users should evaluate technologies such as message queues, tabular storage, and object storage during application design phase. Tools development and research. A number of site-level and user-side tools to manage scientific environments at HPC and cloud environments have evolved in the last few years. However, there are significant gaps and challenges in this space. We identify some key areas for tool development and research related to adoption of cloud models and technologies to science. 1. Virtual machines or provisioned bare metal hardware are useful to many application groups. However, scientists need to handle the management of these provisioned resources, including the software stack, job and data coordination. Tool developers should support tools to simplify and smooth workflow and data management on provisioned resources. 2. Investments are needed in tools that enable automated mechanisms to update images with appropriate patches as they become available and a simple way to organize, share and find these images across user groups and communities. Tool developers should consider developing services that will enable organization and sharing of images. 3. Virtualized cloud environments are limited by networking and I/O options available in the virtual machines. Access to high-performance parallel file systems such as GPFS and Lustre, and low-latency, high-bandwidth interconnects such as InfiniBand within a virtual machine would enable more scientific applications to benefit from virtual environments without sacrificing performance or ease of use. System software developers should explore methods to provide HPC capabilities in virtualized environments. 4. There is a need for new methods to monitor and secure private clouds. Further research is required in this area. Sites typically rely on OS-level controls to implement many security policies. Most of these controls must be shifted into the hypervisor or alternative approaches must be developed. Security developers should explore new methods to secure these environments and, ideally, leverage the advanced mechanisms that virtualization provides. 5. MapReduce can be useful for addressing data-intensive scientific applications, but there is a need for MapReduce implementations that account for characteristics of scientific data and analysis methods. Computer science researchers and developers should explore modifications or extensions to frameworks like Hadoop that would enable the frameworks to understand and exploit the data models of data formats typically used in scientific domains. 6. A number of cloud computing concepts and technologies (e.g., MapReduce, schema-less databases) have evolved around the idea of managing “big data” and associated metadata. Cloud technologies address the issues of automatic scaling, fault-tolerance and data locality, all key to success of large-scale systems. There is a need to investigate the use of cloud technologies and ideas to manage scientific data and metadata. viii
Contents Executive Summary Key Findings Recommendations i iii vi 1 Overview 1 1.1 Magellan Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 NIST Cloud Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.4 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.5 Outline of Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2 Background 5 2.1 Service Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1.1 Infrastructure as a Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1.2 Platform as a Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1.3 Software as a Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1.4 Hardware as a Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Deployment Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.3 Other Related Efforts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3 Magellan Project Activities 8 3.1 Collaborations and Synergistic Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.2 Advanced Networking Initiative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.3 Summary of Project Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 4 Application Characteristics 15 4.1 Computational Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 4.2 Usage Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4.2.1 On-Demand Customized Environments . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4.2.2 Virtual Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4.2.3 Science Gateways . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4.3 Magellan User Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4.3.1 Cloud Computing Attractive Features . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.3.2 Application Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.4 Application Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.4.1 Climate 100 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.4.2 Open Science Grid/STAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.4.3 Supernova Factory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 i
Page 1 and 2: The Magellan Report on Cloud Comput
Page 3 and 4: Executive Summary The goal of Magel
Page 5 and 6: Key Findings The goal of the Magell
Page 7 and 8: Magellan Final Report Finding 8. DO
Page 9: Magellan Final Report role in addre
Page 13 and 14: Magellan Final Report 9.7 Discussio
Page 15 and 16: Chapter 1 Overview Cloud computing
Page 17 and 18: Magellan Final Report • The Argon
Page 19 and 20: Chapter 2 Background The term “cl
Page 21 and 22: Magellan Final Report 2.1.4 Hardwar
Page 23 and 24: Magellan Final Report Table 3.1: Ke
Page 25 and 26: Magellan Final Report Little Magell
Page 27 and 28: Magellan Final Report 3.2 Advanced
Page 29 and 30: Chapter 4 Application Characteristi
Page 31 and 32: Magellan Final Report Table 4.1: Pe
Page 33 and 34: Magellan Final Report Output data
Page 35 and 36: Magellan Final Report of the pipeli
Page 37 and 38: Chapter 5 Magellan Testbed As part
Page 39 and 40: Magellan Final Report Figure 5.1: P
Page 41 and 42: Magellan Final Report Figure 5.2: P
Page 43 and 44: Magellan Final Report NERSC deploye
Page 45 and 46: Magellan Final Report Figure 6.1: A
Page 47 and 48: Magellan Final Report greater than
Page 49 and 50: Magellan Final Report specific QoS
Page 51 and 52: Magellan Final Report configuration
Page 53 and 54: Magellan Final Report 7.4 Summary U
Page 55 and 56: Magellan Final Report Firewalls are
Page 57 and 58: Magellan Final Report Aside from le
Page 59 and 60: Magellan Final Report 9.1 Understan
Page 61 and 62:
Magellan Final Report grid) on 256
Page 63 and 64:
Magellan Final Report Table 9.1: HP
Page 65 and 66:
Magellan Final Report 25  Ping 
Page 67 and 68:
Magellan Final Report 100  12 
Page 69 and 70:
Magellan Final Report case of GTC,
Page 71 and 72:
Magellan Final Report 1.4 IB TCPo
Page 73 and 74:
Magellan Final Report only affects
Page 75 and 76:
Magellan Final Report Figure 9.11:
Page 77 and 78:
Magellan Final Report charted as a
Page 79 and 80:
Magellan Final Report Evaluation Cr
Page 81 and 82:
Magellan Final Report Write Perform
Page 83 and 84:
Magellan Final Report 3500 3000 G
Page 85 and 86:
Magellan Final Report Histogram Plo
Page 87 and 88:
Magellan Final Report SATA devices.
Page 89 and 90:
Magellan Final Report MB/s Virident
Page 91 and 92:
Magellan Final Report and the perfo
Page 93 and 94:
Magellan Final Report (a) Hosts (b)
Page 95 and 96:
Magellan Final Report Routing IP pa
Page 97 and 98:
Chapter 10 MapReduce Programming Mo
Page 99 and 100:
Magellan Final Report 10.3 Hadoop E
Page 101 and 102:
Magellan Final Report 35000  3500
Page 103 and 104:
Magellan Final Report summarize som
Page 105 and 106:
Magellan Final Report Processing ti
Page 107 and 108:
Magellan Final Report in the networ
Page 109 and 110:
Magellan Final Report Workload Patt
Page 111 and 112:
Magellan Final Report This benchmar
Page 113 and 114:
Magellan Final Report Task Tracker
Page 115 and 116:
Magellan Final Report processing ti
Page 117 and 118:
Magellan Final Report Using ESnet
Page 119 and 120:
Magellan Final Report Figure 11.2:
Page 121 and 122:
Magellan Final Report data collecte
Page 123 and 124:
Magellan Final Report comparison to
Page 125 and 126:
Magellan Final Report 11.2.5 Integr
Page 127 and 128:
Magellan Final Report very large (4
Page 129 and 130:
Magellan Final Report for optimizat
Page 131 and 132:
Magellan Final Report One of the ad
Page 133 and 134:
Magellan Final Report commercial cl
Page 135 and 136:
Magellan Final Report Table 12.2: H
Page 137 and 138:
Magellan Final Report Cost per TF t
Page 139 and 140:
Magellan Final Report Productivity.
Page 141 and 142:
Magellan Final Report compute insta
Page 143 and 144:
Chapter 13 Conclusions Cloud comput
Page 145 and 146:
Magellan Final Report Inherently, t
Page 147 and 148:
Bibliography [1] G. Aldering, G. Ad
Page 149 and 150:
Magellan Final Report [30] I. Foste
Page 151 and 152:
Magellan Final Report [67] M. Palan
Page 153 and 154:
Appendix A Publications Selected Pr
Page 155 and 156:
Magellan Final Report Magellan Rese
Page 157 and 158:
Magellan Final Report Selected Mage
Page 159 and 160:
Appendix B Surveys B1
Page 161 and 162:
• Nuclear Physics - Accelarator P
Page 163 and 164:
Allow users to edit responses. What
Page 165 and 166:
Amazon Eucalyptus OpenStack Other:
Page 167 and 168:
Please list any publications/report
Page 169 and 170:
Hadoop Streaming Hadoop Native Prog
show all

Magellan Final Report - Office of Science - U.S. Department of Energy

Create successful ePaper yourself

Delete template?

Save as template?