Magellan Final Report - Office of Science - U.S. Department of Energy

More documents

Recommendations

Info

Magellan Final Report 11.4.2 Customizable Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 11.4.3 On-Demand Bare-Metal Provisioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 11.4.4 User Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 11.4.5 Workflow Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 11.4.6 Data Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 11.4.7 Performance, Reliability and Portability . . . . . . . . . . . . . . . . . . . . . . . . . . 116 11.4.8 Federation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 11.4.9 MapReduce/Hadoop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 12 Cost Analysis 118 12.1 Related Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 12.2 Cost Analysis Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 12.2.1 Assumptions and Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 12.2.2 Computed Hourly Cost of an HPC System . . . . . . . . . . . . . . . . . . . . . . . . 119 12.2.3 Cost of a DOE Center in the Cloud . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 12.2.4 Application Workload Cost and HPL Analysis . . . . . . . . . . . . . . . . . . . . . . 122 12.3 Other Cost Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 12.4 Historical Trends in Pricing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 12.5 Cases where Private and Commercial Clouds may be Cost Effective . . . . . . . . . . . . . . . 125 12.6 Late Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 12.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 13 Conclusions 129 A Publications B Surveys A1 B1 iv
Chapter 1 Overview Cloud computing has served the needs of enterprise web applications for the last few years. The term “cloud computing” has been used to refer to a number of different concepts (e.g., MapReduce, public clouds, private clouds, etc.), technologies (e.g., virtualization, Apache Hadoop), and service models (e.g., Infrastructureas-a-Service [IaaS], Platform-as-a-Service [PaaS], Software-as-a-Service[SaaS]). Clouds have been shown to provide a number of key benefits including cost savings, rapid elasticity, ease of use, and reliability. Cloud computing has been particularly successful with customers lacking significant IT infrastructure or customers who have quickly outgrown their existing capacity. The open-ended nature of scientific exploration and the increasing role of computing in performing science has resulted in a growing need for computing resources. There has been an increasing interest over the last few years in evaluating the use of cloud computing to address these demands. In addition, there are a number of key features of cloud environments that are attractive to some scientific applications. For example, a number of scientific applications have specific software requirements including OS version dependencies, compilers and libraries, and the users require the flexibility associated with custom software environments that virtualized environments can provide. An example of this is the Supernova Factory, which relies on large data volumes for the supernova search and has a code base which consists of a large number of custom modules. The complexity of the pipeline necessitates having specific library and OS versions. Virtualized environments also promise to provide a portable container that will enable scientists to share an environment with collaborators. For example, the ATLAS experiment, a particle physics experiment at the Large Hadron Collider at CERN, is investigating the use of virtual machine images for distribution of all required software [10]. Similarly, the MapReduce model holds promise for data-intensive applications. Thus, cloud computing models promise to be an avenue to address new categories of scientific applications, including data-intensive science applications, on-demand/surge computing, and applications that require customized software environments. A number of groups in the scientific community have investigated and tracked how the cloud software and business model might impact the services offered to the scientific community. However, there is a limited understanding of how to operate and use clouds, how to port scientific workflows, and how to determine the cost/benefit trade-offs of clouds, etc. for scientific applications. The Magellan project was funded by the American Recovery and Reinvestment Act to investigate the applicability of cloud computing for the Department of Energy’s Office of Science (DOE SC) applications. Magellan is a joint project at the Argonne Leadership Computing Facility (ALCF) and the National Energy Research Scientific Computing Center (NERSC). Over the last two years we have evaluated various dimensions of clouds—cloud models such as Infrastructure as a Service (IaaS) and Platform as a Service(PaaS), virtual software stacks, MapReduce and its open source implementation (Hadoop). We evaluated these on various criteria including stability, manageability, and security from a resource provider perspective, and performance and usability from an end-user perspective. Cloud computing has similarities with other distributed computing models such as grid and utility computing. However, the use of virtualization technology, the MapReduce programming model, and tools such 1
Page 1 and 2: The Magellan Report on Cloud Comput
Page 3 and 4: Executive Summary The goal of Magel
Page 5 and 6: Key Findings The goal of the Magell
Page 7 and 8: Magellan Final Report Finding 8. DO
Page 9 and 10: Magellan Final Report role in addre
Page 11 and 12: Contents Executive Summary Key Find
Page 13: Magellan Final Report 9.7 Discussio
Page 17 and 18: Magellan Final Report • The Argon
Page 19 and 20: Chapter 2 Background The term “cl
Page 21 and 22: Magellan Final Report 2.1.4 Hardwar
Page 23 and 24: Magellan Final Report Table 3.1: Ke
Page 25 and 26: Magellan Final Report Little Magell
Page 27 and 28: Magellan Final Report 3.2 Advanced
Page 29 and 30: Chapter 4 Application Characteristi
Page 31 and 32: Magellan Final Report Table 4.1: Pe
Page 33 and 34: Magellan Final Report Output data
Page 35 and 36: Magellan Final Report of the pipeli
Page 37 and 38: Chapter 5 Magellan Testbed As part
Page 39 and 40: Magellan Final Report Figure 5.1: P
Page 41 and 42: Magellan Final Report Figure 5.2: P
Page 43 and 44: Magellan Final Report NERSC deploye
Page 45 and 46: Magellan Final Report Figure 6.1: A
Page 47 and 48: Magellan Final Report greater than
Page 49 and 50: Magellan Final Report specific QoS
Page 51 and 52: Magellan Final Report configuration
Page 53 and 54: Magellan Final Report 7.4 Summary U
Page 55 and 56: Magellan Final Report Firewalls are
Page 57 and 58: Magellan Final Report Aside from le
Page 59 and 60: Magellan Final Report 9.1 Understan
Page 61 and 62: Magellan Final Report grid) on 256
Page 63 and 64: Magellan Final Report Table 9.1: HP
Page 65 and 66:
Magellan Final Report 25  Ping 
Page 67 and 68:
Magellan Final Report 100  12 
Page 69 and 70:
Magellan Final Report case of GTC,
Page 71 and 72:
Magellan Final Report 1.4 IB TCPo
Page 73 and 74:
Magellan Final Report only affects
Page 75 and 76:
Magellan Final Report Figure 9.11:
Page 77 and 78:
Magellan Final Report charted as a
Page 79 and 80:
Magellan Final Report Evaluation Cr
Page 81 and 82:
Magellan Final Report Write Perform
Page 83 and 84:
Magellan Final Report 3500 3000 G
Page 85 and 86:
Magellan Final Report Histogram Plo
Page 87 and 88:
Magellan Final Report SATA devices.
Page 89 and 90:
Magellan Final Report MB/s Virident
Page 91 and 92:
Magellan Final Report and the perfo
Page 93 and 94:
Magellan Final Report (a) Hosts (b)
Page 95 and 96:
Magellan Final Report Routing IP pa
Page 97 and 98:
Chapter 10 MapReduce Programming Mo
Page 99 and 100:
Magellan Final Report 10.3 Hadoop E
Page 101 and 102:
Magellan Final Report 35000  3500
Page 103 and 104:
Magellan Final Report summarize som
Page 105 and 106:
Magellan Final Report Processing ti
Page 107 and 108:
Magellan Final Report in the networ
Page 109 and 110:
Magellan Final Report Workload Patt
Page 111 and 112:
Magellan Final Report This benchmar
Page 113 and 114:
Magellan Final Report Task Tracker
Page 115 and 116:
Magellan Final Report processing ti
Page 117 and 118:
Magellan Final Report Using ESnet
Page 119 and 120:
Magellan Final Report Figure 11.2:
Page 121 and 122:
Magellan Final Report data collecte
Page 123 and 124:
Magellan Final Report comparison to
Page 125 and 126:
Magellan Final Report 11.2.5 Integr
Page 127 and 128:
Magellan Final Report very large (4
Page 129 and 130:
Magellan Final Report for optimizat
Page 131 and 132:
Magellan Final Report One of the ad
Page 133 and 134:
Magellan Final Report commercial cl
Page 135 and 136:
Magellan Final Report Table 12.2: H
Page 137 and 138:
Magellan Final Report Cost per TF t
Page 139 and 140:
Magellan Final Report Productivity.
Page 141 and 142:
Magellan Final Report compute insta
Page 143 and 144:
Chapter 13 Conclusions Cloud comput
Page 145 and 146:
Magellan Final Report Inherently, t
Page 147 and 148:
Bibliography [1] G. Aldering, G. Ad
Page 149 and 150:
Magellan Final Report [30] I. Foste
Page 151 and 152:
Magellan Final Report [67] M. Palan
Page 153 and 154:
Appendix A Publications Selected Pr
Page 155 and 156:
Magellan Final Report Magellan Rese
Page 157 and 158:
Magellan Final Report Selected Mage
Page 159 and 160:
Appendix B Surveys B1
Page 161 and 162:
• Nuclear Physics - Accelarator P
Page 163 and 164:
Allow users to edit responses. What
Page 165 and 166:
Amazon Eucalyptus OpenStack Other:
Page 167 and 168:
Please list any publications/report
Page 169 and 170:
Hadoop Streaming Hadoop Native Prog
show all

Magellan Final Report - Office of Science - U.S. Department of Energy

Create successful ePaper yourself

Delete template?

Save as template?