D2 3 Computing e-Infrastructure cost calculations and business _models_vam1-final

e-­‐FISCAL: www.efiscal.eu
EC Contract Number: 283449
In an effort to get more inputs from more countries and centres during the second period of the project two new
questionnaires were prepared. The first one was a user-­‐friendly spreadsheet version based on the initial one and
the second was a high-­‐level one based on ranges (mores suitable for centres with confidentiality issues). The
questionnaires were further disseminated and the project received four new questionnaires; two of which were
new (from new sites and new countries) and two were updates from the previous ones.
Finally, the excel version of the questionnaire acted as basis for the on-­‐line web-­‐based tool.
3.6 Benchmarking exercise
As shown in the last step of our methodology, we provided for the execution of a benchmarking exercise. The
objective of the benchmarking effort (also described in the D2.2) has been to carry out a performance comparison
of different systems with identical specifications, while focusing on the HPC, HTC and Cloud infrastructures. The
output of the benchmarking exercise would be used in the comparison between the prices of cloud providers and
the costs derived from the e-­‐FISCAL costing model in an effort to compare “like with like”. However, with the
advancements in the computer architecture and infrastructures, it became increasingly difficult to compare
performance of various computer systems by merely looking at their specifications. Therefore, domain-­‐specific
benchmarking tests (such as NAS Parallel Benchmark and HEPSEPC06) are required and were adopted to compare
the systems across HPC, HTC and Cloud infrastructures. In terms of scope, the benchmarking effort focused on the
deployment and testing of the equivalent instances, while comparing the HPC and HTC infrastructures against the
commercial Cloud offerings (e.g. Amazon EC2). Amazon EC2 is chosen for performance comparisons as it is one of
the market leaders and public benchmarking figures are available for the EC2 to compare with. Moreover, it is
important to note here that additional performance tests (e.g. MPPTest, STREAM) can be carried out to
individually measure various performance metrics (e.g. network performance, memory bandwidth, MPI
communication). But, in the context of the e-­‐FISCAL methodology, the aggregated performance factor is sufficient
enough to yield the performance adjusted price figures for various infrastructures. The analysis is presented in
Appendix 7.7.
The benchmarking exercise also had important secondary goals. First of all, setting up the benchmarking
environments within both in-­‐house and Cloud infrastructures gave important tacit information about the practical
migration effort that would be necessary in order to move applications from the in-­‐house infrastructures to the
Cloud and vice versa. Despite the fact that portability and rapid deployment are the key design goals of
benchmarking suites, the effort needed for setting up and maintaining the environments was considerable. This
points out towards a need for further studies with real applications when estimating the feasibility of various
hybrid e-­‐Infrastructure scenarios. Furthermore, the benchmarking study served an important role in building a
common understanding of the e-­‐Infrastructure. Discussing the benchmarking methodologies and their limitations
with financial experts had an important role in enabling efficient trans-­‐disciplinary collaboration; in fact this was
equally important as discussing the strengths and limitations of the different cost assessment methodologies with
e-­‐Infrastructure experts.
e-­‐FISCAL : Financial Study for Sustainable Computing e-­‐Infrastructures
Deliverable D2.3 – Computing e-­‐Infrastructures cost estimation and analysis – Pricing and
Business models
41