Tender Evaluation Matrix Notes v4.0 - STFC's Computational ...

PROTECT ‐ CONTRACT
ICE‐CSE
Computational Science & Engineering Department
Science & Technology Facilities Council
Terry Hewitt 29/8/11 16:50
Style Definition: Normal: Font:Not Italic
Terry Hewitt 29/8/11 16:50
Style Definition: Heading 1: Font:Italic
Annex 5 ‐ Tender Evaluation Matrix ‐ Notes
1 Introduction
These notes are to provide additional background and clarification of the criteria specified in the tender
evaluation matrix.
The tender evaluation will be performed on the responses to the Mandatory and Desirable requirements defined
in the evaluation matrix, marked Mx and Dx.x respectively in the matrix.
All potential suppliers who comply with all the Mandatory requirements in the evaluation matrix, and who also
comply with the other mandatory criteria in the “Selection and Award Criteria” documentation, will have their
tender responses evaluated against the scoring matrix.
We have deliberately aimed to state our requirements and it is the purpose of the open tender exercise that the
supplier who best meets our requirements (as determined by the scoring mechanism) will be the most
appropriate partner in the research collaboration.
Mandatory means Mandatory: any response which does not comply with all the Mandatory requirements will
not be scored.
Points will be awarded for meeting some or all of the level of the Desirable requirements detailed. For each
requirement, a point score in the range 0‐5 will be awarded, and then multiplied by the weighting factor defined
to reach a total point score for each response received.
All compliant tender responses will be scored on the basis of the written submission.
All suppliers who provide compliant responses will be invited to present their solutions to us and respond
verbally (and subsequently in writing) to clarification questions, and the point score for each supplier will be
revised as we see appropriate based on the presentation and question/answer session.
The supplier with the highest point score following this re‐grading exercise will be awarded “preferred supplier”
status, and all other suppliers will be informed that they are not “preferred supplier”.
The specification is divided into 5 major sections:
Section Brief Description Max
Score
Normalised
(100%)
Environmental 20 8.2
ICE‐CSE BASE Provision of an HPC system and file store 35 14.3
I C E ‐ C S E T e n d e r E v a l u a t i o n M a t r i x – A d d i t i o n a l N o t e s Page 1

PROTECT ‐ CONTRACT
ICE‐CSE ADVANCE
ICE‐CSE FUTURES
Provision of an “experimental” system on the suppliers
roadmap to Exascale and forms the basis of the collaboration
Access to future products and input by STFC into the suppliers
Exascale computing developments
75 30.6
25 10.2
ICE‐CSE APPS Establishing a collaborative partnership with the supplier 90 36.7
Total 245 100.0
2 Environmental (M0, D0.1, M1, M2, D1.1, D1.2, D1.3)
This section refers to the complete environmental specification of the solution, hence applies in total to the ICE‐
CSE BASE system, the ICE‐CSE ADVANCE system and the file storage system which works with both systems.
The £6m total figure is inclusive of all taxes, most especially VAT, and covers the entirety of the tender response.
We are not asking for a breakdown of spending by system or by component.
Our power budget stated is derived from a limit we have on recurrent operating costs. We have translated our
budget figure into an average 300kW, but are asking suppliers to quote within the peak envelope of 400kW, and
are taking the risk on electricity price fluctuations ourselves.
STFC are commissioning water cooling systems in the Daresbury machine room which the supplier will be
required to use to cool both the ICE‐CSE BASE and the ICE‐CSE ADVANCE systems. The figure of 75% (or
greater) of the heat expelled by the equipment to be cooled by water accounts for heat expelled to air and chilled
by CRAC units by both compute and storage equipment.
We are very interested in high‐temperature cooling options, which allow us to reduce running costs by
minimising the requirement to operate water chillers.
We are also interested in reducing infrastructure costs by using a single chilled water loop which passes through
the computer equipment and also expels air externally, and therefore needs to be include anti‐freeze.
Suppliers should satisfy themselves that they have included all the necessary pipes, heat exchanging equipment
etc (see M2) to connect and utilize the STFC water supply.
2.1 Additional requirements
Suppliers should satisfy themselves that the equipment can be delivered to and installed in the computer hall
using the designated access route. (See M2.)
Suppliers must provide a risk assessment and method statement for delivery and installation of equipment in
advance for approval by STFC.
Responses for the ICS‐CSE BASE and ICS‐CSE ADVANCE systems may differ.
2.2 Hardware Support
The Supplier shall specify the proposed hardware support arrangements for the initial 3 years including:
• Nominated providers of support for all aspects of the hardware
• Contact details for the nominated providers
• Hours of support throughout a calendar year
• Responsibility and capability for fault diagnosis
I C E ‐ C S E T e n d e r E v a l u a t i o n M a t r i x – A d d i t i o n a l N o t e s Page 2

PROTECT ‐ CONTRACT
• Response times for hardware diagnosis and replacement
• Responsibility for delivery, fitting and despatch of parts
• Policy on EOL hardware disposal
• Escalation procedure
2.3 System Software Support
The Supplier shall specify the proposed system software support arrangements for the initial 3 years including:
• A definition of the scope of system software
• Nominated providers of support for all aspects of the system software
• Contact details for the above
• Hours of support throughout a calendar year
• Responsibility and capability for fault diagnosis
• Mechanism for application of fixes and upgrades
• Responsibility for installation of software upgrades
• Escalation procedure
• Tools and interfaces to allow monitoring of the hardware status, including peripherals, and system
software
3 ICE‐CSE BASE (M3, M4, D4.1, D4.2, M5, M6, D6.1, M7, D7.1, M8, D8.1, D8.2)
ICE‐CSE BASE is a “conventional” HPC system, on which we want to be able to engage industrial partners and run
our current suit of HPC applications.
3.1 Permanent File Store
Whilst the file store requirements are specified in the ICE‐CSE BASE section it is emphasised that the file store
must also support the ICE‐CSE ADVANCE system, and should be sized and configured to support both ICE‐CSE
BASE and ICE‐CSE ADVANCE systems and will be expected to meet similar performance requirements with each
system as detailed in the IO benchmark requirements.
3.1.1 Performance
The assessment of file store performance is based upon the ratio (rp)
rp
= aggregatebandwidth
totalmemory
Where
• aggregate bandwidth is in bytes/second
• total memory is in bytes
A minimum rp of 1/1000 is mandatory.
The aggregate performance will be measured using IOR, running on all nodes in both systems at the same time.
The benchmark file will be provided separately, but will run a set of tests at block sizes from 1MB to 512MB,
with transfer sizes from 64k to 1024k.
3.1.1.1 Minimal systems example
Suppose ICE‐CSE BASE has 2048 cores each with 1GB memory and 2 X 256GB large memory nodes. Total
memory = 2,560GB
I C E ‐ C S E T e n d e r E v a l u a t i o n M a t r i x – A d d i t i o n a l N o t e s Page 3

PROTECT ‐ CONTRACT
Suppose ICE‐CSE ADVANCE has 16,384 cores each with 1GB. Total Memory = 16,384 GB
Total memory = 16,384 + 2,560= 18,944 GB
Minimum combined aggregate performance should be 18,944/1000 = 19GB/S
Minimum usable storage capacity = 10 x Total Memory = 10x18,944GB = 189,440 GB which is less than the
200TiB mandatory, therefore usable storage = 200TiB.
3.1.1.2 Maximal Systems (or Mandatory Plus all desirables)
ICE‐CSE BASE: 3072*1GB + 4x 512GB = 5,120GB Memory
ICE‐CSE ADVANCE: 32,768 cores x 1GB/core = 32,768GB Memory
Total Memory: 37,888GB
Minimum combined aggregate performance should be 37,888GB /1000 = 38 GB/S
Minimum usable storage capacity = 15x Combined System Memory = 15x37,888 = 568,320GB
3.2 Power Consumption
We are keen to minimise operating costs and believe this can be accomplished if we are able to monitor and
control power usage down to the level of single parallel jobs. Tools which enable us to perform this task are
therefore desired.
4 ICE‐CSE ADVANCE System (M9, D9.1, D9.2, D9.3, D9.4, M10, M10.1, M11, D11.1, D11.2, D11.3, M12)
The ICE‐CSE ADVANCE system should represent “state of the art” technology in the supplier’s roadmap along the
path to Exascale computing. We wish to use this system to
• demonstrate extreme parallel scaling and performance over a large number of nodes,
• to optimise existing codes in order to enable them to run on future Exascale systems and
• to develop new codes where existing codes have reached the limit of their capabilities.
5 ICE‐CSE FUTURE (M13, D13.1, D13.2, D13.3)
The reason for establishing the ICE‐CSE collaboration is to give us access to new technology developments when
they become available. We recognise that we cannot pre‐pay for future hardware, and therefore do not require
that new hardware be delivered to us on site in Daresbury, but we want to define part of the collaboration as
giving us access to this sort of equipment.
We also want to be able to contribute to our partner’s product development plans, so having access to preproduction
equipment and a feedback route back into product development will both help enable this.
6 ICE‐CSE APPS (M14, D14.1, D14.2, D14,3, D14.4, D14.5, D14.6, D14.7, D14.8)
We ask that the supplier prepare a business plan, governance structure and examples of similar collaborations in
which various aspects of the collaboration might be included.
We intend that this collection of requirements is consistent with information CSED has published: that the
“collaboration” aspect of ICE‐CSE is vitally important to us and that we intend to recognise input in this area
significantly in determining our preferred supplier.
I C E ‐ C S E T e n d e r E v a l u a t i o n M a t r i x – A d d i t i o n a l N o t e s Page 4

PROTECT ‐ CONTRACT
The ICE‐CSE collaboration should be viewed as a stepping stone to a longer term and larger venture, which we
have been discussing for a number of years. We continue to seek additional funding and urge collaborators to
think of how the role of the ICE‐CSE collaboration could be expanded with additional funding.
6.1 D14.8
Additional capital and recurrent funds are most likely to come from the Department for Business, Innovation &
Skills (BIS). Typically this must be matched by investments from other partners.
Suppliers might like to consider initiatives such as Technology Innovation Centres (TICs) and their funding
models.
I C E ‐ C S E T e n d e r E v a l u a t i o n M a t r i x – A d d i t i o n a l N o t e s Page 5