Rocks+ Mellanox OFED Users Guide - neams

Rocks+ Mellanox OFED Users Guide 

5.4 Edition

Rocks+ Mellanox OFED Users Guide: 

5.4 Edition 

Published May 04 2011 

Copyright © 2011 Clustercorp, Inc.

Table of Contents 

Preface............................................................................................................................................................................v 

1. Overview ....................................................................................................................................................................1 

2. Attributes ...................................................................................................................................................................2 

3. Installing Rocks+ Mellanox OFED .........................................................................................................................3 

3.1. Installation on a New System.........................................................................................................................3 

3.2. Adding to an Existing Frontend .....................................................................................................................3 

4. Configuring InfiniBand ........................................................................................................................................5 

4.1. Setting up InfiniBand IPoIB.......................................................................................................................5 

4.2. Setting up the InfiniBand Subnet Manager ................................................................................................6 

5. Using Rocks+ Mellanox OFED................................................................................................................................7 

5.1. Examining the state of your fabric .................................................................................................................7 

5.2. Debugging ......................................................................................................................................................8 

5.3. Using MPI-Selector........................................................................................................................................8 

5.4. Running IMB (Intel MPI Benchmark) to validate your fabric.......................................................................9 

5.5. Firmware Flash.............................................................................................................................................11 

6. Care and Feeding ....................................................................................................................................................14 

6.1. Updating your kernel....................................................................................................................................14 

6.2. Configuring new appliances with OFED......................................................................................................14 

A. Copyright ................................................................................................................................................................16 

A.1. Rocks+.....................................................................................................................................................16 

A.2. Rocks®........................................................................................................................................................16 

iii

List of Tables 

1-1. Summary..................................................................................................................................................................1 

1-2. Roll Compatibility ...................................................................................................................................................1 

1-3. Platform Compatibility............................................................................................................................................1 

2-1. Roll Attributes .........................................................................................................................................................2 

iv

Preface 

The Rocks+ Mellanox OFED roll installs and configures the driver stack and MPI layer as provided in the 

MLNX_OFED_LINUX-1.5.2-2.1.0-rhel5.5.iso iso (md5sum e7596cb6ab56a043c15d27f469307013) from Mellanox 

Technologies. Support is provided for Infiniband HCA’s from Mellanox Technologies. 

Please visit the Mellanox Technologies site 1 to learn more about the ofed release and its components. 

Notes 

1. http://mellanox.com 

v

Chapter 1. Overview 

Table 1-1. Summary 

Name 

Version 5.4 

Maintained By 

Architecture 

Compatible with Rocks+ 5.4 

Rocks+ Mellanox OFED (rocks+mlnx-ofed) 

Clustercorp 

x86_64 

The Rocks+ Mellanox OFED roll has the following requirements of other rolls. Compatability with all known rolls 

is assured, and all known conflicts are listed. There is no assurance of compatiblity with third-party rolls. 

Table 1-2. Roll Compatibility 

Requires 

base 

ganglia 

hpc 

web-server 

rocks+core 

rocks+kernel 

Conflicts 

xen 

Table 1-3. Platform Compatibility 

Physical Hardware 

Rocks+ Private Cloud 

Amazon EC2 

yes 

no 

no 

1

Chapter 2. Attributes 

Table 2-1. Roll Attributes 

Name Type Default 

ofed a bool TRUE 

Notes: 

a. Default value created using rocks add appliance attr {compute, login} name value and only affects the 

compute, login appliances. 

ofed 

If TRUE then install the OFED driver stack and MPI environments. 

2

Chapter 3. Installing Rocks+ Mellanox OFED 

This roll can be installed during the frontend installation step of your cluster or you can add the roll to a running 

system. The Rocks+ Core roll contains the foundation for all of Rocks+ and must always be installed, and 

licensed, to use any of the other Rocks+ rolls. 

3.1. Installation on a New System 

The Rocks+ Mellanox OFED roll is added to a frontend installation in exactly the same manner as other rolls. 

Refer also to Section 2.2 of the Rocks® Users Guide for more information. 

Upon selecting Submit, the Rocks+ Mellanox OFED roll will be added to the selected rolls on the left. You can then 

select other rolls or start the installation as necessary. 

3.2. Adding to an Existing Frontend 

The Rocks+ Mellanox OFED roll can be added onto an already installed frontend. The following procedure, run 

3

Chapter 3. Installing Rocks+ Mellanox OFED 

as root, will allow you to add the roll. 

# rocks add roll rocks+mlnx-ofed-5.4-*.x86_64.disk1.iso 

# rocks enable roll rocks+mlnx-ofed 

# cd /export/rocks/install 

# rocks create distro 

# rocks run roll rocks+mlnx-ofed | sh 

Reboot your frontend to complete the configuration: 

# reboot 

Lastly, reinstall your compute nodes: 

# rocks run host compute /boot/kickstart/cluster-kickstart 

4

Chapter 4. Configuring InfiniBand 

4.1. Setting up InfiniBand IPoIB 

Follow these instructions for setting up IP over IB (IPoIB). IPoIB is optional, as none of the MPI environments 

provided by the OFED Roll depend on IPoIB. 

Please note that these steps need to be performed before the compute nodes are added to the cluster if you 

want the IP addresses to be assigned automatically. It is always possible to assign IP addresses manually at any 

time. 

4.1.1. Setting up InfiniBand on Compute nodes 

Perform the following steps to have IP addresses allocated to your InfiniBand interfaces (ib0) at compute node 

discovery time. You will need to have functioning InfiniBand fabric with an active subnet manager as the compute 

nodes will attempt to bring up the ib0 interface on first boot. 

rocks add network ipoib rocks set network mtu ipoib 65520 

for example: 

# rocks add network ipoib 172.30.0.0 255.255.0.0 

# rocks set network mtu ipoib 65520 

The IPoIB device defaults to using ib0. If you wish to change this, perform following steps to set the device. 

rocks add appliance attr compute ipoibdevice 

for example: 

# rocks add appliance attr compute ipoibdevice ib1 

4.1.2. Setting up InfiniBand on Headnode 

If you have an InfiniBand hca installed in your headnode, you can setup IPoIB on your headnode by running 

following command (please choose an IP address within the the ipoib network added above, preferrably at the lower 

end of the address range) 

# rocks add host interface localhost ib0 ip=172.30.1.10 subnet=ipoib name=‘hostname -s‘-ib 

5

Chapter 4. Configuring InfiniBand 

bring up the ib0 interface on your headnode 

# rocks sync host network localhost 

# ifup ib0 

4.1.3. Discovering Compute nodes 

At this point you can add your compute nodes as normal 

# insert-ethers 

Choose compute node appliance type and turn on your compute nodes in order you want them assigned in Rocks. 

4.2. Setting up the InfiniBand Subnet Manager 

An InfiniBand network requires a Subnet Manager to be running in either the Infiniband switch itself (switch 

based) or on one of the nodes which is connected to the Infiniband fabric (host based). If the output of ibstat shows 

the port state as "Initializing" then you most likely will need to run a host based subnet manager. 

At this time the subnetmanager attribute should only be set to true for one host. IE: the headnode, or one of your 

compute nodes. 

In this example, Rocks attribute subnetmanager is set to true for node compute-0-0. When compute-0-0 is installed 

next time, the opensmd service (Subnet Manager) will be turned on. 

# rocks add host attr compute-0-0 subnetmanager true 

# ssh compute-0-0 /boot/kickstart/cluster-kickstart 

In this example, Rocks attribute subnetmanager is set to true for the head node. You will need to reboot the head 

node for the setting to take effect. 

# rocks add host attr localhost subnetmanager true 

# reboot 

6

Chapter 5. Using Rocks+ Mellanox OFED 

The Rocks+ Mellanox OFED roll installs Infiniband kernel drivers, libraries, and MPI environments (mvapich, 

OpenMPI) during the provisioning process. A series of debug files are provided containing output from the 

installation. This can be a helpful tool in diagnosing HCA issues. 

5.1. Examining the state of your fabric 

The mlnx-ofed Roll contains a number of tools that can be used to debug issues with the HCAs and Infiniband fabric. 

The ibstat tool will query basic status of Infiniband device(s). If any of the ports show State: Active the hca 

can pass MPI traffic (see Running Intel MPI Benchmark). If ports show State: Initializing, see Setting up the 

InfiniBand Subnet Manager. 

If you are using iWARP 10 Gigabit Ethernet adapters, you may ignore the output of ibstat. As long as the iWARP 

interface (typically eth2) is up and has an IP address it is ready to pass MPI traffic (see Running Intel MPI 

Benchmark). 

# ibstat 

CA ’mlx4_0’ 

CA type: MT26428 

Number of ports: 2 

Firmware version: 2.8.600 

Hardware version: a0 

Node GUID: 0x00237dffff93ef78 

System image GUID: 0x00237dffff93ef7b 

Port 1: 

State: Active 

Physical state: LinkUp 

Rate: 40 

Base lid: 4 

LMC: 0 

SM lid: 1 

Capability mask: 0x02510868 

Port GUID: 0x00237dffff93ef79 

Port 2: 

State: Down 

Physical state: Polling 

Rate: 10 

Base lid: 0 

LMC: 0 

SM lid: 0 

Capability mask: 0x02510868 

Port GUID: 0x00237dffff93ef7a 

The mlnx-ofed roll provides an Infiniband HCA Self Test Utility hca_self_test.ofed which provides firmware 

version, host driver version/state, link status, and other useful information. 

# hca_self_test.ofed 

---- Performing InfiniBand HCA Self Test ---- 

7


Number of HCAs Detected ................ 1 

PCI Device Check ....................... PASS 

Kernel Arch ............................ x86_64 

Host Driver Version .................... OFED-internal-1.5.2-20101219-1546: 1.5.2-2.6.18_194.el5 

Host Driver RPM Check .................. PASS 

HCA Firmware on HCA #0 ................. v2.8.600 

HCA Firmware Check for HCA #0 .......... PASS 

Host Driver Initialization ............. PASS 

Number of HCA Ports Active ............. 1 

Port State of Port #0 on HCA #0 ........ UP 4X QDR 

Port State of Port #1 on HCA #0 ........ DOWN 

Error Counter Check on HCA #0 .......... PASS 

Kernel Syslog Check .................... PASS 

Node GUID on HCA #0 .................... 00:23:7d:ff:ff:93:6e:32 

------------------ DONE --------------------- 

5.2. Debugging 

The files are located at /root/ofed-debug.stage*.out. 

ofed-debug.stage1.out: debug output of any packages which were removed prior to OFED install. 

ofed-debug.stage2.out: debug output of OFED package install and driver reload. 

ofed-debug.stage3.out: debug output of kernel driver rebuild (if required). 

ofed-debug.stage4mft.out: debug output of automatic firmware update. 

ofed-debug.stage5.out: debug output of subnet manager (if applicable). 

5.3. Using MPI-Selector 

The mlnx-ofed Roll allows users to easily switch between default MPI implementations. Here are a few simple 

commands that users can execute: Display current default MPI version: 

$ mpi-selector --query 

openmpi_gcc-1.4.2 

level:user 

Display available versions: 

$ mpi-selector --list 

mvapich_gcc-1.2.0 

mvapich_intel-1.2.0 

mvapich_pgi-1.2.0 

openmpi_gcc-1.4.2 

openmpi_intel-1.4.2 

openmpi_pgi-1.4.2 

8


Switch to another version: 

$ mpi-selector --set openmpi_intel-1.4.2 

Defaults already exist; overwrite them (y/N) y 


default:openmpi_intel-1.4.2 

level:user 

A menu driven version of mpi-selector is also available: 

$ mpi-selector-menu 

Current system default: none 

Current user default: none 

"u" and "s" modifiers can be added to numeric and "U" 

commands to specify "user" or "system-wide". 

1. mvapich_gcc-1.2.0 

2. mvapich_intel-1.2.0 

3. mvapich_pgi-1.2.0 

4. openmpi_gcc-1.4.2 

5. openmpi_intel-1.4.2 

6. openmpi_pgi-1.4.2 

U. Unset default 

Q. Quit 

Selection (1-9[us], U[us], Q): 

More information about using the mpi-selector tool can be found in the man pages (man mpi-selector). Detailed 

information on adding and removing your own MPI implemenations is also referenced in the man page. 

5.4. Running IMB (Intel MPI Benchmark) to validate your 

fabric 

IMB is a good all around test to access Infiniband connectivity and performance. It is pre-built for each of the 

bundled MPI environments in the mlnx-ofed roll (mvapich, mvapich2, openmpi). 

Here is an example IMB run (fill in the names of your compute nodes when creating the hosts file). Note that only the 

PingPong test is run. Remove PingPong from the command line to run the full suite of tests. 

# useradd imb 

# rocks sync users 

# su - imb 

$ mpi-selector --set=openmpi_gcc-1.4.2 

$ logout 

# su - imb 


default:openmpi_gcc-1.4.2 

level:user 

$ cat > hosts


> compute-0-0 

> compute-0-1 

> EOF 

$ mpirun -np ‘wc -l < hosts‘ -machinefile hosts /usr/mpi/gcc/openmpi-1.4.2/tests/IMB-3.2/IMB-MPI1 PingPong 

#--------------------------------------------------- 

# Intel (R) MPI Benchmark Suite V3.2, MPI-1 part 

#--------------------------------------------------- 

# Date : Fri Feb 19 22:49:01 2010 

# Machine : x86_64 

# System : Linux 

# Release : 2.6.18-164.el5 

# Version : #1 SMP Tue Aug 18 15:51:48 EDT 2009 

# MPI Version : 2.1 

# MPI Thread Environment: MPI_THREAD_SINGLE 

# New default behavior from Version 3.2 on: 

# the number of iterations per message size is cut down 

# dynamically when a certain run time (per message size sample) 

# is expected to be exceeded. Time limit is defined by variable 

# "SECS_PER_SAMPLE" (=> IMB_settings.h) 

# or through the flag => -time 

# Calling sequence was: 

# /usr/mpi/gcc/openmpi-1.4.2/tests/IMB-3.2/IMB-MPI1 PingPong 

# Minimum message length in bytes: 0 

# Maximum message length in bytes: 4194304 

# 

# MPI_Datatype : MPI_BYTE 

# MPI_Datatype for reductions : MPI_FLOAT 

# MPI_Op : MPI_SUM 

# 

# 

# List of Benchmarks to run: 

# PingPong 

#--------------------------------------------------- 

# Benchmarking PingPong 

# #processes = 2 

#--------------------------------------------------- 

#bytes #repetitions t[usec] Mbytes/sec 

0 1000 1.58 0.00 

1 1000 1.72 0.56 

2 1000 1.73 1.10 

4 1000 1.66 2.30 

10


8 1000 1.69 4.53 

16 1000 1.76 8.67 

32 1000 1.90 16.03 

64 1000 2.05 29.77 

128 1000 3.52 34.68 

256 1000 3.87 63.15 

512 1000 4.12 118.46 

1024 1000 4.83 202.38 

2048 1000 6.11 319.84 

4096 1000 7.48 522.30 

8192 1000 10.00 781.18 

16384 1000 13.13 1189.80 

32768 1000 19.32 1617.74 

65536 640 29.64 2108.64 

131072 320 55.38 2257.02 

262144 160 99.97 2500.85 

524288 80 187.50 2666.67 

1048576 40 332.99 3003.12 

2097152 20 665.78 3004.00 

4194304 10 1299.94 3077.06 

# All processes entering MPI_Finalize 

5.5. Firmware Flash 

The mlnx-ofed roll installs Infiniband drivers, MVAPICH, Open MPI and includes an auto firmware flashing routine 

during the node kickstart process. A debug file is provided containing output from the auto-installation and update 

process. This can be a helpful tool in diagnosing HCA issues. The file is located at 

/root/ofed-debug-stage4mft.out. 

The following is an example of information included in the debug file: 

The auto firmware flashing routine prints out information at the end of the debug file. Here is an example of a 

successful firmware update: 

version 0.92 

210 ini files registered 

probing devices... 

### 

## 

# 

# hca: mt26428_pci_cr0 

# 

## 

### 

11


standard firmware for PSID[MT_0D90110009] is None 

cx2 firmware for PSID[MT_0D90110009] is 2.8.600 

probed fw version 2.8.600 

probed hardware version 0xb0 

hwversion is 0xb0 and ConnectX2 firmware exists 

firmware up-to-date 

If the process fails to update in a failsafe mode, useful information may be provided. Here is an example of an error 

message related to an invariant sector found on an HCA that will not allow a failsafe update: 

probing devices 

discovered dev: mt25204_pci_cr0 

standard firmware for PSID[MT_03B0120002] is 1.2.0 

probed fw version 1.2.917 

need to update firmware 

mlxburn -dev /dev/mst/mt25204_pci_cr0 -fw fw-25204-rel-1_2_000/fw-25204-rel.mlx -conf fw-25204-rel-1 

-I- Generating image ... 

Current FW version on flash: 1.2.917 

New FW version: 1.2.0 

Note: The new FW version is not newer than the current FW version on flash. 

Do you want to continue (y/n) [n] : y 

Read and verify Invariant Sector 

Invariant sector mismatch. Address 0x40 

- DIFF DETECTED 

in image: 0x15000720, while on flash: 0x14000720 

The invariant sector can not be burnt in a failsafe manner. 

You can perform the FW update without burning the invariant sector by 

by specifying the -skip_is flag. 

See FW release notes for details on invariant sector updates. 

*** ERROR *** Failsafe burn error: Invariant sector mismatch 

-E- Image burn failed: child process exited abnormally 

The firmware update failed due to an Invariant Sector error. 

------------------------------------------------------------ 

This could be due to Cisco firmware loaded on the hca or perhaps old Mellanox firmware 

If you would like to update the firmware, you may use following command. 

This command will update the firmware in NON FAILSAFE mode. 

If the system loses power during firmware update it could render the hca inoperable. 

Proceed with the firmware update at your own risk. 

cd /opt/mlnx-ofed/firmware 

mlxburn -dev /dev/mst/mt25204_pci_cr0 -fw fw-25204-rel-1_2_000/fw-25204-rel.mlx -conf fw-25204-rel-1 

12


In the case above, you have the option of updating the firmware using the provided commands. The instructions 

provided in the above output are only guidelines on updating software images for HCAs. Please consult with your 

hardware supplier support team for more information. Additional information about firmware flashing can be found 

on the Mellanox Technologies Firmware Support and Downloads Site 1 

Notes 

1. http://www.mellanox.com/support/firmware_download.php 

13

Chapter 6. Care and Feeding 

6.1. Updating your kernel 

Be careful to make sure the kernel is compatible with your ofed version. 

6.1.1. Updating the kernel on your compute nodes 

To update the kernel on your compute nodes, you can simply copy the kernel rpm’s to 

/export/rocks/install/contrib/5.4/x86_64/RPMS and rebuild your distribution followed by reinstalling 

your compute nodes. 

For example: 

# ls -1 

kernel-2.6.18-194.32.1.el5.src.rpm 

kernel-2.6.18-194.32.1.el5.x86_64.rpm 

kernel-devel-2.6.18-194.32.1.el5.x86_64.rpm 

kernel-doc-2.6.18-194.32.1.el5.noarch.rpm 

kernel-headers-2.6.18-194.32.1.el5.x86_64.rpm 

# cp -p kernel*.rpm /export/rocks/install/contrib/5.4/x86_64/RPMS 

# cd /export/rocks/install 

# rocks create distro 

# tentakel /boot/kickstart/cluster-kickstart 

6.1.2. Updating the kernel on your head node 

You may perform a yum update to install a new kernel on your headnode. Take care not to update any packages 

which are provided by the mlnx-ofed roll (list may be found in /opt/mlnx-ofed/RPMS/x86_64). 

When the head node is rebooted the ofed kernel drivers will be built for the new kernel and be ready for use. See 

/root/ofed-debug.stage3.out after your head node has rebooted to see the output of ofed kernel driver rebuild. 

If you want to update the kernel on the headnode to the same as the compute nodes from previous example, perform 

following steps. 

# cd /export/rocks/install/contrib/5.4/x86_64/RPMS 

# rpm -Uvh kernel*.rpm 

# reboot 

14

6.2. Configuring new appliances with OFED 

Chapter 6. Care and Feeding 

The Rocks+ standard appliance configuration when using the mlnx-ofed roll enables the ofed attribute for the 

compute appliance (see below). When this attribute is set, the compute nodes are provisioned with the OFED stack to 

provide Infiniband support. 

# rocks list appliance attr 

APPLIANCE ATTR VALUE 

frontend: managed false 

compute: managed true 

compute: ofed true 

nas: managed true 

network: managed false 

power: managed false 

ipmi: managed false 

The ofed attribute can be enabled for any appliance you wish. For example, if you wish to provision your nas 

appliances with ofed then you can add the ofed attribute for the nas appliance. The same technique can be used if you 

define your own compute appliances. 

# rocks add appliance attr nas ofed true 

# rocks list appliance attr 

APPLIANCE ATTR VALUE 

frontend: managed false 

compute: managed true 

compute: ofed true 

nas: managed true 

nas: ofed true 

network: managed false 

power: managed false 

ipmi: managed false 

At this point you can simply reinstall the nodes which are of the appropriate appliance type. 

15

Appendix A. Copyright 

A.1. Rocks+ 

Copyright© 2011 Clustercorp Inc. All rights reserved. 

This product includes software developed by Clustercorp Inc., these portions may not be modified, copied, or 

redistributed without the express written consent of Clustercorp Inc. 

A.2. Rocks® 

This product includes software developed by the Rocks® Cluster Group at the San Diego Supercomputer Center at 

the University of California, San Diego and its contributors. This software is subject to the following additional 

copyright; 

Rocks(r) 

www.rocksclusters.org 

version 5.4 (Maverick) 

Copyright (c) 2000 - 2010 The Regents of the University of California. 

All rights reserved. 

Redistribution and use in source and binary forms, with or without 

modification, are permitted provided that the following conditions are 

met: 

1. Redistributions of source code must retain the above copyright 

notice, this list of conditions and the following disclaimer. 

2. Redistributions in binary form must reproduce the above copyright 

notice unmodified and in its entirety, this list of conditions and the 

following disclaimer in the documentation and/or other materials provided 

with the distribution. 

3. All advertising and press materials, printed or electronic, mentioning 

features or use of this software must display the following acknowledgement: 

"This product includes software developed by the Rocks(r) 

Cluster Group at the San Diego Supercomputer Center at the 

University of California, San Diego and its contributors." 

4. Except as permitted for the purposes of acknowledgment in paragraph 3, 

neither the name or logo of this software nor the names of its 

authors may be used to endorse or promote products derived from this 

software without specific prior written permission. The name of the 

software includes the following terms, and any derivatives thereof: 

"Rocks", "Rocks Clusters", and "Avalanche Installer". For licensing of 

the associated name, interested parties should contact Technology 

16

Transfer & Intellectual Property Services, University of California, 

San Diego, 9500 Gilman Drive, Mail Code 0910, La Jolla, CA 92093-0910, 

Ph: (858) 534-5815, FAX: (858) 534-7345, E-MAIL:invent@ucsd.edu 

THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS “AS IS 

AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, 

THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 

PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS 

BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 

CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 

SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR 

BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, 

WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE 

OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN 

IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 

Appendix A. Copyright 

17

Rocks+ Mellanox OFED Users Guide - neams

Create successful ePaper yourself

Delete template?

Save as template?