===============================================================================
Table of Contents
===============================================================================
-1. General
+1. Overview
2. MVAPICH
3. Open MPI
4. MVAPICH2
===============================================================================
-1. General
+1. Overview
===============================================================================
Three MPI stacks are included in this release of OFED:
-
- MVAPICH 0.9.9
- Open MPI 1.2.2-1
- MVAPICH2 0.9.8p2
are located under: <prefix>/mpi/<compiler>/<mpi stack>/tests/,
where <prefix> is /usr by default.
-1.3 Selecting which MPI to use: mpi-selector
+1.3 Selecting Which MPI to Use: mpi-selector
--------------------------------------------
Depending on how the OFED installer was run, multiple different MPI
implementations may be installed on your system. The OFED installer
which it was run. As such, it is recommended to run the
mpi-selector-menu command on all hosts in a cluster, picking the same
default MPI implementation on each. It may be more convenient,
-however, to use the the mpi-selector command in script-based scenarios
+however, to use the mpi-selector command in script-based scenarios
(such as running on every host in a cluster); mpi-selector effects all
the same functionality as mpi-selector-menu, but is intended for
automated environments. See the mpi-selector(1) manual page for more
===============================================================================
MVAPICH2 is an MPI-2 implementation which includes all MPI-1 features.
-It is based on MPICH2 and MVICH. MVAPICH2 0.9.8 provides many features
+It is based on MPICH2 and MVICH. MVAPICH2 0.9.8 provides many features
including fault-tolerance with checkpoint-restart, RDMA_CM support,
iWARP support, optimized collectives, on-demand connection management,
multi-core optimized and scalable shared memory support, and memory hook
- Default DAPL Provider [default "ib0"]
For non-interactive builds where no MVAPICH2 build options are stored in
-OFED configuration file, the default settings are:
+the OFED configuration file, the default settings are:
Implementation: OFA
ROMIO Support: Y
It is also possible to use the mpirun command in place of mpiexec. They are
actually the same command in MVAPICH2, however using mpiexec is preferred.
-It is possible to run more processes than hosts. In that case, multiple
+It is possible to run more processes than hosts. In this case, multiple
processes will run on some or all of the hosts used. The following examples
demonstrate how to run the MPI tests. The default installation prefix and
gcc version of MVAPICH2 are shown. In each case, it is assumed that a hosts
1. Overview
2. Contents of the OFED Distribution
- 3. HW and SW Requirements
- 4. How to download and extract the OFED
+ 3. Hardware and Software Requirements
+ 4. How to Download and Extract the OFED Distribution
5. Installing OFED Software
6. Building OFED RPMs
7. IPoIB Configuration
This is the OpenFabrics Enterprise Distribution (OFED) version 1.2
software package supporting InfiniBand and iWARP fabrics. It is composed
of several software modules intended for use on a computer cluster
-constructed as an InfiniBand subnet or iWARP network.
+constructed as an InfiniBand subnet or an iWARP network.
This document describes how to install the various modules and test them in
a Linux environment.
The OFED Distribution package generates RPMs for installing the following:
- o OpenFabrics core and ULPs:
+ o OpenFabrics core and ULPs
- HCA drivers (mthca, ipath, ehca)
- iWARP driver (cxgb3)
- core
- Upper Layer Protocols: IPoIB, SDP, SRP Initiator, iSER Initiator
RDS, VNIC and uDAPL
- o OpenFabrics utilities:
+ o OpenFabrics utilities
- OpenSM: InfiniBand Subnet Manager
- Diagnostic tools
- Performance tests
- o MPI:
+ o MPI
- OSU MVAPICH stack supporting the InfiniBand and iWARP interface
- Open MPI stack supporting the InfiniBand and iWARP interface
- OSU MVAPICH2 stack supporting the InfiniBand and iWARP interface
- MPI benchmark tests (OSU BW/LAT, Intel MPI Benchmark, Presta)
- o Extra packages:
+ o Extra packages
- open-iscsi: open-iscsi initiator with iSER support
- ib-bonding: Bonding driver for IPoIB interface
o Sources of all software modules (under conditions mentioned in the
o Documentation
==============================================================================
-3. HW and SW Requirements
+3. Hardware and Software Requirements
==============================================================================
1) Server platform with InfiniBand HCA or iWARP RNIC (see OFED Distribution
Release Notes for details)
-2) Linux OS (see OFED Distribution Release Notes for details)
+2) Linux operating system (see OFED Distribution Release Notes for details)
3) Administrator privileges on your machine(s)
- For Installation only: 200MB
5) For the OFED Distribution to compile on your machine, some software
- packages of your OS distribution are required. These are listed here.
+ packages of your operating system (OS) distribution are required. These
+ are listed here.
OS Distribution Required Packages
--------------- ----------------------------------
o SLES 9.0 kernel-source, udev, rpm
o SLES 10.0 kernel-source, rpm
-Note: To build 32-bit libraries on x86_64 and ppc64 glibc-devel 32bit
- should be installed
+Note: To build 32-bit libraries on x86_64 and ppc64 platforms, the 32-bit
+ glibc-devel should be installed.
Specific Component Requirements:
o OSU MPI a Fortran Compiler (such as gcc-g77)
o ibutils tcl-8.4, tcl-devel-8.4, tk
o tvflash pciutils-devel
- The installer will warn you if you attempt to compile any of the
- above packages and do not have the prerequisites installed.
+Note: The installer will warn you if you attempt to compile any of the
+ above packages and do not have the prerequisites installed.
==============================================================================
-4. How to download and extract the OFED Distribution
+4. How to Download and Extract the OFED Distribution
==============================================================================
1) Download the OFED-X.X.X.tgz file to your target Linux host.
cd /..../OFED-X.X.X
2) Installing the OFED package must be done as root. For a
- menu-driven, first build and installation, run the installer
+ menu-driven first build and installation, run the installer
script:
./install.sh
Interactive menus will direct you through the install process.
Note: After the installer completes, information about the OFED
- installation such as the prefix, kernel version, and
+ installation such as the prefix, the kernel version, and
installation parameters can be found by running
/etc/infiniband/info.
During the interactive installation of OFED, two files are
- generated: ofed.conf and ofed_net.conf. ofed.conf holds the
- installed software modules and configuration settings chosen by the
- user. ofed_net.conf holds the IPoIB settings chosen by the user.
+ generated: ofed.conf and ofed_net.conf.
+ ofed.conf holds the installed software modules and configuration settings
+ chosen by the user. ofed_net.conf holds the IPoIB settings chosen by the
+ user.
If the package is installed on a cluster-shared directory, these
files can then be used to perform an automatic, unattended
Thus it is possible to change user choices (observing the original format).
See examples of ofed.conf and ofed_net.conf under OFED-X.X.X/docs.
-b. To make debugging kernel issues easier, we recommend setting the following
- environment variable before building OFED:
+b. To make debugging kernel issues easier, it is recommended to set the
+ following environment variable before building OFED:
export OPENIB_KERNEL_EXTRA_CFLAGS=-g
This will attach debug info to generated kernel binaries without affecting
their speed.
c. Important note for open-iscsi users:
- Installing iSER as part of OFED installation will also install open-iscsi.
- Before installing OFED, please uninstall any open-iscsi version that may
- be installed on your machine. Installing OFED with iSER support while
- another open-iscsi version is already installed will cause the installation
- process to fail.
+ Installing iSER as part of the OFED installation will also install
+ open-iscsi. Before installing OFED, please uninstall any open-iscsi version
+ that may be installed on your machine. Installing OFED with iSER support
+ while another open-iscsi version is already installed will cause the
+ installation process to fail.
Install Process Results:
/lib/modules/`uname -r`/updates/kernel/net/rds/rds.ko
o The package kernel include files are placed under <prefix>/src/openib.
These includes should be used when building kernel modules which use
- the Openfabrics stack. (Note that these includes, if needed, have
- been "backported" to your kernel).
-o The package raw (unbackported) source files are placed under
+ the Openfabrics stack. (Note that these includes, if needed, are
+ "backported" to your kernel).
+o The raw package (un-backported) source files are placed under
<prefix>/src/ofa_kernel-1.2
o The script "openibd" is installed under /etc/init.d/. This script can
be used to load and unload the software stack.
o The directory /etc/infiniband is created with the files "info" and
"openib.conf". The "info" script can be used to retrieve OFED
installation information. The "openib.conf" file contains the list of
- modules that will be loaded when the "openibd" script is used.
-o The file "90-ib.rules" is installed to /etc/udev/rules.d/
+ modules that are loaded when the "openibd" script is used.
+o The file "90-ib.rules" is installed under /etc/udev/rules.d/
o man pages are installed under /usr/share/man
o If libibverbs-utils is installed, then ofed.sh and ofed.csh are
installed under /etc/profile.d/. These automatically update the PATH
environment variable with <prefix>/bin. In addition, ofed.conf is
- installed under /etc/ld.so.conf.d/ to update dynamic linker's
+ installed under /etc/ld.so.conf.d/ to update the dynamic linker's
run-time search path to find the InfiniBand shared libraries.
o The file /etc/modprobe.conf is updated to include the following:
- "alias ib<n> ib_ipoib" for each ib<n> interface.
- "alias net-pf-27 ib_sdp" for sdp.
o If opensm is installed, the daemon opensmd is installed under /etc/init.d/
and opensm.conf is installed under /etc.
-o If IPoIB configuration files were included, ifcfg-ib<n> files will be
+o If IPoIB configuration files are included, ifcfg-ib<n> files will be
installed at:
- RedHat: /etc/sysconfig/network-scripts/
- SuSE: /etc/sysconfig/network/
-o If iSER was included, open-iscsi userspace files will be also installed:
+o If iSER is included, open-iscsi user-space files will be also installed:
- Configuration files will be installed at /etc/iscsi
- Startup script will be installed at:
- RedHat: /etc/init.d/iscsi
- SuSE: /etc/init.d/open-iscsi
- Other tools (iscsiadm, iscsid, iscsi_discovery, iscsi-iname, iscsistart)
- will be installed at /sbin.
- - Documentation will be installed at:
+ will be installed under /sbin.
+ - Documentation will be installed under:
- RedHat: /usr/share/doc/iscsi-initiator-utils-<version number>
- SuSE: /usr/share/doc/packages/open-iscsi
-o man pages will be installed at /usr/share/man/.
+o man pages will be installed under /usr/share/man/.
==============================================================================
6. Building OFED RPMs
minutes. Installation, however, is a relatively short process
(~5 minutes). A common strategy for OFED installation on large
homogeneous clusters is to extract the tarball on a network
- file system (such as NFS), do the build on NFS, and then run
+ file system (such as NFS), build OFED on NFS, and then run the
installer on each node with the RPMs that were previously built.
==============================================================================
==============================================================================
Configuring IPoIB is an optional step during the installation. During
-interactive installation, the user may choose to insert the ifcfg-ib<n>
+an interactive installation, the user may choose to insert the ifcfg-ib<n>
files. If this option is chosen, the ifcfg-ib<n> files will be
-installed at:
+installed under:
- RedHat: /etc/sysconfig/network-scripts/
- SuSE: /etc/sysconfig/network/
For unattended installations, a configuration file can be provided
with this information. The configuration file must specify the
following information:
-- Fixed values for each IPoIB interface.
+- Fixed values for each IPoIB interface
- Base IPoIB configuration on Ethernet configuration (may be useful for
- cluster configuration).
+ cluster configuration)
Here are some examples of ofed_net.conf:
o Documentation
Notes:
-1. cxgb3 driver is in technology preview state.
+1. The cxgb3 driver is in technology preview state.
2. The Virtual NIC (VNIC) driver is presented as a technology preview on OFED 1.2.
3. All other OFED components are of production quality.
4. See release notes for each package in the docs directory.
1. Intel MPI, Version 3.0 - Package ID: l_mpi_p_3.0.043
2. HP MPI, Version 2.2.5
-1.6 OFED Sources:
------------------
+1.6 OFED Sources
+----------------
Source repositories:
http://www.openfabrics.org/git/
Kernel sources: ~vlad/ofed_1_2/.git
Note: For details regarding the various changes, please see the release notes
for each package in the docs directory.
- 2.1 General changes:
+ 2.1 General changes
o Kernel code based on 2.6.20
o New kernel modules: SA Cache, RDS, VNIC, bonding
o High availability of SRP and IPoIB in GA level
o Added iWARP support (with Chelsio driver)
o MAN pages for libraries (libibverbs and librdmacm)
- 2.1 IPoIB:
+ 2.1 IPoIB
o IPoIB Connected Mode
o High availability support using the bonding module.
- 2.2 SDP:
+ 2.2 SDP
o netstat is now available
- o Improved message BW:
+ o Improved message BW
- 10X for small messages
- 5X for medium messages
- o Scalability:
+ o Scalability
- Added a memory consumption limit
- 2.3 SRP:
+ 2.3 SRP
o High availability is now supported for all systems.
- 2.4 iSER:
+ 2.4 iSER
o Testing more platforms (e.g., ppc64 and ia64)
o Updated packages for ISCSI kernel & user components bundled with OFED.
o Scalability features needed for Intel MPI
2.6 Libraries
- a. libibverbs 1.1:
+ a. libibverbs 1.1
o Fork support (requires apps change)
o Better low-level driver handling, including multiple drivers linked
in statically
o Documentation: man pages
- b. librdmacm (uCMA) 1.0:
+ b. librdmacm (uCMA) 1.0
o Multicast joining from user space
o UD support
o Documentation: man pages
- 2.7 OSM:
+ 2.7 OSM
o Routing improvements
o Performance improvement to min hop and up/down of over an order of magnitude
o New fat-tree and LASH algorithms
o IB router enablement
o SA database dump/restore
- 2.8 Management:
+ 2.8 Management
o Many diagnostic improvements since OFED 1.1 (see detailed RN)
o ibdiagui: A GUI for ibdiagnet
- 2.9 Install:
+ 2.9 Install
o Default prefix directory is now /usr
2.6 MPI:
- a. OSU MVAPICH:
+ a. OSU MVAPICH
o Version was updated to 0.9.9
- b. Open MPI:
+ b. Open MPI
o Version was updated to 1.2.1
o See http://www.open-mpi.org/svn/new.php for details
- c. OSU MVAPICH2:
+ c. OSU MVAPICH2
o MVAPICH2 version 0.98 was added to the OFED package.
- d. Common MPI setup sourcing:
+ d. Common MPI setup sourcing
Simple menu-driven interface to choose which MPI implementation to set as
the default on a per-user and/or system-wide basis
2.7 iWARP Support
o Chelsio NIC supported
- o Verbs and CMA APIs are the same as Infiniband
- o ULPs that are supported:
+ o Verbs and CMA APIs are the same as InfiniBand
+ o ULPs supported
- MPI (mvapich2 tested)
- uDAPL
- o Basic Testing:
+ o Basic Testing
- uDAPL
- mvapich2
- NFS-RDMA
1. OFED installation now supports installing lib32 on 64-bit systems.
2. Hotplug removal does not hang the system when the device is used by
the uverbs interface.
-3. MVAPICH does work on ppc64.
+3. MVAPICH now works on ppc64.
4. libibcm is now thread safe.
-Bugs fixed in each package are reported in the packages release notes.
+Bugs fixed in each package are reported in the package's release notes.
+
===============================================================================
4. Known Issues
stop the old OFED version before uninstalling it.
Workaround: Stop the old OFED stack (/etc/init.d/openibd stop) before
upgrade to OFED 1.2.
-2. Memory registration by user is limited according to the administrator
+1. Memory registration by theuser is limited according to the administrator
setting. See "Pinning (Locking) User Memory Pages" in OFED_tips.txt for
system configuration.
-3. Fork support from kernel 2.6.12 and above is available provided
+2. Fork support from kernel 2.6.12 and above is available provided
that applications do not use threads. The fork() is supported as long
- as parent process does not run before child exits or calls exec().
- The former can be achieved by calling wait(childpid) the later can be
- achieved by application specific means. Posix system() call is
+ as the parent process does not run before the child exits or calls exec().
+ The former can be achieved by calling wait(childpid), and the latter can be
+ achieved by application specific means. The Posix system() call is
supported.
-4. ipath driver is supported only on 64 bit platforms.
-5. There are issues using Intel MPI with Qlogic card driver that cause failures
+3. The ipath driver is supported only on 64-bit platforms.
+4. There are issues using Intel's MPI with the Qlogic card driver that cause
+ failures.
Note: See the release notes of each component for additional issues.
1. OFED Utilities
2. Debug HOWTOs
3. Pinning (Locking) User Memory Pages
-4. External module compilation over OFED-1.2
+4. External Module Compilation Over OFED-1.2
===============================================================================
===============================================================================
The OFED package includes utilities under <prefix>/bin, where <prefix> stands
-for the OFED installation path. To retrieve the this path, run the script
-"/etc/infiniband/info" as explained in 2.2 below.
+for the OFED installation path. To retrieve this path, run the script
+"/etc/infiniband/info" as explained in Section 2.2 below.
Notes:
------
2. Debug HOWTOs
===============================================================================
-2.1 OFED Components and version information
+2.1 OFED Components and Version Information
-------------------------------------------
-The text file BUILD_ID provides data on all OFED components (whether
-installed or not).
-This file is a part of the ofed-docs RPM and installed under
-/usr/share/doc/ofed-docs-1.2 on RedHat and under
-/usr/share/doc/packages/ofed-docs-1.2 on SuSE
+The text file BUILD_ID provides data on all OFED components (whether installed
+or not). This file is a part of the ofed-docs RPM and installed under
+/usr/share/doc/ofed-docs-1.2 on RedHat, and under
+/usr/share/doc/packages/ofed-docs-1.2 on SuSE.
-The same imformation can be obtained by executing 'ofed_info' command.
-For example:
+The same imformation can be obtained by executing the 'ofed_info' command. For
+example:
> ofed_info
OFED-1.2-20070429-0610
2.2 Installed OFED Components
-------------------------------
The script /etc/infiniband/info provides data on the specific OFED installation
-on this machine.
+on the machine.
For example:
/lib/modules/2.6.9-22.ELsmp/build --with-libibcm --with-libibverbs
--with-libipathverbs --with-libmthca --with-mstflint --with-perftest
-2.3 Building/Installing IB Modules with debug information
----------------------------------------------------------
+2.3 Building/Installing InfiniBand (IB) Modules With Debug Information
+----------------------------------------------------------------------
To compile/build/install the IB modules so that they will contain debug
information, set OPENIB_KERNEL_EXTRA_CFLAGS="-g" in your environment
-before running OFED install.sh/build.sh .
+before running OFED's install.sh/build.sh .
===============================================================================
it is allowed to pin.
The user can change the system per-process memory lock limit by adding
-the following two lines to file /etc/security/limits.conf:
+the following two lines to the file /etc/security/limits.conf:
* soft memlock <number>
* hard memlock <number>
Note: The file /etc/security/limits.conf contains further documentation.
+
===============================================================================
-4. External module compilation over OFED-1.2
+4. External Module Compilation Over OFED-1.2
===============================================================================
-In order to build kernel modules depending on OFED's modules take
-Modules.symvers file from <prefix>/src/openib/Module.symvers (part of
-kernel-ib-devel RPM) and copy this to modules subdir and then compile
-your module.
+To build kernel modules depending on OFED's modules, take the Modules.symvers
+file from <prefix>/src/openib/Module.symvers (part of the kernel-ib-devel RPM)
+and copy it to the modules subdir and then compile your module.
-If <prefix>/src/openib/Module.symvers does not exist or empty use
-create_Module.symvers.sh (a part of the ofed-docs RPM) script to create
+If <prefix>/src/openib/Module.symvers does not exist or it is empty, use the
+create_Module.symvers.sh (a part of the ofed-docs RPM) script to create the
Module.symvers file.
See "Module versioning & Module.symvers" in the modules.txt from kernel
Table of Contents
===============================================================================
1. Overview
-2. Notes on Testing Methodology
+2. Notes on Testing Method
3. Test Descriptions
4. Running Tests
===============================================================================
This is a collection of tests written over uverbs intended for use as a
performance micro-benchmark. As an example, the tests can be used for
-HW or SW tuning and/or functional testing.
+hardware or software tuning and/or functional testing.
-Please post results/observations to the openib-general mailing list.
+Please post results and observations to the openib-general mailing list.
See "Contact Us" at http://openib.org/mailman/listinfo/openib-general and
http://www.openib.org.
===============================================================================
-2. Notes on Testing Methodology
+2. Notes on Testing Method
===============================================================================
-- The benchmark used the CPU cycle counter to get time stamps without context
- switch. Some CPU architectures (e.g., Intel's 80486 or older PPC) do NOT
- have such capability.
+- The benchmark uses the CPU cycle counter to get time stamps without a context
+ switch. Some CPU architectures (e.g., Intel's 80486 or older PPC) do NOT have
+ such capability.
- The benchmark measures round-trip time but reports half of that as one-way
latency. This means that it may not be sufficiently accurate for asymmetrical
configurations.
-- Min/Median/Max result is reported.
- The median (vs average) is less sensitive to extreme scores.
- Typically, the "Max" value is the first value measured.
+- Min/Median/Max results are reported.
+ The Median (vs average) is less sensitive to extreme scores.
+ Typically, the Max value is the first value measured.
-- Larger samples help marginally only. The default (1000) is pretty good.
- Note that an array of cycles_t (typically unsigned long) is allocated
+- Larger samples only help marginally. The default (1000) is very satisfactory.
+ Note that an array of cycles_t (typically an unsigned long) is allocated
once to collect samples and again to store the difference between them.
Really big sample sizes (e.g., 1 million) might expose other problems
with the program.
Architectures tested: i686, x86_64, ia64
-
===============================================================================
-4. Test Descriptions
+3. Test Descriptions
===============================================================================
rdma_lat.c latency test with RDMA write transactions
-rdma_bw.c streaming BW test with RDMA write transactions
+rdma_bw.c streaming bandwidth test with RDMA write transactions
-The following tests are mainly useful for HW/SW benchmarking.
+The following tests are mainly useful for hardware/software benchmarking.
They are not intended as actual usage examples.
send_lat.c latency test with send transactions
-send_bw.c BW test with send transactions
+send_bw.c bandwidth test with send transactions
write_lat.c latency test with RDMA write transactions
-write_bw.c BW test with RDMA write transactions
+write_bw.c bandwidth test with RDMA write transactions
read_lat.c latency test with RDMA read transactions
-read_bw.c BW test with RDMA read transactions
+read_bw.c bandwidth test with RDMA read transactions
+
+The executable name of each test starts with the general prefix "ib_";
+for example, ib_write_lat.
-The executable name of each test starts with the general prefix "ib_",
-e.g., ib_write_lat.
-Running Tests
--------------
+===============================================================================
+4. Running Tests
+===============================================================================
Prerequisites:
kernel 2.6
+++ /dev/null
- Open Fabrics Enterprise Distribution (OFED)
- IBUTILS in OFED 1.2 Release Notes
-
- June 2007
-
-
-===============================================================================
-Table of Contents
-===============================================================================
-1. Overview
-2. Requirements
-3. Reports
-4. Known Issues
-
-
-===============================================================================
-1. Overview
-===============================================================================
-
-The IBUTILS package provides diagnostic tools and procedures for InfiniBand
-fabrics. Target users of these utilities are network and data-centers
-managers with basic knowledge of the InfiniBand specification.
-
-The following tools are provided:
-o ibdiagnet - performs a diagnostic check of the entire InfiniBand subnet.
- Should be used on any suspicion of fabric misbehavior.
- The default invocation can be enhanced to perform more advanced checks and
-produce more reports. The following is a partial set of checks it performs:
- - Check for a single master SM
- - Check all routes between hosts are set correctly (include also multicast
- groups)
- - Check for fabric links health
-
-o ibdiagpath - traces a path between two nodes specified by LIDs or a directed
- path. This utility should be used when connectivity between specific two
- hosts is broken.
-
-o ibdiagui - a graphic user interface on top of ibdiagnet
- Mostly suitable for medium size fabrics (<100 nodes) and for users who needs
- to explore an unknown InfiniBand subnet. The main features it provides is an
- automatically generated connectivity graph, an object properties browser and
- hyperlinking of ibdiagnet log to these widgets.
-
-Note: man pages are provided for each tools.
-
-The package tools performs the following diagnostic procedures:
-* Discover the InfiniBand subnet connectivity
-* Determine whether or not a Subnet Manager (SM) is running
-* Identify links which drop packets and/or incur errors by sending MAD
- packets multiple times, across all the links, reporting port monitor counters
-* Identify fabric level mismatches or inconsistencies such as:
- - Duplicate port GUIDs - Two or more different ports with the same GUID
- - Duplicate node GUIDs - Two or more different nodes with the same node GUID
- - Duplicate LIDs - Two or more devices that have the same assigned LID
- - Zero valued LIDs - A device with LID=0 indicates that the SM did not
- assign a LID to this device.
- - Zero valued system GUIDs - A device with system GUID=0 indicates that
- the vendor did not assign it a GUID.
- - An InfiniBand link is in the INIT state, which prevents data transfer
- - Unexpected link width (when using the -lw flag)
- - Unexpected link speed (when using the -ls flag)
- - Check for partitions and SL2VL settings preventing communication between
- specific nodes (ibdiagpath)
-
-===============================================================================
-2. Requirements
-===============================================================================
-
-Software dependency:
-
-1. ibis and ibdm must be installed.
-
-2. ibdiagui also depends on installation of the following packages:
- * Tk8.4 is standard on all Linux distributions, but if not please
- download and install from: http://www.tcl.tk/software/tcltk/
- * Graphviz - an automatic graph layout utility.
- Can be downloaded and installed from: http://www.graphviz.org/
-
-===============================================================================
-3. Reports
-===============================================================================
-The default directory for all generated report files is /tmp .
-
-Both utilities collect summary information regarding all the fabric SM's
-during the run, and then output that information at end of the run in file
-/tmp/ibdiagnet.sm.
-
-Each report message includes:
- - Device Type
- - Device portGUID
- - The direct path to the device
- - If a topology file is provided to be matched with the discovered fabric,
- the node name is also provided in the report message. Otherwise, host
- names are included only in HCA-related report messages.
-
-===============================================================================
-4. Known Issues
-===============================================================================
- ibdiagpath issues:
- - If no subnet manager is initialized in the subnet, FDB tables may be
- incorrectly set. Consequently, PortCounter MADs cannot be sent.
-
- - A link along a LID-routed path in INIT state causes ibdiagpath performance
- queries to fail. The performance queries fail since they cannot proceed via
- non-ACTIVE links.
-
- - ibdiagpath cannot validate the provided topology file against the existing
- fabric topology. If the topology file includes a device/link that does not
- exist, or the device/link information is incorrect, then ibdiagpath may
- -- in name-based routing -- extract a non-existing path based on the
- incorrect topology file.
-
- - If the hostname provided for the -s flag is not the actual local hostname,
- then all the extracted names from the topology file will be incorrect.
- However, all the other information provided will be correct.
-
- - Executing "ibis exit" in order to terminate ibis, while running over RHEL5
- i686, causes ibis to exit uncleanly.
-
-
--- /dev/null
+ Open Fabrics Enterprise Distribution (OFED)
+ IBUTILS in OFED 1.2 Release Notes
+
+ June 2007
+
+
+===============================================================================
+Table of Contents
+===============================================================================
+1. Overview
+2. Requirements
+3. Reports
+4. Known Issues
+
+
+===============================================================================
+1. Overview
+===============================================================================
+
+The IBUTILS package provides diagnostic tools and procedures for InfiniBand
+fabrics. Target users of these utilities are network and data-center managers
+with basic knowledge of the InfiniBand specification.
+
+The following tools are provided:
+o ibdiagnet - performs a diagnostic check of the entire InfiniBand subnet.
+ This utility should be used upon suspicion of fabric misbehavior. The default
+ invocation can be enhanced to perform more advanced checks and to produce
+ additional reports. The following is a partial set of checks it performs:
+ - Checks for a single master Subnet Manager (SM)
+ - Checks that all routes between hosts are set correctly (including multicast
+ groups)
+ - Checks for fabric links health
+
+o ibdiagpath - traces a path between two nodes specified by LIDs or a directed
+ path. This utility should be used when the connectivity between two specific
+ hosts is broken.
+
+o ibdiagui - a graphic user interface on top of ibdiagnet
+ ibdiagui is mostly suitable for medium sized fabrics (<100 nodes) and for
+ users who wish to explore an unknown InfiniBand subnet. The main features it
+ provides are an automatically generated connectivity graph, an object-
+ properties browser, and hyperlinks of the ibdiagnet log to these widgets.
+
+Note: man pages are provided for each tool.
+
+The package tools perform the following diagnostic procedures:
+* Discover the InfiniBand subnet connectivity
+* Determine whether or not an SM is running
+* Identify links which drop packets and/or incur errors by sending MAD
+ packets multiple times, across all the links, reporting port monitor counters
+* Identify fabric level mismatches or inconsistencies such as
+ - Duplicate port GUIDs - Two or more different ports with the same GUID
+ - Duplicate node GUIDs - Two or more different nodes with the same node GUID
+ - Duplicate LIDs - Two or more devices that have the same assigned LID
+ - Zero valued LIDs - A device with LID=0 indicates that the SM did not
+ assign a LID to this device
+ - Zero valued system GUIDs - A device with system GUID=0 indicates that
+ the vendor did not assign it a GUID
+ - An InfiniBand link is in the INIT state which prevents data transfer
+ - Unexpected link width (when using the -lw flag)
+ - Unexpected link speed (when using the -ls flag)
+ - Partitions and SL2VL settings preventing communication between specific
+ nodes (ibdiagpath)
+
+
+===============================================================================
+2. Requirements
+===============================================================================
+
+Software Dependencies:
+
+1. ibis and ibdm must be installed (are part of this OFED package).
+
+2. ibdiagui also depends on the installation of the following packages:
+ - Tk8.4 is standard on all Linux distributions. If it is missing on your
+ machine, download it from http://www.tcl.tk/software/tcltk/
+ - Graphviz - an automatic graph layout utility. It can be downloaded from
+ http://www.graphviz.org/
+
+
+===============================================================================
+3. Reports
+===============================================================================
+The default directory for all generated report files is /tmp.
+
+The ibadiagnet and ibdiagpath utilities collect summary information regarding
+all the fabric SMs during the run, and then output that information at the end
+of the run in the file /tmp/ibdiagnet.sm.
+
+Each report message includes the following items:
+ - Device Type
+ - Device portGUID
+ - The direct path to the device
+ - If a topology file is provided for matching with the discovered InfiniBand
+ fabric, the node name is also provided in the report message. Otherwise,
+ hostnames are included only in HCA-related report messages.
+
+
+===============================================================================
+4. Known Issues
+===============================================================================
+ ibdiagpath issues:
+ - If no subnet manager is initialized in the subnet, FDB tables may be
+ incorrectly set. Consequently, PortCounter MADs cannot be sent.
+
+ - A link along a LID-routed path in INIT state causes ibdiagpath performance
+ queries to fail since the queries cannot proceed via non-ACTIVE links.
+
+ - ibdiagpath cannot validate the provided topology file against the existing
+ fabric topology. If the topology file includes a device/link that does not
+ exist, or the device/link information is incorrect, then ibdiagpath may
+ -- in name-based routing -- extract a non-existing path based on the
+ incorrect topology file.
+
+ - If the hostname provided for the -s flag is not the actual local hostname,
+ then all the extracted names from the topology file will be incorrect.
+ Nevertheless, all the other information provided will be correct.
+
+ - If running on RHEL5, then executing "ibis exit" to terminate ibis yields
+ an unclean exit.
===============================================================================
1. Overview
===============================================================================
-IPoIB is a network driver implementation that enables transmission of IP and
-ARP protocol packets over an InfiniBand UD channel. The implementation conforms
-to the relevant IETF working group's RFCs (http://www.ietf.org).
+IPoIB is a network driver implementation that enables transmitting IP and ARP
+protocol packets over an InfiniBand UD channel. The implementation conforms to
+the relevant IETF working group's RFCs (http://www.ietf.org).
===============================================================================
2. New Features
===============================================================================
+IPoIB now supports "connected mode" (RFC 4755). IPOIB CM is enabled by default
+on hardware that supports the SRQ optional feature (mthca, ipath).
+The maximum MTU for connected mode has been increased to 65520. By default, MTU
+will be configured to this maximum value.
-IPoIB now supports connected mode (RFC 4755). IPOIB CM is enabled by default
-on hardware that supports SRQ optional feature (mthca, ipath).
-Max MTU for connected mode is increased to 65520.
-By default, MTU will be configured to this maximum value.
+1. IPoIB will accept incoming connected mode connections unless connected mode
+ is disabled at compile time.
+2. IPoIB will use connected mode for all outgoing traffic to unicast destina-
+ tions that support connected mode if and only if connected mode is enabled
+ at run time.
+3. For destinations that do not support connected mode, IPoIB will fall back on
+ datagram mode.
+4. For multicast traffic, IPoIB always uses datagram mode.
-1. IPoIB will accept incoming connected mode connections
- unless disabled at compile time
-2. IPoIB will use connected mode for all outgoing traffic
- to unicast destinations that support connected mode,
- if and only if connected mode is enabled at run time.
-3. For destinations that do not support connected mode, IPoIB
- will fall back on datagram mode
-4. For multicast traffic, IPoIB always uses datagram mode
Usage and configuration:
-1. To check current mode used for outgoing connections:
+1. To check the current mode used for outgoing connections, enter:
cat /sys/class/net/ib0/mode
-2. To disable IPoIB CM at compile time:
+2. To disable IPoIB CM at compile time, enter:
cd OFED-1.2
-export OFA_KERNEL_PARAMS="--without-ipoib-cm"
+ export OFA_KERNEL_PARAMS="--without-ipoib-cm"
./install.sh
-3. To change run time configuration for IPoIB:
+3. To change the run-time configuration for IPoIB, enter:
edit /etc/infiniband/openib.conf, change the following parameters:
# Enable IPoIB Connected Mode
SET_IPOIB_CM=yes
# Set IPoIB MTU
IPOIB_MTU=65520
-4. You can also change mode and MTU for a specific interface manually:
- To enable connected mode for interface ib0:
+4. You can also change the mode and MTU for a specific interface manually.
+
+ To enable connected mode for interface ib0, enter:
echo connected > /sys/class/net/ib0/mode
- To increase MTU:
+
+ To increase MTU, enter:
ifconfig ib0 mtu 65520
+
===============================================================================
3. Known Issues
===============================================================================
-1. A host that has multiple interfaces, each belonging to a different IP subnet,
- yet they use the same InfiniBand Partition and are connected to the same IB
- Switch, violates the IP rule requiring different broadcast domains and hence
- may build an incorrect ARP table.
-
- Thus the correct setting of multi-homed IPoIB host would be to use a different
- PKEY for each IP subnet. When a host has multiple interfaces on the same IP
- subnet, to avoid the case of a peer building an incorrect ARP entry (neighbor),
- set the net.ipv4.conf.X.arp_ignore value to 1 or 2, where X stands for the IPoIB
- (non child) interfaces (eg ib0, ib1, etc), this would cause the network stack to
- send ARP replies only on the interface with the IP address specified in the ARP
- request:
+1. If a host has multiple interfaces and (a) each interface belongs to a
+ different IP subnet, (b) they all use the same InfiniBand Partition, and (c)
+ they are connected to the same IB Switch, then the host violates the IP rule
+ requiring different broadcast domains. Consequently, the host may build an
+ incorrect ARP table.
+
+ The correct setting of a multi-homed IPoIB host is achieved by using a
+ different PKEY for each IP subnet. If a host has multiple interfaces on the
+ same IP subnet, then to prevent a peer from building an incorrect ARP entry
+ (neighbor) set the net.ipv4.conf.X.arp_ignore value to 1 or 2, where X
+ stands for the IPoIB (non-child) interfaces (e.g., ib0, ib1, etc). This
+ causes the network stack to send ARP replies only on the interface with the
+ IP address specified in the ARP request:
sysctl -w net.ipv4.conf.ib0.arp_ignore=1
sysctl -w net.ipv4.conf.ib1.arp_ignore=1
- or globally,
+ Or, globally,
sysctl -w net.ipv4.conf.all.arp_ignore=1
- To learn more on the arp_ignore param, see Documentation/networking/ip-sysctl.txt
- Note that distributions have the means to make kernel params persistent.
+ To learn more about the arp_ignore parameter, see Documentation/networking/ip-sysctl.txt.
+ Note that distributions have the means to make kernel parameters persistent.
2. On SuSE 10 and SLES 10:
a. There are IPoIB alias lines in modprobe.conf which prevent stopping/
These alias lines cause the drivers to be loaded again by udev scripts.
Workaround: Change modprobe.conf to set
- OPENIB_PARAMS="--without-modprobe" before running install.sh,
+ OFA_KERNEL_PARAMS="--without-modprobe" before running install.sh,
or remove the alias lines from modprobe.conf.
b. The ib1 interface uses the configuration script of ib0.
Workaround: Invoke ifup/ifdown using both the interface name and the
- configuration script name (example: ifup ib1 ib1).
+ configuration script name (example: ifup ib1 ib1).
-3. After a hotplug event, IPoIB interface falls back on datagram mode,
+3. After a hotplug event, the IPoIB interface falls back to datagram mode, and
MTU is reduced to 2K.
Workarounds:
- - Set up IPoIB HA service as documented below
- - Re-enable connected mode and increase the mtu manually:
- echo connected > /sys/class/net/ib0/mode
- ifconfig ib0 mtu 65520
+ a. Set up the IPoIB HA service as documented below
+ b. Re-enable connected mode and increase MTU manually:
+ echo connected > /sys/class/net/ib0/mode
+ ifconfig ib0 mtu 65520
-4. Since the IPoIB configuration files (ifcfg-ib<n>) are installed at the
+4. Since the IPoIB configuration files (ifcfg-ib<n>) are installed under the
standard networking scripts location (RedHat:/etc/sysconfig/network-scripts/
and SuSE: /etc/sysconfig/network/), the option IPOIB_LOAD=no in openib.conf
does not prevent the loading of IPoIB on boot.
-5. On RedHat EL 4 up4, IPOIB implementation is not spec compliant:
+5. On RedHat EL 4 up4, the IPOIB implementation is not spec-compliant:
- ipoib multicast does not work
- - ipoib can not inter-operate between RHEL4U4 and other hosts
- This is due to missing code in the kernel which was available in u3 and u5 and removed
- in u4. As a work around, upgrade to RHEL4U5.
-
-6. IPoIB connected mode, if enabled, uses large MTU for connected mode messages
- and small MTU for datagram (in particular, multicast) messages, and relies on
- path mtu discovery to adjust MTU appropriately. Packets sent in the window
- before MTU discovery automatically reduces the MTU for a specific destination
- will be dropped, producing the following message in system log:
+ - ipoib cannot interoperate between RHEL4U4 and other hosts. This is due to
+ missing code in the kernel which was available in U3 and U5 but removed in
+ U4. As a workaround, upgrade to RHEL4U5.
+
+6. If IPoIB connected mode is enabled, it uses a large MTU for connected mode
+ messages and a small MTU for datagram (in particular, multicast) messages,
+ and relies on path MTU discovery to adjust MTU appropriately. Packets sent
+ in the window before MTU discovery automatically reduces the MTU for a
+ specific destination will be dropped, producing the following message in the
+ system log:
"packet len <actual length> (> <max allowed length>) too long to send, dropping"
- To warn about this, a message is produced in system log each time
- MTU is set to a value higher than 2K.
+ To warn about this, a message is produced in the system log each time MTU is
+ set to a value higher than 2K.
7. In connected mode, TCP latency for short messages is larger by approx. 1usec
- (~5%) than in datagram mode. As a work-around, use datagram mode.
+ (~5%) than in datagram mode. As a workaround, use datagram mode.
8. Single-socket TCP bandwidth for kernels < 2.6.18 is lower than with
newer kernels. We recommend kernels from 2.6.18 and up for
----------
1. It may be required to run over different UDP ports than the well known ports
(67 and 68). Free port numbers greater than 0x8000 must be chosen. To
- specify a server or client port number, use the option -p <port number>.
+ specify a server or
a client port number, use the option -p <port number>.
The client's port number must be the chosen server's port number plus one.
-2. For IPoIB to use DHCP, it is required to patch ISC's DHCP. The patch file can
- be found under OFED-1.2/docs/dhcp after extracting the distribution file
- (after installation it can also be found under <prefix>/docs/dhcp). The patch
- should be applied for the server and for each client. Tests were run on
- version 3.0.4 of the DHCP package.
+2. For IPoIB to use DHCP, you must patch ISC's DHCP. The patch file can be
+ found under OFED-1.2/docs/dhcp after extracting the distribution file.
+ (After installation it can also be found under <prefix>/docs/dhcp.) The
+ patch should be applied for the server and for each client. Tests were run
+ on version 3.0.4 of the DHCP package.
===============================================================================
5. High Availability (HA) Service
===============================================================================
High Availability (HA) service for IPoIB interfaces is provided via the
-ipoibtools package. Ipoibtools currently includes a perl script, ipoib_ha.pl,
-and two executables, arpingib and mcasthandle.
+ipoibtools package. The package currently includes a perl script, ipoib_ha.pl,
+and two executables: arpingib and mcasthandle.
-The HA service operates as follows: A user-level daemon runs in background to
-detect failure of the primary IPoIB interface. If such a failure is detected
-(e.g., port down), the daemon configures the secondary IPoIB interface with the
-configuration parameters of the primary IPoIB interface (so that the secondary
-interface assumes the IP identity of the primary interface).
+The HA service operates as follows: a user-level daemon runs in background to
+detect failure of the primary IPoIB interface. If such a failure is detected,
+(e.g., a port is down), the daemon configures the secondary IPoIB interface
+with the configuration parameters of the primary IPoIB interface. Thus, the
+secondary interface assumes the IP identity of the primary interface.
Enabling the HA Service
-----------------------
-p primary IPoIB interface (default: ib0)
-s secondary IPoIB interface (default: ib1)
- --with-arping use modified arping utility to send an unsolicited
+ --with-arping use a modified arping utility to send an unsolicited
ARP REPLY
--with-multicast support applications that are using multicast
-v verbose output
6. The ib-bonding driver
===============================================================================
The ib-bonding driver is a High Availability solution for IPoIB interfaces.
-It is based on the Linux Ethernet Bonding Driver and was adopted to work with
-IPoIB. ib-bonding package contains a bonding driver and a utility called
+It is based on the Linux Ethernet Bonding Driver and was adapted to work with
+IPoIB. The ib-bonding package contains a bonding driver and a utility called
ib-bond to manage and control the driver operation.
The ib-bonding driver comes with the ib-bonding package (run rpm -qi ib-bonding
to get the package information).
Using the ib-bonding driver
---------------------------
-The ib-bonding driver can be loaded manually or automatically
+The ib-bonding driver can be loaded manually or automatically.
1. Manual operation:
-Use the utility ib-bond to stop/query/stop the driver. To get help about this
+Use the utility ib-bond to stop, query, or stop the driver. For details on this
utility, read the documentation that comes with the ib-bonding package.
2. Automatic operation:
-Edit file '/etc/infiniband/openib.conf' as follows:
+Edit the file '/etc/infiniband/openib.conf' as follows:
# Enable the bonding driver on startup.
IPOIBBOND_ENABLE=yes
# # Set bond interface names
bond1_SLAVES=ib2,ib3,ib4
Notes:
-* The ib-bonding driver doesn't load when the HA service is configured to load.
-* If bondX name is defined but one of bondX_SLAVES or bondX_IP is missing, then that
- specific bond will not be created.
+* The ib-bonding driver does not load when the HA service is configured to load
+* If the bondX name is defined but one of bondX_SLAVES or bondX_IPs is missing,
+ then that specific bond will not be created.
===============================================================================
1. Overview
===============================================================================
-mthca is the low level driver implementation for Mellanox Technologies HCAs:
-InfiniHost, InfiniHost III Ex and InfiniHost III Lx.
+mthca is the low level driver implementation for the following Mellanox Techno-
+logies HCAs: InfiniHost, InfiniHost III Ex and InfiniHost III Lx.
mthca Available Parameters
--------------------------
mthca parameters:
- - tune_pci increase PCI burst from the default set by BIOS if nonzero
- - msi attempt to use MSI if nonzero
- - msi_x attempt to use MSI-X if nonzero
- - fw_cmd_doorbell post FW commands through doorbell page if nonzero (and
- supported by FW)
- - catas_reset_disable disable device reset on a catastrophic event if nonzero
+ tune_pci - increase PCI burst from the default set by BIOS if
+ nonzero
+ msi - attempt to use MSI if nonzero
+ msi_x - attempt to use MSI-X if nonzero
+ fw_cmd_doorbell - post firmware commands through doorbell page if non-
+ zero (and supported by firmware)
+ catas_reset_disable - disable device reset on a catastrophic event if non-
+ zero
===============================================================================
2. Fixed Bugs
===============================================================================
-1. Allocating many MRs (which uses many memory pages) causes kernel Oops. (490)
-2. Improve scheduling of kernel QPs in mthca. (542)
-3. Wrong mthca errors (e.g. CQ entry for unknown QP bd0406). (394)
+1. Allocating many MRs uses many memory pages and causes a kernel oops. (490)
+2. Improved scheduling of kernel QPs in mthca. (542)
+3. Wrong mthca error reporting (e.g., CQ entry for unknown QP bd0406). (394)
4. Kernel panic on boot when using Hide-DDR FW. (544)
-5. Off-by-one in FMR handling on MemFree can cause memory corruption. (424)
-6. InfiniHost III Lx: Data corruption when working with FMR (Influence on the
- ULPs: iSER, SRP and Lustre).
-7. Fix RESET to ERROR transition
-8. Set cleaned CQEs back to HW ownership when cleaning CQ
-9. Fix posting >255 receive WRs for InfiniHost
-10. Fix error path in mthca_alloc_memfree()
-11. Allow the QP state transition RESET->RESET
-12. Fix access to MTT and MPT tables on non-cache-coherent CPUs
+5. Off-by-one FMR handling can cause memory corruption for mem-free devices.
+ (424)
+6. InfiniHost III Lx: Data corruption when working with FMR (Affected ULPs:
+ iSER, SRP and Lustre).
+7. RESET to ERROR transition
+8. Set cleaned CQEs back to HW ownership upon cleaning the CQ
+9. (InfiniHost) Posting more than 255 receive Work Requests
+10. Error path in mthca_alloc_memfree()
+11. The QP state transition RESET->RESET was not allowed
+12. Access to MTT and MPT tables on non-cache-coherent CPUs
===============================================================================
3. Known Issues
===============================================================================
-1. UAR size other than 8MB prevents mthca driver loading. The default UAR
- size is 8MB. If it is changed, the following error message will be logged to
- /var/log/messages upon attempt to load the mthca driver:
+1. A UAR size other than 8MB prevents mthca driver loading. The default UAR
+ size is 8MB. If the size is changed, the following error message will be
+ logged to /var/log/messages upon attempting to load the mthca driver:
ib_mthca 0000:04:00.0: Missing UAR, aborting.
-2. If a user level application that uses multicast receives a control signal
+2. If a user level application using multicast receives a control signal
in the process of detaching from a multicast group, its QP may remain a
member of the multicast group (in HCA).
Workaround: Destroy the multicast group after detaching the QP from it.
-3. On MemFree devices, RC QPs can be created with a maximum of (max_sge - 3)
+3. In mem-free devices, RC QPs can be created with a maximum of (max_sge - 3)
entries only.
-4. Performance degradation due to wrong BIOS configuration:
- The PCI Express spec. requires BIOS to set the MaxReadReq register
- for each card for maximum performance and stability.
+4. Performance can be degraded due to a wrong BIOS configuration:
+ The PCI Express specification requires the BIOS to set the MaxReadReq
+ register for each HCA card for maximum performance and stability.
- If you are seeing bandwidth performance degradation, you can try forcing
- the card to behave out of PCI Express spec. by setting the tune_pci=1 module
- parameter. This tune_pci=1 option was the default setting in OFED
- 1.0, which might have masked performance degradation on some systems.
+ If you experience bandwidth performance degradation, try forcing the card to
+ behave not according to the PCI Express specification by setting the
+ tune_pci=1 module parameter. This tune_pci=1 assignment was the default
+ setting in OFED 1.0; therefore, it may have masked performance degradation
+ on some systems.
- If tune_pci=1 improves bandwidth, please report the issue to your
- BIOS vendor. Please note that Mellanox Technologies does not recommend using
- tune_pci=1 in production systems: working with tune_pci=1 option set is
- untested and is known to trigger stability issues on some platforms.
+ If tune_pci=1 improves bandwidth, please report the issue to your BIOS
+ vendor. Please note that Mellanox Technologies does not recommend using
+ tune_pci=1 in production systems: working with tune_pci=1 set is untested
+ and is known to trigger instability issues on some platforms.
June 2007
-Overview
---------
+===============================================================================
+Table of Contents
+===============================================================================
+1. Overview
+2. Software Dependencies
+3. New Features
+4. Bug Fixes
+5. Known Issues
+6. Main Verification Flows
+
+
+===============================================================================
+1. Overview
+===============================================================================
These are the release notes for OSU MPI MVAPICH-0.9.9.
OSU MPI is an MPI channel implementation over InfiniBand
-from Ohio State University (OSU).
+by Ohio State University (OSU).
-(http://nowlab.cse.ohio-state.edu/projects/mpi-iba/).
+See http://nowlab.cse.ohio-state.edu/projects/mpi-iba/.
-Software Dependencies
----------------------
-OSU MPI depends on the installation of the OFED Distribution stack with OpenSM
-running. The MPI module also requires an established network interface (either
+
+===============================================================================
+2. Software Dependencies
+===============================================================================
+OSU MPI depends on the installation of the OFED stack with OpenSM running.
+The MPI module also requires an established network interface (either
InfiniBand IPoIB or Ethernet).
-New Features ( compared to mvapich 0.9.7-mlnx2.2.0 )
-------------
+
+===============================================================================
+3. New Features ( Compared to mvapich 0.9.7-mlnx2.2.0 )
+===============================================================================
This version for OFED has the following additional features:
-- Improved message coalescing:
+- Improved message coalescing
* Reduction of per QP send queues for reduction in memory requirement
- * Increases the small message messaging rate significantly
-- Multi-core optimizations:
+ * Significant increase in small message messaging rate
+- Multi-core optimizations
* Optimized scalable shared memory design
- * Optimized, high-performance shared memory aware collective operations
+ * Optimized, high-performance shared-memory-aware collective operations
* Multi-port support for enabling user processes to bind to different IB ports
for balanced communication performance
- On-demand connection management using native IB UD support
- Multi-path support for hot-spot avoidance in large scale clusters
- Memory Hook Support provided by integration with ptmalloc2 library
-Bug Fixes
----------
-- Error report fixes.
-- PKEY support fix.
-Known Issues
-------------
+===============================================================================
+4. Bug Fixes
+===============================================================================
+- Error report fixes
+- PKEY support fix
+
+
+===============================================================================
+5. Known Issues
+===============================================================================
- A process running MPI cannot fork after MPI_Init. Using fork might cause a
segmentation fault.
- Using mpirun with ssh has a signal collection problem. Killing the run
(using CTRL-C) might leave some of the processes running on some of the
nodes. This can also happen if one of the processes exits with an error.
Note: This problem does not exist with rsh.
-- The MPD job launcher feature of OSU MPI module has not been tested by Mellanox
- Technologies. See http://nowlab.cse.ohio-state.edu/projects/mpi-iba/ for more
- details.
+- The MPD job launcher feature of the OSU MPI module has not been tested by
+ Mellanox Technologies.
+ See http://nowlab.cse.ohio-state.edu/projects/mpi-iba/ for more details.
- For users of Mellanox Technologies firmware fw-23108 or fw-25208 only:
OSU MPI might fail in its default configuration if your HCA is burnt with an
fw-23108 version that is earlier than 3.4.000, or with an fw-25208 version
Workaround: Edit /etc/hosts and comment-out/remove the line that maps
IP address 127.0.0.2 to the system's fully qualified hostname.
-Main Verification Flows
------------------------
+
+===============================================================================
+5. Main Verification Flows
+===============================================================================
In order to verify the correctness of OSU MPI, the following tests and
parameters were run.
Test Description
-===================================================================
-Intel's test suite 1400 Intel tests
+-------------------------------------------------------------------
+Intel's Test suite - 1400 Intel tests
BW/LT OSU's test for bandwidth latency
IMB Intel's MPI Benchmark test
mpitest b_eff test
===============================================================================
1. Overview
2. Supported Platforms
-3. Known Issues
+3. Installation & Configuration
+4. New Features
===============================================================================
1. Overview
SLES 10
===============================================================================
-3. Known Issues
+3. Installation & Configuration
===============================================================================
-1. RDS HA fails
-
-
+To install RDS select rds in OFED's manual installation or put 'rds=y' in the
+ofed.conf for unattended installation.
+To load RDS module upon boot edit file '/etc/infiniband/openib.conf' as
+follows:
+
+# Load RDS module
+RDS_LOAD=yes
+
+===============================================================================
+4. New Features
+===============================================================================
+RDS socket send buffers are now per socket, not per flow, and the new recv
+buffer value is per socket. So we should use the core stack sndbuf and rcvbuf
+values instead of offering a separate interface.
+
+Use the following script to change the default and max socket buffer sysctls
+to a 1 MB:
+
+#!/bin/bash
+
+nr=$((1 * 1024 * 1024))
+sys="/proc/sys/net/core"
+
+for f in {r,w}mem_{default,max}; do
+ echo $nr > $sys/$f
+done
SDP in OFED is at GA level for OFED 1.2.
===============================================================================
-2. new features
+2. New Features
===============================================================================
-* add proc fs support.
+* Added proc fs support
-* OFED is shipped with patched version of netstat (sdpnetstat).
- SDP info is printed by default. To view only SDP sockets use the "-S" flag.
- Info given from sdpnetstat (for each SDP socket):
+* OFED is shipped with a patched version of netstat (sdpnetstat).
+ SDP info is printed by default. To view only SDP sockets use the "-S" flag.
+ Info provided by sdpnetstat (for each SDP socket):
Recv-Q
Send-Q
Local Address
Inode
PID/Program name
-* Allow user, via module parameter, to bound SDP's memory use
+* The user can now bound SDP's memory use via a module parameter
+
+* Implemented slow start on receive buffers. The socket is created with small
+ buffers, and when needed the socket requests the other side to resize its
+ buffers to allow for more data.
-* Implement slow start on recv buffers. The socket is created with small buffers
- and when needed the socket requests the other side so resize it's buffers to
- allow for more data.
===============================================================================
3. Bug Fixes
===============================================================================
-* merge small skbs on receive side into larger ones (bugzilla 276)
+* Merge of small SKBs on receive side into larger ones (bugzilla 276)
+
+* DMA leak and DMA direction (556)
-* fix DMA leak, and DMA direction (556)
+* Improved bandwidth
-* improve BW
+* Kernel oops when device is removed (bugzilla 579)
-* kernel oops when device is removed (bugzilla 579)
+* Non-compliance with CA4-119 (bugzilla 596)
-* compliant with CA4-119 (bugzilla 596)
+* A CM disconnect does not wake up sleeping processes (bugzzila 492)
-* cm disconnect should wake up any sleeping processes (bugzzila num. 492)
+* Interoperability bugs in login messages
-* fix interop bugs in login messages
+* Incorrect handling of receive side shutdown on listening socket
-* handle shutdown recv on listening socket
+* Credit starvation (eliminated by modifying buffer use calculation)
-* modify buffer use calculation to eliminate credit starvation
===============================================================================
4. Known Issues
===============================================================================
-- Each SDP socket currently consumes up to 2 MBytes of memory. If this value
- is high for your installation, it is possible to trade off performance
+- Each SDP socket currently consumes up to 2 MB memory. If this value
+ is too high for your installation, it is possible to trade off performance
for lower memory utilization per socket by reducing the value of the
"rcvbuf_scale" module parameter (default: 16).
- Note: the minimum legal value for this parameter is 1.
- At this parameter value, each socket will consume approximately 128 KBytes.
+ Note: The minimum legal value for this parameter is 1.
+ If rcvbuf_scale=1 then each socket will consume approximately 128 KB.
- Note: memory usage of recv buffers can be limited via the "top_mem_usage"
+ Note: Memory usage of receive buffers can be limited via the "top_mem_usage"
module parameter. Setting the parameter (default: 0) will cause socket
- memory usage for recv buffers to be the maximum allowed under the system
+ memory usage for receive buffers to be the maximum allowed under the system-
wide limit.
-- Small message size performance is low when messages are sent by client
- at a rate lower than the rate at which they are consumed by server,
- and when TCP_CORK is not set. This is observed, for example, with iperf
- benchmark. As a workaround, set the TCP_CORK socket option
- to ensure data is sent in at least 32K byte chunks.
+- Small message size performance is low when the rate of Client sent messages
+ is lower than the rate at which they are consumed by Server, and when
+ TCP_CORK is not set. This is observed, for example, with the iperf benchmark.
+ As a workaround, set the TCP_CORK socket option to ensure data is sent in at
+ least 32KB chunks.
-- Performance is low on 32-bit kernels, as SDP utilizes high memory
- to ease memory pressure. Moving to a 64-bit kernel solves this
- problem even if the application remains a 32-bit one.
+- Performance is low on 32-bit kernels as SDP utilizes high memory to ease
+ memory pressure. Moving to a 64-bit kernel solves this problem even if the
+ application remains a 32-bit one.
-- By default, SDP utilizes a 2 Kbyte MTU size. This may cause PCI-X cards
- using Mellanox Technologies "Infinihost" HCAs to experience low bandwidth.
- Workaround: reset the MTU size to 1K in this situation, using either of
- the two methods below:
+- By default, SDP utilizes a 2KB MTU size. This may cause PCI-X cards
+ using Mellanox Technologies InfiniHost HCAs to experience low bandwidth.
+ Workaround: Reset the MTU size to 1K in this situation, using one of
+ the two following methods:
- 1. Activate the "tavor quirk" workaround in opensm:
+ 1. Activate the "tavor quirk" workaround in opensm
a. Create an opensm options cache file (/var/cache/osm/opensm.opts):
> opensm --cache-options -o
b. Add the following line to /var/cache/osm/opensm.opts:
c. Rerun opensm using your usual command line options to activate
the opensm quirk option.
- 2. Activate the "tavor quirk" workaround in cma:
- set the tavor_quirk module parameter of the rdma_cm module to value 1
+ 2. Activate the "tavor quirk" workaround in cma
+ Set the tavor_quirk module parameter of the rdma_cm module to value 1
(default: 0).
-- Connection rate in lower than TCP. This may cause clients to get connection
- refused when server is under connection stress.
+- Connection rate in lower than TCP. This may cause Clients to get connection
+ refused when Server is under connection stress.
-- Due to limited dma pool on PPC, sdp cannot sustain more than about 1000 sockets
- with default parameters. Limiting max_large_sockets and rcvbuf_scale allows
- more sockets to be opened but at a performance cost.
+- Due to a limited DMA pool on PPC, SDP cannot sustain more than about 1000
+ sockets with default parameters. Limiting max_large_sockets and rcvbuf_scale
+ allows more sockets to be opened but at a performance cost.
-- http using a client over curl has problems connecting using libsdp matching
- rules. Using SIMPLE_LIBSDP causes data tranfer to stall.
+- http using a Client over curl has problems connecting using libsdp matching
+ rules. Using SIMPLE_LIBSDP causes data transfer to stall.
===============================================================================
- LTP socket tests
- iperf-2.0.2
- ttcp
-- Threaded and forking echo client server examples
-- Various Java client server applications (SUN:jre, BEA:jrockit/WebLogic, GNU:gij/gcj)
+- Threaded and forking echo Client Server examples
+- Various Java Client Server applications (SUN:jre, BEA:jrockit/WebLogic, GNU:gij/gcj)
- Many UNIX utilities to verify that pre-load did not harm the applications
- shutdown/close/accept
- lowat/timeo
===============================================================================
2. New Features
===============================================================================
-* change default rules to "both" for server and client
+* Changed default rules to "both" for Server and Client
-* Add support for two new APIs: pselect and poll
+* Added support for two new APIs: pselect and poll
-* If no match is made, default to "both"
+* If no match is made, defaults to "both"
===============================================================================
3 Bug Fixes
The following list of bugs were fixed. Note that other less critical
or visible bugs were also fixed.
-* Handle the case where another lib is init before libsdp but calls API
+* Incorrect handling in case another lib is init before libsdp but calls API
cloned by libsdp (bugzilla 304)
-* libsdp.conf is not being overwritten (bugzilla 303)
+* libsdp.conf does not get overwritten (bugzilla 303)
-* fix security issue of log file:
- 1. root file is under /var/log - so users can not play with it
- 2. if the file maps into a link it is not used (stderr instead)
- 3. file name extended with effective user ID
+* Fixed the log file security issue:
+ 1. Root file is now under /var/log so users can not play with it
+ 2. If the file maps into a link then it is not used (stderr instead)
+ 3. The file name is extended with an effective user ID
-* install failed if the target sysconfig directory did not exist
+* Install fails if the target sysconfig directory does not exist
-* accept when the FD is marked NONBLOCK
+* SDP now accepts when the FD is marked NONBLOCK
-* failing to handle async connect in both mode, default to SDP only
+* Failure to handle async connect in both mode, default to SDP only
-* document libsdp.conf log levels 3 and 4 (bugzilla 559)
+* Document libsdp.conf log levels 3 and 4 (bugzilla 559)
-* in "both" mode, only close the SDP socket in case the TCP connection
+* In "both" mode, only the SDP socket is closed in case the TCP connection
succeeds (bugzilla 576)
-* getsockname returns invalid address for sockets bind to ipv4 address
+* getsockname returns an invalid address for sockets bound to an ipv4 address
encapsulated in ipv6 (bugzilla 613)
===============================================================================
4. Known Issues
===============================================================================
-* When using server to listen on both SDP and TCP, the number of sockets and
+* When using Server to listen on both SDP and TCP, the number of sockets and
file descriptors is doubled.
* A rare race still exists when performing bind/listen on ANY_PORT. The race
- is between applications and has been greatly minimized. A test to reproduce it
- has not been found yet. The race is between libsdp running the sequence
+ is between applications and has been greatly minimized. A test to reproduce
+ it has not been found yet. The race is between libsdp running the sequence
close(fd1) and bind(fd2, port), and another application/thread explicitly
trying to bind(fd3, port) to the same port.
- To resolve this race a change in SDP/CMA behavior is required (provide
+ To resolve this race a change in SDP/CMA behavior is required: provide a
different port number in successive calls to bind (ANY_PORT) and SDP support
- for "unbind").
+ for "unbind".
-* libsdp does not support non-blocking connect in "both" mode. In such case,
+* libsdp does not support a non-blocking connect in "both" mode. In this case,
libsdp defaults to SDP and a warning is printed to the log file.
-* Applications statically linked with libc will not allow dynamic pre-loading
- to port these applications to use SDP one needs to replace specific socket
+* Applications statically linked with libc will not allow dynamic pre-loading.
+ To port these applications to use SDP you need to replace specific socket
calls (within the application) with AF_INET_SDP and recompile.
===============================================================================
==============================================================================
2. Changes and Bug Fixes
==============================================================================
-* Support HA in Red Hat
-* Use a rules file for filtering targets and adding attributes values
+* Support for High Availability in Red Hat
+* Added a rules file for filtering targets and adding attribute values
(max_cmd_per_lun, max_sect).
-* Decrease log size in ./var/log/messages
-* support CM dgid redirection
-* Handle a rare race condition.
-* Fix Bugs: fix some memory leaks and potential out of buffer writes,
- correct line in error messages in rule files
+* Decreased the log size of ./var/log/messages
+* Support for CM dgid redirection
+* Bug fixes:
+ - Memory leaks and potential out of buffer writes
+ - Correct line in error messages in rule files
+ - A rare race condition
+
==============================================================================
3. Software Dependencies
srp_daemon
----------
The srp_daemon utility is based on ibsrpdm and extends its functionality.
-In addition to the ibsrpdm functionality described above, srp_daemon can also:
+In addition to the ibsrpdm functionality described above, srp_daemon can also
- Establish an SRP connection by itself (without the need to issue the "echo"
command described in Section 6)
- Continue running in background, detecting new targets and establishing SRP
- Discover reachable SRP targets given an infiniband HCA name and port, rather
than just by /dev/umad<N> where <N> is a digit
- Enable High Availability operation (together with Device-Mapper Multipath)
-- Have a configuration file that can decide to which targets to connect.
+- Have a configuration file that determines the targets to connect to
a. srp_daemon commands equivalent to ibsrpdm:
- To both discover the SRP Targets and establish connections with them, just
add the -e option to the above command.
- - Executing srp_daemon without -a option will display only the reachable
- Targets to which the initiator is not connected (via the port upon which
- srp_daemon was activated). When executing with the -e option it is better
- to omit -a.
+ - Executing srp_daemon over a port without the -a option will only display
+ the reachable targets via the port and to which the initiator is not
+ connected. If executing with the -e option it is better to omit -a.
- It is recommended to use the -n option. This option adds the initiator_ext
to the connecting string. (See Section 9 for more details).
- - One can set a configuration file for srp_daemon. The default
- configuration file is /etc/srp_daemon.conf but it is possible to supply
- another configuration file using the -f option. The configuration file
- configure to which targets srp_daemon is allowed to connect. The
- configuration file can be used also to set values for additional
+ - srp_daemon has a configuration file that can be set, where the default is
+ /etc/srp_daemon.conf. Use the -f to supply a different configuration file
+ that configures the targets srp_daemon is allowed to connect to. The
+ configuration file can also be used to set values for additional
parameters (e.g., max_cmd_per_lun, max_sect).
- - Continuous background (daemon) operation, providing automatic ongoing
- detection and connection capability -- see the next section.
+ - A continuous background (daemon) operation, providing an automatic ongoing
+ detection and connection capability. See Section 8.
==============================================================================
8. Automatic Discovery and Connecting to Targets
srp_daemon -e -o. This utility will scan the fabric once, connect to
every Target it detects, and then exit.
- - Note that srp_daemon will follow the configuration it finds in
- /etc/srp_daemon.conf. It can ignore a target if this target is disallowed
- by the configuration file.
+ NOTE: srp_daemon will follow the configuration it finds in
+ /etc/srp_daemon.conf. Thus, it will ignore a target that is disallowed in
+ the configuration file.
- To connect to all the existing Targets in the fabric and to connect
to new targets that will join the fabric, execute srp_daemon -e. This utility
When the failed path recovers, it will be detected by the SRP daemon. The SRP
daemon will then request ib_srp to connect to this target. Once the connection
is up, there will be a new scsi_host for this target. Multipath will be
-executed on the devices of this host, and we will return to the original state
-(before the path failed).
+executed on the devices of this host, returning to the original state (prior to
+the failed path).
High Availability Prerequisites
-------------------------------
Installation for RHEL4: (Execute once)
- - Verify that the standard device-mapper-multipath rpm is installed. If not, install it from the RHEL distribution.
+ - Verify that the standard device-mapper-multipath rpm is installed. If not,
+ install it from the RHEL distribution.
Installation for SLES10: (Execute once)
- - Verify that multipath is installed. If not, install it from the
- installation. It is also possible to download it
- from http://christophe.varoqui.free.fr/multipath-tools/multipath-tools-0.4.7.tar.bz2
- and then compile and install it.
+ - Verify that multipath is installed. If not, take it from the installation
+ or download it from http://christophe.varoqui.free.fr/multipath-tools/multipath-tools-0.4.7.tar.bz2 and then compile and install it.
- Update udev: (Execute once - for manual activation of High Availability only)
This file should have one line:
ACTION=="add", KERNEL=="sd*[!0-9]", RUN+="/sbin/multipath %M:%m"
- Note that when SRPHA_ENABLE is set to "yes" (see below in Automatic activation
- of High Availability subsection), this file is created upon each boot of
- the driver and deleted when the driver is unloaded.
+ Note: When SRPHA_ENABLE is set to "yes" (see Automatic Activation of High
+ Availability below), this file is created upon each boot of the driver and
+ is deleted when the driver is unloaded.
Manual Activation of High Availability
as described above
4) Execute for each port and each HCA:
srp_daemon -c -e -R 300 -i <InfiniBand HCA name> -p <port number>
- (You can use another value for -R. See in Known Issues section the
- workaround for the rare race condition.)
+ (You can use another value for -R. See under the Known Issues section
+ the workaround for the rare race condition.)
This step can be performed by executing srp_daemon.sh, which sends
its log to /var/log/srp_daemon.log.
Now it is possible to access the SRP LUNs on /dev/mapper/.
- NOTE: It is possible that regular (not SRP) LUNs may also be present;
- the SRP LUNs may be identified by their name. you may want to
- configure the /etc/multipath.conf file to change multipath behavior.
+ NOTE: It is possible for regular (non-SRP) LUNs to also be present;
+ the SRP LUNs may be identified by their names. You can configure the
+ /etc/multipath.conf file to change multipath behavior.
Automatic Activation of High Availability
from a target during run-time.
- There is a very rare race condition which can cause the SRP daemon to miss a
- target that joins the fabric. The race can occur if a target join and leaves
- the fabric several times in a short time. (This can happen if the cable is
- not connected well). In such a case the SM may ignore this quick change of
- state and may not send an InformInfo to the srp_daemon.
+ target that joins the fabric. The race can occur if a target joins and leaves
+ the fabric several times in a short time (e.g., if the cable is not connected
+ well). In such a case, the SM may ignore this quick change of state and may
+ not send an InformInfo to the srp_daemon.
Workaround: Execute the srp_daemon command with the -R <sec> option. This
option causes the SRP daemon to perform a full rescan of the fabric every
to the Target" for details.
- The srp_daemon tool reads by default the configuration file
- /etc/srp_daemon.conf. In case this configuration file disallows to connect
- to a certain target, srp_daemon will ignore this target. If you find out
- that srp_daemon ignores a target, please check /etc/srp_daemon.conf file.
+ /etc/srp_daemon.conf. In case this configuration file disallows connecting
+ to a certain target, srp_daemon will ignore the target. If you find out
+ that srp_daemon ignores a target, please check the /etc/srp_daemon.conf file.
==============================================================================
13. Vendor Specific Notes
/sys/class/inifiniband_srp/srp-mthca0-1/add_target
-2. Change the SRP map on the Qlogic SRP Target to set the expected
- initiator extension to 0. For details on how to change the SRP map on a
- Qlogic SRP Target, please refer to product documentation.
+2. Change the SRP map on the Qlogic SRP Target to set the expected initiator
+ extension to 0. For details on how to change the SRP map on a Qlogic SRP
+ Target, please refer to product documentation.
projects / compat-rdma / docs.git / commitdiff