Open Fabrics Enterprise Distribution (OFED)
- MPI in OFED 1.3 README
+ MPI in OFED 1.3.1 README
- February 2008
+ June 2008
===============================================================================
1. Overview
===============================================================================
Three MPI stacks are included in this release of OFED:
-- MVAPICH 1.0.0
-- Open MPI 1.2.5-1
-- MVAPICH2 1.0.2
+- MVAPICH 1.0.1-2533
+- Open MPI 1.2.6-1
+- MVAPICH2 1.0.3-1
Setup, compilation and run information of MVAPICH, Open MPI and MVAPICH2 is
provided below in sections 2, 3 and 4 respectively.
1. Overview
==============================================================================
-This is the OpenFabrics Enterprise Distribution (OFED) version 1.3
+This is the OpenFabrics Enterprise Distribution (OFED) version 1.3.1
software package supporting InfiniBand and iWARP fabrics. It is composed
of several software modules intended for use on a computer cluster
constructed as an InfiniBand subnet or an iWARP network.
the Openfabrics stack. (Note that these includes, if needed, are
"backported" to your kernel).
o The raw package (un-backported) source files are placed under
- <prefix>/src/ofa_kernel-1.3
+ <prefix>/src/ofa_kernel-1.3.1
o The script "openibd" is installed under /etc/init.d/. This script can
be used to load and unload the software stack.
o The directory /etc/infiniband is created with the files "info" and
Setting IPoIB Configuration:
----------------------------
+There is no default configuration for IPoIB interfaces.
-The default IPoIB interface configuration is based on DHCP. Note that
-a special patch for DHCP servers is required for supporting IPoIB
-clients. A patch for dhcp v3.0.4 is is available under
-OFED-X.X.X/docs/dhcp.
-
-If you are not using DHCP to obtain IP addresses for clients using
-IPoIB, you must manually specify the full IP configuration during the
+One should manually specify the full IP configuration during the
interactive installation: IP address, network address, netmask, and
-broadcast address.
+broadcast address, or use the ofed_net.conf file.
For bonding setting please see "ipoib_release_notes.txt"
1) Via the installation menu.
2) Using the script ofed_uninstall.sh. The script is part of ofed-scripts
package.
+3) ofed_uninstall.sh script supports an option to executes 'openibd stop'
+ before removing the RPMs using the flag: --unload-modules
==============================================================================
9. Upgrading OFED
OFED documentation is located in the ofed-docs RPM. After
installation the documents are located under the directory:
-/usr/share/doc/ofed-docs-1.3 for RedHat
-/usr/share/doc/packages/ofed-docs-1.3 for SuSE
+/usr/share/doc/ofed-docs-1.3.1 for RedHat
+/usr/share/doc/packages/ofed-docs-1.3.1 for SuSE
Document list:
Open Fabrics Enterprise Distribution (OFED)
- Version 1.3.1-rc4
+ Version 1.3.1
Release Notes
June 2008
Table of Contents
===============================================================================
1. Overview, which includes:
- - OFED Distribution Rev 1.3 Contents
+ - OFED Distribution Rev 1.3.1 Contents
- Supported Platforms and Operating Systems
- Supported HCA and RNIC Adapter Cards and Firmware Versions
- Tested Switch Platforms
===============================================================================
3. Changes and Fixed Bugs since OFED 1.3
===============================================================================
-- MVAPICH 1.0.1
-- Open MPI 1.2.6
-- uDAPL v1 (1.2.6) and v2 (2.0.8)
-- librdmacm 1.0.7
-- IPoIB bug fixes
-- SRP bug fixes
-- RDS bug fixes
-- Bonding fixes and support for SM failover.
-- Low level drivers update:
- - cxgb3 (Chelsio)
- - ehca (IBM)
- - mlx4 (Mellanox)
- - nes (NetEffect)
-- Added backport to support SLES 10 SP2 and RHEL 5 up2 beta
-
Note: For details regarding the various fixed bugs, please see the
release notes for each component in the docs directory.
+New OS support:
+ - SLES 10 SP2
+ - RHEL 5 up2
+
+Install scripts:
+ - There is no default configuration for IPoIB interfaces: One should manually
+ specify the full IP configuration or use the ofed_net.conf file.
+ - A new option was added to the ofed_uninstall.sh script: --unload-modules.
+ This option executes 'openibd stop' before removing the OFED RPMs.
+
+Libraries:
+ - uDAPL v1 (1.2.7-1) and v2 (2.0.9-1)
+ - librdmacm 1.0.7
+
+Kernel changes:
+ - IPoIB bug fixes
+ - SRP bug fixes
+ - RDS bug fixes
+ - Bonding fixes and support for SM failover.
+ - Core: SA module fix
+
+Utilities:
+ - opensm version 3.1.11
+ - mstflint
+ - ibutils
+
+Low level drivers:
+ - cxgb3 (Chelsio)
+ - ehca (IBM)
+ - mlx4 (Mellanox)
+ - nes (NetEffect)
+ - ipath (Qlogic)
+
+MPI packages:
+ - MVAPICH 1.0.1
+ - Open MPI 1.2.6
+ - MVAPICH2 1.0.3
===============================================================================
4. Known Issues
Open Fabrics Enterprise Distribution (OFED)
- Tips for Working with OFED 1.3
+ Tips for Working with OFED 1.3.1
- February 2008
+ June 2008
===============================================================================
Table of Contents
1. OFED Utilities
2. Debug HOWTOs
3. Pinning (Locking) User Memory Pages
-4. External Module Compilation Over OFED-1.3
+4. External Module Compilation Over OFED-1.3.1
5. Adding/Deleting a patch to OFED package
6. Adding vendor specific actions to the installation of OFED
7. How to compile OFED sources manually
To learn about other utilities, use their --help flag.
2. The sources for all utilities are not part of the RPM installation. However,
- all sources exist in the openib-1.3.tgz tarball.
+ all sources exist in the openib-1.3.1.tgz tarball.
1.1 Device Information
-------------------------------------------
The text file BUILD_ID provides data on all OFED components (whether installed
or not). This file is a part of the ofed-docs RPM and installed under
-/usr/share/doc/ofed-docs-1.3 on RedHat, and under
-/usr/share/doc/packages/ofed-docs-1.3 on SuSE.
+/usr/share/doc/ofed-docs-1.3.1 on RedHat, and under
+/usr/share/doc/packages/ofed-docs-1.3.1 on SuSE.
The same information can be obtained by executing the 'ofed_info' command. For
example:
> ofed_info
-OFED-1.3-rc6
+OFED-1.3.1
libibverbs:
git://git.openfabrics.org/ofed_1_3/libibverbs.git ofed_1_3
commit 40b771aa6a9c0ad092b2e20775b4723d3b173792
commit 82be4d81859d1fd2edf830220fe65a9923b80a46
libcxgb3:
git://git.openfabrics.org/ofed_1_3/libcxgb3.git ofed_1_3
-commit 8218482d03898f859e1fa272649fafc64f91f05f
+commit 6f7485feb244d8571fcab2292ef92c97bea48df0
libnes:
git://git.openfabrics.org/ofed_1_3/libnes.git ofed_1_3
commit 471fa2e5a7bb2f8946119396358c31adcc6c2fb3
commit 53ec35f544bbc1838bbadc2210909c25a954a5e2
librdmacm:
git://git.openfabrics.org/ofed_1_3/librdmacm.git ofed_1_3
-commit e7c55c78f8a8537966feba3ae61bcd5f043cbf97
+commit a0ef80a1e0d5debdae48a844fbc8d09aec5b24b1
dapl1:
git://git.openfabrics.org/ofed_1_3/dapl1.git ofed_1_3
-commit 93eafe464227375d162e1fc764c2df5eab1ad50f
+commit 7a9b58d6c50fc0a357de540ec3eb2ab2e07f8779
dapl2:
git://git.openfabrics.org/ofed_1_3/dapl2.git ofed_1_3
-commit ee46aa47e2a4deeded347fe18bd6321db61c6594
+commit 2583f07d9d0f55eee14e0b0e6074bc6fd0712177
libsdp:
git://git.openfabrics.org/ofed_1_3/libsdp.git ofed_1_3
commit c8102dccc502930442b23de658674d386456b350
commit e0ce2d42eeb25f8e89b8f6daaa32a630c9b64f0d
perftest:
git://git.openfabrics.org/ofed_1_3/perftest.git ofed_1_3
-commit 398da3f1d503b24141afa990fd46eb55c42a11cf
+commit 6321b5468f7293088cc003809049c02b176130d8
qlvnictools:
git://git.openfabrics.org/ofed_1_3/qlvnictools.git ofed_1_3
-commit 41a148393a602810df80109e71086970a91c1d8d
+commit 086f9cb80ee790d61bddaf201ecbae32a2ff21dd
tvflash:
git://git.openfabrics.org/ofed_1_3/tvflash.git ofed_1_3
-commit 39a63301f0344b6b3d45bc4b16d76be81f4377c0
+commit f5e7407a7f2058448df5e5320d9843f944427429
mstflint:
git://git.openfabrics.org/ofed_1_3/mstflint.git ofed_1_3
-commit 260ae655d80fa3ed15c4544519a4c7f0cf768bd6
+commit 78bbd3d521a9078553a991111ffb6f76665b9ee9
qperf:
git://git.openfabrics.org/ofed_1_3/qperf.git ofed_1_3
commit 6221aabd038df0b7033e035378ca190641ed2295
management:
git://git.openfabrics.org/ofed_1_3/management.git ofed_1_3
-commit 2a37e8eb439553e448bdcf9e5905d5673d3b2820
+commit d9c852406dae14e8284f9cfb1c7f495bbb55fddf
ibutils:
git://git.openfabrics.org/ofed_1_3/ibutils.git ofed_1_3
-commit b7e9a2df90544591aadb9caa28d670c000c13691
+commit 7daf94fab6eaf307316326f3f49704e6080a1508
ibsim:
git://git.openfabrics.org/ofed_1_3/ibsim.git ofed_1_3
commit 55113d9f919709c7c97ea41d29991941b9c8be70
-ofa_kernel-1.3:
+ofa_kernel-1.3.1:
Git:
git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel
-commit 2942aaa279dc11281b114e23d843e306dd6696b8
+commit 39e1dc833f98e5134f91fcf7f33df402adf4bc0c
# MPI
-mvapich-1.0.0-2106.src.rpm
-mvapich2-1.0.2-4.src.rpm
-openmpi-1.2.5-1.src.rpm
+mvapich-1.0.1-2533.src.rpm
+mvapich2-1.0.3-1.src.rpm
+openmpi-1.2.6-1.src.rpm
mpitests-3.0-773.src.rpm
===============================================================================
-4. External Module Compilation Over OFED-1.3
+4. External Module Compilation Over OFED-1.3.1
===============================================================================
To build kernel modules depending on OFED's modules, take the Modules.symvers
--type|-t <kernel|backport <kernel tag>|addons <kernel tag>>
Examples:
-ofed_patch.sh --add --ofed /tmp/OFED-1.3/ --patch /tmp/cma_fix.patch --type kernel
+ofed_patch.sh --add --ofed /tmp/OFED-1.3.1/ --patch /tmp/cma_fix.patch --type kernel
-ofed_patch.sh --remove --ofed /tmp/OFED-1.3/ --patch cma_fix.patch --type kernel
+ofed_patch.sh --remove --ofed /tmp/OFED-1.3.1/ --patch cma_fix.patch --type kernel
===============================================================================
6. Adding vendor specific actions to the installation of OFED
7.1 Compiling the kernel modules
--------------------------------
-1. tar xzf OFED-1.3.tgz
-2. rpm -ihv OFED-1.3/SRPMS/ofa_kernel-1.3-ofed20071122.src.rpm
+1. tar xzf OFED-1.3.1.tgz
+2. rpm -ihv OFED-1.3.1/SRPMS/ofa_kernel-1.3.1-ofed1.3.1.src.rpm
3. cd /usr/src/redhat/SOURCES
-4. tar xzvf ofa_kernel-1.3.tgz
-5. cd ofa_kernel-1.3
+4. tar xzvf ofa_kernel-1.3.1.tgz
+5. cd ofa_kernel-1.3.1
6. configure:
run ./configure --help for a list of options.
basic invocation is:
7.2 Compiling the user space libraries
--------------------------------------
-To install user space library from the source RPM provided by OFED-1.3 manually,i
+To install user space library from the source RPM provided by OFED-1.3.1 manually,
do the following:
Example for libibverbs:
-1. tar xzf OFED-1.3.tgz
-2. rpm -ihv OFED-1.3/SRPMS/libibverbs-1.1.1-0.10.ofed20080205.src.rpm
+1. tar xzf OFED-1.3.1.tgz
+2. rpm -ihv OFED-1.3.1/SRPMS/libibverbs-1.1.1-0.10.ofed20080205.src.rpm
3. cd /usr/src/redhat/SOURCES (for RedHat)
or
cd /usr/src/packages/SOURCES (for SuSE)
========================
1) Release Notes for OFED Distribution components are to be found under
OFED-1.3.1/docs and, after the package installation, under
- /usr/share/doc/ofed-docs-1.3 for RedHat
- /usr/share/doc/packages/ofed-docs-1.3 for SuSE.
+ /usr/share/doc/ofed-docs-1.3.1 for RedHat
+ /usr/share/doc/packages/ofed-docs-1.3.1 for SuSE.
2) For a detailed installation guide, see OFED_Installation_Guide.txt.
3) For more information, please visit the OFED web-page http://www.openfabrics.org
Open Fabrics Enterprise Distribution (OFED)
- ehca in OFED 1.3 Release Notes
+ ehca in OFED 1.3.1 Release Notes
- February 2008
+ June 2008
Overview
Open Fabrics Enterprise Distribution (OFED)
- ipath in OFED 1.3 Release Notes
+ ipath in OFED 1.3.1 Release Notes
- February 2008
+ June 2008
Overview
===============================================================================
4. DHCP Support of IPoIB
===============================================================================
-IPoIB is configured by default to use information obtained dynamically from a
-DHCP server, at driver startup time, to configure its interfaces.
-
Note: To use DHCP the user must apply a special patch (see "DHCP Notes" below).
DHCP Supported Operating Systems
===============================================================================
7. Bug Fixes and Enhancements Since OFED 1.3
===============================================================================
+- There is no default configuration for IPoIB interfaces: One should manually
+ specify the full IP configuration or use the ofed_net.conf file. See
+ OFED_Installation_Guide.txt for details on ipoib configuration.
- Don't drop multicast sends when they can be queued
- IPoIB panics with RHEL5U1, RHEL4U6 and RHEL4U5: Bug fix when copying small
SKBs (bug 989)
- Kernel Oops during "port up/down test" (bug 1040)
- Restart the stack during iperf 2.0.4 ver2.0.4 in client side cause to kernel
panic (bug 985)
+- Fix neigh destructor oops on kernel versions between 2.6.17 and 2.6.20
- Set max CM MTU when moving to CM mode, instead of setting it in openibd script
- Fix CQ size calculations for ipoib
- Bonding: Enable build for SLES10 SP2
Open Fabrics Enterprise Distribution (OFED)
- mlx4 in OFED 1.3 Release Notes
+ mlx4 in OFED 1.3.1 Release Notes
- February 2008
+ June 2008
===============================================================================
1. Overview
2. Supported Firmware Versions
3. Fixed Bugs and Enhancements
-4. Known Issues
+4. Bug Fixes and Enhancements Since OFED 1.3
+5. Known Issues
===============================================================================
1. Overview
- Can now create a QP using the value of max_qp_wr returned by ibv_query_device.
- QP limits returned to caller of ibv_create_qp now always lie within device limits.
+==============================================================================
+4. Bug Fixes and Enhancements Since OFED 1.3
===============================================================================
-4. Known Issues
+- Fix race when detaching a QP from a MCG
+- FMRs: Use device minimum page size from the device capabilities, and not hard-coded.
+- FMRs: Avoid recycling old FMR R_Keys too soon
+- Change default value of enable_qos module param to 0.
+
+===============================================================================
+5. Known Issues
===============================================================================
- The following commands and/or features are not supported:
o Resize CQ
member of the multicast group (in HCA).
Workaround: Destroy the multicast group after detaching the QP from it.
-3. In mem-free devices, RC QPs can be created with a maximum of (max_sge - 3)
- entries only.
+3. In mem-free devices, RC QPs can be created with a maximum of (max_sge - 1)
+ entries only; UD QPs can be created with a maximum of (max_sge - 3) entries.
4. Performance can be degraded due to a wrong BIOS configuration:
The PCI Express specification requires the BIOS to set the MaxReadReq
Open Fabrics Enterprise Distribution (OFED)
- RDS in OFED 1.3 Release Notes
+ RDS in OFED 1.3.1 Release Notes
- February 2008
+ June 2008
===============================================================================
3. Installation & Configuration
4. New Features
5. Bug fixes and Enhancements since OFED 1.2
-6. Known Issues
+6. Bug fixes and Enhancements since OFED 1.3
+7. Known Issues
===============================================================================
1. Overview
containing the bitmap on the socket.
===============================================================================
-6. Known Issues
+6. Bug fixes and Enhancements since OFED 1.3
+===============================================================================
+- Fix a bug in RDMA signaling
+- Add 3 more stats counters
+- Fix kernel oops: swiotlb_unmap_sg+0x35/0x126
+- Fixes for RDMA API
+
+===============================================================================
+7. Known Issues
===============================================================================
1. RDS is not supported on ppc64 platforms
2. RDS is not supported on 2.6.24 kernel
Open Fabrics Enterprise Distribution (OFED)
- SRP in OFED 1.3 Release Notes
+ SRP in OFED 1.3.1 Release Notes
- February 2008
+ June 2008
==============================================================================
==============================================================================
1. Overview
- 2. Changes and Bug Fixes
+ 2. Changes and Bug Fixes since OFED 1.2
+ 3. Changes and Bug Fixes since OFED 1.3
3. Software Dependencies
4. Major Features
5. Loading SRP Initiator
==============================================================================
-2. Changes and Bug Fixes
+2. Changes and Bug Fixes since OFED 1.2
==============================================================================
* Fix a bug - In the previous version only the last attributes in the allow
line of the rules file was used.
SRP_DAEMON_ENABLE in /etc/infiniband/openib.conf to "yes".
==============================================================================
-3. Software Dependencies
+3. Changes and Bug Fixes since OFED 1.3
+==============================================================================
+* Adding target_oofabric and target_infabric fields in sysfs to receive those
+ events from srp_daemon.
+* Adding async event_handler for local port up/down events
+* Reset the req and set scsi command with different statuses
+* Do not fail-over if the dev_loss_tmo has not expired starting from
+ target_out_of_fabric or port_error events
+
+==============================================================================
+4. Software Dependencies
==============================================================================
The SRP Initiator depends on the installation of the OFED Distribution stack
with OpenSM running.
==============================================================================
-4. Major Features
+5. Major Features
==============================================================================
This SRP Initiator is based on source taken from openib.org gen2 implementing
- Basic functionality, task management and limited error handling
==============================================================================
-5. Loading SRP Initiator
+6. Loading SRP Initiator
==============================================================================
To load the SRP module, either execute the "modprobe ib_srp" command after the
==============================================================================
-6. Manually Establishing an SRP Connection
+7. Manually Establishing an SRP Connection
==============================================================================
The following steps describe how to manually load an SRP connection between
==============================================================================
-7. SRP Tools - ibsrpdm and srp_daemon
+8. SRP Tools - ibsrpdm and srp_daemon
==============================================================================
To assist in performing the steps in Section 6, the OFED 1.3 distribution
detection and connection capability. See Section 8.
==============================================================================
-8. Automatic Discovery and Connecting to Targets
+9. Automatic Discovery and Connecting to Targets
==============================================================================
- Make sure that the ib_srp module is loaded, the SRP Initiator can reach an
Availability section).
==============================================================================
-9. Multiple Connections from Initiator IB Port to the Target
+10. Multiple Connections from Initiator IB Port to the Target
==============================================================================
Some system configurations may need multiple SRP connections from
SRP_DAEMON_ENABLE to yes).
==============================================================================
-10. High Availability (HA)
+11. High Availability (HA)
==============================================================================
High Availability Overview
==============================================================================
-11. Shutting Down SRP
+12. Shutting Down SRP
==============================================================================
SRP can be shutdown by using "rmmod ib_srp", or by stopping the OFED driver
==============================================================================
-12. Known Issues
+13. Known Issues
==============================================================================
- The SCSI device is sent offline when a link goes down for several seconds,
that srp_daemon ignores a target, please check the /etc/srp_daemon.conf file.
==============================================================================
-13. Vendor Specific Notes
+14. Vendor Specific Notes
==============================================================================
Hosts connected to Qlogic SRP Targets must perform one of the following