===============================================================================
MLNX_EN driver for Mellanox Adapter Cards with 10GigE Support
- README for MLNX_OFED 1.5
+ README for OFED 1.5
December 2009
===============================================================================
- Open Fabrics Enterprise Distribution (OFED)
- MPI in OFED 1.5 README
- December 2009
+ MPI in OFED 1.5 README
+
+ December 2009
===============================================================================
===============================================================================
1. Overview
===============================================================================
-Three MPI stacks are included in this release of OFED:
-- MVAPICH 1.1.0-3143
+Open Fabrics Enterprise Distribution (OFED)Three MPI stacks are included in this release of OFED:
+- MVAPICH 1.2.0
- Open MPI 1.4
-- MVAPICH2 1.2p1
+- MVAPICH2 1.4
Setup, compilation and run information of MVAPICH, Open MPI and MVAPICH2 is
provided below in sections 2, 3 and 4 respectively.
see http://modules.sourceforge.net/ for details).
Note that the site-wide default is set in a file that is typically not
-on a networked filesystem, and is therefore specific to the host on
+on a networked file system, and is therefore specific to the host on
which it was run. As such, it is recommended to run the
mpi-selector-menu command on all hosts in a cluster, picking the same
default MPI implementation on each. It may be more convenient,
details.
Additionally, per-user defaults are set in a file in the user's $HOME
-directory. If this directory is not on a network-shared filesystem
+directory. If this directory is not on a network-shared file system
between all hosts that will be used for MPI applications, then it also
needs to be propagated to all relevant hosts.
1.4 Updating MPI Installations
------------------------------
Note that all of the MPI implementations included in the OFED software
-package are the versions that were available when OFED v1.4 was
+package are the versions that were available when OFED v1.5 was
released. They have been QA tested with this version of OFED and are
fully supported.
OFED-included MPI software packages that prohibit installing
newer/other MPI implementations.
-It should be also noted that versions of MPI released after OFED v1.4
+It should be also noted that versions of MPI released after OFED v1.5
are not supported by OFED. But since each MPI has its own release
schedule and QA process (each of which involves testing with the OFED
stack), it may sometimes be desirable -- or even advisable, depending
2. MVAPICH MPI
===============================================================================
-This package is a 1.1.0 version of the MVAPICH software package,
+This package is a 1.2.0 version of the MVAPICH software package,
and is the officially supported MPI stack for this release of OFED.
See http://mvapich.cse.ohio-state.edu for more details.
*** Running OSU tests ***
-/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/osu_benchmarks-3.0/osu_bw
-/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/osu_benchmarks-3.0/osu_latency
-/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/osu_benchmarks-3.0/osu_bibw
-/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/osu_benchmarks-3.0/osu_bcast
+/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/osu_benchmarks-3.1.1/osu_bw
+/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/osu_benchmarks-3.1.1/osu_latency
+/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/osu_benchmarks-3.1.1/osu_bibw
+/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/osu_benchmarks-3.1.1/osu_bcast
*** Running Intel MPI Benchmark test (Full test) ***
-/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/IMB-3.1/IMB-MPI1
+/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/IMB-3.2/IMB-MPI1
*** Running Presta test ***
-/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/presta-1.4.0/com -o 100
-/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/presta-1.4.0/glob -o 100
-/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/presta-1.4.0/globalop
+/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/presta-1.4.0/com -o 100
+/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/presta-1.4.0/glob -o 100
+/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/presta-1.4.0/globalop
===============================================================================
Example 1: Running the OSU bandwidth:
- > cd /usr/mpi/gcc/openmpi-1.4/tests/osu_benchmarks-3.0
+ > cd /usr/mpi/gcc/openmpi-1.4/tests/osu_benchmarks-3.1.1
> mpirun -np <N> -hostfile <HOSTFILE> osu_bw
Example 2: Running the Intel MPI Benchmark benchmarks:
- > cd /usr/mpi/gcc/openmpi-1.4/tests/IMB-3.1
+ > cd /usr/mpi/gcc/openmpi-1.4/tests/IMB-3.2
> mpirun -np <N> -hostfile <HOSTFILE> IMB-MPI1
--> Note that the version of IMB-EXT that ships in this version of
be enabled across all nodes.
Note: ssh will be used by default. In order to use rsh, use the -rsh option on
-the mpirun_rsh commandline. For more options, see mpirun_rsh -help or the
+the mpirun_rsh command line. For more options, see mpirun_rsh -help or the
MVAPICH2 user guide.
*** Running 4 processes on 4 nodes ***
*** Running OSU tests ***
-/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.0/osu_bw
-/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.0/osu_latency
-/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.0/osu_bibw
-/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.0/osu_bcast
+/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.1.1/osu_bw
+/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.1.1/osu_latency
+/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.1.1/osu_bibw
+/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.1.1/osu_bcast
*** Running Intel MPI Benchmark test (Full test) ***
-/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/IMB-3.1/IMB-MPI1
+/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/IMB-3.2/IMB-MPI1
*** Running Presta test ***
OSU Tests Example:
-$ cd /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.0
+$ cd /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.1.1
$ mpdboot -n 2 -f ./hosts
$ mpiexec -n 2 ./osu_bcast
$ mpiexec -n 2 ./osu_bibw
Intel MPI Benchmark Example:
-$ cd /usr/mpi/gcc/mvapich2-1.2p1/tests/IMB-3.1
+$ cd /usr/mpi/gcc/mvapich2-1.2p1/tests/IMB-3.2
$ mpdboot -n 2 -f ./hosts
$ mpiexec -n 2 ./IMB-MPI1
$ mpdallexit
3) Installation
a) Build the mstflint utility. This package is built using a standard
autotools method.
-
+
Example:
> ./configure
> make
# List all Mellanox devices
> /sbin/lspci -d 15b3:
02:00.0 Ethernet controller: Mellanox Technologies Unknown device 6368 (rev a0)
-
+
# Use mstflint tool to query the firmware on this device
> mstflint -d 02:00.0 q
# List all Mellanox devices
> /sbin/lspci -d 15b3:
02:00.0 Ethernet controller: Mellanox Technologies Unknown device 6368 (rev a0)
-
+
# Use mstregdump to dump HW registers, using PCI config cycles
> mstregdump /proc/bus/pci/02/00.0 > crdump.log
Note: Typically, you will need root privileges for hardware access
-
+
5) Usage (mstflint):
Read mstflint usage. Enter "./mstflint -h" for a short help message, or
"./mstflint -hh" for a detailed help message.
> mstregdump mthca0 > dumpfile
7) Usage (mstvpd):
- A VPD dump is displayed to the standard output.
- A list of keywords to dump can be supplied after the -- flag
- to apply an output filter.
-
- Examples:
- > mstvpd mthca0
- ID: Lion cub DDR
- PN: MHGA28-1T
- EC: A3
- SN: MT0551X00740
- V0: PCIe x8
- V1: N/A
- YA: R R
- RW:
-
- > mstvpd mthca0 -- PN ID
- PN: MHGA28-1T
- ID: Lion cub DDR
+ A VPD dump is displayed to the standard output.
+ A list of keywords to dump can be supplied after the -- flag
+ to apply an output filter.
+
+ Examples:
+ > mstvpd mthca0
+ ID: Lion cub DDR
+ PN: MHGA28-1T
+ EC: A3
+ SN: MT0551X00740
+ V0: PCIe x8
+ V1: N/A
+ YA: R R
+ RW:
+
+ > mstvpd mthca0 -- PN ID
+ PN: MHGA28-1T
+ ID: Lion cub DDR
8) Problem Reporting:
- Please collect the following information when reporting issues:
+ Please collect the following information when reporting issues:
- uname -a
- cat /etc/issue
- cat /proc/bus/pci/devices
- mstflint -vv
- lspci
+ uname -a
+ cat /etc/issue
+ cat /proc/bus/pci/devices
+ mstflint -vv
+ lspci
mstflint -d 02:00.0 v
mstflint -d 02:00.0 q
mstvpd 02:00.0
The OFED Distribution package generates RPMs for installing the following:
o OpenFabrics core and ULPs:
- - HCA drivers (mthca, mlx4, ipath, ehca)
+ - HCA drivers (mthca, mlx4, qib, ehca)
- iWARP driver (cxgb3, nes)
- core
- Upper Layer Protocols: IPoIB, SDP, SRP Initiator and target, iSER
- Initiator and target, RDS, qlgc_vnic, uDAPL and NFS-RDMA
+ Initiator and target, RDS, qlgc_vnic, uDAPL and NFS-RDMA
o OpenFabrics utilities
- OpenSM: InfiniBand Subnet Manager
- Diagnostic tools
installation parameters can be found by running
/etc/infiniband/info.
- Information on the driver version and source git trees can be found
- using the ofed_info utility
+ Information on the driver version and source git trees can be found
+ using the ofed_info utility
During the interactive installation of OFED, two files are
--without-depcheck Skip Distro's libraries check
-v|-vv|-vvv. Set verbosity level
-q. Set quiet - no messages will be printed
+ --force Force uninstall RPM coming with Distribution
--all|--hpc|--basic Install all,hpc or basic packages
correspondingly
Open Fabrics Enterprise Distribution (OFED)
- Version 1.5-rc6
- Release Notes
- December 2009
+ Version 1.5
+ Release Notes
+ December 2009
===============================================================================
Table of Contents
===============================================================================
1. Overview, which includes:
- - OFED Distribution Rev 1.4 Contents
+ - OFED Distribution Rev 1.5 Contents
- Supported Platforms and Operating Systems
- Supported HCA and RNIC Adapter Cards and Firmware Versions
- Tested Switch Platforms
- Third party Test Packages
- OFED sources
-4. Main Changes from OFED 1.4.1
+4. Main Changes from OFED 1.4.2
5. Known Issues
-----------------------
The OFED package contains the following components:
- OpenFabrics core and ULPs:
- - IB HCA drivers (mthca, mlx4, qib, ehca)
+ - IB HCA drivers (mthca, mlx4, qib, ehca)
- iWARP RNIC driver (cxgb3, nes)
- - core
- - Upper Layer Protocols: IPoIB, SDP, SRP Initiator and target, iSER
+ - core
+ - Upper Layer Protocols: IPoIB, SDP, SRP Initiator and target, iSER
Initiator and target, RDS, uDAPL, qlgc_vnic and NFS-RDMA.
- OpenFabrics utilities:
- - OpenSM (OSM): InfiniBand Subnet Manager
- - Diagnostic tools
- - Performance tests
+ - OpenSM (OSM): InfiniBand Subnet Manager
+ - Diagnostic tools
+ - Performance tests
- MPI:
- - OSU MPI stack supporting the InfiniBand and iWARP interface
- - Open MPI stack supporting the InfiniBand and iWARP interface
+ - OSU MPI stack supporting the InfiniBand and iWARP interface
+ - Open MPI stack supporting the InfiniBand and iWARP interface
- OSU MVAPICH2 stack supporting the InfiniBand and iWARP interface
- - MPI benchmark tests (OSU benchmarks, Intel MPI benchmarks, Presta)
+ - MPI benchmark tests (OSU benchmarks, Intel MPI benchmarks, Presta)
- Extra packages:
- - open-iscsi: open-iscsi initiator with iSER support
+ - open-iscsi: open-iscsi initiator with iSER support
- ib-bonding: Bonding driver for IPoIB interface
- Sources of all software modules (under conditions mentioned in the modules'
- LICENSE files)
+ LICENSE files)
- Documentation
Notes:
-1. iSER target is of Beta quality.
+1. iSER target and NFS-RDMA is of Beta quality.
2. All other OFED components are of production quality.
3. See release notes for each package in the docs directory.
4. Any Topspin copyright belongs to Cisco Systems, Inc.
- ia64
o Linux Operating Systems:
- - RedHat EL4 up7: 2.6.9-78.ELsmp
- - RedHat EL4 up8: 2.6.9-89.ELsmp
- - RedHat EL5 up3: 2.6.18-128.el5
- - RedHat EL5 up4: 2.6.18-164.el5
- - SLES10 SP2: 2.6.16.60-0.21-smp
- - SLES10 SP3: 2.6.16.60-0.54-smp
- - SLES11: 2.6.27.19-5-default
+ - RedHat EL4 up7 2.6.9-78.ELsmp
+ - RedHat EL4 up8 2.6.9-89.ELsmp
+ - RedHat EL5 up3 2.6.18-128.el5
+ - RedHat EL5 up4 2.6.18-164.el5
+ - SLES10 SP2 2.6.16.60-0.21-smp
+ - SLES10 SP3 2.6.16.60-0.54-smp
+ - SLES11 2.6.27.19-5-default
- OEL 4 up7 2.6.9-78.ELsmp
- OEL 4 up8 2.6.9-89.ELsmp
- CentOS5.3 2.6.18-128.el5
- CentOS5.4 2.6.18-164.el5
- Fedora Core12 2.6.31.5-127.fc12 *
- OpenSuSE 11.2 2.6.31.5-0.1-default *
- - kernel.org: 2.6.29, 2.6.30,
+ - kernel.org 2.6.29, 2.6.30,
2.6.31 and 2.6.32 *
* Minimal QA for these versions
- InfiniHost III Ex (MemFree: fw-25218 Rev 5.3.000
with memory: fw-25208 Rev 4.8.200)
- InfiniHost III Lx (fw-25204 Rev 1.2.000)
- - ConnectX IB (fw-25408 Rev 2.6.000)
+ - ConnectX IB (fw-25408 Rev 2.7.000)
For official firmware versions please see:
http://www.mellanox.com/content/pages.php?pg=firmware_download
- R310/R320 10GbE iWARP Adapters
o Intel RNICs:
- - NE020 10Gb iWARP Adapter
+ - NE020 10Gb iWARP Adapter
1.4 Switches Supported
1.5 Third Party Packages
------------------------
The following third party packages have been tested with OFED 1.5:
-1. Intel MPI, Version 3.0 - Package ID: l_mpi_p_3.0.043
-2. HP-MPI 2.2.5.1 (without XRC support)
-3. HP-MPI 2.2.7 (with XRC support)
+- Intel MPI, Version 3.2.2
+- Intel MPI, Version 4.0 beta
1.6 OFED Sources
----------------
All sources are located under git://git.openfabrics.org/
Kernel sources: git://git.openfabrics.org/ofed_1_5/linux-2.6.git ofed_kernel
-User level Sources are located in all git trees as written in the BUILD_ID
+User level Sources are downloaded from http://www.openfabrics.org/downloads/
+as written in the BUILD_ID
The kernel sources are based on Linux 2.6.30 mainline kernel. Its patches
are included in the OFED sources directory.
For details see HOWTO.build_ofed.
===============================================================================
-2. Main Changes from OFED 1.4.1
+2. Main Changes from OFED 1.4.2
===============================================================================
Note: For details regarding the various changes, please see the release notes
for each package in the docs directory.
2.1 General changes
o Kernel code based on 2.6.30
+ o libraries location - all userspace libraries can be downloaded from
+ http://www.openfabrics.org/downloads/
+ See BUILD_ID for exact location
+ o Qlogic replaced low level driver from ipath to qib.
2.2 SDP
o Performance improvements
o Zero copy in beta level
2.3 uDAPL
- o ???
+ o New UCM provider (ofa-v2-mlx4_0-1u) with IB UD based CM per process. More
+ scalable then rdma_cm (cma) or socket cm (scm).
+ o Common code base with WinOF 2.1.
+ o Bug fixes.
+
2.4 perftest
- o Renamed tests:
- ib_rdma_bw -> rdma_bw
- ib_rdma_lat -> rdma_lat
+ o Renamed tests:
+ ib_rdma_bw -> rdma_bw
+ ib_rdma_lat -> rdma_lat
2.5 Management
o OpenSM
o Diagnostic tools:
- 2.6 MPI:
+ 2.6 MPI:
a. OSU MVAPICH 1.2.0
- b. Open MPI 1.3.4
+ b. Open MPI 1.4
c. OSU MVAPICH2 1.4
d. MPI tests 3.2
+ 2.7 iSER:
+ o Available only on kernel.org 2.6.30, 2.6.31 and 2.6.32
+
+ 2.8 NFS-RDMA
+ o Added support for RHEL5.4, SLES10 SP3, kernel.org 2.6.25 and 2.6.30.
+ kernels 2.6.26 and 2.6.27 are not supported
+ o NFS-RDMA is in beta level. Will be GA in OFED-1.5.1
===============================================================================
3. Known Issues
===============================================================================
The following is a list of general limitations and known issues of the various
-components of the OFED 1.4 release.
+components of the OFED 1.5 release.
1. When upgrading from an earlier OFED version, the installation script does not
stop the earlier OFED version prior to uninstalling it.
4. The qib driver is supported only on 64-bit platforms.
5. When installing OFED on OpenSuse or Ubuntu one should use the
--without-depcheck option of the install.pl script
-6. To install OFED 1.4 on Fedora Core 8 one should:
- 1. Install libtool RPM (required by libibcommon)
- 2. Install tcsh RPM (required by mpi-selector)
- 3. Create the file '.rpmmacros' (required by mvapich):
- echo "%__arch_install_post %{nil}" >> /root/.rpmmacros
-7. IPoIB: brctl utilities do not work on IPoIB interfaces. The reason for that
+6. IPoIB: brctl utilities do not work on IPoIB interfaces. The reason for that
is that these utilities support devices of type Ethernet only.
-8. "openibd stop" can sometime fail with the error:
+7. "openibd stop" can sometime fail with the error:
Unloading ib_cm [FAILED]
ERROR: Module ib_cm is in use by ib_ipoib
Workaround: run "openibd stop" again.
-9. When working with ISCSI over IPoIB or mlx4_en, you must disable LRO (even
+8. When working with ISCSI over IPoIB or mlx4_en, you must disable LRO (even
if IPoIB is set to connected mode). This is because there is a bug in older
kernels which causes a kernel panic.
-10. On SLES11 in case that uninstall is failing, need to look at the error log
+9. On SLES11 in case that uninstall is failing, need to look at the error log
and remove the RPMs manually using 'rpm -e <rpms list>'
-11. On SLES11 one should set allow_unsupported_modules parameter to 1 in file:
+10. On SLES11 one should set allow_unsupported_modules parameter to 1 in file:
/etc/modprobe.d/unsupported-modules. Without this the modules will not
load.
+11. iSER is supported on kernel.org: 2.6.30, 2.6.31 and 2.6.32 only
+ OFED-1.5 will not install iSER on other kernels and original iSER
+ module coming with Linux Distribution will stop working because of
+ mismatch in symbols version.
Note: See the release notes of each component for additional issues.
- ib_write_bw_postlist : RDMA write , posting a list of WQEs.
3.Other:
- - ib_clock_test : tests cpu clock samples.
+ - ib_clock_test : tests CPU clock samples.
Usage:
Server: <test name> <options>
> ofed_info
OFED-1.5
management:
-http://www.openfabrics.org/downloads/management/daily/libibumad-1.3.3_20091204_86353ad.tar.gz
-http://www.openfabrics.org/downloads/management/daily/libibmad-1.3.3_20091204_86353ad.tar.gz
-http://www.openfabrics.org/downloads/management/daily/opensm-3.3.3_20091204_86353ad.tar.gz
-http://www.openfabrics.org/downloads/management/daily/infiniband-diags-1.5.3_20091204_86353ad.tar.gz
+http://www.openfabrics.org/downloads/management/daily/libibumad-1.3.4_20091214.tar.gz
+http://www.openfabrics.org/downloads/management/daily/libibmad-1.3.4_20091214.tar.gz
+http://www.openfabrics.org/downloads/management/daily/opensm-3.3.4_20091221_033cf4c.tar.gz
+http://www.openfabrics.org/downloads/management/daily/infiniband-diags-1.5.4_20091221_033cf4c.tar.gz
libibverbs:
-http://www.openfabrics.org/downloads/rdmaoe/libibverbs/libibverbs-1.1.2-0.8.g66ece2f.tar.gz
+http://www.openfabrics.org/downloads/libibverbs/libibverbs-1.1.2-0.2.g81705cb.tar.gz
libmthca:
http://www.openfabrics.org/downloads/libmthca/libmthca-1.0.5-0.1.gbe5eef3.tar.gz
libmlx4:
-http://www.openfabrics.org/downloads/rdmaoe/libmlx4/libmlx4-1.0-0.3.g1531245.tar.gz
+http://www.openfabrics.org/downloads/libmlx4/libmlx4-1.0-0.1.gfa9c3d4.tar.gz
libcxgb3:
http://www.openfabrics.org/downloads/cxgb3/libcxgb3-1.2.5.tar.gz
libehca:
libipathverbs:
http://www.openfabrics.org/downloads/libipathverbs/libipathverbs-1.2.tar.gz
libnes:
-http://www.openfabrics.org/downloads/nes/libnes-0.9.0.tar.gz
+http://www.openfabrics.org/downloads/nes/libnes-1.0.0.tar.gz
libibcm:
http://www.openfabrics.org/downloads/rdmacm/libibcm-1.0.5.tar.gz
librdmacm:
srptools:
http://www.openfabrics.org/downloads/srptools/srptools-0.0.4-0.1.gce1f64c.tar.gz
perftest:
-http://www.openfabrics.org/downloads/perftest/perftest-1.2.2-0.6.g3a2b795.tar.gz
+http://www.openfabrics.org/downloads/perftest/perftest-1.2.3-0.8.g196d994.tar.gz
qlvnictools:
http://www.openfabrics.org/downloads/qlvnictools/qlvnictools-0.0.1-0.1.ge27eef7.tar.gz
tvflash:
qperf:
http://www.openfabrics.org/downloads/qperf/qperf-0.4.6-0.1.gb81434e.tar.gz
ibutils:
-http://www.openfabrics.org/downloads/ibutils/ibutils-1.2-0.1.ge8e69b7.tar.gz
+http://www.openfabrics.org/downloads/ibutils/ibutils-1.5.tar.gz
ibsim:
http://www.openfabrics.org/downloads/ibsim/ibsim-0.5-0.1.g327c3d8.tar.gz
dapl:
ofa_kernel-1.5:
Git:
-git://git.openfabrics.org/ofed_1_5/linux-2.6.git rdmaoe
-commit 5ed038c4a28a8d9ea99d7999af178ee80ec2f663
+git://git.openfabrics.org/ofed_1_5/linux-2.6.git ofed_kernel_1_5
+commit 1b6ce93a1ccee8d38bc88a7114abd77dae3410be
# MPI
-mvapich-1.1.0-3478.src.rpm
+mvapich-1.2.0-3567.src.rpm
mvapich2-1.4-2.src.rpm
-openmpi-1.3.4-1.src.rpm
+openmpi-1.4-1.src.rpm
mpitests-3.2-916.src.rpm
ib_bonding:
http://www.openfabrics.org/~monis/ofed_1_5/ib-bonding-0.9.0-41.src.rpm
rnfs-utils:
http://www.openfabrics.org/~swise/ofed_1_5/rnfs-utils/rnfs-utils-1.1.5-10.OFED.src.rpm
-
2.2 Installed OFED Components
-------------------------------
The script /etc/infiniband/info provides data on the specific OFED installation
If <prefix>/src/ofa_kernel/Module.symvers does not exist or it is empty, use the
create_Module.symvers.sh (a part of the ofed-docs RPM) script to create the
Module.symvers file.
+Example:
+MODULES_DIR=/lib/modules/`uname -r`/updates ./create_Module.symvers.sh
See "Module versioning & Module.symvers" in the modules.txt from kernel
documentation (e.g. linux-2.6.30/Documentation/kbuild/modules.txt).
Open Fabrics Enterprise Distribution (OFED)
- Performance Tests README for OFED 1.5
-
+ Performance Tests README for OFED 1.5
+
December 2009
simultaneously -- they will not conflict with each other.
Three MPI stacks are included in this release of OFED:
-- MVAPICH 1.2.0
-- Open MPI 1.3.4
-- MVAPICH2 1.4
+- MVAPICH
+- Open MPI
+- MVAPICH2
OFED also includes 4 basic tests that can be run against each MPI
stack: bandwidth (bw), latency (lt), Intel MPI Benchmark and Presta. The tests
\r
Prerequisites\r
-------------\r
-0. Supported distributions: RHEL 5.2/5.3/5.4, SLES 10 sp2, SLES 11 \r
+0. Supported distributions: RHEL 5.3/5.4, SLES 10 sp2, SLES 11 \r
\r
Note: On distribution default kernels you can run scst_vdisk blockio mode\r
to have good performance. You can also run scst_disk ie. scsi pass-thru\r
#
# Description: creates Module.symvers file for InfiniBand modules
-K_VER=${K_VER:-$(uname -r)}
-MOD_SYMVERS_IB=./Module.symvers
+KVERSION=${KVERSION:-$(uname -r)}
+MOD_SYMVERS=./Module.symvers
SYMS=/tmp/syms
-if [ -d /lib/modules/$K_VER/updates/kernel/drivers/infiniband ]; then
- MODULES_DIR=/lib/modules/$K_VER/updates/kernel/drivers/infiniband
-elif [ -d /lib/modules/$K_VER/kernel/drivers/infiniband ]; then
- MODULES_DIR=/lib/modules/$K_VER/kernel/drivers/infiniband
-else
- echo "No infiniband modules found"
- exit 1
-fi
-
-echo MODULES_DIR=${MODULES_DIR}
+echo MODULES_DIR=${MODULES_DIR-:./}
-if [ -f ${MOD_SYMVERS_IB} -a ! -f ${MOD_SYMVERS_IB}.save ]; then
- mv ${MOD_SYMVERS_IB} ${MOD_SYMVERS_IB}.save
+if [ -f ${MOD_SYMVERS} -a ! -f ${MOD_SYMVERS}.save ]; then
+ mv ${MOD_SYMVERS} ${MOD_SYMVERS}.save
fi
-rm -f $MOD_SYMVERS_IB
+rm -f $MOD_SYMVERS
rm -f $SYMS
for mod in $(find ${MODULES_DIR} -name '*.ko') ; do
done
n_syms=$(wc -l $SYMS |cut -f1 -d" ")
-echo Found $n_syms InfiniBand symbols in $n_mods InfiniBand modules
+echo Found $n_syms OFED kernel symbols in $n_mods modules
n=1
-
while [ $n -le $n_syms ] ; do
line=$(head -$n $SYMS|tail -1)
line1=$(echo $line|cut -f1 -d:)
line2=$(echo $line|cut -f2 -d:)
- file=$(echo $line1|cut -f6- -d/)
- file=$(echo $file|cut -f1 -d.)
-
+ file=$(echo $line1| sed -e 's@./@@' -e 's@.ko@@' -e "s@$PWD/@@")
crc=$(echo $line2|cut -f1 -d" ")
- crc=${crc:8}
sym=$(echo $line2|cut -f3 -d" ")
- sym=${sym:6}
- echo -e "0x$crc\t$sym\t$file" >> $MOD_SYMVERS_IB
- if [ -z $allsyms ] ; then
- allsyms=$sym
- else
- allsyms="$allsyms|$sym"
- fi
+ echo -e "0x$crc\t$sym\t$file" >> $MOD_SYMVERS
n=$((n+1))
done
-echo ${MOD_SYMVERS_IB} created.
+echo ${MOD_SYMVERS} created.
Open Fabrics Enterprise Distribution (OFED)
- Diagnostic Tools in OFED 1.4 Release Notes
+ Diagnostic Tools in OFED 1.5 Release Notes
- December 2008
+ December 2009
-Repo: git://git.openfabrics.org/~ofed_1_3/management.git (release)
- git://git.openfabrics.org/~sashak/management/management.git (development)
+Repo: git://git.openfabrics.org/~sashak/management/management.git
+URL: http://www.openfabrics.org/downloads/management
General
===============================================================================
1. Introduction
-2. How to work with ib-bond
-3. How to work with interface configuration scripts
-3.1 Configuration with initscripts support
-3.1.1 Writing network scripts under Redhat-AS4 (Update 4, 5, 6 or 7)
-3.1.2 Writing network scripts under Redhhat-EL5
-3.2 Configuration with sysconfig support
-3.2.1 Writing network scripts under SLES-10
-3.3 Configuring Ethernet slaves
+2. How to work with interface configuration scripts
+2.1 Configuration with initscripts support
+2.1.1 Writing network scripts under Redhat-AS4 (Update 6, 7 or 8)
+2.1.2 Writing network scripts under Redhhat-EL5
+2.2 Configuration with sysconfig support
+2.2.1 Writing network scripts under SLES-10
+2.3 Configuring Ethernet slaves
1. Introduction
-------------------------------------------------------------------------------
on the Linux Ethernet Bonding Driver and was adopted to work with IPoIB.
However, the support for for IPoIB interfaces is only for the active-backup
mode, other modes should not be used.
-ib-bonding package contains a bonding driver and a utility called ib-bond to
-manage and control the driver operation.
-2. How to work with ib-bond
+2. How to work with interface configuration scripts
-------------------------------------------------------------------------------
+To create an interface configuration script for the ibX and bondX interfaces,
+you should use the standard syntax (depending on your OS).
-* Creating a bonding network interface
- --bond-name: sets the name of the bonding network interface. Default is bond0
- --bond-ip <IP/[MASK]>: sets the IP address of bond0. If MASK is not given it
- is set to 255.255.255.0 (24 bits). Note that MASK should be the number of 1
- bits in the netmask.
- --slaves: a comma separated list of slave ib devices. If not given ib0 and
- ib1 will be used as slaves. Child interfaces are allowed.
- --miimon: the MII monitoring interval in mSec. Default is 100
-* Deleting a bonding network interface
- --stop: unenslave slaves and delete a specific bonding network interface (use with --bond-name)
- --stop-all: unenslave slaves and delete all bonding network interfaces
-* Querying a bonding network interface
- --status: show the status of a specific bonding network interface (use with --bond-name)
- --status-all: show the status of all bonding network interfaces
-
-Examples:
-
-* To bring up bond0 with ib0 and ib2 as slaves (assumes 2 HCAs)
- ib-bond --bond-ip 192.186.10.100 --slaves ib0,ib2
-* To bring up bond1 with ib0.f1f1 1and ib1.f1f1 as slaves with non default
- netmask
- ib-bond --bond-name bond1 --bond-ip 192.186.10.100/25 --slaves ib0.f1f1,ib1.f1f1
-* To query the status of bond1
- ib-bond --bond-name bond1 --status
-* To query the status of all bonding interfaces
- ib-bond --status-all
-* To stop bond1
- ib-bond --bond-name bond1 --stop
-* To stop all bonding interfaces
- ib-bond --stop-all
-
-3. How to work with interface configuration scripts
--------------------------------------------------------------------------------
-Using ib-bond to configure interfaces doesn't save the configuration anywhere,
-so whenever the master or one of the slaves is destroyed the configuration
-should be restored by running ib-bond again (e.g. after system reboot).
-It is possible to avoid that if you create an interface configuration script for
-the ibX and bondX interfaces. To do that, you should use the standard syntax to
-create the bonding configuration (depending on your OS).
-
-3.1 Configuration with initscripts support
+2.1 Configuration with initscripts support
------------------------------------------
Note: This feature is available only for Redhat-AS4 (Update 4, Update 5,
Update 6 or Update 7) and for Redhat-EL5 and above.
-3.1.1 Writing network scripts under Redhat-AS4 (Update 4, 5, 6 or 7)
+2.1.1 Writing network scripts under Redhat-AS4 (Update 4, 5, 6 or 7)
-----------------------------------------------------------------
* In the master (bond) interface script add the line:
TYPE=Bonding
After the configuration is saved, restart the network service by running:
/etc/init.d/network restart
-3.1.2 Writing network scripts under Redhhat-EL5
+2.1.2 Writing network scripts under Redhhat-EL5
-----------------------------------------------
Follow the instructions in 3.1.1 (Writing network scripts under Redhat-AS4)
with the following changes:
You have to restart the network service in order to recreate the bonding
interface.
-3.2 Configuration with sysconfig support
+2.2 Configuration with sysconfig support
----------------------------------------
Note: This feature is available only for SLES-10 and above.
-3.2.1 Writing network scripts under SLES-10
+2.2.1 Writing network scripts under SLES-10
-----------------------------------------------
* In the master (bond) interface script add the lins:
After the configuration is saved, restart the network service by running:
/etc/init.d/network restart
-3.3 Configuring Ethernet slaves
+2.3 Configuring Ethernet slaves
-------------------------------
It is not possible to have a mix of Ethernt slaves and IPoIB slaves under the
same bonding master. It is possible however that a bonding master of Ethernet
Table of Contents
===============================================================================
1. Overview
-2. New Features
-3. Known Issues
-4. DHCP Support of IPoIB
-5. The ib-bonding driver
-6. Bug Fixes and Enhancements Since OFED 1.3
-7. Bug Fixes and Enhancements Since OFED 1.3.1
-8. Bug Fixes and Enhancements Since OFED 1.4
-9. Bug Fixes and Enhancements Since OFED 1.4.2
-10. Performance tuning
+2. Known Issues
+3. DHCP Support of IPoIB
+4. The ib-bonding driver
+5. Bug Fixes and Enhancements Since OFED 1.3
+6. Bug Fixes and Enhancements Since OFED 1.3.1
+7. Bug Fixes and Enhancements Since OFED 1.4
+8. Bug Fixes and Enhancements Since OFED 1.4.2
+9. Performance tuning
===============================================================================
1. Overview
the relevant IETF working group's RFCs (http://www.ietf.org).
-===============================================================================
-2. New Features
-===============================================================================
-1. This version of ofed introduces improvements to IPOIB by cutting the CPU
- overhead in handling receive packets. This will improve operation
- in datagram mode:
- Large Receive Offload (LRO) - aggregating multiple incoming packets from a
- single stream into a larger buffer before they are passed higher up the
- networking stack, thus reducing the number of packets that have to be
- processed.
- This feature is enabled on HCAs that can support LRO, e.g. ConnectX.
-2. Datagram mode: LSO (large send offload) allows the networking stack to pass
- SKBs with data size larger than the MTU to the IPoIB driver and have the HCA
- HW fragment the data to multiple MSS-sized packets. Add a device capability
- flag IB_DEVICE_UD_TSO for devices that can perform TCP segmentation offload,
- a new send work request opcode IB_WR_LSO, header, hlen and mss fields for
- the work request structure, and a new IB_WC_LSO completion type.
- This feature is enabled on HCAs that can support LSO, e.g. ConnectX.
-
-
Usage and configuration:
========================
1. To check the current mode used for outgoing connections, enter:
===============================================================================
-3. Known Issues
+2. Known Issues
===============================================================================
1. If a host has multiple interfaces and (a) each interface belongs to a
different IP subnet, (b) they all use the same InfiniBand Partition, and (c)
layer address at an offset of two bytes with respect to older kernels. This
causes the other host to misinterpret the hardware address resulting in failure
to resolve path which are based on wrong GIDs. As an example, RH 4.x and RH
- 5.x cannot interoperate.
+ 5.x cannot inter-operate.
8. In connected mode, TCP latency for short messages is larger by approx. 1usec
(~5%) than in datagram mode. As a workaround, use datagram mode.
kernel. To avoid this leave the IPoIB interfaces enslaved when unloading
ib_ipoib.
-17. On SLES 11, sysconfig scripts overide the interface mode and set it to
+17. On SLES 11, sysconfig scripts override the interface mode and set it to
datagram on each call to ifup, ifdown, etc. To avoid this, add the line
IPOIB_MODE=connected
to the interface configuration file (e.g. ifcfg-ib0)
kernel has all the functionality to support IPoIB slaves. This approach
however doesn't patch the sysconfig (SuSE) or initscripts (RedHat) package
so the network configuration scrips may not work properly.
- For example, if you install OFED on RHEL5.2 that runns kernel 2.6.30 and
+ For example, if you install OFED on RHEL5.2 that runs kernel 2.6.30 and
you try to configure and run bonding, you won't be able to restart the
network and see bond0 up and running with IPoIB slaves.
A workaround to this problem would be this
- a. Compile ib-bonding source rpm (under SRPMS directory) separately on a
- machine with RHEL5.2 and kernel 2.6.18-92.el5 (default for this OS)
- 2. Install the binary rpm while machine runs kernel 2.6.18-92.el5.
- This will patch the OS configuration scripts and install the bonding module.
- 3. Switch to kernel 2.6.30. The module that was compiled in (2) will not be
- loaded since it was compiled and installed for a different kernel.
- 4. Configure bonding and restart the network. The bonding interface
- should be up and running afterwards.
+ a. Compile ib-bonding source rpm (under SRPMS directory) separately on a
+ machine with RHEL5.2 and kernel 2.6.18-92.el5 (default for this OS)
+ 2. Install the binary rpm while machine runs kernel 2.6.18-92.el5.
+ This will patch the OS configuration scripts and install the bonding module.
+ 3. Switch to kernel 2.6.30. The module that was compiled in (2) will not be
+ loaded since it was compiled and installed for a different kernel.
+ 4. Configure bonding and restart the network. The bonding interface
+ should be up and running afterwards.
19. On RHEL5.X, '/etc/init.d/openibd start' prints the following messages while
bringing up IPoIB interfaces:
This does not affect IPoIB configuration and interfaces are configured as
expected.
+20. In IPoIB connected mode packages larger than 2016 bytes are not sent.
+ https://bugs.openfabrics.org/show_bug.cgi?id=1839
===============================================================================
-4. IPoIB Configuration Based on DHCP
+3. IPoIB Configuration Based on DHCP
===============================================================================
Setting an IPoIB interface configuration based on DHCP (v3.1.2 which is available
===============================================================================
-5. The ib-bonding driver
+4. The ib-bonding driver
===============================================================================
The ib-bonding driver is a High Availability solution for IPoIB interfaces.
It is based on the Linux Ethernet Bonding Driver and was adapted to work with
-IPoIB. The ib-bonding package contains a bonding driver and a utility called
-ib-bond to manage and control the driver operation.
-The ib-bonding driver comes with the ib-bonding package (run rpm -qi ib-bonding
-to get the package information).
+IPoIB. The ib-bonding driver comes with the ib-bonding package
+(run rpm -qi ib-bonding to get the package information).
Using the ib-bonding driver
---------------------------
-The ib-bonding driver can be loaded manually or automatically.
-
-1. Manual operation:
-Use the utility ib-bond to start, query, or stop the driver. For details on this
-utility, read the documentation for the ib-bonding package.
+The ib-bonding driver is loaded automatically.
-2. Automatic operation:
- Use standard OS tools (sysconfig in SuSE and initscripts in Redhat)
- to create a configuration that will come up with network restart. For details
- on this, read the documentation for the ib-bonding package.
+Automatic operation:
+Use standard OS tools (sysconfig in SuSE and initscripts in RedHat)
+to create a configuration that will come up with network restart. For details
+on this, read the documentation for the ib-bonding package.
Notes:
* Using /etc/infiniband/openib.conf to create a persistent configuration is
that is >= 2.6.27 (e.g. SLES11). The bonding driver that comes with those kernels
already supports enslaving of IPoIB interfaces. In addition, an OS can come
with an older kernel but with a patched bonding driver that also doesn't
- require modification (e.g. RHEL5.4). OFEED will not replace the bonding
+ require modification (e.g. RHEL5.4). OFED will not replace the bonding
module in such cases also.
However, there still might be a issue of OS configuration tools (like sysconfig or
initscripts) that needs a fix but such issues were not observed yet.
===============================================================================
-6. Bug Fixes and Enhancements Since OFED 1.3
+5. Bug Fixes and Enhancements Since OFED 1.3
===============================================================================
- There is no default configuration for IPoIB interfaces: One should manually
specify the full IP configuration or use the ofed_net.conf file. See
documentation for details)
===============================================================================
-7. Bug Fixes and Enhancements Since OFED 1.3.1
+6. Bug Fixes and Enhancements Since OFED 1.3.1
===============================================================================
- IPoIB: Refresh paths instead of flushing them on SM change events to improve
failover respond
- Bonding: Set default number of grat. ARP after failover to three (was one)
===============================================================================
-8. Bug Fixes and Enhancements Since OFED 1.4
+7. Bug Fixes and Enhancements Since OFED 1.4
===============================================================================
- Performance tuning is enabled by default for IPOIB CM.
- Clear IPOIB_FLAG_ADMIN_UP if ipoib_open fails
- fix unprotected use of priv->broadcast in ipoib_mcast_join_task.
- Do not join broadcast group if interface is brought down
-
+
===============================================================================
-9. Bug Fixes and Enhancements Since OFED 1.4.2
+8. Bug Fixes and Enhancements Since OFED 1.4.2
===============================================================================
- Check that the format of multicast link addresses is correct before taking
them from dev->mc_list to priv->multicast_list. This way we never try to
- send a bogus address to the SA, which prevents badness from erronous
+ send a bogus address to the SA, which prevents badness from erroneous
'ip maddr addr add', broken bonding drivers, etc. (bugzilla #1664)
- IPoIB: Don't turn on carrier for a non-active port.
If a bonding interface uses this IPoIB interface as a slave it might
not detect that this slave is almost useless and failover
functionality will be damaged. The fix checks the state of the IB
port in the carrier_task before calling netif_carrier_on(). (bugzilla #1726)
+- Clear ipoib_neigh.dgid in ipoib_neigh_alloc()
+ IPoIB can miss a change in destination GID under some conditions. The
+ problem is caused when ipoib_neigh->dgid contains a stale address.
+ The fix is to set ipoib_neigh->dgid to zero in ipoib_neigh_alloc().
===============================================================================
-10. Performance tuning
+9. Performance tuning
===============================================================================
When IPoIB is configured to run in connected mode, tcp parameter tuning is
performed at driver startup -- to improve the throughput of medium and large
Open Fabrics Enterprise Distribution (OFED)
- iSER initiator in OFED 1.4 Release Notes
+ iSER initiator in OFED 1.5 Release Notes
- December 2008
+ December 2009
* Background
iSER allows iSCSI to be layered over RDMA transports (including
InfiniBand and iWARP (RNIC)).
- The OpenFabrics iSER initiator implementation is interoperable with
+ The OpenFabrics iSER initiator implementation is inter-operable with
open-iscsi (http://www.open-iscsi.org/). It provides an alternative
transport to iscsi_tcp in the open-iscsi framework. The iSER transport
exposes a transport API to scsi_transport_iscsi, and a SCSI LLD API to
* Supported platforms
- SLES 10
- SLES 10 sp1
- SLES 10 sp2
- RHAS 4 up4
- RHAS 4 up5
- RHAS 4 up6
- RHAS 4 up7
- RHEL 5
- RHEL 5.1
- RHEL 5.2
-
- The release has been tested against Voltaire iSCSI over iSER target
- running in Voltaire's IB/Fibre-Channel router (SR4G) and the STGT
- target.
+ kernel.org: 2.6.30, 2.6.31 and 2.6.32
+
+ OFED-1.5 will not install iSER on other kernels and original iSER
+ module coming with Linux Distribution will stop working because of
+ mismatch in symbols version.
* Fixed Bugs and Enhancements since OFED 1.3
iSER:
- Add change_queue_depth method
- Fix list iteration bug
- Handle iser_device allocation error gracefully
- - Don't change ITT endianness
+ - Don't change ITT endianess
- Move high-volume debug output to higher debug level
- Count FMR alignment violations per session
Open-iSCSI:
* Known Issues
Open-iSCSI:
- - modifing node transport_name while session is active
+ - modifying node transport_name while session is active
will create stale session. It will be deleted only after reboot.
- - This issue is scheduled for OFED1.4 as part of new open-iscsi version.
* Installation/upgrade of open-iscsi
If iSER is selected to be installed with OFED, open-iscsi will be also
Open Fabrics Enterprise Distribution (OFED)
- STGT/iSER target in OFED 1.4 Release Notes
+ STGT/iSER target in OFED 1.5 Release Notes
- December 2008
+ December 2009
* Background
Open Fabrics Enterprise Distribution (OFED)
- ConnectX driver (mlx4) in OFED 1.5.0 Release Notes
-
- Decenber 2009
+ ConnectX driver (mlx4) in OFED 1.5 Release Notes
+
+ December 2009
===============================================================================
5. Infiniband (mlx4_ib) new features and bug fixes since OFED 1.4
6. Eth (mlx4_en) new features and bug fixes since OFED 1.4
7. New features and bug fixes since OFED 1.4.1
-8. New features and bug fixes since OFED 1.5.0
+8. New features and bug fixes since OFED 1.4.2
9. Known Issues
10. mlx4 Available Parameters
"ib" - Always Infiniband.
"auto" - Link sensing mode - detect port type based on the attached
network type. If no link is detected, the driver retries link
- sensing every few seconds.
+ sensing every few seconds.
Port link type can be configured for each device in the system at run time
using the "/sbin/connectx_port_config" script.
- Fixed recovery flows from memory allocation failures.
- When the system is low on memory, the mlx4_en driver now allocates smaller RX
rings.
-- The mlx4_coredriver now retries to obtain MSI-X vectors if the initial request is
+- The mlx4_core driver now retries to obtain MSI-X vectors if the initial request is
rejected by the OS
===============================================================================
options mlx4_en parameter=<value>
mlx4_core parameters:
- set_4k_mtu: attempt to set 4K MTU to all ConnectX ports (default 0)
- msi_x: attempt to use MSI-X if nonzero (default 1)
- enable_qos: Enable Quality of Service support in the HCA if > 0, (default 0)
- block_loopback Block multicast loopback packets if > 0 (default: 1)
- internal_err_reset: Reset device on internal errors if non-zero (default 1)
- debug_level: Enable debug tracing if > 0 (default 0)
- log_num_qp: log maximum number of QPs per HCA (default is 17; max is 20)
- log_num_srq: log maximum number of SRQs per HCA (default is 16; max is 20)
- log_rdmarc_per_qp: log number of RDMARC buffers per QP (default is 4; max is 7)
- log_num_cq: log maximum number of CQs per HCA (default is 16 max is 19)
- log_num_mcg: log maximum number of multicast groups per HCA (default is 13; max is 21)
- log_num_mpt: log maximum number of memory protection table entries per HCA
- (default is 17; max is 20)
- log_num_mtt: log maximum number of memory translation table segments per HCA
- (default is 20; max is 20)
- log_num_mac: log maximum number of MACs per ETH port (1-7) (int)
- log_num_vlan: log maximum number of VLANs per ETH port (0-7) (int)
- log_mtts_per_seg Log2 number of MTT entries per segment (1-5; default is 3)
- use_prio: Enable steering by VLAN priority on ETH ports (0/1, default 0) (bool)
+ set_4k_mtu: attempt to set 4K MTU to all ConnectX ports (int)
+ debug_level: Enable debug tracing if > 0 (int)
+ block_loopback: Block multicast loopback packets if > 0 (int)
+ msi_x: attempt to use MSI-X if nonzero (int)
+ log_num_mac: Log2 max number of MACs per ETH port (1-7) (int)
+ log_num_vlan: Log2 max number of VLANs per ETH port (0-7) (int)
+ use_prio: Enable steering by VLAN priority on ETH ports (0/1, default 0)
+ (bool)
+ log_num_qp: log maximum number of QPs per HCA (int)
+ log_num_srq: log maximum number of SRQs per HCA (int)
+ log_rdmarc_per_qp: log number of RDMARC buffers per QP (int)
+ log_num_cq: log maximum number of CQs per HCA (int)
+ log_num_mcg: log maximum number of multicast groups per HCA (int)
+ log_num_mpt: log maximum number of memory protection table entries per HCA
+ (int)
+ log_num_mtt: log maximum number of memory translation table segments per HCA
+ (int)
+ log_mtts_per_seg: Log2 number of MTT entries per segment (1-5) (int)
+ enable_qos: Enable Quality of Service support in the HCA (default: off)
+ (bool)
+ internal_err_reset: Reset device on internal errors if non-zero
+ (default 1) (int)
mlx4_ib parameters:
- debug_level: Enable debug tracing if > 0 (default 0)
+ debug_level: Enable debug tracing if > 0 (default 0)
mlx4_en parameters:
- rss_xor: Use XOR hash function for RSS 0 (default is xor)
- rss_mask: RSS hash type bitmask (default is 0xf)
- num_lro: Number of LRO sessions per ring or disabled (0) (default is 32)
- ip_reasm: Allow reassembly of fragmented IP packets (default is enabled)
- pfctx: Priority based Flow Control policy on TX[7:0].
- Per priority bit mask (default is 0)
- pfcrx: Priority based Flow Control policy on RX[7:0].
- Per priority bit mask (default is 0)
- inline_thold: threshold for using inline data (default is 128)
+ rss_xor: Use XOR hash function for RSS 0 (default is xor)
+ rss_mask: RSS hash type bitmask (default is 0xf)
+ num_lro: Number of LRO sessions per ring or disabled (0) (default is 32)
+ ip_reasm: Allow reassembly of fragmented IP packets (default is enabled)
+ pfctx: Priority based Flow Control policy on TX[7:0].
+ Per priority bit mask (default is 0)
+ pfcrx: Priority based Flow Control policy on RX[7:0].
+ Per priority bit mask (default is 0)
+ inline_thold: threshold for using inline data (default is 128)
MPI Selector 1.0 release notes
- December 2008
+ December 2009
==============================
OFED contains a simple mechanism for system administrators and end
--------------------------------------------
*******************************************************************************
-RELEASE: OFED 1.4
-DATE: Dec 2008
+RELEASE: OFED 1.5
+DATE: Dec 2009
===============================================================================
Table of Contents
for later analysis by Mellanox.
e) mstvpd
This utility dumps the on-card VPD.
- f) hca_self_test.ofed
- This scripts checks the status of software, firmware and hardware
- of the HCAs installed on the local host.
+ f) hca_self_test.ofed
+ This scripts checks the status of software, firmware and hardware
+ of the HCAs installed on the local host.
===============================================================================
2. New Features
Open Fabrics Enterprise Distribution (OFED)
- mthca in OFED 1.4.1 Release Notes
+ mthca in OFED 1.5 Release Notes
- May 2009
+ December 2009
===============================================================================
===============================================================================
1. Overview
===============================================================================
-mthca is the low level driver implementation for the following Mellanox Techno-
-logies HCAs: InfiniHost, InfiniHost III Ex and InfiniHost III Lx.
+mthca is the low level driver implementation for the following Mellanox
+Technologies HCAs: InfiniHost, InfiniHost III Ex and InfiniHost III Lx.
mthca Available Parameters
--------------------------
options ib_mthca parameter=<value>
mthca parameters:
-
- tune_pci - increase PCI burst from the default set by BIOS if
- nonzero
- msi - attempt to use MSI if nonzero
- msi_x - attempt to use MSI-X if nonzero
- fw_cmd_doorbell - post firmware commands through doorbell page if non-
- zero (and supported by firmware)
- catas_reset_disable - disable device reset on a catastrophic event if non-
- zero
- debug_level - Enable debug tracing if > 0 (int)
- num_qp - maximum number of QPs per HCA (int)
- rdb_per_qp - number of RDB buffers per QP (int)
- num_cq - maximum number of CQs per HCA (int)
- num_mcg - maximum number of multicast groups per HCA (int)
- num_mpt - maximum number of memory protection table entries
- per HCA (int)
- num_mtt - maximum number of memory translation table segments
- per HCA (int)
- num_udav - maximum number of UD address vectors per HCA (int)
- fmr_reserved_mtts - number of memory translation table segments reserved
- for FMR (int)
- log_mtts_per_seg - log2 number of MTT entries per segment (1-5)
+ catas_reset_disable: disable reset on catastrophic event if nonzero
+ (int)
+ fw_cmd_doorbell: post FW commands through doorbell page if
+ nonzero (and supported by FW) (int)
+ debug_level: Enable debug tracing if > 0 (int)
+ msi_x: attempt to use MSI-X if nonzero (int)
+ tune_pci: increase PCI burst from the default set by BIOS if nonzero (int)
+ num_qp: maximum number of QPs per HCA (int)
+ rdb_per_qp: number of RDB buffers per QP (int)
+ num_cq: maximum number of CQs per HCA (int)
+ num_mcg: maximum number of multicast groups per HCA (int)
+ num_mpt: maximum number of memory protection table entries per HCA (int)
+ num_mtt: maximum number of memory translation table segments per HCA (int)
+ num_udav: maximum number of UD address vectors per HCA (int)
+ fmr_reserved_mtts: number of memory translation table segments reserved for
+ FMR (int)
+ log_mtts_per_seg: Log2 number of MTT entries per segment (1-5) (int)
===============================================================================
2. Fixed Bugs
===============================================================================
-- IB_EVENT_LID_CHANGE is generated more appropriately.
-- Improved MTT buddy allocator (free count per order).
-- Fix check of max_send_sge for special QPs.
-- RESET->ERR QP state transition no longer supported (IB Spec 1.2.1).
-- Clear ICM pages before handing to FW.
-- Fixed race condition between create QP and destroy QP (bugzilla 1389)
+- Fix access to freed memory in catastrophic processing
+ catas_reset() uses pointer to mthca_dev, but mthca_dev is not valid after
+ call __mthca_restart_one().
===============================================================================
solve an issue when had more than 2^18 max qp's configured.
===============================================================================
-3. Known Issues
+4. Known Issues
===============================================================================
1. A UAR size other than 8MB prevents mthca driver loading. The default UAR
size is 8MB. If the size is changed, the following error message will be
Open Fabrics Enterprise Distribution (OFED)
- OSU MPI MVAPICH-1.1.0, in OFED 1.4.r10 Release Notes
+ OSU MPI MVAPICH-1.2.0, in OFED 1.5 Release Notes
- May 2009
+ December 2009
===============================================================================
===============================================================================
1. Overview
===============================================================================
-These are the release notes for OSU MPI MVAPICH-1.1.0.
+These are the release notes for OSU MPI MVAPICH-1.2.0.
OSU MPI is an MPI channel implementation over InfiniBand
by Ohio State University (OSU).
===============================================================================
-3. New Features ( Compared to mvapich 1.0.0 )
+3. New Features ( Compared to mvapich 1.1.0 )
===============================================================================
-MVAPICH-1.1.0 has the following additional features:
-- eXtended Reliable Connection (XRC) support
-- Lock-free design to provide support for asynchronous
- progress at both sender and receiver to overlap
- computation and communication
-- Optimized MPI_allgather collective
+MVAPICH-1.2.0 has the following additional features:
+- Advanced network recovery support
+- mpirun launcher improvements
- Efficient intra-node shared memory communication
support for diskless clusters
-- Enhanced Totalview Support with the new mpirun_rsh framework
+- RoCEE (RDMAoE) networks support
===============================================================================
-4. Bug Fixes ( Compared to mvapich 1.0.0 )
+4. Bug Fixes ( Compared to mvapich 1.1.0 )
===============================================================================
-- De-register stale memory regions earlier to prevent
- excess allocations of physical memory
-- Fixes for MPI_Query_thread and MPI_Is_thread_main
-- Fixes for PGI compiler support
-- Compilation warnings cleanup
-- Fixes for optimized colletives
-- Fix data types for memory allocations
- Multiple fixes for mpirun_rsh launcher
===============================================================================
fw-23108 version that is earlier than 3.4.000, or with an fw-25208 version
4.7.400 or earlier.
- NOTE: There is no issue if you chose to update firmware during Mellanox
- OFED installation as newer firmware versions were burnt.
+ NOTE: There is no issue if you chose to update firmware during Mellanox
+ OFED installation as newer firmware versions were burnt.
Workaround:
Option 1 - Update the firmware. For instructions, see Mellanox Firmware Tools
- (MFT) User's Manual under the docs/ folder.
+ (MFT) User's Manual under the docs/ folder.
Option 2 - In mvapich.conf, set VIADEV_SRQ_ENABLE=0
- MVAPICH may fail to run on some SLES 10 machines due to problems in resolving
-prefix=/usr
core=y
mthca=y
mlx4=y
mlx4_en=y
cxgb3=y
nes=y
-ipath=y
ipoib=y
sdp=y
srp=y
libibverbs-devel=y
libibverbs-devel-static=y
libibverbs-utils=y
+libibverbs-debuginfo=y
libmthca=y
libmthca-devel-static=y
+libmthca-debuginfo=y
libmlx4=y
libmlx4-devel=y
+libmlx4-debuginfo=y
libcxgb3=y
libcxgb3-devel=y
+libcxgb3-debuginfo=y
libnes=y
libnes-devel-static=y
-libipathverbs=y
-libipathverbs-devel=y
+libnes-debuginfo=y
libibcm=y
libibcm-devel=y
-libibcommon=y
-libibcommon-devel=y
-libibcommon-static=y
+libibcm-debuginfo=y
libibumad=y
libibumad-devel=y
libibumad-static=y
+libibumad-debuginfo=y
libibmad=y
libibmad-devel=y
libibmad-static=y
+libibmad-debuginfo=y
ibsim=y
+ibsim-debuginfo=y
librdmacm=y
librdmacm-utils=y
librdmacm-devel=y
+librdmacm-debuginfo=y
libsdp=y
libsdp-devel=y
+libsdp-debuginfo=y
opensm=y
opensm-libs=y
opensm-devel=y
+opensm-debuginfo=y
opensm-static=y
compat-dapl=y
compat-dapl-devel=y
dapl-devel=y
dapl-devel-static=y
dapl-utils=y
+dapl-debuginfo=y
perftest=y
mstflint=y
tvflash=y
ibutils=y
infiniband-diags=y
qperf=y
+qperf-debuginfo=y
ofed-docs=y
ofed-scripts=y
mpi-selector=y
mpitests_mvapich_gcc=y
mpitests_mvapich2_gcc=y
mpitests_openmpi_gcc=y
-mvapich2_conf_impl=ofa
-mvapich2_conf_romio=1
-mvapich2_conf_shared_libs=1
-mvapich2_conf_ckpt=0
-mvapich2_conf_vcluster=small
-mvapich2_conf_dapl_provider=ib0
Open Fabrics Enterprise Distribution (OFED)
Open MPI in OFED 1.5 Copyrights, License, and Release Notes
-
+
December 2009
Open MPI Copyrights
Version: OpenSM 3.2.x
Repo: git://git.openfabrics.org/~sashak/management.git
-Date: May 2009
+Date: December 2009
1 Overview
----------
Distribution
- Open Fabrics Enterprise Distribution (OFED) 1.4, December 2008
+ Open Fabrics Enterprise Distribution (OFED) 1.5, December 2009
Summary
qperf - Measure RDMA and IP performance
Open Fabrics Enterprise Distribution (OFED)
- RDMA CM in OFED 1.4 Release Notes
+ RDMA CM in OFED 1.5 Release Notes
- December 2008
+ December 2009
===============================================================================
2. New Features
3. Known Issues
4. Fixed bugs since OFED 1.3
+5. Fixed bugs since OFED 1.4.2
===============================================================================
1. Overview
- The reject status fix has been inserted for DAPL.
===============================================================================
-5. Fixed bugs since OFED 1.3.1
+5. Fixed bugs since OFED 1.4.2
===============================================================================
-- Non
+- examples: print errno on failures
+- return errors from the library consistently
+- initialize correct pthread condition
+- fix race initializing library
Open Fabrics Enterprise Distribution (OFED)
- RDS in OFED 1.4.1 Release Notes
- May 2009
+ RDS in OFED 1.5 Release Notes
+ December 2009
===============================================================================
2. supported platforms
===============================================================================
- RHEL4.0 Update 4,5,6
- RHEL5.0 Update 1,2
- SLES 10
+ RHEL5.0 Update 3,4
+ SLES 10 SP2,3
+ SLES 11
+ kernel.org 2.6.29, 2.6.30, 2.6.31 and 2.6.32
===============================================================================
3. Installation & Configuration
4. New Features
===============================================================================
-RDS protocol version 3.1.
-
-RDS v3.1 is backwards compatible with v3.0 via protocol negotiation.
-
-Support for iWARP (bcopy mode only).
-
-Locking and scalability improvements.
-
-Credit-based flow control for iWARP transport.
-
-TCP transport removed.
+Added TCP transport.
===============================================================================
5. Bug fixes and Enhancements since OFED 1.3.1
- Open Fabrics Enterprise Distribution (OFED)
- SRP in OFED 1.4 Release Notes
-
- December 2008
+ Open Fabrics Enterprise Distribution (OFED)
+ SRP in OFED 1.5 Release Notes
+
+ December 2009
==============================================================================
> max_cmd_per_lun - Default: 63
> max_sect (short for max_sectors) - sets the request size of a command
> io_class - Default: 0x100 as in rev 16A of the specification
- Note: In rev 10 the default was 0xff00
+ Note: In rev 10 the default was 0xff00
> initiator_ext - Please refer to Section 9 (Multiple Connections...)
d. See SRP Tools below for instructions on how the parameters in the
echo command above may be obtained.
Initialization: (Execute after each boot of the driver)
1) Execute modprobe dm-multipath
2) Execute modprobe ib-srp
- 3) Make sure you have created file /etc/udev/rules.d/91-srp.rules
- as described above
+ 3) Make sure you have created file /etc/udev/rules.d/91-srp.rules
+ as described above
4) Execute for each port and each HCA:
srp_daemon -c -e -R 300 -i <InfiniBand HCA name> -p <port number>
(You can use another value for -R. See under the Known Issues section
- the workaround for the rare race condition.)
+ the workaround for the rare race condition.)
- This step can be performed by executing srp_daemon.sh, which sends
+ This step can be performed by executing srp_daemon.sh, which sends
its log to /var/log/srp_daemon.log.
- Now it is possible to access the SRP LUNs on /dev/mapper/.
+ Now it is possible to access the SRP LUNs on /dev/mapper/.
- NOTE: It is possible for regular (non-SRP) LUNs to also be present;
- the SRP LUNs may be identified by their names. You can configure the
+ NOTE: It is possible for regular (non-SRP) LUNs to also be present;
+ the SRP LUNs may be identified by their names. You can configure the
/etc/multipath.conf file to change multipath behavior.
echo -n \
id_ext=0000000000000001,ioc_guid=00066a0138000165,\
- dgid=fe8000000000000000066a0260000165,pkey=ffff,\
- service_id=0000494353535250,io_class=ff00,\
+ dgid=fe8000000000000000066a0260000165,pkey=ffff,\
+ service_id=0000494353535250,io_class=ff00,\
initiator_ext=0000000000000001 > \
/sys/class/inifiniband_srp/srp-mthca0-1/add_target
projects / ~aditr / compat-rdma-docs.git / commitdiff