From 578723cbd8a9f6899d7338ecaea472948fd2652b Mon Sep 17 00:00:00 2001 From: Vladimir Sokolovsky Date: Tue, 29 Dec 2009 19:42:45 +0200 Subject: [PATCH] Updated documentation Signed-off-by: Vladimir Sokolovsky --- MLNX_EN_README.txt | 2 +- MPI_README.txt | 58 ++++++++--------- MSTFLINT_README.txt | 56 ++++++++--------- OFED_Installation_Guide.txt | 9 +-- OFED_release_notes.txt | 112 ++++++++++++++++++--------------- OFED_tips.txt | 31 ++++----- PERF_TEST_README.txt | 4 +- README.txt | 6 +- SRPT_README.txt | 2 +- create_Module.symvers.sh | 39 +++--------- diags_release_notes.txt | 8 +-- ib-bonding.txt | 73 +++++---------------- ipoib_release_notes.txt | 107 +++++++++++++------------------ iser_release_notes.txt | 30 +++------ iser_target_release_notes.txt | 4 +- mlx4_release_notes.txt | 73 ++++++++++----------- mpi-selector_release_notes.txt | 2 +- mstflint_release_notes.txt | 10 +-- mthca_release_notes.txt | 58 ++++++++--------- mvapich_release_notes.txt | 34 ++++------ ofed.conf-example | 27 ++++---- open_mpi_release_notes.txt | 2 +- opensm_release_notes.txt | 2 +- qperf_release_notes.txt | 2 +- rdma_cm_release_notes.txt | 12 ++-- rds_release_notes.txt | 23 +++---- srp_release_notes.txt | 28 ++++----- 27 files changed, 360 insertions(+), 454 deletions(-) diff --git a/MLNX_EN_README.txt b/MLNX_EN_README.txt index bca7586..9220051 100644 --- a/MLNX_EN_README.txt +++ b/MLNX_EN_README.txt @@ -1,6 +1,6 @@ =============================================================================== MLNX_EN driver for Mellanox Adapter Cards with 10GigE Support - README for MLNX_OFED 1.5 + README for OFED 1.5 December 2009 =============================================================================== diff --git a/MPI_README.txt b/MPI_README.txt index 4833f23..fa82cf4 100644 --- a/MPI_README.txt +++ b/MPI_README.txt @@ -1,7 +1,7 @@ - Open Fabrics Enterprise Distribution (OFED) - MPI in OFED 1.5 README - December 2009 + MPI in OFED 1.5 README + + December 2009 =============================================================================== @@ -16,10 +16,10 @@ Table of Contents =============================================================================== 1. Overview =============================================================================== -Three MPI stacks are included in this release of OFED: -- MVAPICH 1.1.0-3143 +Open Fabrics Enterprise Distribution (OFED)Three MPI stacks are included in this release of OFED: +- MVAPICH 1.2.0 - Open MPI 1.4 -- MVAPICH2 1.2p1 +- MVAPICH2 1.4 Setup, compilation and run information of MVAPICH, Open MPI and MVAPICH2 is provided below in sections 2, 3 and 4 respectively. @@ -72,7 +72,7 @@ software package (which is not included in the OFED software package; see http://modules.sourceforge.net/ for details). Note that the site-wide default is set in a file that is typically not -on a networked filesystem, and is therefore specific to the host on +on a networked file system, and is therefore specific to the host on which it was run. As such, it is recommended to run the mpi-selector-menu command on all hosts in a cluster, picking the same default MPI implementation on each. It may be more convenient, @@ -83,7 +83,7 @@ automated environments. See the mpi-selector(1) manual page for more details. Additionally, per-user defaults are set in a file in the user's $HOME -directory. If this directory is not on a network-shared filesystem +directory. If this directory is not on a network-shared file system between all hosts that will be used for MPI applications, then it also needs to be propagated to all relevant hosts. @@ -96,7 +96,7 @@ convenient set of command line tools and menus. 1.4 Updating MPI Installations ------------------------------ Note that all of the MPI implementations included in the OFED software -package are the versions that were available when OFED v1.4 was +package are the versions that were available when OFED v1.5 was released. They have been QA tested with this version of OFED and are fully supported. @@ -106,7 +106,7 @@ been successfully installed. There is nothing specific about the OFED-included MPI software packages that prohibit installing newer/other MPI implementations. -It should be also noted that versions of MPI released after OFED v1.4 +It should be also noted that versions of MPI released after OFED v1.5 are not supported by OFED. But since each MPI has its own release schedule and QA process (each of which involves testing with the OFED stack), it may sometimes be desirable -- or even advisable, depending @@ -123,7 +123,7 @@ The web sites of each MPI implementation are listed below: 2. MVAPICH MPI =============================================================================== -This package is a 1.1.0 version of the MVAPICH software package, +This package is a 1.2.0 version of the MVAPICH software package, and is the officially supported MPI stack for this release of OFED. See http://mvapich.cse.ohio-state.edu for more details. @@ -173,20 +173,20 @@ rsh, add to the mpirun_rsh command the parameter: -rsh *** Running OSU tests *** -/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/osu_benchmarks-3.0/osu_bw -/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/osu_benchmarks-3.0/osu_latency -/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/osu_benchmarks-3.0/osu_bibw -/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/osu_benchmarks-3.0/osu_bcast +/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/osu_benchmarks-3.1.1/osu_bw +/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/osu_benchmarks-3.1.1/osu_latency +/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/osu_benchmarks-3.1.1/osu_bibw +/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/osu_benchmarks-3.1.1/osu_bcast *** Running Intel MPI Benchmark test (Full test) *** -/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/IMB-3.1/IMB-MPI1 +/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/IMB-3.2/IMB-MPI1 *** Running Presta test *** -/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/presta-1.4.0/com -o 100 -/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/presta-1.4.0/glob -o 100 -/usr/mpi/gcc/mvapich-1.1.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.1.0/tests/presta-1.4.0/globalop +/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/presta-1.4.0/com -o 100 +/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/presta-1.4.0/glob -o 100 +/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/presta-1.4.0/globalop =============================================================================== @@ -343,12 +343,12 @@ http://www.open-mpi.org/faq/). Example 1: Running the OSU bandwidth: - > cd /usr/mpi/gcc/openmpi-1.4/tests/osu_benchmarks-3.0 + > cd /usr/mpi/gcc/openmpi-1.4/tests/osu_benchmarks-3.1.1 > mpirun -np -hostfile osu_bw Example 2: Running the Intel MPI Benchmark benchmarks: - > cd /usr/mpi/gcc/openmpi-1.4/tests/IMB-3.1 + > cd /usr/mpi/gcc/openmpi-1.4/tests/IMB-3.2 > mpirun -np -hostfile IMB-MPI1 --> Note that the version of IMB-EXT that ships in this version of @@ -517,7 +517,7 @@ on mpirun_rsh. To launch a MPI job with mpirun_rsh, password-less ssh needs to be enabled across all nodes. Note: ssh will be used by default. In order to use rsh, use the -rsh option on -the mpirun_rsh commandline. For more options, see mpirun_rsh -help or the +the mpirun_rsh command line. For more options, see mpirun_rsh -help or the MVAPICH2 user guide. *** Running 4 processes on 4 nodes *** @@ -531,14 +531,14 @@ $ mpirun_rsh -np 4 -hostfile hostfile /path/to/my_mpi_app *** Running OSU tests *** -/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.0/osu_bw -/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.0/osu_latency -/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.0/osu_bibw -/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.0/osu_bcast +/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.1.1/osu_bw +/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.1.1/osu_latency +/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.1.1/osu_bibw +/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.1.1/osu_bcast *** Running Intel MPI Benchmark test (Full test) *** -/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/IMB-3.1/IMB-MPI1 +/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/IMB-3.2/IMB-MPI1 *** Running Presta test *** @@ -582,7 +582,7 @@ file has been created in the specific directory with two hosts. OSU Tests Example: -$ cd /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.0 +$ cd /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.1.1 $ mpdboot -n 2 -f ./hosts $ mpiexec -n 2 ./osu_bcast $ mpiexec -n 2 ./osu_bibw @@ -592,7 +592,7 @@ $ mpdallexit Intel MPI Benchmark Example: -$ cd /usr/mpi/gcc/mvapich2-1.2p1/tests/IMB-3.1 +$ cd /usr/mpi/gcc/mvapich2-1.2p1/tests/IMB-3.2 $ mpdboot -n 2 -f ./hosts $ mpiexec -n 2 ./IMB-MPI1 $ mpdallexit diff --git a/MSTFLINT_README.txt b/MSTFLINT_README.txt index 886cdaf..75bdb53 100644 --- a/MSTFLINT_README.txt +++ b/MSTFLINT_README.txt @@ -31,7 +31,7 @@ MSTFLINT Package - Firmware Burning and Diagnostics Tools 3) Installation a) Build the mstflint utility. This package is built using a standard autotools method. - + Example: > ./configure > make @@ -51,7 +51,7 @@ MSTFLINT Package - Firmware Burning and Diagnostics Tools # List all Mellanox devices > /sbin/lspci -d 15b3: 02:00.0 Ethernet controller: Mellanox Technologies Unknown device 6368 (rev a0) - + # Use mstflint tool to query the firmware on this device > mstflint -d 02:00.0 q @@ -78,12 +78,12 @@ MSTFLINT Package - Firmware Burning and Diagnostics Tools # List all Mellanox devices > /sbin/lspci -d 15b3: 02:00.0 Ethernet controller: Mellanox Technologies Unknown device 6368 (rev a0) - + # Use mstregdump to dump HW registers, using PCI config cycles > mstregdump /proc/bus/pci/02/00.0 > crdump.log Note: Typically, you will need root privileges for hardware access - + 5) Usage (mstflint): Read mstflint usage. Enter "./mstflint -h" for a short help message, or "./mstflint -hh" for a detailed help message. @@ -107,33 +107,33 @@ MSTFLINT Package - Firmware Burning and Diagnostics Tools > mstregdump mthca0 > dumpfile 7) Usage (mstvpd): - A VPD dump is displayed to the standard output. - A list of keywords to dump can be supplied after the -- flag - to apply an output filter. - - Examples: - > mstvpd mthca0 - ID: Lion cub DDR - PN: MHGA28-1T - EC: A3 - SN: MT0551X00740 - V0: PCIe x8 - V1: N/A - YA: R R - RW: - - > mstvpd mthca0 -- PN ID - PN: MHGA28-1T - ID: Lion cub DDR + A VPD dump is displayed to the standard output. + A list of keywords to dump can be supplied after the -- flag + to apply an output filter. + + Examples: + > mstvpd mthca0 + ID: Lion cub DDR + PN: MHGA28-1T + EC: A3 + SN: MT0551X00740 + V0: PCIe x8 + V1: N/A + YA: R R + RW: + + > mstvpd mthca0 -- PN ID + PN: MHGA28-1T + ID: Lion cub DDR 8) Problem Reporting: - Please collect the following information when reporting issues: + Please collect the following information when reporting issues: - uname -a - cat /etc/issue - cat /proc/bus/pci/devices - mstflint -vv - lspci + uname -a + cat /etc/issue + cat /proc/bus/pci/devices + mstflint -vv + lspci mstflint -d 02:00.0 v mstflint -d 02:00.0 q mstvpd 02:00.0 diff --git a/OFED_Installation_Guide.txt b/OFED_Installation_Guide.txt index 9b4689a..0c3e0d3 100644 --- a/OFED_Installation_Guide.txt +++ b/OFED_Installation_Guide.txt @@ -52,11 +52,11 @@ General Notes: The OFED Distribution package generates RPMs for installing the following: o OpenFabrics core and ULPs: - - HCA drivers (mthca, mlx4, ipath, ehca) + - HCA drivers (mthca, mlx4, qib, ehca) - iWARP driver (cxgb3, nes) - core - Upper Layer Protocols: IPoIB, SDP, SRP Initiator and target, iSER - Initiator and target, RDS, qlgc_vnic, uDAPL and NFS-RDMA + Initiator and target, RDS, qlgc_vnic, uDAPL and NFS-RDMA o OpenFabrics utilities - OpenSM: InfiniBand Subnet Manager - Diagnostic tools @@ -162,8 +162,8 @@ Note: The installer will warn you if you attempt to compile any of the installation parameters can be found by running /etc/infiniband/info. - Information on the driver version and source git trees can be found - using the ofed_info utility + Information on the driver version and source git trees can be found + using the ofed_info utility During the interactive installation of OFED, two files are @@ -202,6 +202,7 @@ Note: The installer will warn you if you attempt to compile any of the --without-depcheck Skip Distro's libraries check -v|-vv|-vvv. Set verbosity level -q. Set quiet - no messages will be printed + --force Force uninstall RPM coming with Distribution --all|--hpc|--basic Install all,hpc or basic packages correspondingly diff --git a/OFED_release_notes.txt b/OFED_release_notes.txt index 5fd6110..9deb640 100644 --- a/OFED_release_notes.txt +++ b/OFED_release_notes.txt @@ -1,20 +1,20 @@ Open Fabrics Enterprise Distribution (OFED) - Version 1.5-rc6 - Release Notes - December 2009 + Version 1.5 + Release Notes + December 2009 =============================================================================== Table of Contents =============================================================================== 1. Overview, which includes: - - OFED Distribution Rev 1.4 Contents + - OFED Distribution Rev 1.5 Contents - Supported Platforms and Operating Systems - Supported HCA and RNIC Adapter Cards and Firmware Versions - Tested Switch Platforms - Third party Test Packages - OFED sources -4. Main Changes from OFED 1.4.1 +4. Main Changes from OFED 1.4.2 5. Known Issues @@ -34,29 +34,29 @@ all of its nodes to this new version. ----------------------- The OFED package contains the following components: - OpenFabrics core and ULPs: - - IB HCA drivers (mthca, mlx4, qib, ehca) + - IB HCA drivers (mthca, mlx4, qib, ehca) - iWARP RNIC driver (cxgb3, nes) - - core - - Upper Layer Protocols: IPoIB, SDP, SRP Initiator and target, iSER + - core + - Upper Layer Protocols: IPoIB, SDP, SRP Initiator and target, iSER Initiator and target, RDS, uDAPL, qlgc_vnic and NFS-RDMA. - OpenFabrics utilities: - - OpenSM (OSM): InfiniBand Subnet Manager - - Diagnostic tools - - Performance tests + - OpenSM (OSM): InfiniBand Subnet Manager + - Diagnostic tools + - Performance tests - MPI: - - OSU MPI stack supporting the InfiniBand and iWARP interface - - Open MPI stack supporting the InfiniBand and iWARP interface + - OSU MPI stack supporting the InfiniBand and iWARP interface + - Open MPI stack supporting the InfiniBand and iWARP interface - OSU MVAPICH2 stack supporting the InfiniBand and iWARP interface - - MPI benchmark tests (OSU benchmarks, Intel MPI benchmarks, Presta) + - MPI benchmark tests (OSU benchmarks, Intel MPI benchmarks, Presta) - Extra packages: - - open-iscsi: open-iscsi initiator with iSER support + - open-iscsi: open-iscsi initiator with iSER support - ib-bonding: Bonding driver for IPoIB interface - Sources of all software modules (under conditions mentioned in the modules' - LICENSE files) + LICENSE files) - Documentation Notes: -1. iSER target is of Beta quality. +1. iSER target and NFS-RDMA is of Beta quality. 2. All other OFED components are of production quality. 3. See release notes for each package in the docs directory. 4. Any Topspin copyright belongs to Cisco Systems, Inc. @@ -70,20 +70,20 @@ Notes: - ia64 o Linux Operating Systems: - - RedHat EL4 up7: 2.6.9-78.ELsmp - - RedHat EL4 up8: 2.6.9-89.ELsmp - - RedHat EL5 up3: 2.6.18-128.el5 - - RedHat EL5 up4: 2.6.18-164.el5 - - SLES10 SP2: 2.6.16.60-0.21-smp - - SLES10 SP3: 2.6.16.60-0.54-smp - - SLES11: 2.6.27.19-5-default + - RedHat EL4 up7 2.6.9-78.ELsmp + - RedHat EL4 up8 2.6.9-89.ELsmp + - RedHat EL5 up3 2.6.18-128.el5 + - RedHat EL5 up4 2.6.18-164.el5 + - SLES10 SP2 2.6.16.60-0.21-smp + - SLES10 SP3 2.6.16.60-0.54-smp + - SLES11 2.6.27.19-5-default - OEL 4 up7 2.6.9-78.ELsmp - OEL 4 up8 2.6.9-89.ELsmp - CentOS5.3 2.6.18-128.el5 - CentOS5.4 2.6.18-164.el5 - Fedora Core12 2.6.31.5-127.fc12 * - OpenSuSE 11.2 2.6.31.5-0.1-default * - - kernel.org: 2.6.29, 2.6.30, + - kernel.org 2.6.29, 2.6.30, 2.6.31 and 2.6.32 * * Minimal QA for these versions @@ -98,7 +98,7 @@ well as iWARP RNICs by Chelsio Communications and Intel. - InfiniHost III Ex (MemFree: fw-25218 Rev 5.3.000 with memory: fw-25208 Rev 4.8.200) - InfiniHost III Lx (fw-25204 Rev 1.2.000) - - ConnectX IB (fw-25408 Rev 2.6.000) + - ConnectX IB (fw-25408 Rev 2.7.000) For official firmware versions please see: http://www.mellanox.com/content/pages.php?pg=firmware_download @@ -120,7 +120,7 @@ well as iWARP RNICs by Chelsio Communications and Intel. - R310/R320 10GbE iWARP Adapters o Intel RNICs: - - NE020 10Gb iWARP Adapter + - NE020 10Gb iWARP Adapter 1.4 Switches Supported @@ -136,40 +136,48 @@ companies: 1.5 Third Party Packages ------------------------ The following third party packages have been tested with OFED 1.5: -1. Intel MPI, Version 3.0 - Package ID: l_mpi_p_3.0.043 -2. HP-MPI 2.2.5.1 (without XRC support) -3. HP-MPI 2.2.7 (with XRC support) +- Intel MPI, Version 3.2.2 +- Intel MPI, Version 4.0 beta 1.6 OFED Sources ---------------- All sources are located under git://git.openfabrics.org/ Kernel sources: git://git.openfabrics.org/ofed_1_5/linux-2.6.git ofed_kernel -User level Sources are located in all git trees as written in the BUILD_ID +User level Sources are downloaded from http://www.openfabrics.org/downloads/ +as written in the BUILD_ID The kernel sources are based on Linux 2.6.30 mainline kernel. Its patches are included in the OFED sources directory. For details see HOWTO.build_ofed. =============================================================================== -2. Main Changes from OFED 1.4.1 +2. Main Changes from OFED 1.4.2 =============================================================================== Note: For details regarding the various changes, please see the release notes for each package in the docs directory. 2.1 General changes o Kernel code based on 2.6.30 + o libraries location - all userspace libraries can be downloaded from + http://www.openfabrics.org/downloads/ + See BUILD_ID for exact location + o Qlogic replaced low level driver from ipath to qib. 2.2 SDP o Performance improvements o Zero copy in beta level 2.3 uDAPL - o ??? + o New UCM provider (ofa-v2-mlx4_0-1u) with IB UD based CM per process. More + scalable then rdma_cm (cma) or socket cm (scm). + o Common code base with WinOF 2.1. + o Bug fixes. + 2.4 perftest - o Renamed tests: - ib_rdma_bw -> rdma_bw - ib_rdma_lat -> rdma_lat + o Renamed tests: + ib_rdma_bw -> rdma_bw + ib_rdma_lat -> rdma_lat 2.5 Management o OpenSM @@ -181,18 +189,25 @@ for each package in the docs directory. o Diagnostic tools: - 2.6 MPI: + 2.6 MPI: a. OSU MVAPICH 1.2.0 - b. Open MPI 1.3.4 + b. Open MPI 1.4 c. OSU MVAPICH2 1.4 d. MPI tests 3.2 + 2.7 iSER: + o Available only on kernel.org 2.6.30, 2.6.31 and 2.6.32 + + 2.8 NFS-RDMA + o Added support for RHEL5.4, SLES10 SP3, kernel.org 2.6.25 and 2.6.30. + kernels 2.6.26 and 2.6.27 are not supported + o NFS-RDMA is in beta level. Will be GA in OFED-1.5.1 =============================================================================== 3. Known Issues =============================================================================== The following is a list of general limitations and known issues of the various -components of the OFED 1.4 release. +components of the OFED 1.5 release. 1. When upgrading from an earlier OFED version, the installation script does not stop the earlier OFED version prior to uninstalling it. @@ -210,24 +225,23 @@ components of the OFED 1.4 release. 4. The qib driver is supported only on 64-bit platforms. 5. When installing OFED on OpenSuse or Ubuntu one should use the --without-depcheck option of the install.pl script -6. To install OFED 1.4 on Fedora Core 8 one should: - 1. Install libtool RPM (required by libibcommon) - 2. Install tcsh RPM (required by mpi-selector) - 3. Create the file '.rpmmacros' (required by mvapich): - echo "%__arch_install_post %{nil}" >> /root/.rpmmacros -7. IPoIB: brctl utilities do not work on IPoIB interfaces. The reason for that +6. IPoIB: brctl utilities do not work on IPoIB interfaces. The reason for that is that these utilities support devices of type Ethernet only. -8. "openibd stop" can sometime fail with the error: +7. "openibd stop" can sometime fail with the error: Unloading ib_cm [FAILED] ERROR: Module ib_cm is in use by ib_ipoib Workaround: run "openibd stop" again. -9. When working with ISCSI over IPoIB or mlx4_en, you must disable LRO (even +8. When working with ISCSI over IPoIB or mlx4_en, you must disable LRO (even if IPoIB is set to connected mode). This is because there is a bug in older kernels which causes a kernel panic. -10. On SLES11 in case that uninstall is failing, need to look at the error log +9. On SLES11 in case that uninstall is failing, need to look at the error log and remove the RPMs manually using 'rpm -e ' -11. On SLES11 one should set allow_unsupported_modules parameter to 1 in file: +10. On SLES11 one should set allow_unsupported_modules parameter to 1 in file: /etc/modprobe.d/unsupported-modules. Without this the modules will not load. +11. iSER is supported on kernel.org: 2.6.30, 2.6.31 and 2.6.32 only + OFED-1.5 will not install iSER on other kernels and original iSER + module coming with Linux Distribution will stop working because of + mismatch in symbols version. Note: See the release notes of each component for additional issues. diff --git a/OFED_tips.txt b/OFED_tips.txt index c5b89f2..e352a97 100644 --- a/OFED_tips.txt +++ b/OFED_tips.txt @@ -97,7 +97,7 @@ c. Using sysfs file system - ib_write_bw_postlist : RDMA write , posting a list of WQEs. 3.Other: - - ib_clock_test : tests cpu clock samples. + - ib_clock_test : tests CPU clock samples. Usage: Server: @@ -193,16 +193,16 @@ example: > ofed_info OFED-1.5 management: -http://www.openfabrics.org/downloads/management/daily/libibumad-1.3.3_20091204_86353ad.tar.gz -http://www.openfabrics.org/downloads/management/daily/libibmad-1.3.3_20091204_86353ad.tar.gz -http://www.openfabrics.org/downloads/management/daily/opensm-3.3.3_20091204_86353ad.tar.gz -http://www.openfabrics.org/downloads/management/daily/infiniband-diags-1.5.3_20091204_86353ad.tar.gz +http://www.openfabrics.org/downloads/management/daily/libibumad-1.3.4_20091214.tar.gz +http://www.openfabrics.org/downloads/management/daily/libibmad-1.3.4_20091214.tar.gz +http://www.openfabrics.org/downloads/management/daily/opensm-3.3.4_20091221_033cf4c.tar.gz +http://www.openfabrics.org/downloads/management/daily/infiniband-diags-1.5.4_20091221_033cf4c.tar.gz libibverbs: -http://www.openfabrics.org/downloads/rdmaoe/libibverbs/libibverbs-1.1.2-0.8.g66ece2f.tar.gz +http://www.openfabrics.org/downloads/libibverbs/libibverbs-1.1.2-0.2.g81705cb.tar.gz libmthca: http://www.openfabrics.org/downloads/libmthca/libmthca-1.0.5-0.1.gbe5eef3.tar.gz libmlx4: -http://www.openfabrics.org/downloads/rdmaoe/libmlx4/libmlx4-1.0-0.3.g1531245.tar.gz +http://www.openfabrics.org/downloads/libmlx4/libmlx4-1.0-0.1.gfa9c3d4.tar.gz libcxgb3: http://www.openfabrics.org/downloads/cxgb3/libcxgb3-1.2.5.tar.gz libehca: @@ -210,7 +210,7 @@ http://www.openfabrics.org/downloads/libehca/libehca-1.2.1-0.1.g0a82a52.tar.gz libipathverbs: http://www.openfabrics.org/downloads/libipathverbs/libipathverbs-1.2.tar.gz libnes: -http://www.openfabrics.org/downloads/nes/libnes-0.9.0.tar.gz +http://www.openfabrics.org/downloads/nes/libnes-1.0.0.tar.gz libibcm: http://www.openfabrics.org/downloads/rdmacm/libibcm-1.0.5.tar.gz librdmacm: @@ -222,7 +222,7 @@ http://www.openfabrics.org/downloads/sdpnetstat/sdpnetstat-1.60-0.1.g798e44f.tar srptools: http://www.openfabrics.org/downloads/srptools/srptools-0.0.4-0.1.gce1f64c.tar.gz perftest: -http://www.openfabrics.org/downloads/perftest/perftest-1.2.2-0.6.g3a2b795.tar.gz +http://www.openfabrics.org/downloads/perftest/perftest-1.2.3-0.8.g196d994.tar.gz qlvnictools: http://www.openfabrics.org/downloads/qlvnictools/qlvnictools-0.0.1-0.1.ge27eef7.tar.gz tvflash: @@ -232,7 +232,7 @@ http://www.openfabrics.org/downloads/mstflint/mstflint-1.4-0.1.g899ead3.tar.gz qperf: http://www.openfabrics.org/downloads/qperf/qperf-0.4.6-0.1.gb81434e.tar.gz ibutils: -http://www.openfabrics.org/downloads/ibutils/ibutils-1.2-0.1.ge8e69b7.tar.gz +http://www.openfabrics.org/downloads/ibutils/ibutils-1.5.tar.gz ibsim: http://www.openfabrics.org/downloads/ibsim/ibsim-0.5-0.1.g327c3d8.tar.gz dapl: @@ -242,13 +242,13 @@ http://www.openfabrics.org/downloads/dapl/compat-dapl-1.2.15.tar.gz ofa_kernel-1.5: Git: -git://git.openfabrics.org/ofed_1_5/linux-2.6.git rdmaoe -commit 5ed038c4a28a8d9ea99d7999af178ee80ec2f663 +git://git.openfabrics.org/ofed_1_5/linux-2.6.git ofed_kernel_1_5 +commit 1b6ce93a1ccee8d38bc88a7114abd77dae3410be # MPI -mvapich-1.1.0-3478.src.rpm +mvapich-1.2.0-3567.src.rpm mvapich2-1.4-2.src.rpm -openmpi-1.3.4-1.src.rpm +openmpi-1.4-1.src.rpm mpitests-3.2-916.src.rpm ib_bonding: http://www.openfabrics.org/~monis/ofed_1_5/ib-bonding-0.9.0-41.src.rpm @@ -257,7 +257,6 @@ http://www.openfabrics.org/~vlad/ofed_1_5/rds-tools/rds-tools-1.5-1.src.rpm rnfs-utils: http://www.openfabrics.org/~swise/ofed_1_5/rnfs-utils/rnfs-utils-1.1.5-10.OFED.src.rpm - 2.2 Installed OFED Components ------------------------------- The script /etc/infiniband/info provides data on the specific OFED installation @@ -318,6 +317,8 @@ and copy it to the modules subdir and then compile your module. If /src/ofa_kernel/Module.symvers does not exist or it is empty, use the create_Module.symvers.sh (a part of the ofed-docs RPM) script to create the Module.symvers file. +Example: +MODULES_DIR=/lib/modules/`uname -r`/updates ./create_Module.symvers.sh See "Module versioning & Module.symvers" in the modules.txt from kernel documentation (e.g. linux-2.6.30/Documentation/kbuild/modules.txt). diff --git a/PERF_TEST_README.txt b/PERF_TEST_README.txt index cc57bb9..3c6b3fe 100644 --- a/PERF_TEST_README.txt +++ b/PERF_TEST_README.txt @@ -1,6 +1,6 @@ Open Fabrics Enterprise Distribution (OFED) - Performance Tests README for OFED 1.5 - + Performance Tests README for OFED 1.5 + December 2009 diff --git a/README.txt b/README.txt index b34976d..c57c4c1 100644 --- a/README.txt +++ b/README.txt @@ -193,9 +193,9 @@ install one or more MPI stacks. Multiple MPI stacks can be installed simultaneously -- they will not conflict with each other. Three MPI stacks are included in this release of OFED: -- MVAPICH 1.2.0 -- Open MPI 1.3.4 -- MVAPICH2 1.4 +- MVAPICH +- Open MPI +- MVAPICH2 OFED also includes 4 basic tests that can be run against each MPI stack: bandwidth (bw), latency (lt), Intel MPI Benchmark and Presta. The tests diff --git a/SRPT_README.txt b/SRPT_README.txt index c3e982f..8d13e8e 100644 --- a/SRPT_README.txt +++ b/SRPT_README.txt @@ -21,7 +21,7 @@ to *real* devices Prerequisites ------------- -0. Supported distributions: RHEL 5.2/5.3/5.4, SLES 10 sp2, SLES 11 +0. Supported distributions: RHEL 5.3/5.4, SLES 10 sp2, SLES 11 Note: On distribution default kernels you can run scst_vdisk blockio mode to have good performance. You can also run scst_disk ie. scsi pass-thru diff --git a/create_Module.symvers.sh b/create_Module.symvers.sh index 2c6e6a9..5b2d76d 100755 --- a/create_Module.symvers.sh +++ b/create_Module.symvers.sh @@ -28,25 +28,16 @@ # # Description: creates Module.symvers file for InfiniBand modules -K_VER=${K_VER:-$(uname -r)} -MOD_SYMVERS_IB=./Module.symvers +KVERSION=${KVERSION:-$(uname -r)} +MOD_SYMVERS=./Module.symvers SYMS=/tmp/syms -if [ -d /lib/modules/$K_VER/updates/kernel/drivers/infiniband ]; then - MODULES_DIR=/lib/modules/$K_VER/updates/kernel/drivers/infiniband -elif [ -d /lib/modules/$K_VER/kernel/drivers/infiniband ]; then - MODULES_DIR=/lib/modules/$K_VER/kernel/drivers/infiniband -else - echo "No infiniband modules found" - exit 1 -fi - -echo MODULES_DIR=${MODULES_DIR} +echo MODULES_DIR=${MODULES_DIR-:./} -if [ -f ${MOD_SYMVERS_IB} -a ! -f ${MOD_SYMVERS_IB}.save ]; then - mv ${MOD_SYMVERS_IB} ${MOD_SYMVERS_IB}.save +if [ -f ${MOD_SYMVERS} -a ! -f ${MOD_SYMVERS}.save ]; then + mv ${MOD_SYMVERS} ${MOD_SYMVERS}.save fi -rm -f $MOD_SYMVERS_IB +rm -f $MOD_SYMVERS rm -f $SYMS for mod in $(find ${MODULES_DIR} -name '*.ko') ; do @@ -55,29 +46,19 @@ for mod in $(find ${MODULES_DIR} -name '*.ko') ; do done n_syms=$(wc -l $SYMS |cut -f1 -d" ") -echo Found $n_syms InfiniBand symbols in $n_mods InfiniBand modules +echo Found $n_syms OFED kernel symbols in $n_mods modules n=1 - while [ $n -le $n_syms ] ; do line=$(head -$n $SYMS|tail -1) line1=$(echo $line|cut -f1 -d:) line2=$(echo $line|cut -f2 -d:) - file=$(echo $line1|cut -f6- -d/) - file=$(echo $file|cut -f1 -d.) - + file=$(echo $line1| sed -e 's@./@@' -e 's@.ko@@' -e "s@$PWD/@@") crc=$(echo $line2|cut -f1 -d" ") - crc=${crc:8} sym=$(echo $line2|cut -f3 -d" ") - sym=${sym:6} - echo -e "0x$crc\t$sym\t$file" >> $MOD_SYMVERS_IB - if [ -z $allsyms ] ; then - allsyms=$sym - else - allsyms="$allsyms|$sym" - fi + echo -e "0x$crc\t$sym\t$file" >> $MOD_SYMVERS n=$((n+1)) done -echo ${MOD_SYMVERS_IB} created. +echo ${MOD_SYMVERS} created. diff --git a/diags_release_notes.txt b/diags_release_notes.txt index 709fdd9..9db4cbf 100644 --- a/diags_release_notes.txt +++ b/diags_release_notes.txt @@ -1,11 +1,11 @@ Open Fabrics Enterprise Distribution (OFED) - Diagnostic Tools in OFED 1.4 Release Notes + Diagnostic Tools in OFED 1.5 Release Notes - December 2008 + December 2009 -Repo: git://git.openfabrics.org/~ofed_1_3/management.git (release) - git://git.openfabrics.org/~sashak/management/management.git (development) +Repo: git://git.openfabrics.org/~sashak/management/management.git +URL: http://www.openfabrics.org/downloads/management General diff --git a/ib-bonding.txt b/ib-bonding.txt index c6266f4..1727d6b 100644 --- a/ib-bonding.txt +++ b/ib-bonding.txt @@ -2,14 +2,13 @@ IB Bonding =============================================================================== 1. Introduction -2. How to work with ib-bond -3. How to work with interface configuration scripts -3.1 Configuration with initscripts support -3.1.1 Writing network scripts under Redhat-AS4 (Update 4, 5, 6 or 7) -3.1.2 Writing network scripts under Redhhat-EL5 -3.2 Configuration with sysconfig support -3.2.1 Writing network scripts under SLES-10 -3.3 Configuring Ethernet slaves +2. How to work with interface configuration scripts +2.1 Configuration with initscripts support +2.1.1 Writing network scripts under Redhat-AS4 (Update 6, 7 or 8) +2.1.2 Writing network scripts under Redhhat-EL5 +2.2 Configuration with sysconfig support +2.2.1 Writing network scripts under SLES-10 +2.3 Configuring Ethernet slaves 1. Introduction ------------------------------------------------------------------------------- @@ -17,58 +16,18 @@ ib-bonding is a High Availability solution for IPoIB interfaces. It is based on the Linux Ethernet Bonding Driver and was adopted to work with IPoIB. However, the support for for IPoIB interfaces is only for the active-backup mode, other modes should not be used. -ib-bonding package contains a bonding driver and a utility called ib-bond to -manage and control the driver operation. -2. How to work with ib-bond +2. How to work with interface configuration scripts ------------------------------------------------------------------------------- +To create an interface configuration script for the ibX and bondX interfaces, +you should use the standard syntax (depending on your OS). -* Creating a bonding network interface - --bond-name: sets the name of the bonding network interface. Default is bond0 - --bond-ip : sets the IP address of bond0. If MASK is not given it - is set to 255.255.255.0 (24 bits). Note that MASK should be the number of 1 - bits in the netmask. - --slaves: a comma separated list of slave ib devices. If not given ib0 and - ib1 will be used as slaves. Child interfaces are allowed. - --miimon: the MII monitoring interval in mSec. Default is 100 -* Deleting a bonding network interface - --stop: unenslave slaves and delete a specific bonding network interface (use with --bond-name) - --stop-all: unenslave slaves and delete all bonding network interfaces -* Querying a bonding network interface - --status: show the status of a specific bonding network interface (use with --bond-name) - --status-all: show the status of all bonding network interfaces - -Examples: - -* To bring up bond0 with ib0 and ib2 as slaves (assumes 2 HCAs) - ib-bond --bond-ip 192.186.10.100 --slaves ib0,ib2 -* To bring up bond1 with ib0.f1f1 1and ib1.f1f1 as slaves with non default - netmask - ib-bond --bond-name bond1 --bond-ip 192.186.10.100/25 --slaves ib0.f1f1,ib1.f1f1 -* To query the status of bond1 - ib-bond --bond-name bond1 --status -* To query the status of all bonding interfaces - ib-bond --status-all -* To stop bond1 - ib-bond --bond-name bond1 --stop -* To stop all bonding interfaces - ib-bond --stop-all - -3. How to work with interface configuration scripts -------------------------------------------------------------------------------- -Using ib-bond to configure interfaces doesn't save the configuration anywhere, -so whenever the master or one of the slaves is destroyed the configuration -should be restored by running ib-bond again (e.g. after system reboot). -It is possible to avoid that if you create an interface configuration script for -the ibX and bondX interfaces. To do that, you should use the standard syntax to -create the bonding configuration (depending on your OS). - -3.1 Configuration with initscripts support +2.1 Configuration with initscripts support ------------------------------------------ Note: This feature is available only for Redhat-AS4 (Update 4, Update 5, Update 6 or Update 7) and for Redhat-EL5 and above. -3.1.1 Writing network scripts under Redhat-AS4 (Update 4, 5, 6 or 7) +2.1.1 Writing network scripts under Redhat-AS4 (Update 4, 5, 6 or 7) ----------------------------------------------------------------- * In the master (bond) interface script add the line: TYPE=Bonding @@ -118,7 +77,7 @@ required and can be omitted. After the configuration is saved, restart the network service by running: /etc/init.d/network restart -3.1.2 Writing network scripts under Redhhat-EL5 +2.1.2 Writing network scripts under Redhhat-EL5 ----------------------------------------------- Follow the instructions in 3.1.1 (Writing network scripts under Redhat-AS4) with the following changes: @@ -143,11 +102,11 @@ Note: restarting OFED doesn't keep the bonding configuration via initscripts. You have to restart the network service in order to recreate the bonding interface. -3.2 Configuration with sysconfig support +2.2 Configuration with sysconfig support ---------------------------------------- Note: This feature is available only for SLES-10 and above. -3.2.1 Writing network scripts under SLES-10 +2.2.1 Writing network scripts under SLES-10 ----------------------------------------------- * In the master (bond) interface script add the lins: @@ -192,7 +151,7 @@ PRE_DOWN_SCRIPT=/etc/sysconfig/network/unenslave.sh After the configuration is saved, restart the network service by running: /etc/init.d/network restart -3.3 Configuring Ethernet slaves +2.3 Configuring Ethernet slaves ------------------------------- It is not possible to have a mix of Ethernt slaves and IPoIB slaves under the same bonding master. It is possible however that a bonding master of Ethernet diff --git a/ipoib_release_notes.txt b/ipoib_release_notes.txt index 5a7a238..4c4f135 100644 --- a/ipoib_release_notes.txt +++ b/ipoib_release_notes.txt @@ -8,15 +8,14 @@ Table of Contents =============================================================================== 1. Overview -2. New Features -3. Known Issues -4. DHCP Support of IPoIB -5. The ib-bonding driver -6. Bug Fixes and Enhancements Since OFED 1.3 -7. Bug Fixes and Enhancements Since OFED 1.3.1 -8. Bug Fixes and Enhancements Since OFED 1.4 -9. Bug Fixes and Enhancements Since OFED 1.4.2 -10. Performance tuning +2. Known Issues +3. DHCP Support of IPoIB +4. The ib-bonding driver +5. Bug Fixes and Enhancements Since OFED 1.3 +6. Bug Fixes and Enhancements Since OFED 1.3.1 +7. Bug Fixes and Enhancements Since OFED 1.4 +8. Bug Fixes and Enhancements Since OFED 1.4.2 +9. Performance tuning =============================================================================== 1. Overview @@ -26,26 +25,6 @@ protocol packets over an InfiniBand UD channel. The implementation conforms to the relevant IETF working group's RFCs (http://www.ietf.org). -=============================================================================== -2. New Features -=============================================================================== -1. This version of ofed introduces improvements to IPOIB by cutting the CPU - overhead in handling receive packets. This will improve operation - in datagram mode: - Large Receive Offload (LRO) - aggregating multiple incoming packets from a - single stream into a larger buffer before they are passed higher up the - networking stack, thus reducing the number of packets that have to be - processed. - This feature is enabled on HCAs that can support LRO, e.g. ConnectX. -2. Datagram mode: LSO (large send offload) allows the networking stack to pass - SKBs with data size larger than the MTU to the IPoIB driver and have the HCA - HW fragment the data to multiple MSS-sized packets. Add a device capability - flag IB_DEVICE_UD_TSO for devices that can perform TCP segmentation offload, - a new send work request opcode IB_WR_LSO, header, hlen and mss fields for - the work request structure, and a new IB_WC_LSO completion type. - This feature is enabled on HCAs that can support LSO, e.g. ConnectX. - - Usage and configuration: ======================== 1. To check the current mode used for outgoing connections, enter: @@ -75,7 +54,7 @@ Usage and configuration: =============================================================================== -3. Known Issues +2. Known Issues =============================================================================== 1. If a host has multiple interfaces and (a) each interface belongs to a different IP subnet, (b) they all use the same InfiniBand Partition, and (c) @@ -142,7 +121,7 @@ Usage and configuration: layer address at an offset of two bytes with respect to older kernels. This causes the other host to misinterpret the hardware address resulting in failure to resolve path which are based on wrong GIDs. As an example, RH 4.x and RH - 5.x cannot interoperate. + 5.x cannot inter-operate. 8. In connected mode, TCP latency for short messages is larger by approx. 1usec (~5%) than in datagram mode. As a workaround, use datagram mode. @@ -180,7 +159,7 @@ Usage and configuration: kernel. To avoid this leave the IPoIB interfaces enslaved when unloading ib_ipoib. -17. On SLES 11, sysconfig scripts overide the interface mode and set it to +17. On SLES 11, sysconfig scripts override the interface mode and set it to datagram on each call to ifup, ifdown, etc. To avoid this, add the line IPOIB_MODE=connected to the interface configuration file (e.g. ifcfg-ib0) @@ -190,18 +169,18 @@ Usage and configuration: kernel has all the functionality to support IPoIB slaves. This approach however doesn't patch the sysconfig (SuSE) or initscripts (RedHat) package so the network configuration scrips may not work properly. - For example, if you install OFED on RHEL5.2 that runns kernel 2.6.30 and + For example, if you install OFED on RHEL5.2 that runs kernel 2.6.30 and you try to configure and run bonding, you won't be able to restart the network and see bond0 up and running with IPoIB slaves. A workaround to this problem would be this - a. Compile ib-bonding source rpm (under SRPMS directory) separately on a - machine with RHEL5.2 and kernel 2.6.18-92.el5 (default for this OS) - 2. Install the binary rpm while machine runs kernel 2.6.18-92.el5. - This will patch the OS configuration scripts and install the bonding module. - 3. Switch to kernel 2.6.30. The module that was compiled in (2) will not be - loaded since it was compiled and installed for a different kernel. - 4. Configure bonding and restart the network. The bonding interface - should be up and running afterwards. + a. Compile ib-bonding source rpm (under SRPMS directory) separately on a + machine with RHEL5.2 and kernel 2.6.18-92.el5 (default for this OS) + 2. Install the binary rpm while machine runs kernel 2.6.18-92.el5. + This will patch the OS configuration scripts and install the bonding module. + 3. Switch to kernel 2.6.30. The module that was compiled in (2) will not be + loaded since it was compiled and installed for a different kernel. + 4. Configure bonding and restart the network. The bonding interface + should be up and running afterwards. 19. On RHEL5.X, '/etc/init.d/openibd start' prints the following messages while bringing up IPoIB interfaces: @@ -214,9 +193,11 @@ Usage and configuration: This does not affect IPoIB configuration and interfaces are configured as expected. +20. In IPoIB connected mode packages larger than 2016 bytes are not sent. + https://bugs.openfabrics.org/show_bug.cgi?id=1839 =============================================================================== -4. IPoIB Configuration Based on DHCP +3. IPoIB Configuration Based on DHCP =============================================================================== Setting an IPoIB interface configuration based on DHCP (v3.1.2 which is available @@ -286,27 +267,21 @@ host1# dhclient cf dhclient.conf ib1 =============================================================================== -5. The ib-bonding driver +4. The ib-bonding driver =============================================================================== The ib-bonding driver is a High Availability solution for IPoIB interfaces. It is based on the Linux Ethernet Bonding Driver and was adapted to work with -IPoIB. The ib-bonding package contains a bonding driver and a utility called -ib-bond to manage and control the driver operation. -The ib-bonding driver comes with the ib-bonding package (run rpm -qi ib-bonding -to get the package information). +IPoIB. The ib-bonding driver comes with the ib-bonding package +(run rpm -qi ib-bonding to get the package information). Using the ib-bonding driver --------------------------- -The ib-bonding driver can be loaded manually or automatically. - -1. Manual operation: -Use the utility ib-bond to start, query, or stop the driver. For details on this -utility, read the documentation for the ib-bonding package. +The ib-bonding driver is loaded automatically. -2. Automatic operation: - Use standard OS tools (sysconfig in SuSE and initscripts in Redhat) - to create a configuration that will come up with network restart. For details - on this, read the documentation for the ib-bonding package. +Automatic operation: +Use standard OS tools (sysconfig in SuSE and initscripts in RedHat) +to create a configuration that will come up with network restart. For details +on this, read the documentation for the ib-bonding package. Notes: * Using /etc/infiniband/openib.conf to create a persistent configuration is @@ -316,14 +291,14 @@ Notes: that is >= 2.6.27 (e.g. SLES11). The bonding driver that comes with those kernels already supports enslaving of IPoIB interfaces. In addition, an OS can come with an older kernel but with a patched bonding driver that also doesn't - require modification (e.g. RHEL5.4). OFEED will not replace the bonding + require modification (e.g. RHEL5.4). OFED will not replace the bonding module in such cases also. However, there still might be a issue of OS configuration tools (like sysconfig or initscripts) that needs a fix but such issues were not observed yet. =============================================================================== -6. Bug Fixes and Enhancements Since OFED 1.3 +5. Bug Fixes and Enhancements Since OFED 1.3 =============================================================================== - There is no default configuration for IPoIB interfaces: One should manually specify the full IP configuration or use the ofed_net.conf file. See @@ -343,7 +318,7 @@ Notes: documentation for details) =============================================================================== -7. Bug Fixes and Enhancements Since OFED 1.3.1 +6. Bug Fixes and Enhancements Since OFED 1.3.1 =============================================================================== - IPoIB: Refresh paths instead of flushing them on SM change events to improve failover respond @@ -353,7 +328,7 @@ Notes: - Bonding: Set default number of grat. ARP after failover to three (was one) =============================================================================== -8. Bug Fixes and Enhancements Since OFED 1.4 +7. Bug Fixes and Enhancements Since OFED 1.4 =============================================================================== - Performance tuning is enabled by default for IPOIB CM. - Clear IPOIB_FLAG_ADMIN_UP if ipoib_open fails @@ -366,23 +341,27 @@ Notes: - fix unprotected use of priv->broadcast in ipoib_mcast_join_task. - Do not join broadcast group if interface is brought down - + =============================================================================== -9. Bug Fixes and Enhancements Since OFED 1.4.2 +8. Bug Fixes and Enhancements Since OFED 1.4.2 =============================================================================== - Check that the format of multicast link addresses is correct before taking them from dev->mc_list to priv->multicast_list. This way we never try to - send a bogus address to the SA, which prevents badness from erronous + send a bogus address to the SA, which prevents badness from erroneous 'ip maddr addr add', broken bonding drivers, etc. (bugzilla #1664) - IPoIB: Don't turn on carrier for a non-active port. If a bonding interface uses this IPoIB interface as a slave it might not detect that this slave is almost useless and failover functionality will be damaged. The fix checks the state of the IB port in the carrier_task before calling netif_carrier_on(). (bugzilla #1726) +- Clear ipoib_neigh.dgid in ipoib_neigh_alloc() + IPoIB can miss a change in destination GID under some conditions. The + problem is caused when ipoib_neigh->dgid contains a stale address. + The fix is to set ipoib_neigh->dgid to zero in ipoib_neigh_alloc(). =============================================================================== -10. Performance tuning +9. Performance tuning =============================================================================== When IPoIB is configured to run in connected mode, tcp parameter tuning is performed at driver startup -- to improve the throughput of medium and large diff --git a/iser_release_notes.txt b/iser_release_notes.txt index 90653c8..0a023b6 100644 --- a/iser_release_notes.txt +++ b/iser_release_notes.txt @@ -1,8 +1,8 @@ Open Fabrics Enterprise Distribution (OFED) - iSER initiator in OFED 1.4 Release Notes + iSER initiator in OFED 1.5 Release Notes - December 2008 + December 2009 * Background @@ -10,7 +10,7 @@ iSER allows iSCSI to be layered over RDMA transports (including InfiniBand and iWARP (RNIC)). - The OpenFabrics iSER initiator implementation is interoperable with + The OpenFabrics iSER initiator implementation is inter-operable with open-iscsi (http://www.open-iscsi.org/). It provides an alternative transport to iscsi_tcp in the open-iscsi framework. The iSER transport exposes a transport API to scsi_transport_iscsi, and a SCSI LLD API to @@ -19,20 +19,11 @@ * Supported platforms - SLES 10 - SLES 10 sp1 - SLES 10 sp2 - RHAS 4 up4 - RHAS 4 up5 - RHAS 4 up6 - RHAS 4 up7 - RHEL 5 - RHEL 5.1 - RHEL 5.2 - - The release has been tested against Voltaire iSCSI over iSER target - running in Voltaire's IB/Fibre-Channel router (SR4G) and the STGT - target. + kernel.org: 2.6.30, 2.6.31 and 2.6.32 + + OFED-1.5 will not install iSER on other kernels and original iSER + module coming with Linux Distribution will stop working because of + mismatch in symbols version. * Fixed Bugs and Enhancements since OFED 1.3 iSER: @@ -41,7 +32,7 @@ - Add change_queue_depth method - Fix list iteration bug - Handle iser_device allocation error gracefully - - Don't change ITT endianness + - Don't change ITT endianess - Move high-volume debug output to higher debug level - Count FMR alignment violations per session Open-iSCSI: @@ -55,9 +46,8 @@ * Known Issues Open-iSCSI: - - modifing node transport_name while session is active + - modifying node transport_name while session is active will create stale session. It will be deleted only after reboot. - - This issue is scheduled for OFED1.4 as part of new open-iscsi version. * Installation/upgrade of open-iscsi If iSER is selected to be installed with OFED, open-iscsi will be also diff --git a/iser_target_release_notes.txt b/iser_target_release_notes.txt index fc597d8..b865944 100644 --- a/iser_target_release_notes.txt +++ b/iser_target_release_notes.txt @@ -1,7 +1,7 @@ Open Fabrics Enterprise Distribution (OFED) - STGT/iSER target in OFED 1.4 Release Notes + STGT/iSER target in OFED 1.5 Release Notes - December 2008 + December 2009 * Background diff --git a/mlx4_release_notes.txt b/mlx4_release_notes.txt index 917edf6..37c1c89 100644 --- a/mlx4_release_notes.txt +++ b/mlx4_release_notes.txt @@ -1,7 +1,7 @@ Open Fabrics Enterprise Distribution (OFED) - ConnectX driver (mlx4) in OFED 1.5.0 Release Notes - - Decenber 2009 + ConnectX driver (mlx4) in OFED 1.5 Release Notes + + December 2009 =============================================================================== @@ -14,7 +14,7 @@ Table of Contents 5. Infiniband (mlx4_ib) new features and bug fixes since OFED 1.4 6. Eth (mlx4_en) new features and bug fixes since OFED 1.4 7. New features and bug fixes since OFED 1.4.1 -8. New features and bug fixes since OFED 1.5.0 +8. New features and bug fixes since OFED 1.4.2 9. Known Issues 10. mlx4 Available Parameters @@ -82,7 +82,7 @@ o Port type management: "ib" - Always Infiniband. "auto" - Link sensing mode - detect port type based on the attached network type. If no link is detected, the driver retries link - sensing every few seconds. + sensing every few seconds. Port link type can be configured for each device in the system at run time using the "/sbin/connectx_port_config" script. @@ -199,7 +199,7 @@ Bug Fixes - Fixed recovery flows from memory allocation failures. - When the system is low on memory, the mlx4_en driver now allocates smaller RX rings. -- The mlx4_coredriver now retries to obtain MSI-X vectors if the initial request is +- The mlx4_core driver now retries to obtain MSI-X vectors if the initial request is rejected by the OS =============================================================================== @@ -250,36 +250,39 @@ In order to set mlx4 parameters, add the following line(s) to /etc/modpobe.conf: options mlx4_en parameter= mlx4_core parameters: - set_4k_mtu: attempt to set 4K MTU to all ConnectX ports (default 0) - msi_x: attempt to use MSI-X if nonzero (default 1) - enable_qos: Enable Quality of Service support in the HCA if > 0, (default 0) - block_loopback Block multicast loopback packets if > 0 (default: 1) - internal_err_reset: Reset device on internal errors if non-zero (default 1) - debug_level: Enable debug tracing if > 0 (default 0) - log_num_qp: log maximum number of QPs per HCA (default is 17; max is 20) - log_num_srq: log maximum number of SRQs per HCA (default is 16; max is 20) - log_rdmarc_per_qp: log number of RDMARC buffers per QP (default is 4; max is 7) - log_num_cq: log maximum number of CQs per HCA (default is 16 max is 19) - log_num_mcg: log maximum number of multicast groups per HCA (default is 13; max is 21) - log_num_mpt: log maximum number of memory protection table entries per HCA - (default is 17; max is 20) - log_num_mtt: log maximum number of memory translation table segments per HCA - (default is 20; max is 20) - log_num_mac: log maximum number of MACs per ETH port (1-7) (int) - log_num_vlan: log maximum number of VLANs per ETH port (0-7) (int) - log_mtts_per_seg Log2 number of MTT entries per segment (1-5; default is 3) - use_prio: Enable steering by VLAN priority on ETH ports (0/1, default 0) (bool) + set_4k_mtu: attempt to set 4K MTU to all ConnectX ports (int) + debug_level: Enable debug tracing if > 0 (int) + block_loopback: Block multicast loopback packets if > 0 (int) + msi_x: attempt to use MSI-X if nonzero (int) + log_num_mac: Log2 max number of MACs per ETH port (1-7) (int) + log_num_vlan: Log2 max number of VLANs per ETH port (0-7) (int) + use_prio: Enable steering by VLAN priority on ETH ports (0/1, default 0) + (bool) + log_num_qp: log maximum number of QPs per HCA (int) + log_num_srq: log maximum number of SRQs per HCA (int) + log_rdmarc_per_qp: log number of RDMARC buffers per QP (int) + log_num_cq: log maximum number of CQs per HCA (int) + log_num_mcg: log maximum number of multicast groups per HCA (int) + log_num_mpt: log maximum number of memory protection table entries per HCA + (int) + log_num_mtt: log maximum number of memory translation table segments per HCA + (int) + log_mtts_per_seg: Log2 number of MTT entries per segment (1-5) (int) + enable_qos: Enable Quality of Service support in the HCA (default: off) + (bool) + internal_err_reset: Reset device on internal errors if non-zero + (default 1) (int) mlx4_ib parameters: - debug_level: Enable debug tracing if > 0 (default 0) + debug_level: Enable debug tracing if > 0 (default 0) mlx4_en parameters: - rss_xor: Use XOR hash function for RSS 0 (default is xor) - rss_mask: RSS hash type bitmask (default is 0xf) - num_lro: Number of LRO sessions per ring or disabled (0) (default is 32) - ip_reasm: Allow reassembly of fragmented IP packets (default is enabled) - pfctx: Priority based Flow Control policy on TX[7:0]. - Per priority bit mask (default is 0) - pfcrx: Priority based Flow Control policy on RX[7:0]. - Per priority bit mask (default is 0) - inline_thold: threshold for using inline data (default is 128) + rss_xor: Use XOR hash function for RSS 0 (default is xor) + rss_mask: RSS hash type bitmask (default is 0xf) + num_lro: Number of LRO sessions per ring or disabled (0) (default is 32) + ip_reasm: Allow reassembly of fragmented IP packets (default is enabled) + pfctx: Priority based Flow Control policy on TX[7:0]. + Per priority bit mask (default is 0) + pfcrx: Priority based Flow Control policy on RX[7:0]. + Per priority bit mask (default is 0) + inline_thold: threshold for using inline data (default is 128) diff --git a/mpi-selector_release_notes.txt b/mpi-selector_release_notes.txt index 6677624..95944dc 100644 --- a/mpi-selector_release_notes.txt +++ b/mpi-selector_release_notes.txt @@ -1,5 +1,5 @@ MPI Selector 1.0 release notes - December 2008 + December 2009 ============================== OFED contains a simple mechanism for system administrators and end diff --git a/mstflint_release_notes.txt b/mstflint_release_notes.txt index e200d95..ba1e12e 100644 --- a/mstflint_release_notes.txt +++ b/mstflint_release_notes.txt @@ -2,8 +2,8 @@ -------------------------------------------- ******************************************************************************* -RELEASE: OFED 1.4 -DATE: Dec 2008 +RELEASE: OFED 1.5 +DATE: Dec 2009 =============================================================================== Table of Contents @@ -32,9 +32,9 @@ Table of Contents for later analysis by Mellanox. e) mstvpd This utility dumps the on-card VPD. - f) hca_self_test.ofed - This scripts checks the status of software, firmware and hardware - of the HCAs installed on the local host. + f) hca_self_test.ofed + This scripts checks the status of software, firmware and hardware + of the HCAs installed on the local host. =============================================================================== 2. New Features diff --git a/mthca_release_notes.txt b/mthca_release_notes.txt index cb2b1b5..40f3c4e 100644 --- a/mthca_release_notes.txt +++ b/mthca_release_notes.txt @@ -1,7 +1,7 @@ Open Fabrics Enterprise Distribution (OFED) - mthca in OFED 1.4.1 Release Notes + mthca in OFED 1.5 Release Notes - May 2009 + December 2009 =============================================================================== @@ -15,8 +15,8 @@ Table of Contents =============================================================================== 1. Overview =============================================================================== -mthca is the low level driver implementation for the following Mellanox Techno- -logies HCAs: InfiniHost, InfiniHost III Ex and InfiniHost III Lx. +mthca is the low level driver implementation for the following Mellanox +Technologies HCAs: InfiniHost, InfiniHost III Ex and InfiniHost III Lx. mthca Available Parameters -------------------------- @@ -25,38 +25,30 @@ In order to set mthca parameters, add the following line to /etc/modpobe.conf: options ib_mthca parameter= mthca parameters: - - tune_pci - increase PCI burst from the default set by BIOS if - nonzero - msi - attempt to use MSI if nonzero - msi_x - attempt to use MSI-X if nonzero - fw_cmd_doorbell - post firmware commands through doorbell page if non- - zero (and supported by firmware) - catas_reset_disable - disable device reset on a catastrophic event if non- - zero - debug_level - Enable debug tracing if > 0 (int) - num_qp - maximum number of QPs per HCA (int) - rdb_per_qp - number of RDB buffers per QP (int) - num_cq - maximum number of CQs per HCA (int) - num_mcg - maximum number of multicast groups per HCA (int) - num_mpt - maximum number of memory protection table entries - per HCA (int) - num_mtt - maximum number of memory translation table segments - per HCA (int) - num_udav - maximum number of UD address vectors per HCA (int) - fmr_reserved_mtts - number of memory translation table segments reserved - for FMR (int) - log_mtts_per_seg - log2 number of MTT entries per segment (1-5) + catas_reset_disable: disable reset on catastrophic event if nonzero + (int) + fw_cmd_doorbell: post FW commands through doorbell page if + nonzero (and supported by FW) (int) + debug_level: Enable debug tracing if > 0 (int) + msi_x: attempt to use MSI-X if nonzero (int) + tune_pci: increase PCI burst from the default set by BIOS if nonzero (int) + num_qp: maximum number of QPs per HCA (int) + rdb_per_qp: number of RDB buffers per QP (int) + num_cq: maximum number of CQs per HCA (int) + num_mcg: maximum number of multicast groups per HCA (int) + num_mpt: maximum number of memory protection table entries per HCA (int) + num_mtt: maximum number of memory translation table segments per HCA (int) + num_udav: maximum number of UD address vectors per HCA (int) + fmr_reserved_mtts: number of memory translation table segments reserved for + FMR (int) + log_mtts_per_seg: Log2 number of MTT entries per segment (1-5) (int) =============================================================================== 2. Fixed Bugs =============================================================================== -- IB_EVENT_LID_CHANGE is generated more appropriately. -- Improved MTT buddy allocator (free count per order). -- Fix check of max_send_sge for special QPs. -- RESET->ERR QP state transition no longer supported (IB Spec 1.2.1). -- Clear ICM pages before handing to FW. -- Fixed race condition between create QP and destroy QP (bugzilla 1389) +- Fix access to freed memory in catastrophic processing + catas_reset() uses pointer to mthca_dev, but mthca_dev is not valid after + call __mthca_restart_one(). =============================================================================== @@ -68,7 +60,7 @@ mthca parameters: solve an issue when had more than 2^18 max qp's configured. =============================================================================== -3. Known Issues +4. Known Issues =============================================================================== 1. A UAR size other than 8MB prevents mthca driver loading. The default UAR size is 8MB. If the size is changed, the following error message will be diff --git a/mvapich_release_notes.txt b/mvapich_release_notes.txt index a78fad5..8c872ae 100644 --- a/mvapich_release_notes.txt +++ b/mvapich_release_notes.txt @@ -1,7 +1,7 @@ Open Fabrics Enterprise Distribution (OFED) - OSU MPI MVAPICH-1.1.0, in OFED 1.4.r10 Release Notes + OSU MPI MVAPICH-1.2.0, in OFED 1.5 Release Notes - May 2009 + December 2009 =============================================================================== @@ -18,7 +18,7 @@ Table of Contents =============================================================================== 1. Overview =============================================================================== -These are the release notes for OSU MPI MVAPICH-1.1.0. +These are the release notes for OSU MPI MVAPICH-1.2.0. OSU MPI is an MPI channel implementation over InfiniBand by Ohio State University (OSU). @@ -34,28 +34,18 @@ InfiniBand IPoIB or Ethernet). =============================================================================== -3. New Features ( Compared to mvapich 1.0.0 ) +3. New Features ( Compared to mvapich 1.1.0 ) =============================================================================== -MVAPICH-1.1.0 has the following additional features: -- eXtended Reliable Connection (XRC) support -- Lock-free design to provide support for asynchronous - progress at both sender and receiver to overlap - computation and communication -- Optimized MPI_allgather collective +MVAPICH-1.2.0 has the following additional features: +- Advanced network recovery support +- mpirun launcher improvements - Efficient intra-node shared memory communication support for diskless clusters -- Enhanced Totalview Support with the new mpirun_rsh framework +- RoCEE (RDMAoE) networks support =============================================================================== -4. Bug Fixes ( Compared to mvapich 1.0.0 ) +4. Bug Fixes ( Compared to mvapich 1.1.0 ) =============================================================================== -- De-register stale memory regions earlier to prevent - excess allocations of physical memory -- Fixes for MPI_Query_thread and MPI_Is_thread_main -- Fixes for PGI compiler support -- Compilation warnings cleanup -- Fixes for optimized colletives -- Fix data types for memory allocations - Multiple fixes for mpirun_rsh launcher =============================================================================== @@ -78,12 +68,12 @@ MVAPICH-1.1.0 has the following additional features: fw-23108 version that is earlier than 3.4.000, or with an fw-25208 version 4.7.400 or earlier. - NOTE: There is no issue if you chose to update firmware during Mellanox - OFED installation as newer firmware versions were burnt. + NOTE: There is no issue if you chose to update firmware during Mellanox + OFED installation as newer firmware versions were burnt. Workaround: Option 1 - Update the firmware. For instructions, see Mellanox Firmware Tools - (MFT) User's Manual under the docs/ folder. + (MFT) User's Manual under the docs/ folder. Option 2 - In mvapich.conf, set VIADEV_SRQ_ENABLE=0 - MVAPICH may fail to run on some SLES 10 machines due to problems in resolving diff --git a/ofed.conf-example b/ofed.conf-example index d2fdafd..2ab3399 100644 --- a/ofed.conf-example +++ b/ofed.conf-example @@ -1,11 +1,9 @@ -prefix=/usr core=y mthca=y mlx4=y mlx4_en=y cxgb3=y nes=y -ipath=y ipoib=y sdp=y srp=y @@ -17,36 +15,43 @@ libibverbs=y libibverbs-devel=y libibverbs-devel-static=y libibverbs-utils=y +libibverbs-debuginfo=y libmthca=y libmthca-devel-static=y +libmthca-debuginfo=y libmlx4=y libmlx4-devel=y +libmlx4-debuginfo=y libcxgb3=y libcxgb3-devel=y +libcxgb3-debuginfo=y libnes=y libnes-devel-static=y -libipathverbs=y -libipathverbs-devel=y +libnes-debuginfo=y libibcm=y libibcm-devel=y -libibcommon=y -libibcommon-devel=y -libibcommon-static=y +libibcm-debuginfo=y libibumad=y libibumad-devel=y libibumad-static=y +libibumad-debuginfo=y libibmad=y libibmad-devel=y libibmad-static=y +libibmad-debuginfo=y ibsim=y +ibsim-debuginfo=y librdmacm=y librdmacm-utils=y librdmacm-devel=y +librdmacm-debuginfo=y libsdp=y libsdp-devel=y +libsdp-debuginfo=y opensm=y opensm-libs=y opensm-devel=y +opensm-debuginfo=y opensm-static=y compat-dapl=y compat-dapl-devel=y @@ -54,6 +59,7 @@ dapl=y dapl-devel=y dapl-devel-static=y dapl-utils=y +dapl-debuginfo=y perftest=y mstflint=y tvflash=y @@ -63,6 +69,7 @@ rds-tools=y ibutils=y infiniband-diags=y qperf=y +qperf-debuginfo=y ofed-docs=y ofed-scripts=y mpi-selector=y @@ -72,9 +79,3 @@ openmpi_gcc=y mpitests_mvapich_gcc=y mpitests_mvapich2_gcc=y mpitests_openmpi_gcc=y -mvapich2_conf_impl=ofa -mvapich2_conf_romio=1 -mvapich2_conf_shared_libs=1 -mvapich2_conf_ckpt=0 -mvapich2_conf_vcluster=small -mvapich2_conf_dapl_provider=ib0 diff --git a/open_mpi_release_notes.txt b/open_mpi_release_notes.txt index 63ac611..1ade320 100644 --- a/open_mpi_release_notes.txt +++ b/open_mpi_release_notes.txt @@ -1,6 +1,6 @@ Open Fabrics Enterprise Distribution (OFED) Open MPI in OFED 1.5 Copyrights, License, and Release Notes - + December 2009 Open MPI Copyrights diff --git a/opensm_release_notes.txt b/opensm_release_notes.txt index a7352d0..f87bb6b 100644 --- a/opensm_release_notes.txt +++ b/opensm_release_notes.txt @@ -3,7 +3,7 @@ Version: OpenSM 3.2.x Repo: git://git.openfabrics.org/~sashak/management.git -Date: May 2009 +Date: December 2009 1 Overview ---------- diff --git a/qperf_release_notes.txt b/qperf_release_notes.txt index cc5a83b..28a5114 100644 --- a/qperf_release_notes.txt +++ b/qperf_release_notes.txt @@ -1,5 +1,5 @@ Distribution - Open Fabrics Enterprise Distribution (OFED) 1.4, December 2008 + Open Fabrics Enterprise Distribution (OFED) 1.5, December 2009 Summary qperf - Measure RDMA and IP performance diff --git a/rdma_cm_release_notes.txt b/rdma_cm_release_notes.txt index 1c71526..977ebfa 100644 --- a/rdma_cm_release_notes.txt +++ b/rdma_cm_release_notes.txt @@ -1,7 +1,7 @@ Open Fabrics Enterprise Distribution (OFED) - RDMA CM in OFED 1.4 Release Notes + RDMA CM in OFED 1.5 Release Notes - December 2008 + December 2009 =============================================================================== @@ -11,6 +11,7 @@ Table of Contents 2. New Features 3. Known Issues 4. Fixed bugs since OFED 1.3 +5. Fixed bugs since OFED 1.4.2 =============================================================================== 1. Overview @@ -91,6 +92,9 @@ following: - The reject status fix has been inserted for DAPL. =============================================================================== -5. Fixed bugs since OFED 1.3.1 +5. Fixed bugs since OFED 1.4.2 =============================================================================== -- Non +- examples: print errno on failures +- return errors from the library consistently +- initialize correct pthread condition +- fix race initializing library diff --git a/rds_release_notes.txt b/rds_release_notes.txt index 1d41241..0381f80 100644 --- a/rds_release_notes.txt +++ b/rds_release_notes.txt @@ -1,6 +1,6 @@ Open Fabrics Enterprise Distribution (OFED) - RDS in OFED 1.4.1 Release Notes - May 2009 + RDS in OFED 1.5 Release Notes + December 2009 =============================================================================== @@ -26,9 +26,10 @@ For details see RDS_README.txt and man 7 rds. 2. supported platforms =============================================================================== - RHEL4.0 Update 4,5,6 - RHEL5.0 Update 1,2 - SLES 10 + RHEL5.0 Update 3,4 + SLES 10 SP2,3 + SLES 11 + kernel.org 2.6.29, 2.6.30, 2.6.31 and 2.6.32 =============================================================================== 3. Installation & Configuration @@ -46,17 +47,7 @@ RDS_LOAD=yes 4. New Features =============================================================================== -RDS protocol version 3.1. - -RDS v3.1 is backwards compatible with v3.0 via protocol negotiation. - -Support for iWARP (bcopy mode only). - -Locking and scalability improvements. - -Credit-based flow control for iWARP transport. - -TCP transport removed. +Added TCP transport. =============================================================================== 5. Bug fixes and Enhancements since OFED 1.3.1 diff --git a/srp_release_notes.txt b/srp_release_notes.txt index 4061498..80cc543 100644 --- a/srp_release_notes.txt +++ b/srp_release_notes.txt @@ -1,8 +1,8 @@ - Open Fabrics Enterprise Distribution (OFED) - SRP in OFED 1.4 Release Notes - - December 2008 + Open Fabrics Enterprise Distribution (OFED) + SRP in OFED 1.5 Release Notes + + December 2009 ============================================================================== @@ -105,7 +105,7 @@ automatically. > max_cmd_per_lun - Default: 63 > max_sect (short for max_sectors) - sets the request size of a command > io_class - Default: 0x100 as in rev 16A of the specification - Note: In rev 10 the default was 0xff00 + Note: In rev 10 the default was 0xff00 > initiator_ext - Please refer to Section 9 (Multiple Connections...) d. See SRP Tools below for instructions on how the parameters in the echo command above may be obtained. @@ -371,20 +371,20 @@ Manual Activation of High Availability Initialization: (Execute after each boot of the driver) 1) Execute modprobe dm-multipath 2) Execute modprobe ib-srp - 3) Make sure you have created file /etc/udev/rules.d/91-srp.rules - as described above + 3) Make sure you have created file /etc/udev/rules.d/91-srp.rules + as described above 4) Execute for each port and each HCA: srp_daemon -c -e -R 300 -i -p (You can use another value for -R. See under the Known Issues section - the workaround for the rare race condition.) + the workaround for the rare race condition.) - This step can be performed by executing srp_daemon.sh, which sends + This step can be performed by executing srp_daemon.sh, which sends its log to /var/log/srp_daemon.log. - Now it is possible to access the SRP LUNs on /dev/mapper/. + Now it is possible to access the SRP LUNs on /dev/mapper/. - NOTE: It is possible for regular (non-SRP) LUNs to also be present; - the SRP LUNs may be identified by their names. You can configure the + NOTE: It is possible for regular (non-SRP) LUNs to also be present; + the SRP LUNs may be identified by their names. You can configure the /etc/multipath.conf file to change multipath behavior. @@ -517,8 +517,8 @@ successfully: echo -n \ id_ext=0000000000000001,ioc_guid=00066a0138000165,\ - dgid=fe8000000000000000066a0260000165,pkey=ffff,\ - service_id=0000494353535250,io_class=ff00,\ + dgid=fe8000000000000000066a0260000165,pkey=ffff,\ + service_id=0000494353535250,io_class=ff00,\ initiator_ext=0000000000000001 > \ /sys/class/inifiniband_srp/srp-mthca0-1/add_target -- 2.41.0