Open Fabrics Enterprise Distribution (OFED)
- Version 3.5-rc6
+ Version 3.5
Release Notes
- January 2013
+ February 2013
===============================================================================
These are the release notes of OpenFabrics Enterprise Distribution (OFED)
release 3.5. The OFED software package is composed of several software modules,
and is intended for use on a computer cluster constructed as an InfiniBand
-subnet or iWARP network.
+Fabric, an iWARP Network or a RoCE Fabric.
Note: If you plan to upgrade the OFED package on your cluster, please upgrade
all of its nodes to this new version.
- OpenFabrics core and ULPs:
- IB HCA drivers (mthca, mlx4, qib, ehca)
- iWARP RNIC driver (cxgb3, cxgb4, nes)
- - core
- - Upper Layer Protocols: IPoIB, SRP Initiator and target, iSER
+ - RoCE drivers (mlx4)
+- ib core
+ - Upper Layer Protocols: IPoIB, SRP Initiator and target, iSER
Initiator and target, RDS, uDAPL, qlgc_vnic and NFS-RDMA*.
- OpenFabrics utilities:
- OpenSM (OSM): InfiniBand Subnet Manager
- Diagnostic tools
- Performance tests
- Extra packages:
- - infinipath-psm: Performance-Scaled Messaging API, an accelerated
+ - infinipath-psm: Performance-Scaled Messaging API, an accelerated
interface to Intel(R) HCAs
- Sources of all software modules (under conditions mentioned in the modules'
LICENSE files)
- Documentation
+
1.2 Supported Platforms and Operating Systems
---------------------------------------------
o CPU architectures:
- x86_64
- x86
- ppc64
- - ia64
-
+
o Linux Operating Systems:
- RedHat EL6.2 2.6.32-220.el6
- RedHat EL6.3 2.6.32-279.el6
- kernel.org 3.5 *
* Minimal QA for these versions.
-
+
1.3 HCAs and RNICs Supported
----------------------------
-This release supports IB HCAs by Mellanox Technologies, Intel and IBM as
-well as iWARP RNICs by Chelsio Communications and Intel.
-
- o Mellanox Technologies HCAs (SDR, DDR and QDR Modes are Supported):
- - ConnectX(R) and ConnectX EN (fw-25408 Rev 2.9.1000)
- - ConnectX-2 (fw-ConnectX2 Rev 2.9.1000)
- - ConnectX-2 EN (fw-ConnectX2 Rev 2.9.1200)
- - ConnectX-3 (fw-ConnectX3 Rev 2.10.0700) (FDR and FDR10 Modes are Supported)
- - ConnectX-3 EN (fw-ConnectX3 Rev 2.10.0800)
-
- For official firmware versions please see:
- http://www.mellanox.com/content/pages.php?pg=firmware_download
-
- o Intel (formerly QLogic) HCAs:
- - Intel(R) True Scale DDR PCIe x8 and x16 HCAs
- - Intel(R) True Scale QDR PCIe x8 Gen2 HCAs
+This release supports IB HCAs by IBM, Intel and Mellanox Technologies, iWARP RNICs by Chelsio Communications and Intel and RoCE adapters by IBM and Mellanox.
+InfiniBand Adapters
o IBM HCAs:
- GX Dual-port SDR 4x IB HCA
- GX Dual-port SDR 12x IB HCA
- GX Dual-port DDR 4x IB HCA
- GX Dual-port DDR 12x IB HCA
+ o Intel (formerly QLogic) HCAs:
+ - Intel(R) True Scale DDR PCIe x8 and x16 HCAs
+ - Intel(R) True Scale QDR PCIe x8 Gen2 HCAs
+
+ o Mellanox Technologies HCAs (SDR, DDR and QDR Modes are Supported):
+- ConnectX(R) and ConnectX EN (fw-25408 Rev 2.9.1000)
+ - ConnectX-2 (fw-ConnectX2 Rev 2.9.1000)
+ - ConnectX-3 (fw-ConnectX3 Rev 2.11.0500) (FDR and FDR10 Modes are Supported)
+
+ For official firmware versions please see:
+ http://www.mellanox.com/content/pages.php?pg=firmware_download
+
+iWARP Adapters
o Chelsio RNICs:
- S310/S320 10GbE Storage Accelerators
- R310/R320 10GbE iWARP Adapters
+- T4 Based Adapters with the exception of the T420-SO-CR
o Intel RNICs:
- NE020 10Gb iWARP Adapter
+RoCE Adapters
+ o IBM
+- IBM Flex System EN4132 2-port 10 GbE RoCE
+- IBM EL27 PCIe LP 2-Port 10GbE RoCE SFP+ adapter
+- IBM EC28 PCIe 2-Port 10GbE RoCE SFP+ adapter
+
+ o Mellanox
+- ConnectX-2 EN (fw-ConnectX2 Rev 2.9.1200)
+ - ConnectX-3 EN (fw-ConnectX3 Rev 2.11.0500)
+
1.4 Switches Supported
----------------------
This release was tested with switches and gateways provided by the following
companies:
- - Intel (formerly QLogic)
- - Flextronics
- - Oracle (Sun)
- - Mellanox
+InfiniBand Switches
+ o Flextronics
+ - F-X430044
+ o Intel (formerly QLogic)
+- 12200
+ o Mellanox
+- IS-5030
+- SX6025
+- SX6036
+
+iWARP Switches
+ o Fujitisu
+- XG2000C 10Gb Ethernet Switch
+
+RoCE Switches
+ o Arista
+ o BLADE Network Technologies (BNT)
+ o Mellanox
+- SX1036
+- SX1024
+- SX1016
1.5 Third Party Packages
------------------------
The following third party packages have been tested with OFED 3.5:
- MVAPICH2 - 1.8
-- Intel MPI - 4.0.3/4.1
-- Open MPI - 1.6/1.6.1/1.6.3
+- Intel MPI - 4.1
+- Open MPI - 1.6/1.6.1
1.6 OFED Sources
----------------
URL: git://git.openfabrics.org/compat-rdma/linux-3.5.git
Branch: for-ofed-3.5
-- Linux kernel subtree that includes files relevant for the OFED project only.
+- Linux kernel sub-tree that includes files relevant for the OFED project only.
Based on v3.5. Used to shorten git clone time.
Note: the regular Linux git tree can be used as well.
===============================================================================
2. Change log
===============================================================================
-OFED-3.5-rc6 Main Changes from OFED 3.5-rc5
--------------------------------------------------------------------------------
-IB/rds: correct IB api usage
-perftest-2.0-0.27.g2edd80a
-
-OFED-3.5-rc5 Main Changes from OFED 3.5-rc4
--------------------------------------------------------------------------------
-perftest-2.0-0.25.g971259d
-
-OFED-3.5-rc4 Main Changes from OFED 3.5-rc3
--------------------------------------------------------------------------------
-RDMA/nes: Fixes for OFED-3.5 RC3
-RDMA/cxgb4: Keep the maximum number of stag limited to T4_MAX_NUM_STAG
-Updated packages:
- infinipath-psm-3.1-364.1140_open
-OFED-3.5-rc3 Main Changes from OFED 3.5-rc2
--------------------------------------------------------------------------------
-RDMA/nes: Fix for terminate timer crash
-RDMA/nes: Fix for crash when adding timer and timer is pending
-RDMA/nes: Fix for incorrect multicast address
-RDMA/nes: Fix for crash when adding timer
-RDMA/nes: Locking fix to prevent race conditions
-RDMA/nes: Changing nes driver version
-RDMA/nes: Fixes for PAU handling
-cxgb4: Initialize data structures before using.
-cxgb4: Fix initialization of SGE_CONTROL register
-cxgb4: Add embedded firmware and its configuration file support
-cxgb4/iw_cxgb4: Add LE hash collision bug fix for active and passive offloaded connections
-cxgb4: Fix unable to get UP event from the LLD
-cxgb4: Remove unnecessary #ifdef condition
-cxgb4: Fix build warnings introduced after T4 configuration file support
-cxgb4: Add support for T4 configuration file
-iw_cxgb4: only log rx_data warnings if cpl status is non zero.
-RDMA/cxgb4: fix mpa version mismatch
-RDMA/cxgb4: abort connections that receive unexpected streaming mode data
-RDMA/cxgb4: Insert hwtid in pass_accept_req instead in pass_establish
-RDMA/cxgb4: Remove the limitation on maximum number of STAGs
-linux-next-cherry-picks: Fix build failure for kernel < 3.2
-IB/qib: remove EBUSY patch
-ib_srp: Avoid that module removal can trigger a deadlock
-ib_srp: Unbreak build on SLES 11 SP2
-ib_srp: Add post-3.5 upstream patches
-ib_srp: Avoid use-after-free during module unload on pre-2.6.36 kernels
-
-Updated packages:
- mstflint-1.4-1.23.g5c2032a
- perftest-2.0-0.8.gaeb562e
- qperf-0.4.9
-
-OFED-3.5-rc2 Main Changes from OFED 3.5-rc1
--------------------------------------------------------------------------------
-compat-rdma: Add SRP backport
-compat-rdma: /etc/init.d/openibd: Fix LSB header
-compat-rdma: IB/qib: linux-3.6 patches backported
-compat-rdma: iw_cxgb4: Fix bug 2369 in OFED bugzilla
-compat-rdma: IB/qib: fix compliance regression in 3.5
-compat-rdma: RDMA/nes: Add linux-next-pending patches
-compat-rdma: RDMA/nes: Updated backports
-compat-rdma: NFSRDMA RHEL6.3 backport
-compat-rdma: NFSRDMA SLES11SP2 backport
-compat-rdma: linux-next-cherry-picks: RDMA/ucma.c: Different fix for ucma context uid=0, causing iWarp RDMA applications to fail in connection establishment
-
-Updated packages:
- infinipath-psm-3.0.1-115.1015_open
- perftest-1.4.0-0.80.gd1763bd
- qperf-0.4.7-0.2.gf3f7001
-
-OFED-3.5-rc1 Main Changes from OFED 3.2-rc1
--------------------------------------------------------------------------------
-1. Based on linux-3.5 kernel
-2. Updated packages:
- ibacm-1.0.7
- librdmacm-1.0.16
- dapl-2.0.36
-
- Management:
- libibumad-1.3.8
- infiniband-diags-1.6.1
- opensm-3.3.15
-
-3. Removed libsdp, sdpnetstat
-
-OFED-3.5-rc1 Main Changes from OFED 1.5.4.1
+OFED-3.5 Main Changes from OFED 1.5.4.1 and 1.5.4
-------------------------------------------------------------------------------
1. Based on linux-3.5 kernel
2. kernel-ib RPM renamed to compat-rdma
-3. Removed MPI packages
-4. Updated packages:
- libibverbs-1.1.6
- libmlx4-1.0.4
- dapl-2.0.35
- ibacm-1.0.6
- perftest-1.4.0-0.66.g9074687
-
- Management:
- libibmad-1.3.9
- opensm-3.3.14
- infiniband-diags-1.6.0
-
-
-OFED-1.5.4 Main Changes from OFED 1.5.3.2
--------------------------------------------------------------------------------
-1. Add kernel 3.0 support
-2. Add RHEL6.2 support
-3. Add NFSRDMA to the list of packages to be installed by default on supported
- platforms
-4. Add new InfiniBand link speeds: EDR/FDR/FDR10
-5. OFA Kernel: See ofa_kernel.changelog.txt
-
-6. Updated packages:
- libibverbs-1.1.4-1.24.gb89d4d7
- libmlx4-1.0.1-1.20.g6771d22
- libnes-1.1.3
- libcxgb3-1.3.1
- libcxgb4-1.2.0
- dapl-2.0.34
- ibacm-1.0.5
- librdmacm-1.0.15
- infinipath-psm-2.9-926.1005_open
- ibutils-1.5.7-0.1.g05a9d1a
- perftest-1.3.0-0.56.gd6f6093
- mstflint-1.4-1.18.g1adcfbf
-
- Management:
- libibmad-1.3.8
- infiniband-diags-1.5.12
- opensm-3.3.12
-
- MPI:
- mvapich2-1.7-5.2011svn5031
-
-OFED-1.5.3.2 Main Changes from OFED 1.5.3.1
--------------------------------------------------------------------------------
-1. Add RHEL5.7 support
-2. Add RHEL6.1 support
-3. Add kernel.org 2.6.37 - 2.6.39 support
-4. ib-bonding-0.9.0-43 (SLES10 SP4 support)
+3. Removed MPI packages and tests
+4. Removed SDP kernel module and libsdp, sdpnetstat
+5. XRC is supported in kernel only
-OFED-1.5.3.1 Main Changes from OFED 1.5.3
--------------------------------------------------------------------------------
-1. Fix QIB support on SLES10 SP3
-
-OFED-1.5.3 Main Changes from OFED 1.5.2
--------------------------------------------------------------------------------
-1. Updated packages:
- dapl-2.0.32
- librdmacm-1.0.14.1
- ibacm-1.0.4
- libibmad-1.3.7
- libibumad-1.3.7
- infiniband-diags-1.5.8
- opensm-3.3.9
- ibutils-1.5.7
- libibverbs-1.1.4-1.22.g7257cd3.tar.gz
- libmlx4-1.0.1-1.18.gb810a27.tar.gz
- libnes-1.1.1
- infinipath-psm-1.14
- libsdp-1.1.108-0.15.gd7fdb72
- mstflint-1.4-0.9.gc683022
- sdpnetstat-1.60-0.3.gb6e7425
- perftest-1.3.0-0.42.gf350d3d
-
- - MPI
- openmpi-1.4.3-1
- mvapich2-1.6-2
-
-2. Added RHEL6 support
-3. Added RHEL5.6 support
-4. Improved RAW Ethernet QP support (requires ConnectX/ConnectX2 FW >= 2.8.0600)
-5. NFSoRDMA was removed from the list of packages in the install script to
- prevent its installation by default.
- To force NFSoRDMA installation, add the following lines to the ofed.conf:
- nfsrdma=y
- rnfs-utils=y
-
- To get ofed.conf, run
- # ./install.pl -p
-
- After adding the lines above to ofed.conf, to install OFED run
- # ./install.pl -c ofed.conf
-
-OFED-1.5.2 Main Changes from OFED 1.5.1
--------------------------------------------------------------------------------
-1. Added RHEL5.5, SLES11 SP1 and kernel 2.6.33 support
-2. /etc/modprobe.conf deprecated
- ib_ipoib use /etc/modprobe.d/ib_ipoib.conf
- ib_sdp use /etc/modprobe.d/ib_sdp.conf
- mlx4_en use /etc/modprobe.d/mlx4_en.conf
-3. Added RAW Ethernet QP support in libibverbs
-4. Added multicast support in perftest
-5. Added new package: ibacm. See ibacm_release_notes.txt
-6. Added new package infinipath-psm.
+6. Removed support for the OSes:
+ - RHEL 5.X
+ - SLES 10 SPx
7. Updated packages:
- libibverbs to rev 1.1.4-0.14.gb6c138b
- libehca to rev 1.2.2-0.1.g69e1a88
- libmlx4 to rev 1.0-0.13.g4e5c43f
- libnes to rev 1.1.0
- librdmacm to rev 1.0.13
- dapl to rev 2.0.30
- compat-depl to rev 1.2.19
- libsdp to rev 1.1.103
- mstflint to rev 1.4-0.6.g63179f4
- perftest to rev 1.3.0-0.27
- rds-tools to rev 2.0.4
- Management:
- opensm to rev 3.3.7
- libibmad to rev 1.3.6
- libibumad to rev 1.3.6
- infiniband-diags to rev 1.5.7
- MPI:
- openmpi to rev 1.4.2-1
- mvapich2 to rev 1.5.1-2.p1
- mpitests to rev 3.2-923
-
-OFED-1.5.1 Main Changes from OFED 1.5
--------------------------------------------------------------------------------
-1. Added RoCEE support - see RoCEE_README.txt
-2. Added enhanced atomic operations to ConnectX (kernel only).
- See mlx4_release_notes.txt.
-3. Updated Open MPI to rev 1.4.1-2ofed
-4. Updated MVAPICH2 to rev 1.4.1
-5. Updated DAPL to rev 2.0.27
-6. Updated libnes to rev 1.0.1
-7. Updated librdmacm to rev 1.0.11
-8. Removed tvflash RPM
-9. NFS-RDMA is not supported on SLES10 SP3
-10. Fixed IPv6 support and IPv4 routing corner cases for RDMA CM
-11. Bug fixes
-
-OFED-1.5 Main Changes from OFED 1.4.2
--------------------------------------------------------------------------------
-Note: For details regarding the various changes, please see the release notes
-for each package in the docs directory.
-
- 2.1 General changes
- o Kernel code based on 2.6.30
- o libraries location - all userspace libraries can be downloaded from
- http://www.openfabrics.org/downloads/
- See BUILD_ID for exact locations
- o Qlogic moved the low level driver from ipath to qib.
-
- 2.2 SDP
- o Performance improvements
- o Zero copy in beta level
-
- 2.3 uDAPL
- o New UCM provider (ofa-v2-mlx4_0-1u) with IB UD-based CM per process.
- It is more scalable than rdma_cm (cma) or socket cm (scm).
- o Common code base with WinOF 2.1
- o Bug fixes
-
-
- 2.4 perftest
- o Renamed tests:
- ib_rdma_bw -> rdma_bw
- ib_rdma_lat -> rdma_lat
-
- 2.5 Management
- o OpenSM
- - Support for Mesh Analysis for LASH routing algorithm
- - Reloadable OpenSM configuration (preliminary implementation)
- - Routing paths sorted balancing (for UpDown and MinHops)
- - Weighted LID matrices calculation (for UpDown, MinHop and DOR)
- - I/O nodes connectivity (for FatTree)
-
- 2.6 MPI:
- a. OSU MVAPICH 1.2.0
- b. Open MPI 1.4
- c. OSU MVAPICH2 1.4
- d. MPI tests 3.2
-
- 2.7 iSER:
- o Available on:
- - kernel.org: 2.6.30 and higher
- - RHEL5.4
-
- 2.8 NFS-RDMA
- o Added support for RHEL5.4, SLES10 SP3, kernel.org 2.6.25 and 2.6.30.
- Kernels 2.6.26 and 2.6.27 are not supported
- o NFS-RDMA is in Beta level
+ - libibverbs-1.1.6
+ - librdmacm-1.0.16
+ - ibacm-1.0.7
+ - dapl-2.0.36
+ - opensm-3.3.15
+ - infiniband-diags-1.6.1
+ - libibmad-1.3.9
+ - libibumad-1.3.8
+ - infinipath-psm-3.1-364.1140_open
+ - libmlx4-1.0.4
+ - mstflint-1.4-1.23.g5c2032a
+ - perftest-2.0-0.27.g2edd80a
+ - qperf-0.4.9
+
===============================================================================
3. Known Issues
components of the OFED 3.5-rc2 release.
1. When upgrading from an earlier OFED version, the installation script does
- not stop the earlier OFED version prior to uninstalling it.
- Workaround: Stop the old OFED stack (/etc/init.d/openibd stop) before
+ not stop the earlier OFED version prior to uninstalling it.
+ Workaround: Stop the old OFED stack (/etc/init.d/openibd stop) before
upgrading to OFED 3.5 or reboot the server after OFED installation.
2. Memory registration by the user is limited according to administrator
setting. See "Pinning (Locking) User Memory Pages" in OFED_tips.txt for
7. On SLES11.x, set allow_unsupported_modules parameter to 1 in file:
/etc/modprobe.d/unsupported-modules. Without this the modules will not
load.
+8. RDS is supported only on kernel.org 3.5 and not on any distro OS.
Note: See the release notes of each component for additional issues.
+
Open Fabrics Enterprise Distribution (OFED)
Tips for Working with OFED 3.5
- August 2012
+ February 2013
===============================================================================
Table of Contents
2. Debug HOWTOs
3. Pinning (Locking) User Memory Pages
4. External Module Compilation Over OFED-3.5
-5. Adding/Deleting a patch to OFED package
-6. Adding vendor specific actions to the installation of OFED
-7. How to compile OFED sources manually
+5. Adding vendor specific actions to the installation of OFED
+6. How to compile OFED sources manually
===============================================================================
1. OFED Utilities
ofed-scripts:
git://git.openfabrics.org/~vlad/ofed_scripts.git compat-rdma
-ar.gz
compat-rdma:
git://git.openfabrics.org/compat-rdma/compat-rdma.git master
===============================================================================
-4. External Module Compilation Over OFED-1.5.1
+4. External Module Compilation Over OFED-3.5
===============================================================================
To build kernel modules depending on OFED's modules, take the Modules.symvers
See "Module versioning & Module.symvers" in the modules.txt from kernel
documentation (e.g. linux-2.6.30/Documentation/kbuild/modules.txt).
-
-===============================================================================
-5. Adding/Deleting a patch to OFED package
-===============================================================================
-If there is a need to add or delete a patch to OFED package one should use the
-ofed_patch.sh script, that is available under the docs directory.
-
-This script supports kernel sources only.
-
-Usage:
-Add patch to OFED:
-ofed_patch.sh --add
- --ofed|-o <path_to_ofed>
- --patch|-p <path_to_patch>
- --type|-t <kernel|backport <kernel tag>|addons <kernel tag>>
-
-Remove patch from OFED:
-ofed_patch.sh --remove
- --ofed|-o <path_to_ofed>
- --patch|-p <patch name>
- --type|-t <kernel|backport <kernel tag>|addons <kernel tag>>
-
-Examples:
-ofed_patch.sh --add --ofed /tmp/OFED-1.5.1/ --patch /tmp/cma_fix.patch --type kernel
-
-ofed_patch.sh --remove --ofed /tmp/OFED-1.5.1/ --patch cma_fix.patch --type kernel
-
===============================================================================
-6. Adding vendor specific actions to the installation of OFED
+5. Adding vendor specific actions to the installation of OFED
===============================================================================
Vendors that want to add actions to the install/uninstall process of OFED can
bind external scripts to hooks in install.pl and ofed_uninstall.sh.
-6.1 Specifying vendor scripts and configuration parameters
+5.1 Specifying vendor scripts and configuration parameters
-----------------------------------------------------------
This option is only available when installing ofed in non interactive mode.
Edit the OFED configuration file (ofed.conf ) and add the lines below (you don't have to use all of them).
Running ./install.pl -c ofed.conf in the OFED directory will now invoke the
relevant vendor specific actions.
-6.2. Requirements from vendor scripts
+5.2. Requirements from vendor scripts
-------------------------------------
The script files that are given to install.pl in ofed.conf should
- be located in the root directory of OFED
If vendor script fails it fails the entire installation.
-6.3 Skeleton for pre/post install vendor script
+5.3 Skeleton for pre/post install vendor script
-----------------------------------------------
install.pl passes some useful installation variables to the vendor pre/post
install scripts environment. See the example below for a typical usage.
exit 0
===============================================================================
-7. How to compile OFED sources manually
+6. How to compile OFED sources manually
===============================================================================
These are the instructions how to compile and install kernel and user parts "manually"
meaning without building the RPMs and without using the install.pl script.
-7.1 Compiling the kernel modules
+6.1 Compiling the kernel modules
--------------------------------
1. tar xzf OFED-3.5.tgz
2. rpm -ihv OFED-3.5/SRPMS/compat-rdma-3.5-OFED.3.5.src.rpm
2. INSTALL_MOD_DIR = "extra/ofa_kernel" for RedHat base Distros
INSTALL_MOD_DIR = "updates" for other Distros
-7.2 Compiling the user space libraries
+6.2 Compiling the user space libraries
--------------------------------------
To install user space library from the source RPM provided by OFED-1.5.1 manually,
do the following:
Open Fabrics Enterprise Distribution (OFED)
- Version 3.2-rc1
+ Version 3.5
README
- June 2012
+ February 2013
==============================================================================
Table of contents
==============================================================================
1. Overview
- 2. Contents of the OFED Distribution
- 3. Hardware and Software Requirements
- 4. How to Download and Extract the OFED Distribution
- 5. Installing OFED Software
- 6. Building OFED RPMs
- 7. IPoIB Configuration
- 8. Uninstalling OFED
- 9. Upgrading OFED
- 10. Configuration
- 11. Starting and Verifying the IB Fabric
- 12. Related Documentation
+ 2. How to Download and Extract the OFED Distribution
+ 3. Installing OFED Software
+ 4. Building OFED RPMs
+ 5. IPoIB Configuration
+ 6. Uninstalling OFED
+ 7. Upgrading OFED
+ 8. Configuration
+ 9. Starting and Verifying the IB Fabric
+ 10. Related Documentation
==============================================================================
1. Overview
==============================================================================
-This is the OpenFabrics Enterprise Distribution (OFED) version 3.2
+This is the OpenFabrics Enterprise Distribution (OFED) version 3.5
software package supporting InfiniBand and iWARP fabrics. It is composed
of several software modules intended for use on a computer cluster
constructed as an InfiniBand subnet or an iWARP network.
nodes in the cluster using any cluster-aware tools (such as pdsh).
==============================================================================
-2. OFED Package Contents
-==============================================================================
-
-The OFED Distribution package generates RPMs for installing the following:
-
- o OpenFabrics core and ULPs:
- - HCA drivers (mthca, mlx4, qib, ehca)
- - iWARP driver (cxgb3, cxgb4, nes)
- - core
- - Upper Layer Protocols: IPoIB, SRP Initiator and target, iSER
- Initiator and target, RDS, qlgc_vnic, uDAPL and NFS-RDMA
- o OpenFabrics utilities
- - OpenSM: InfiniBand Subnet Manager
- - Diagnostic tools
- - Performance tests
- o Extra packages
- - infinipath-psm: Performance-Scaled Messaging API, an accelerated
- interface to QLogic HCAs
- o Sources of all software modules (under conditions mentioned in the
- modules' LICENSE files)
- o Documentation
-
-==============================================================================
-3. Hardware and Software Requirements
-==============================================================================
-
-1) Server platform with InfiniBand HCA or iWARP RNIC (see OFED Distribution
- Release Notes for details)
-
-2) Linux operating system (see OFED Distribution Release Notes for details)
-
-3) Administrator privileges on your machine(s)
-
-4) Disk Space: - For Build & Installation: 300MB
- - For Installation only: 200MB
-
-5) For the OFED Distribution to compile on your machine, some software
- packages of your operating system (OS) distribution are required. These
- are listed here.
-
-OS Distribution Required Packages
---------------- ----------------------------------
-General:
-o Common to all gcc, glib, glib-devel, glibc, glibc-devel,
- glibc-devel-32bit (to build 32-bit libraries on x86_86
- and ppc64), zlib-devel, libstdc++-devel
-o RedHat, Fedora kernel-devel, rpm-build, redhat-rpm-config
-o SLES kernel-source, rpm
-
-Note: To build 32-bit libraries on x86_64 and ppc64 platforms, the 32-bit
- glibc-devel should be installed.
-
-Specific Component Requirements:
-o ibutils tcl-8.4, tcl-devel-8.4, tk, libstdc++-devel
-o mstflint libstdc++-devel (32-bit on ppc64), gcc-c++
-
-Note: The installer will warn you if you attempt to compile any of the
- above packages and do not have the prerequisites installed.
- On SLES, some of required RPMs can be found on SLES SDK DVD.
- E.g.: libgfortran43, kernel-source, ...
-
-==============================================================================
-4. How to Download and Extract the OFED Distribution
+2. How to Download and Extract the OFED Distribution
==============================================================================
1) Download the OFED-X.X.X.tgz file to your target Linux host.
tar xzvf OFED-X.X.X.tgz
==============================================================================
-5. Installing OFED Software
+3. Installing OFED Software
==============================================================================
1) Go to the directory into which the package was extracted:
o The OFED package is installed under <prefix> directory. Default prefix is /usr
o The kernel modules are installed under:
- /lib/modules/`uname -r`/updates/ - on SUSE Distros
- /lib/modules/`uname -r`/extra/ofa_kernel/ - on RedHat distros
+ /lib/modules/`uname -r`/updates/
o The package kernel include files are placed under <prefix>/src/compat-rdma/.
These includes should be used when building kernel modules which use
the Openfabrics stack. (Note that these includes, if needed, are
o man pages will be installed under /usr/share/man/.
==============================================================================
-6. Building OFED RPMs
+4. Building OFED RPMs
==============================================================================
1) Go to the directory into which the package was extracted:
installer on each node with the RPMs that were previously built.
==============================================================================
-7. IP-over-IB (IPoIB) Configuration
+5. IP-over-IB (IPoIB) Configuration
==============================================================================
Configuring IPoIB is an optional step during the installation. During
ONBOOT_ib0=1
==============================================================================
-8. Uninstalling OFED
+6. Uninstalling OFED
==============================================================================
There are two ways to uninstall OFED:
before removing the RPMs using the flag: --unload-modules
==============================================================================
-9. Upgrading OFED
+7. Upgrading OFED
==============================================================================
If an old OFED version is installed, it may be upgraded by installing a
your machine in order to start the new OFED stack.
==============================================================================
-10. Configuration
+8. Configuration
==============================================================================
Most of the OFED components can be configured or reconfigured after
list of the modules that will be loaded automatically upon boot can be
found in the /etc/infiniband/openib.conf file. Other configuration
files include:
-- SDP configuration file: /etc/libsdp.conf
- OpenSM configuration file: /etc/ofa/opensm.conf (for RedHat)
/etc/sysconfig/opensm (for SuSE) - should be
created manually if required.
==============================================================================
-11. Starting and Verifying the IB Fabric
+9. Starting and Verifying the IB Fabric
==============================================================================
1) If you rebooted your machine after the installation process completed,
IB interfaces should be up. If you did not reboot your machine, please
==============================================================================
-12. Related Documentation
+10. Related Documentation
==============================================================================
OFED documentation is located in the ofed-docs RPM. After
o README.txt
o OFED_Installation_Guide.txt
- o MPI_README.txt
o Examples of configuration files
o OFED_tips.txt
o HOWTO.build_ofed
Version: @VERSION@
Release: 0
License: GPL/BSD
-Url: http://git.openfabrics.org/git?p=ofed_1_5/docs.git;a=summary
+Url: http://git.openfabrics.org/git?p=compat-rdma/docs.git;a=summary
Group: Documentation/Man
Source: %{name}-%{version}.tar.gz
BuildRoot: %{?build_root:%{build_root}}%{!?build_root:/var/tmp/%{name}-%{version}-root}
cxgb3=y
cxgb4=y
nes=y
+qib=y
ipoib=y
srp=y
-srpt=y
-rds=y
+nfsrdma=y
compat-rdma=y
compat-rdma-devel=y
libibverbs=y
libibverbs-devel=y
libibverbs-devel-static=y
libibverbs-utils=y
+libibverbs-debuginfo=y
libmthca=y
libmthca-devel-static=y
+libmthca-debuginfo=y
libmlx4=y
libmlx4-devel=y
+libmlx4-debuginfo=y
libcxgb3=y
libcxgb3-devel=y
+libcxgb3-debuginfo=y
libcxgb4=y
libcxgb4-devel=y
+libcxgb4-debuginfo=y
libnes=y
libnes-devel-static=y
+libnes-debuginfo=y
+libipathverbs=y
+libipathverbs-devel=y
+libipathverbs-debuginfo=y
libibcm=y
libibcm-devel=y
+libibcm-debuginfo=y
libibumad=y
libibumad-devel=y
libibumad-static=y
+libibumad-debuginfo=y
libibmad=y
libibmad-devel=y
libibmad-static=y
+libibmad-debuginfo=y
ibsim=y
+ibsim-debuginfo=y
ibacm=y
librdmacm=y
librdmacm-utils=y
librdmacm-devel=y
+librdmacm-debuginfo=y
opensm=y
opensm-libs=y
opensm-devel=y
+opensm-debuginfo=y
opensm-static=y
compat-dapl=y
compat-dapl-devel=y
dapl-devel=y
dapl-devel-static=y
dapl-utils=y
+dapl-debuginfo=y
perftest=y
mstflint=y
srptools=y
rds-devel=y
ibutils=y
infiniband-diags=y
+qperf=y
+qperf-debuginfo=y
ofed-docs=y
ofed-scripts=y
infinipath-psm=y
Open Fabrics Enterprise Distribution (OFED)
- How To Build OFED 1.5.x
+ How To Build OFED 3.5
- December 2011
+ February 2013
==============================================================================
Example:
- ./build.pl --version 1.5.3-rc1 -p packages-ofed
+ ./build.pl --version 3.5-rc1 -p packages-ofed
- This command will create a package (i.e., subtree) called OFED-1.5.3-rc1
+ This command will create a package (i.e., subtree) called OFED-3.5-rc1
under /tmp/$USER/
==============================================================================
+++ /dev/null
-===============================================================================
- MLNX_EN driver for Mellanox Adapter Cards with 10GigE Support
- README for OFED 1.5.3
-
- March 2011
-===============================================================================
-
-Contents:
-=========
-1. Overview
-2. Ethernet Driver Usage and Configuration
-
-
-1. Overview
-===========
-MLNX_EN driver is composed from mlx4_core and mlx4_en kernel modules.
-
-The MLNX_EN driver release exposes the following capabilities:
-- Single/Dual port
-- Fibre Channel over Ethernet (FCoE)
-- Up to 16 Rx queues per port
-- 5 TX queues per port
-- Rx steering mode: Receive Core Affinity (RCA)
-- Tx arbitration mode: VLAN user-priority (off by default)
-- MSI-X or INTx
-- Adaptive interrupt moderation
-- HW Tx/Rx checksum calculation
-- Large Send Offload (i.e., TCP Segmentation Offload)
-- Large Receive Offload
-- IP reassembly offload for fragmented IP packets
-- Multi-core NAPI support
-- VLAN Tx/Rx acceleration (HW VLAN stripping/insertion)
-- HW VLAN filtering
-- HW multicast filtering
-- ifconfig up/down + mtu changes (up to 10K)
-- Ethtool support
-- Net device statistics
-
-
-2. Ethernet Driver Usage and Configuration
-==========================================
-
-- To assign an IP address to the interface run:
- #> ifconfig eth<x> <ip>
-
- where 'x' is the OS assigned interface number.
-
-- To check driver and device information run:
- #> ethtool -i eth<x>
-
- Example:
- #> ethtool -i eth2
- driver: mlx4_en (MT_0BD0110004)
- version: 1.5.2 (March 2010)
- firmware-version: 2.8.000
- bus-info: 0000:0e:00.0
-
-- To query stateless offload status run:
- #> ethtool -k eth<x>
-
-- To set stateless offload status run:
- #> ethtool -K eth<x> [rx on|off] [tx on|off] [sg on|off] [tso on|off]
-
-- To query interrupt coalescing settings run:
- #> ethtool -c eth<x>
-
-- By default, the driver uses adaptive interrupt moderation for the receive path,
- which adjusts the moderation time according to the traffic pattern.
- Adaptive moderation settings can be set by:
- #> ethtool -C eth<x> adaptive-rx on|off
-
-- To set interrupt coalescing settings run:
- #> ethtool -C eth<x> [rx-usecs N] [rx-frames N] [tx-usecs N] [tx-frames N]
-
- Note: usec settings correspond to the time to wait after the *last* packet
- sent/received before triggering an interrupt
-
-- To query pause frame settings run:
- #> ethtool -a eth<x>
-
-- To set pause frame settings run:
- #> ethtool -A eth<x> [rx on|off] [tx on|off]
-
-- To query ring size values run:
- #> ethtool -g eth<x>
-
-- To modify rings size run:
- #> ethtool -G eth<x> [rx <N>] [tx <N>]
-
-- To obtain additional device statistics, run:
- #> ethtool -S eth<x>
-
-- To perform a self diagnostics test, run:
- #> ethtool -t eth<x>
-
-
-The driver defaults to the following parameters:
-- Both ports are activated (i.e., a net device is created for each port)
-- The number of Rx rings for each port is the number of on-line CPUs
-- Per-core NAPI is enabled
-- LRO is enabled with 32 concurrent sessions per Rx ring
-
-Some of these values can be changed using module parameters, which are
-detailed by running:
-#> modinfo mlx4_en
-
-To set non-default values to module parameters, the following line should be
-added to /etc/modprobe.conf file:
- "options mlx4_en <param_name>=<value> <param_name>=<value> ..."
-
-Values of all parameters can be observed in /sys/module/mlx4_en/parameters/.
-
-
+++ /dev/null
-
- MPI in OFED 1.5.2 README
-
- September 2010
-
-===============================================================================
-Table of Contents
-===============================================================================
-1. Overview
-2. MVAPICH
-3. Open MPI
-4. MVAPICH2
-
-
-===============================================================================
-1. Overview
-===============================================================================
-Open Fabrics Enterprise Distribution (OFED)Three MPI stacks are included in
-this release of OFED:
-- MVAPICH 1.2.0
-- Open MPI 1.4.2
-- MVAPICH2 1.5.1
-
-Setup, compilation and run information of MVAPICH, Open MPI and MVAPICH2 is
-provided below in sections 2, 3 and 4 respectively.
-
-1.1 Installation Note
----------------------
-In Step 2 of the main menu of install.pl, options 2, 3 and 4 can install
-one or more MPI stacks. Please refer to docs/OFED_Installation_Guide.txt
-to learn about the different options.
-
-The installation script allows each MPI to be compiled using one or
-more compilers. Users need to set, per MPI stack installed, the PATH
-and/or LD_LIBRARY_PATH so as to install the desired compiled MPI stacks.
-
-1.2 MPI Tests
--------------
-OFED includes four basic tests that can be run against each MPI stack:
-bandwidth (bw), latency (lt), Intel MPI Benchmark, and Presta. The tests
-are located under: <prefix>/mpi/<compiler>/<mpi stack>/tests/,
-where <prefix> is /usr by default.
-
-1.4 Selecting Which MPI to Use: mpi-selector
---------------------------------------------
-Depending on how the OFED installer was run, multiple different MPI
-implementations may be installed on your system. The OFED installer
-will run an MPI selector tool during the installation process,
-presenting a menu-based interface to select which MPI implementation
-is set as the default for all users. This MPI selector tool can be
-re-run at any time by the administrator after the OFED installer
-completes to modify the site-wide default MPI implementation selection
-by invoking the "mpi-selector-menu" command (root access is typically
-required to change the site-wide default).
-
-The mpi-selector-menu command can also be used by non-administrative
-users to override the site-wide default MPI implementation selection
-by setting a per-user default. Specifically: unless a user runs the
-MPI selector tool to set a per-user default, their environment will be
-setup for the site-wide default MPI implementation.
-
-Note that the default MPI selection does *not* affect the shell from
-which the command was invoked (or any other shells that were already
-running when the MPI selector tool was invoked). The default
-selection is only changed for *new* shells started after the selector
-tool was invoked. It is recommended that once the default MPI
-implementation is changed via the selector tool, users should logout
-and login again to ensure that they have a consistent view of the
-default MPI implementation. Other tools can be used to change the MPI
-environment in the current shell, such as the environment modules
-software package (which is not included in the OFED software package;
-see http://modules.sourceforge.net/ for details).
-
-Note that the site-wide default is set in a file that is typically not
-on a networked file system, and is therefore specific to the host on
-which it was run. As such, it is recommended to run the
-mpi-selector-menu command on all hosts in a cluster, picking the same
-default MPI implementation on each. It may be more convenient,
-however, to use the mpi-selector command in script-based scenarios
-(such as running on every host in a cluster); mpi-selector effects all
-the same functionality as mpi-selector-menu, but is intended for
-automated environments. See the mpi-selector(1) manual page for more
-details.
-
-Additionally, per-user defaults are set in a file in the user's $HOME
-directory. If this directory is not on a network-shared file system
-between all hosts that will be used for MPI applications, then it also
-needs to be propagated to all relevant hosts.
-
-Note: The MPI selector tool typically sets the PATH and/or
-LD_LIBRARY_PATH for a given MPI implementation. This step can, of
-course, also be performed manually by a user or on a site-wide basis.
-The MPI selector tool simply bundles up this functionality in a
-convenient set of command line tools and menus.
-
-1.4 Updating MPI Installations
-------------------------------
-Note that all of the MPI implementations included in the OFED software
-package are the versions that were available when OFED v1.5 was
-released. They have been QA tested with this version of OFED and are
-fully supported.
-
-However, note that administrators can go to the web sites of each MPI
-implementation and download / install newer versions after OFED has
-been successfully installed. There is nothing specific about the
-OFED-included MPI software packages that prohibit installing
-newer/other MPI implementations.
-
-It should be also noted that versions of MPI released after OFED v1.5
-are not supported by OFED. But since each MPI has its own release
-schedule and QA process (each of which involves testing with the OFED
-stack), it may sometimes be desirable -- or even advisable, depending
-on how old the MPI implementations are that are included in OFED -- to
-download install a newer version of MPI.
-
-The web sites of each MPI implementation are listed below:
-
-- Open MPI: http://www.open-mpi.org/
-- MVAPICH: http://mvapich.cse.ohio-state.edu/
-- MVAPICH2: http://mvapich.cse.ohio-state.edu/overview/mvapich2/
-
-===============================================================================
-2. MVAPICH MPI
-===============================================================================
-
-This package is a 1.2.0 version of the MVAPICH software package,
-and is the officially supported MPI stack for this release of OFED.
-See http://mvapich.cse.ohio-state.edu for more details.
-
-
-2.1 Setting up for MVAPICH
---------------------------
-To launch MPI jobs, its installation directory needs to be included
-in PATH and LD_LIBRARY_PATH. To set them, execute one of the following
-commands:
- source <prefix>/mpi/<compiler>/<mpi stack>/bin/mpivars.sh
- -- when using sh for launching MPI jobs
- or
- source <prefix>/mpi/<compiler>/<mpi stack>/bin/mpivars.csh
- -- when using csh for launching MPI jobs
-
-
-2.2 Compiling MVAPICH Applications:
------------------------------------
-***Important note***:
-A valid Fortran compiler must be present in order to build the MVAPICH MPI
-stack and tests.
-
-The default gcc-g77 Fortran compiler is provided with all RedHat Linux
-releases. SuSE distributions earlier than SuSE Linux 9.0 do not provide
-this compiler as part of the default installation.
-
-The following compilers are supported by OFED's MVAPICH package: Gcc,
-Intel, Pathscale and PGI. The install script prompts the user to choose
-the compiler with which to build the MVAPICH RPM. Note that more
-than one compiler can be selected simultaneously, if desired.
-
-For details see:
- http://mvapich.cse.ohio-state.edu/support
-
-To review the default configuration of the installation, check the default
-configuration file: <prefix>/mpi/<compiler>/<mpi stack>/etc/mvapich.conf
-
-2.3 Running MVAPICH Applications:
----------------------------------
-Requirements:
-o At least two nodes. Example: mtlm01, mtlm02
-o Machine file: Includes the list of machines. Example: /root/cluster
-o Bidirectional rsh or ssh without a password
-
-Note: ssh will be used unless -rsh is specified. In order to use
-rsh, add to the mpirun_rsh command the parameter: -rsh
-
-*** Running OSU tests ***
-
-/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/osu_benchmarks-3.1.1/osu_bw
-/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/osu_benchmarks-3.1.1/osu_latency
-/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/osu_benchmarks-3.1.1/osu_bibw
-/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/osu_benchmarks-3.1.1/osu_bcast
-
-*** Running Intel MPI Benchmark test (Full test) ***
-
-/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/IMB-3.2/IMB-MPI1
-
-*** Running Presta test ***
-
-/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/presta-1.4.0/com -o 100
-/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/presta-1.4.0/glob -o 100
-/usr/mpi/gcc/mvapich-1.2.0/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich-1.2.0/tests/presta-1.4.0/globalop
-
-
-===============================================================================
-3. Open MPI
-===============================================================================
-
-Open MPI is a next-generation MPI implementation from the Open MPI
-Project (http://www.open-mpi.org/). Version 1.4 of Open MPI is
-included in this release, which is also available directly from the
-main Open MPI web site.
-
-A working Fortran compiler is not required to build Open MPI, but some
-of the included MPI tests are written in Fortran. These tests will
-not compile/run if Open MPI is built without Fortran support.
-
-The following compilers are supported by OFED's Open MPI package: GNU,
-Pathscale, Intel, or Portland. The install script prompts the user
-for the compiler with which to build the Open MPI RPM. Note that more
-than one compiler can be selected simultaneously, if desired.
-
-Users should check the main Open MPI web site for additional
-documentation and support. (Note: The FAQ file considers OpenFabrics
-tuning among other issues.)
-
-3.1 Setting up for Open MPI
----------------------------
-Selecting to use Open MPI via the mpi-selector-mpi and mpi-selector
-tools will perform all the necessary setup for users to build and run
-Open MPI applications. If you use the MPI selector tools, you can
-skip the rest of this section.
-
-If you do not wish to use the MPI selector tools, the Open MPI team
-strongly advises users to put the Open MPI installation directory in
-their PATH and LD_LIBRARY_PATH. This can be done at the system level
-if all users are going to use Open MPI. Specifically:
-
-- add <prefix>/bin to PATH
-- add <prefix>/lib to LD_LIBRARY_PATH
-
-<prefix> is the directory where the desired Open MPI instance was
-installed ("instance" refers to the compiler used for Open MPI
-compilation at install time.).
-
-If you are using a job scheduler to launch MPI jobs (e.g., SLURM,
-Torque), setting the PATH and LD_LIBRARY_PATH is still required, but
-it does not need to be set in your shell startup files. Procedures
-describing how to add these values to PATH and LD_LIBRARY_PATH are
-described in detail at:
-
- http://www.open-mpi.org/faq/?category=running
-
-3.2 Open MPI Installation Support / Updates
--------------------------------------------
-The OFED package will install Open MPI with support for TCP, shared
-memory, and the OpenFabrics network stacks. No other networks are
-supported by the OFED Open MPI installation.
-
-Open MPI supports a wide variety of run-time environments. The OFED
-installer will not include support for all of them, however (e.g.,
-Torque and PBS-based environments are not supported by the
-OFED-installed Open MPI).
-
-The ompi_info command can be used to see what support was installed;
-look for plugins for your specific environment / network / etc. If
-you do not see them, the OFED installer did not include support for
-them.
-
-As described above, administrators or users can go to the Open MPI web
-site and download / install either a newer version of Open MPI (if
-available), or the same version with different configuration options
-(e.g., support for Torque / PBS-based environments).
-
-3.3 Compiling Open MPI Applications
------------------------------------
-(copied from http://www.open-mpi.org/faq/?category=mpi-apps -- see
-this web page for more details)
-
-The Open MPI team strongly recommends that you simply use Open MPI's
-"wrapper" compilers to compile your MPI applications. That is, instead
-of using (for example) gcc to compile your program, use mpicc. Open
-MPI provides a wrapper compiler for four languages:
-
- Language Wrapper compiler name
- ------------- --------------------------------
- C mpicc
- C++ mpiCC, mpicxx, or mpic++
- (note that mpiCC will not exist
- on case-insensitive file-systems)
- Fortran 77 mpif77
- Fortran 90 mpif90
- ------------- --------------------------------
-
-Note that if no Fortran 77 or Fortran 90 compilers were found when
-Open MPI was built, Fortran 77 and 90 support will automatically be
-disabled (respectively).
-
-If you expect to compile your program as:
-
- > gcc my_mpi_application.c -lmpi -o my_mpi_application
-
-Simply use the following instead:
-
- > mpicc my_mpi_application.c -o my_mpi_application
-
-Specifically: simply adding "-lmpi" to your normal compile/link
-command line *will not work*. See
-http://www.open-mpi.org/faq/?category=mpi-apps if you cannot use the
-Open MPI wrapper compilers.
-
-Note that Open MPI's wrapper compilers do not do any actual compiling
-or linking; all they do is manipulate the command line and add in all
-the relevant compiler / linker flags and then invoke the underlying
-compiler / linker (hence, the name "wrapper" compiler). More
-specifically, if you run into a compiler or linker error, check your
-source code and/or back-end compiler -- it is usually not the fault of
-the Open MPI wrapper compiler.
-
-3.4 Running Open MPI Applications:
-----------------------------------
-Open MPI uses either the "mpirun" or "mpiexec" commands to launch
-applications. If your cluster uses a resource manager (such as
-SLURM), providing a hostfile is not necessary:
-
- > mpirun -np 4 my_mpi_application
-
-If you use rsh/ssh to launch applications, they must be set up to NOT
-prompt for a password (see http://www.open-mpi.org/faq/?category=rsh
-for more details on this topic). Moreover, you need to provide a
-hostfile containing a list of hosts to run on.
-
-Example:
-
- > cat hostfile
- host1.example.com
- host2.example.com
- host3.example.com
- host4.example.com
-
- > mpirun -np 4 -hostfile hostfile my_mpi_application
- (application runs on all 4 hosts)
-
-In the following examples, replace <N> with the number of hosts to run on,
-and <HOSTFILE> with the filename of a valid hostfile listing the hosts
-to run on (unless you are running under a supported resource manager,
-in which case a hostfile is unnecessary).
-
-Also note that Open MPI is highly run-time tunable. There are many
-options that can be tuned to obtain optimal performance of your MPI
-applications (see the Open MPI web site / FAQ for more information:
-http://www.open-mpi.org/faq/).
-
- - <N> is an integer indicating how many MPI processes to run (e.g., 2)
- - <HOSTFILE> is the filename of a hostfile, as described above
-
-Example 1: Running the OSU bandwidth:
-
- > cd /usr/mpi/gcc/openmpi-1.4.1/tests/osu_benchmarks-3.1.1
- > mpirun -np <N> -hostfile <HOSTFILE> osu_bw
-
-Example 2: Running the Intel MPI Benchmark benchmarks:
-
- > cd /usr/mpi/gcc/openmpi-1.4.1/tests/IMB-3.2
- > mpirun -np <N> -hostfile <HOSTFILE> IMB-MPI1
-
- --> Note that the version of IMB-EXT that ships in this version of
- OFED contains a bug that will cause it to immediately error
- out when run with Open MPI.
-
-Example 3: Running the Presta benchmarks:
-
- > cd /usr/mpi/gcc/openmpi-1.4.1/tests/presta-1.4.0
- > mpirun -np <N> -hostfile <HOSTFILE> com -o 100
-
-NOTE: In order to run Open MPI over RoCCE (RDMAoE) network, follow MCA parameter
- is required:
- --mca btl_openib_cpc_include rdmacm
-
-
-3.5 More Open MPI Information
------------------------------
-Much, much more information is available about using and tuning Open
-MPI (to include OpenFabrics-specific tunable parameters) on the Open
-MPI web site FAQ:
-
- http://www.open-mpi.org/faq/
-
-Users who cannot find the answers that they are looking for, or are
-experiencing specific problems should consult the "how to get help" web
-page for more information:
-
- http://www.open-mpi.org/community/help/
-
-
-===============================================================================
-4. MVAPICH2 MPI
-===============================================================================
-
-MVAPICH2 is an MPI-2 implementation which includes all MPI-1 features.
-It is based on MPICH2 and MVICH. MVAPICH2 provides many features including
-fault-tolerance with checkpoint-restart, RDMA_CM support, iWARP support,
-optimized collectives, on-demand connection management, multi-core optimized
-and scalable shared memory support, and memory hook with ptmalloc2 library
-support. The ADI-3-level design of MVAPICH2 supports many features including:
-MPI-2 functionalities (one-sided, collectives and data-type), multi-threading
-and all MPI-1 functionalities. It also supports a wide range of platforms
-(architecture, OS, compilers, InfiniBand adapters and iWARP adapters). More
-information can be found on the MVAPICH2 project site:
-
-http://mvapich.cse.ohio-state.edu/overview/mvapich2/
-
-A valid Fortran compiler must be present in order to build the MVAPICH2
-MPI stack and tests. The following compilers are supported by OFED's
-MVAPICH2 MPI package: gcc, intel, pgi, and pathscale. The install script
-prompts the user to choose the compiler with which to build the MVAPICH2
-MPI RPM. Note that more than one compiler can be selected simultaneously,
-if desired.
-
-The install script prompts for various MVAPICH2 build options as detailed
-below:
-
-
-- Implementation (OFA or uDAPL) [default "OFA"]
- - OFA (IB and iWARP) Options:
- - ROMIO Support [default Y]
- - Shared Library Support [default Y]
- - Checkpoint-Restart Support [default N]
- * requires an installation of BLCR and prompts for the
- BLCR installation directory location
- - uDAPL Options:
- - ROMIO Support [default Y]
- - Shared Library Support [default Y]
- - Cluster Size [default "Small"]
- - I/O Bus [default "PCI-Express"]
- - Link Speed [default "SDR"]
- - Default DAPL Provider [default ""]
- * the default provider is determined based on detected OS
-
-For non-interactive builds where no MVAPICH2 build options are stored in
-the OFED configuration file, the default settings are:
-
-Implementation: OFA
-ROMIO Support: Y
-Shared Library Support: Y
-Checkpoint-Restart Support: N
-
-
-4.1 Setting up for MVAPICH2
----------------------------
-Selecting to use MVAPICH2 via the MPI selector tools will perform
-most of the setup necessary to build and run MPI applications with
-MVAPICH2. If one does not wish to use the MPI Selector tools, using
-the following settings should be enough:
-
- - add <prefix>/bin to PATH
-
-The <prefix> above is the directory where the desired MVAPICH2
-instance was installed ("instance" refers to the path based on
-the RPM package name, including the compiler chosen during the
-install). It is also possible to source the following files
-in order to setup the proper environment:
-
-source <prefix>/bin/mpivars.sh [for Bourne based shells]
-source <prefix>/bin/mpivars.csh [for C based shells]
-
-In addition to the user environment settings handled by the MPI selector
-tools, some other system settings might need to be modified. MVAPICH2
-requires the memlock resource limit to be modified from the default
-in /etc/security/limits.conf:
-
-* soft memlock unlimited
-
-MVAPICH2 requires bidirectional rsh or ssh without a password to work.
-The default is ssh, and in this case it will be required to add the
-following line to the /etc/init.d/sshd script before sshd is started:
-
-ulimit -l unlimited
-
-It is also possible to specify a specific size in kilobytes instead
-of unlimited if desired.
-
-The MVAPICH2 OFA build requires an /etc/mv2.conf file specifying the
-IP address of an Infiniband HCA (IPoIB) for RDMA-CM functionality
-or the IP address of an iWARP adapter for iWARP functionality if
-either of those are desired. This is not required by default, unless
-either of the following runtime environment variables are set when
-using the OFA MVAPICH2 build:
-
-RDMA-CM
--------
-MV2_USE_RDMA_CM=1
-
-iWARP
------
-MV2_USE_IWARP_MODE=1
-
-Otherwise, the OFA build will work without an /etc/mv2.conf file using
-only the Infiniband HCA directly.
-
-The MVAPICH2 uDAPL build requires an /etc/dat.conf file specifying the
-DAPL provider information. The default DAPL provider is chosen at
-build time, with a default value of "ib0", however it can also be
-specified at runtime by setting the following environment variable:
-
-MV2_DEFAULT_DAPL_PROVIDER=<interface>
-
-More information about MVAPICH2 can be found in the MVAPICH2 User Guide:
-
-http://mvapich.cse.ohio-state.edu/support/
-
-
-4.2 Compiling MVAPICH2 Applications
------------------------------------
-The MVAPICH2 compiler command for each language are:
-
-Language Compiler Command
--------- ----------------
-C mpicc
-C++ mpicxx
-Fortran 77 mpif77
-Fortran 90 mpif90
-
-The system compiler commands should not be used directly. The Fortran 90
-compiler command only exists if a Fortran 90 compiler was used during the
-build process.
-
-
-4.3 Running MVAPICH2 Applications
----------------------------------
-4.3.1 Running MVAPICH2 Applications with mpirun_rsh
----------------------------------------------------
->From release 1.2, MVAPICH2 comes with a faster and more scalable startup based
-on mpirun_rsh. To launch a MPI job with mpirun_rsh, password-less ssh needs to
-be enabled across all nodes.
-
-Note: ssh will be used by default. In order to use rsh, use the -rsh option on
-the mpirun_rsh command line. For more options, see mpirun_rsh -help or the
-MVAPICH2 user guide.
-
-*** Running 4 processes on 4 nodes ***
-
-$ cat > hostfile
-node1
-node2
-node3
-node4
-$ mpirun_rsh -np 4 -hostfile hostfile /path/to/my_mpi_app
-
-*** Running OSU tests ***
-
-/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.1.1/osu_bw
-/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.1.1/osu_latency
-/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.1.1/osu_bibw
-/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.1.1/osu_bcast
-
-*** Running Intel MPI Benchmark test (Full test) ***
-
-/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/IMB-3.2/IMB-MPI1
-
-*** Running Presta test ***
-
-/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/presta-1.4.0/com -o 100
-/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/presta-1.4.0/glob -o 100
-/usr/mpi/gcc/mvapich2-1.2p1/bin/mpirun_rsh -np 2 -hostfile /root/cluster /usr/mpi/gcc/mvapich2-1.2p1/tests/presta-1.4.0/globalop
-
-4.3.2 Running MVAPICH2 Applications with mpd and mpiexec
---------------------------------------------------------
-Launching processes in MVAPICH2 is a two step process. First, mpdboot must
-be used to launch MPD daemons on the desired hosts. Second, the mpiexec
-command is used to launch the processes. MVAPICH2 requires bidirectional
-ssh or rsh without a password. This is specified when the MPD daemons are
-launched with the mpdboot command through the --rsh command line option.
-The default is ssh. Once the processes are finished, stopping the MPD
-daemons with the mpdallexit command should be done. The following example
-shows the basic procedure:
-
-4 Processes on 4 Hosts Example:
-
-$ cat >hostsfile
-node1.example.com
-node2.example.com
-node3.example.com
-node4.example.com
-
-$ mpdboot -n 4 -f ./hostsfile
-
-$ mpiexec -n 4 ./my_mpi_application
-
-$ mpdallexit
-
-It is also possible to use the mpirun command in place of mpiexec. They are
-actually the same command in MVAPICH2, however using mpiexec is preferred.
-
-It is possible to run more processes than hosts. In this case, multiple
-processes will run on some or all of the hosts used. The following examples
-demonstrate how to run the MPI tests. The default installation prefix and
-gcc version of MVAPICH2 are shown. In each case, it is assumed that a hosts
-file has been created in the specific directory with two hosts.
-
-OSU Tests Example:
-
-$ cd /usr/mpi/gcc/mvapich2-1.2p1/tests/osu_benchmarks-3.1.1
-$ mpdboot -n 2 -f ./hosts
-$ mpiexec -n 2 ./osu_bcast
-$ mpiexec -n 2 ./osu_bibw
-$ mpiexec -n 2 ./osu_bw
-$ mpiexec -n 2 ./osu_latency
-$ mpdallexit
-
-Intel MPI Benchmark Example:
-
-$ cd /usr/mpi/gcc/mvapich2-1.2p1/tests/IMB-3.2
-$ mpdboot -n 2 -f ./hosts
-$ mpiexec -n 2 ./IMB-MPI1
-$ mpdallexit
-
-Presta Benchmarks Example:
-
-$ cd /usr/mpi/gcc/mvapich2-1.2p1/tests/presta-1.4.0
-$ mpdboot -n 2 -f ./hosts
-$ mpiexec -n 2 ./com -o 100
-$ mpiexec -n 2 ./glob -o 100
-$ mpiexec -n 2 ./globalop
-$ mpdallexit
This package contains a burning tool and diagnostic tools for Mellanox
manufactured HCA/NIC cards. It also provides access to the relevant source
code. Please see the file LICENSE for licensing details.
- This package is based on a subset of the Mellanox Firmware Tools (MFT) package.
- For a full documentation of the MFT package, please refer to the downloads page
- in Mellanox web site.
+ This package is based on a subset of the Mellanox Firmware Tools (MFT) package.
+ For a full documentation of the MFT package, please refer to the downloads page
+ in Mellanox web site.
----------------------------------------------------------------------------
NOTE:
for later analysis by Mellanox.
e) mstvpd
This utility dumps the on-card VPD.
- f) mstmcra
- This debug utility reads a single word from the device configuration space.
+ f) mstmcra
+ This debug utility reads a single word from the device configuration space.
3) Installation
a) Build the mstflint utility. This package is built using a standard
autotools method.
-
+
Example:
> ./configure
> make
# List all Mellanox devices
> /sbin/lspci -d 15b3:
02:00.0 Ethernet controller: Mellanox Technologies MT25448 [ConnectX EN 10GigE, PCIe 2.0 2.5GT/s] (rev a0)
-
+
# Use mstregdump to dump HW registers, using PCI config cycles
> mstregdump /proc/bus/pci/02/00.0 > crdump.log
Note: Typically, you will need root privileges for hardware access
- d) Accessing a multi-function device:
+ d) Accessing a multi-function device:
- In some configuration, the CA device identifies as a multi-function device on PCI. E.G.:
- > /sbin/lspci -d 15b3:
- 07:00.0 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
- 07:00.1 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
- 07:00.2 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
- 07:00.3 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
- 07:00.4 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
- 07:00.5 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
- 07:00.6 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
- 07:00.7 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
-
- These multiple "logical" devices are actually a single physical device, so firmware update or "physical"
- diagnostics should be run only on one of the functions.
-
- When the device driver is loaded, only the primary physical function of the device can be accessed.
- In Linux that would typically be function 0. This function can be accessed using memory mapping, aas
- described in sub section a) above. E.G.:
- > mstflint -d 07:00.0 q
-
- When the device driver is not loaded, all the functions can be accessed using configuration cycles, as
- described in sub section c) above. It is recommended to use function 0 for FW update or diagnostics, E.G.:
- > mstflint -d /proc/bus/pci/07/00.0 q
+ In some configuration, the CA device identifies as a multi-function device on PCI. E.G.:
+ > /sbin/lspci -d 15b3:
+ 07:00.0 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
+ 07:00.1 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
+ 07:00.2 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
+ 07:00.3 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
+ 07:00.4 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
+ 07:00.5 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
+ 07:00.6 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
+ 07:00.7 Ethernet controller: Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
+
+ These multiple "logical" devices are actually a single physical device, so firmware update or "physical"
+ diagnostics should be run only on one of the functions.
+
+ When the device driver is loaded, only the primary physical function of the device can be accessed.
+ In Linux that would typically be function 0. This function can be accessed using memory mapping, as
+ described in sub section a) above. E.G.:
+ > mstflint -d 07:00.0 q
+
+ When the device driver is not loaded, all the functions can be accessed using configuration cycles, as
+ described in sub section c) above. It is recommended to use function 0 for FW update or diagnostics, E.G.:
+ > mstflint -d /proc/bus/pci/07/00.0 q
5) Usage (mstflint):
- Read mstflint usage. Enter "./mstflint -h" for a short help message, or
- "./mstflint -hh" for a detailed help message.
-
- Obtaining firmware files:
- If you purchased your card from Mellanox Technologies, please use the
- Mellanox website (www.mellanox.com, under 'Firmware' downloads) to
- download the firmware for your card.
- If you purchased your card from a vendor other than Mellanox, get a
- specific firmware configuration (INI) file from your HCA card vendor and
- generate the binary image.
-
- Use mstflint to burn a device according to the burning instructions in
- "mstflint -hh" and in Mellanox web site firmware page.
+ Read mstflint usage. Enter "./mstflint -h" for a short help message, or
+ "./mstflint -hh" for a detailed help message.
+
+ Obtaining firmware files:
+ If you purchased your card from Mellanox Technologies, please use the
+ Mellanox website (www.mellanox.com, under 'Firmware' downloads) to
+ download the firmware for your card.
+ If you purchased your card from a vendor other than Mellanox, get a
+ specific firmware configuration (INI) file from your HCA card vendor and
+ generate the binary image.
+
+ Use mstflint to burn a device according to the burning instructions in
+ "mstflint -hh" and in Mellanox web site firmware page.
6) Usage (mstregdump):
- An internal register dump is displayed to the standard output.
- Please store it in a file for analysis by Mellanox.
+ An internal register dump is displayed to the standard output.
+ Please store it in a file for analysis by Mellanox.
- Example:
- > mstregdump mthca0 > dumpfile
+ Example:
+ > mstregdump mthca0 > dumpfile
7) Usage (mstvpd):
A VPD dump is displayed to the standard output.
- A list of keywords to dump can be supplied after the -- flag
- to apply an output filter.
-
- Examples:
- > mstvpd mlx4_0
- ID: Hawk Dual Port
- PN: MNPH29C-XTR
- EC: X2
- SN: MT1001X00749
- V0: PCIe Gen2 x8
- V1: N/A
- YA: N/A
- RW:
+ A list of keywords to dump can be supplied after the -- flag
+ to apply an output filter.
+
+ Examples:
+ > mstvpd mlx4_0
+ ID: Hawk Dual Port
+ PN: MNPH29C-XTR
+ EC: X2
+ SN: MT1001X00749
+ V0: PCIe Gen2 x8
+ V1: N/A
+ YA: N/A
+ RW:
> mstvpd mlx4_0 -- PN ID
- PN: MNPH29C-XTR
- ID: Hawk Dual Port
+ PN: MNPH29C-XTR
+ ID: Hawk Dual Port
8) Problem Reporting:
- Please collect the following information when reporting issues:
-
- uname -a
- cat /etc/issue
- cat /proc/bus/pci/devices
- mstflint -vv
- lspci
- mstflint -d 02:00.0 v
- mstflint -d 02:00.0 q
- mstvpd 02:00.0
+ Please collect the following information when reporting issues:
+
+ uname -a
+ cat /etc/issue
+ cat /proc/bus/pci/devices
+ mstflint -vv
+ lspci
+ mstflint -d 02:00.0 v
+ mstflint -d 02:00.0 q
+ mstvpd 02:00.0
Open Fabrics Enterprise Distribution (OFED)
- Performance Tests README for OFED 1.5
+ Performance Tests README for OFED 2.0
- December 2010
+ January 2013
2. Notes on Testing Methodology
3. Test Descriptions
4. Running Tests
+5. Known Issues
===============================================================================
1. Overview
The collection conatains a set of BW and latency benchmark such as :
- * Read - ib_read_bw and ib_read_lat.
- * Write - ib_write_bw, ib_write_bw_postlist and ib_write_lat.
- * Send - ib_send_bw and ib_send_lat.
- * RDMA - rdma_bw and rdma_lat.
- * Additional benchmark: ib_clock_test.
+ * Read - ib_read_bw and ib_read_lat.
+ * Write - ib_write_bw and ib_wriet_lat.
+ * Send - ib_send_bw and ib_send_lat.
+ * Atomic - ib_atomic_bw and ib_atomic_lat
+ * Raw Etherent (when working with MOFED2) - raw_ethernet_bw.
-Please post results/observations/bugs/remarks to the mailing list specified below:
- * Maintainer - idos@dev.mellanox.co.il
- * OFED mailing list - ewg@lists.openfabrics.org
- or linux-rdma@vger.kernel.org
- * http://openib.org/mailman/listinfo/openib-general
+Please post results/observations to the openib-general mailing list.
+See "Contact Us" at http://openib.org/mailman/listinfo/openib-general and
+http://www.openib.org.
===============================================================================
2. Notes on Testing Methodology
===============================================================================
-The bencmarks specified below are tested of the following architectures:
-- i686
-- x86_64
-- ia64
-
-- The benchmark uses the CPU cycle counter to get time stamps without context
- switch. Some CPU architectures (e.g., Intel's 80486 or older PPC) do NOT
+- The benchmark used the CPU cycle counter to get time stamps without context
+ switch. Some CPU architectures (e.g., Intel's 80486 or older PPC) do NOT
have such capability.
-- The benchmark measures round-trip time but reports half of that as one-way
- latency. Thus, it may not be sufficiently accurate for asymmetrical
+- The latency benchmarks measures round-trip time but reports half of that as one-way
+ latency. This means that it may not be sufficiently accurate for asymmetrical
configurations.
-- On BW benchmarks , the BW is calculated on the send side only, as it calculates
- the BW after collecting completion from the receive side.
- If using the bidirectional flag , BW is calculated on both sides
+- On Bw benchmarks , We calculate the BW on send side only, as he calculates
+ the Bw after collecting completion from the receive side.
+ In case we use the bidirectional flag , BW is calculated on both sides.
+ in ib_send_bw, server side also calculate the received throughput.
-- Min/Median/Max result is reported.
+- Min/Median/Max result is reported in latency tests.
The median (vs average) is less sensitive to extreme scores.
Typically, the "Max" value is the first value measured.
-- Larger samples help marginally only. The default (1000) is sufficient.
+- Larger samples help marginally only. The default (1000) is pretty good.
Note that an array of cycles_t (typically unsigned long) is allocated
once to collect samples and again to store the difference between them.
- Large sample sizes (e.g., 1 million) might expose other problems
- with the program.
+ Really big sample sizes (e.g., 1 million) might expose other problems
+ with the program. In this case you can use -N flag (No Peak) to instruct
+ the test sample only 2 times (begining and end).
+
+- All throughput tests now have duration feature as well (-D <seconds to run>)
+ to instruct the test to run for <seconds to run>.
+ Another feature added is --run_infinitely, which instruct the test to run
+ all te time and print throughput every 5 seconds.
-- The "-H" option will dump the histogram for additional statistical analysis.
+- The "-H" option (latency) will dump the histogram for additional statistical analysis.
See xgraph, ygraph, r-base (http://www.r-project.org/), pspp, or other
statistical math programs.
+Architectures tested: i686, x86_64, ia64
+
+
===============================================================================
4. Test Descriptions
===============================================================================
-rdma_lat.c latency test with RDMA write transactions
-rdma_bw.c streaming BW test with RDMA write transactions
-
-
The following tests are mainly useful for HW/SW benchmarking.
They are not intended as actual usage examples.
send_lat.c latency test with send transactions
send_bw.c BW test with send transactions
-write_lat.c latency test with RDMA write transactions
+write_lat.c latency test with RDMA write transactions
write_bw.c BW test with RDMA write transactions
read_lat.c latency test with RDMA read transactions
read_bw.c BW test with RDMA read transactions
+atomic_lat.c latency test with atomic transactions
+atomic_bw.c BW test atomic transactions
+raw_ethernet_send_bw.c BW test with Raw Etherent client and receiver.
-The executable name of each test starts with the general prefix "ib_",
-e.g., ib_write_lat, except for those of RDMA tests,
-their excutable have the same name except of the .c.
+The executable name of each test starts with the general prefix "ib_",e.g., ib_write_lat.
+Except of raw_ethernet_send_bw.c case, which the binary called raw_ethernet_bw.
Running Tests
-------------
Prerequisites:
kernel 2.6
- ib_uverbs (kernel module) matches libibverbs
- ("match" means binary compatible, but ideally of the same SVN rev)
+ (kernel module) matches libibverbs
+ (kernel module) matches librdmacm
+ (kernel module) matches libibumad
+ (kernel module) matches libmath (lm).
+
Server: ./<test name> <options>
Client: ./<test name> <options> <server IP address>
o --help lists the available <options>
*** IMPORTANT NOTE: The SAME OPTIONS must be passed to both server and client.
-
+ *** IMPORTANT NOTE: You need to be running a Subnet Manager on the switch or on one of the nodes in your fabric, in case you are in IB fabric.
Common Options to all tests:
- -p, --port=<port> Listen on/connect to port <port> (default: 18515).
- -m, --mtu=<mtu> Mtu size (default: 1024).
- -d, --ib-dev=<dev> Use IB device <dev> (default: first device found).
- -i, --ib-port=<port> Use port <port> of IB device (default: 1).
- -o, --out=<num_of_out> Number of outstanding reads. only in READ.
- -q, --qp=<num_of_qps> Number of Qps to perform. only in write_bw.
- -c, --connection=<c> Connection type : RC,UC,UD according to spec.
- -g, --mcg=<num_of_qps> Number of Qps in MultiCast group. in SEND only
- -M, --MGID=<addr> <addr> as the group MGID in format '255:1:X:X:X:X:X:X:X:X:X:X:X:X:X:X'.
- -s, --size=<size> Size of message to exchange (default: 1).
- -a, --all Run sizes from 2 till 2^23.
- -t, --tx-depth=<dep> Size of tx queue (default: 50).
- -r, --rx-depth=<dep> Make rx queue bigger than tx (default 600).
- -n, --iters=<iters> Number of exchanges (at least 100, default: 1000).
- -I, --inline_size=<size> Max size of message to be sent in inline mode.
- On Bw tests default is 1,latency tests is 400.
- -C, --report-cycles Report times in cpu cycle units.
- -u, --qp-timeout=<timeout> QP timeout, timeout value is 4 usec*2 ^(timeout).
- Default is 14.
- -S, --sl=<sl> SL (default 0).
- -H, --report-histogram Print out all results (Default: summary only).
- Only on Latnecy tests.
- -x, --gid-index=<index> Test uses GID with GID index taken from command
- Line (for RDMAoE index should be 0).
- -b, --bidirectional Measure bidirectional bandwidth (default uni).
- On BW tests only (Implicit on latency tests).
- -V, --version Display version number.
- -e, --events Sleep on CQ events (default poll).
- -N, --no peak-bw Cancel peak-bw calculation (default with peak-bw)
- -F, --CPU-freq Do not fail even if cpufreq_ondemand module.
-
- *** IMPORTANT NOTE: You need to be running a Subnet Manager on the switch or
- on one of the nodes in your fabric.
+----------------------------
+ -h, --help Display this help message screen.
+ -p, --port=<port> Listen on/connect to port <port> (default: 18515) when exchaning data.
+ -R, --rdma_cm Connect QPs with rdma_cm and run test on those QPs.
+ -z, --com_rdma_cm Communicate with rdma_cm module to exchange data - use regular QPs.
+ -m, --mtu=<mtu> QP Mtu size (default: active_mtu from ibv_devinfo).
+ -c, --connection=<RC/UC/UD> Connection type RC/UC/UD (default RC)
+ -d, --ib-dev=<dev> Use IB device <dev> (default: first device found).
+ -i, --ib-port=<port> Use port <port> of IB device (default: 1).
+ -s, --size=<size> Size of message to exchange (default: 1).
+ -a, --all Run sizes from 2 till 2^23.
+ -n, --iters=<iters> Number of exchanges (at least 100, default: 1000).
+ -x, --gid-index=<index> Test uses GID with GID index taken from command
+ -V, --version Display version number.
+ -e, --events Sleep on CQ events (default poll).
+ -F, --CPU-freq Do not fail even if cpufreq_ondemand module.
+ -I, --inline_size=<size> Max size of message to be sent in inline mode.
+ -u, --qp-timeout=<timeout> QP timeout, timeout value is 4 usec*2 ^timeout (default: 14).
+ -S, --sl=<sl> SL - Service Level (default 0)
+
+ -r, --rx-depth=<dep> Make rx queue bigger than tx (default 600).
+
+Options for latency tests:
+--------------------------
+
+ -C, --report-cycles Report times in cpu cycle units.
+ -H, --report-histogram Print out all results (Default: summary only).
+ -U, --report-unsorted Print out unsorted results (default sorted).
+
+Options for BW tests:
+---------------------
+
+ -b, --bidirectional Measure bidirectional bandwidth (default uni).
+ -N, --no peak-bw Cancel peak-bw calculation (default with peak-bw)
+ -Q, --cq-mod Generate Cqe only after <cq-mod> completion
+ -t, --tx-depth=<dep> Size of tx queue (default: 128).
+ -O, --dualport Run test in dual-port mode (2 QPs). both ports must be active (default OFF).
+ -D, --duration=<sec> Run test for <sec> period of seconds.
+ -f, --margin=<sec> When in Duration, measure results within margins (default: 2)
+ -l, --post_list=<list size> Post list of WQEs of <list size> size (instead of single post).
+ -q, --qp=<num of qp's> Num of QPs running in the process (default: 1).
+ --run_infinitely Run test forever, print results every 5 seconds.
+
+SEND tests (ib_send_lat or ib_send_bw) flags:
+---------------------------------------------
+
+ -r, --rx-depth=<dep> Size of RX queue (default: 512 in BW test).
+ -g, --mcg=<num_of_qps> Send messages to multicast group with <num_of_qps> qps attached to it.
+ -M, --MGID=<multicast_gid> In multicast, uses <multicast_gid> as the group MGID.
+
+ATOMIC tests (ib_atomic_lat or ib_atomic_bw) flags:
+---------------------------------------------------
+
+ -A, --atomic_type=<type> type of atomic operation from {CMP_AND_SWAP,FETCH_AND_ADD}.
+ -o, --outs=<num> Number of outstanding read/atomic requests - also on READ tests.
+
+Options for Raw Ethernet BW test:
+---------------------------------
+ -B, --source_mac source MAC address by this format XX:XX:XX:XX:XX:XX (default take the MAC address form GID).
+ -E, --dest_mac destination MAC address by this format XX:XX:XX:XX:XX:XX **MUST** be entered.
+ -J, --server_ip server ip address by this format X.X.X.X (using to send packets with IP header).
+ -j, --client_ip client ip address by this format X.X.X.X (using to send packets with IP header).
+ -K, --server_port server udp port number (using to send packets with UPD header).
+ -k, --client_port client udp port number (using to send packets with UDP header).
+ -Z, --server choose server side for the current machine (--server/--client must be selected ).
+ -P, --client choose client side for the current machine (--server/--client must be selected).
+
+----------------------------------------------
+Special feature detailed explination in tests:
+----------------------------------------------
+
+ 1. Usage of post_list feature (-l, --post_list=<list size>).
+ In this case, each QP will prepare <list size> ibv_send_wr (instead of 1), and will chain them to each other.
+ In Chaning we mean allocating <list_size> array, and setting 'next' pointer of each ibv_send_wr in the array
+ to point to the following element in the array. the last ibv_send_wr in the array will point to NULL.
+ In this case, when post sending the first ibv_send_wr in the list, will instruct the HW to post all of those WQEs.
+ Which means each post_send will post <list_size> messages.
+ This feature is good we want to know the maximum message rate of QPs in a single process.
+ Since we are limited to SW post_send (~ 10 Mpps, since we have ~ 500 ns between each SW post_send), we can see
+ the true HW message rate when setting <list_size> of 64 (for example) since it's not depended on SW limitations.
+
+ 2. RDMA_CM feature.
+ You can add to all tests the "-R" flag to connect the QPs from each side with the rdma_cm library.
+ In this case, the library will connect the QPs and will use the IPoIB interface for doing it.
+ It helps when you don't have Ethernet connection between the 2 nodes.
+ You must supply the IPoIB interface as the server IP.
+
+ 3. Multicast feauture in ib_send_lat and in ib_send_bw.
+ Send tests have built in feature of testing multicast performance, in verbs level.
+ You can use "-g" to specify the number of QPs to attach to this multicast group.
+ "-M" flag allows you to choose the multicast group address.
+
+
+===============================================================================
+5. Known Issues
+===============================================================================
+Up until now, we still have known (unsolved) issues in the package.
+Here is a list of the main issues:
+
+ 1. Multicast feauture in ib_send_lat and in ib_send_bw still have many problems!
+ Will increase the support and bug fixes in this Q, but now the tests may stuck
+ and could produce undefine behaviours.
+
+ 2. Bidirectional feature in ib_send_bw test, when running in UD or UC mode.
+ The algorithm we use for the bidirectional measurement is designed for RC connection type.
+ When running in UC or UD connection types, there is a small probablity the test will be stuck.
+ We are working now to fix it.
+
+ 3. RDMA_CM feature in read tests still doesn't work.
+
+ 4. Dual-port support currently works only with ib_write_bw.
+ 5. Compabilty issues may occur between different versions of perftest.
+ Please make sure you work with the same version on both sides to ensure consistency of the test.
+++ /dev/null
-This is a release of the QLogic VNIC driver on OFED 1.4. This driver is
-currently supported on Intel x86 32 and 64 bit machines.
-Supported OS are:
-- RHEL 4 Update 4.
-- RHEL 4 Update 5.
-- RHEL 4 Update 6.
-- SLES 10.
-- SLES 10 Service Pack 1.
-- SLES 10 Service Pack 1 Update 1.
-- SLES 10 Service Pack 2.
-- RHEL 5.
-- RHEL 5 Update 1.
-- RHEL 5 Update 2.
-- vanilla 2.6.27 kernel.
-
-The VNIC driver in conjunction with the QLogic Ethernet Virtual I/O Controller
-(EVIC) provides Ethernet interfaces on a host with IB HCA(s) without the need
-for any physical Ethernet NIC.
-
-This file describes the use of the QLogic VNIC ULP service on an OFED stack
-and covers the following points:
-
-A) Creating QLogic VNIC interfaces
-B) Discovering VEx/EVIC IOCs present on the fabric using ib_qlgc_vnic_query
-C) Starting the QLogic VNIC driver and the VNIC interfaces
-D) Assigning IP addresses etc for the QLogic VNIC interfaces
-E) Information about the QLogic VNIC interfaces
-F) Deleting a specific QLogic VNIC interface
-G) Forced Failover feature for QLogic VNIC.
-H) Infiniband Quality of Service for VNIC.
-I) QLogic VNIC Dynamic Update Daemon Tool and Hot Swap support
-J) Information about creating VLAN interfaces
-K) Information about enabling IB Multicast for QLogic VNIC interface
-L) Basic Troubleshooting
-
-A) Creating QLogic VNIC interfaces
-
-The VNIC interfaces can be created with the help of
-the configuration file which must be placed at /etc/infiniband/qlgc_vnic.cfg.
-
-Please take a look at /etc/infiniband/qlgc_vnic.cfg.sample file (available also
-as part of the documentation) to see how VNIC configuration files are written.
-You can use this configuration file as the basis for creating a VNIC configuration
-file by copying it to /etc/infiniband/qlgc_vnic.cfg. Of course you will have to
-replace the IOCGUID, IOCSTRING values etc in the sample configuration file
-with those of the EVIC IOCs present on your fabric.
-
-(For backward compatibilty, if this file is missing,
-/etc/infiniband/qlogic_vnic.cfg or /etc/sysconfig/ics_inic.cfg
-will be used for configuration)
-
-Please note that using DGID of the EVIC/VEx IOC is
-recommended as it will ensure the quickest startup of the
-VNIC service. If DGID is specified then you must also
-specify the IOCGUID. More details can be found in
-the qlgc_vnic.cfg.sample file.
-
-In case of a host consisting of more than 1 HCAs plugged in, VNIC
-interfaces can be configured based on HCA no and Port No or PORTGUID.
-
-B) Discovering EVIC/VEx IOCs present on the fabric using ib_qlgc_vnic_query
-
-For writing the configuration file, you will need information
-about the EVIC/VEx IOCs present on the fabric like their IOCGUID,
-IOCSTRING etc. The ib_qlgc_vnic_query tool should be used to get this
-information.
-
-When ib_qlgc_vnic_query is executed without any options, it scans through ALL
-active IB ports on the host and obtains the detailed information about all the
-EVIC/VEx IOCs reachable through each active IB port:
-
-# ib_qlgc_vnic_query
-
-HCA No = 0, HCA = mlx4_0, Port = 1, Port GUID = 0x0002c903000010f5, State = Active
-
- IO Unit Info:
- port LID: 0008
- port GID: fe8000000000000000066a11de000070
- change ID: 0003
- max controllers: 0x02
-
-
- controller[ 1]
- GUID: 00066a01de000070
- vendor ID: 00066a
- device ID: 000030
- IO class : 2000
- ID: EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1
- service entries: 2
- service[ 0]: 1000066a00000001 / InfiniNIC.InfiniConSys.Control:01
- service[ 1]: 1000066a00000101 / InfiniNIC.InfiniConSys.Data:01
-
- IO Unit Info:
- port LID: 0009
- port GID: fe8000000000000000066a21de000070
- change ID: 0003
- max controllers: 0x02
-
-
- controller[ 2]
- GUID: 00066a02de000070
- vendor ID: 00066a
- device ID: 000030
- IO class : 2000
- ID: EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 2
- service entries: 2
- service[ 0]: 1000066a00000002 / InfiniNIC.InfiniConSys.Control:02
- service[ 1]: 1000066a00000102 / InfiniNIC.InfiniConSys.Data:02
-
-HCA No = 0, HCA = mlx4_0, Port = 2, Port GUID = 0x0002c903000010f6, State = Active
-
- IO Unit Info:
- port LID: 0008
- port GID: fe8000000000000000066a11de000070
- change ID: 0003
- max controllers: 0x02
-
-
- controller[ 1]
- GUID: 00066a01de000070
- vendor ID: 00066a
- device ID: 000030
- IO class : 2000
- ID: EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1
- service entries: 2
- service[ 0]: 1000066a00000001 / InfiniNIC.InfiniConSys.Control:01
- service[ 1]: 1000066a00000101 / InfiniNIC.InfiniConSys.Data:01
-
- IO Unit Info:
- port LID: 0009
- port GID: fe8000000000000000066a21de000070
- change ID: 0003
- max controllers: 0x02
-
-
- controller[ 2]
- GUID: 00066a02de000070
- vendor ID: 00066a
- device ID: 000030
- IO class : 2000
- ID: EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 2
- service entries: 2
- service[ 0]: 1000066a00000002 / InfiniNIC.InfiniConSys.Control:02
- service[ 1]: 1000066a00000102 / InfiniNIC.InfiniConSys.Data:02
-
-HCA No = 1, HCA = mlx4_1, Port = 1, Port GUID = 0x0002c90300000785, State = Down
-
- Port State is Down. Skipping search of DM nodes on this port.
-
-HCA No = 1, HCA = mlx4_1, Port = 2, Port GUID = 0x0002c90300000786, State = Active
-
- IO Unit Info:
- port LID: 0008
- port GID: fe8000000000000000066a11de000070
- change ID: 0003
- max controllers: 0x02
-
-
- controller[ 1]
- GUID: 00066a01de000070
- vendor ID: 00066a
- device ID: 000030
- IO class : 2000
- ID: EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1
- service entries: 2
- service[ 0]: 1000066a00000001 / InfiniNIC.InfiniConSys.Control:01
- service[ 1]: 1000066a00000101 / InfiniNIC.InfiniConSys.Data:01
-
- IO Unit Info:
- port LID: 0009
- port GID: fe8000000000000000066a21de000070
- change ID: 0003
- max controllers: 0x02
-
-
- controller[ 2]
- GUID: 00066a02de000070
- vendor ID: 00066a
- device ID: 000030
- IO class : 2000
- ID: EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 2
- service entries: 2
- service[ 0]: 1000066a00000002 / InfiniNIC.InfiniConSys.Control:02
- service[ 1]: 1000066a00000102 / InfiniNIC.InfiniConSys.Data:02
-
-This is meant to help the network administrator to know about HCA/Port information
-on host along with EVIC IOCs reachable through given IB ports on fabric. When
-ib_qlgc_vnic_query is run with -e option, it reports the IOCGUID information
-and with -s option it reports the IOCSTRING information for the EVIC/VEx IOCs
-present on the fabric:
-
-# ib_qlgc_vnic_query -e
-
-HCA No = 0, HCA = mlx4_0, Port = 1, Port GUID = 0x0002c903000010f5, State = Active
-
- ioc_guid=00066a01de000070,dgid=fe8000000000000000066a11de000070,pkey=ffff
- ioc_guid=00066a02de000070,dgid=fe8000000000000000066a21de000070,pkey=ffff
-HCA No = 0, HCA = mlx4_0, Port = 2, Port GUID = 0x0002c903000010f6, State = Active
-
- ioc_guid=00066a01de000070,dgid=fe8000000000000000066a11de000070,pkey=ffff
- ioc_guid=00066a02de000070,dgid=fe8000000000000000066a21de000070,pkey=ffff
-HCA No = 1, HCA = mlx4_1, Port = 1, Port GUID = 0x0002c90300000785, State = Down
-
- Port State is Down. Skipping search of DM nodes on this port.
-
-HCA No = 1, HCA = mlx4_1, Port = 2, Port GUID = 0x0002c90300000786, State = Active
-
- ioc_guid=00066a01de000070,dgid=fe8000000000000000066a11de000070,pkey=ffff
- ioc_guid=00066a02de000070,dgid=fe8000000000000000066a21de000070,pkey=ffff
-
-# ib_qlgc_vnic_query -s
-
-HCA No = 0, HCA = mlx4_0, Port = 1, Port GUID = 0x0002c903000010f5, State = Active
-
-"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1"
-"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 2"
-HCA No = 0, HCA = mlx4_0, Port = 2, Port GUID = 0x0002c903000010f6, State = Active
-
-"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1"
-"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 2"
-HCA No = 1, HCA = mlx4_1, Port = 1, Port GUID = 0x0002c90300000785, State = Down
-
- Port State is Down. Skipping search of DM nodes on this port.
-
-HCA No = 1, HCA = mlx4_1, Port = 2, Port GUID = 0x0002c90300000786, State = Active
-
-"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1"
-"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 2"
-
-# ib_qlgc_vnic_query -es
-
-HCA No = 0, HCA = mlx4_0, Port = 1, Port GUID = 0x0002c903000010f5, State = Active
-
- ioc_guid=00066a01de000070,dgid=fe8000000000000000066a11de000070,pkey=ffff,"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1"
- ioc_guid=00066a02de000070,dgid=fe8000000000000000066a21de000070,pkey=ffff,"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 2"
-HCA No = 0, HCA = mlx4_0, Port = 2, Port GUID = 0x0002c903000010f6, State = Active
-
- ioc_guid=00066a01de000070,dgid=fe8000000000000000066a11de000070,pkey=ffff,"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1"
- ioc_guid=00066a02de000070,dgid=fe8000000000000000066a21de000070,pkey=ffff,"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 2"
-HCA No = 1, HCA = mlx4_1, Port = 1, Port GUID = 0x0002c90300000785, State = Down
-
- Port State is Down. Skipping search of DM nodes on this port.
-
-HCA No = 1, HCA = mlx4_1, Port = 2, Port GUID = 0x0002c90300000786, State = Active
-
- ioc_guid=00066a01de000070,dgid=fe8000000000000000066a11de000070,pkey=ffff,"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1"
- ioc_guid=00066a02de000070,dgid=fe8000000000000000066a21de000070,pkey=ffff,"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 2"
-
-ib_qlgc_vnic_query can be used to discover EVIC IOCs on the fabric based on
-umad device, HCA no/Port no and PORTGUID as follows:
-
-For umad devices, it takes the name of the umad device mentioned with '-d'
-option:
-
-# ib_qlgc_vnic_query -es -d /dev/infiniband/umad0
-
-HCA No = 0, HCA = mlx4_0, Port = 1
-
- ioc_guid=00066a01de000070,dgid=fe8000000000000000066a11de000070,pkey=ffff,"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1"
- ioc_guid=00066a02de000070,dgid=fe8000000000000000066a21de000070,pkey=ffff,"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 2"
-
-If the name of the HCA and its port no is known, then ib_qlgc_vnic_query can
-make use of this information to discover EVIC IOCs on the fabric. HCA name
-and port no is specified with '-C' and '-P' options respectively.
-
-# ib_qlgc_vnic_query -es -C mlx4_1 -P 2
-
- ioc_guid=00066a01de000070,dgid=fe8000000000000000066a11de000070,pkey=ffff,"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1"
- ioc_guid=00066a02de000070,dgid=fe8000000000000000066a21de000070,pkey=ffff,"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 2"
-
-In case, if HCA name is not specified but port no is specified, HCA 0 is
-selected as default HCA to discover IOCs and if Port no is missing then,
-Port 1 of HCA name mentioned is used to discover the IOCs. If both are
-missing, the behaviour is default and ib_qlgc_vnic_query will scan all the
-IB ports on the host to discover IOCs reachable through each one of them.
-
-PORTGUID information about the IB ports on given host can be obtained using
-the option '-L':
-
-# ib_qlgc_vnic_query -L
-
-0,mlx4_0,1,0x0002c903000010f5
-0,mlx4_0,2,0x0002c903000010f6
-1,mlx4_1,1,0x0002c90300000785
-1,mlx4_1,2,0x0002c90300000786
-
-This actually lists different configurable parameters of IB ports present on
-given host in the order: HCA No, HCA Name, Port No, PORTGUID separated by
-commas. PORTGUID value obtained thus, can be used to discover EVIC IOCs
-reachable through it using '-G' option as follows:
-
-# ib_qlgc_vnic_query -es -G 0x0002c903000010f5
-
-HCA No = 0, HCA = mlx4_0, Port = 1, Port GUID = 0x0002c903000010f5, State = Active
-
- ioc_guid=00066a01de000070,dgid=fe8000000000000000066a11de000070,pkey=ffff,"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1"
- ioc_guid=00066a02de000070,dgid=fe8000000000000000066a21de000070,pkey=ffff,"EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 2"
-
-C) Starting the QLogic VNIC driver and the QLogic VNIC interfaces
-
-To start the QLogic VNIC service as a part of startup of OFED stack, set
-
-QLGC_VNIC_LOAD=yes
-
-in /etc/infiniband/openib.conf file. With this actually, the QLogic VNIC
-service will also be stopped when the OFED stack is stopped. Also, if OFED
-stack has been marked to start on boot, QLogic VNIC service will also start
-on boot.
-
-The rest of the discussion in this subsection C) is valid only if
-
-QLGC_VNIC_LOAD=no
-
-is set into /etc/infiniband/openib.conf.
-
-Once you have created a configuration file, you can start the VNIC driver
-and create the VNIC interfaces specified in the configuration file with:
-
-#/sbin/service qlgc_vnic start
-
-You can stop the VNIC driver and bring down the VNIC interfaces with
-
-#/sbin/service qlgc_vnic stop
-
-To restart the QLogic VNIC driver, you can use
-
-#/sbin/service qlgc_vnic restart
-
-If you have not started the Infiniband network stack (Infinipath or OFED),
-then running "/sbin/service qlgc_vnic start" command will also cause the
-Infiniband network stack to be started since the QLogic VNIC service requires
-the Infiniband stack.
-
-On the other hand if you start the Infiniband network stack separately, then
-the correct order of starting is:
-
-- Start the Infiniband stack
-- Start QLogic VNIC service
-
-For example, if you use OFED, correct order of starting is:
-
-/sbin/service openibd start
-/sbin/service qlgc_vnic start
-
-Correct order of stopping is:
-
-- Stop QLogic VNIC service
-- Stop the Infiniband stack
-
-For example, if you use OFED, correct order of stopping is:
-
-/sbin/service qlgc_vnic stop
-/sbin/service openibd stop
-
-If you try to stop the Infiniband stack when the QLogic VNIC service is
-running,
-you will get an error message that some of the modules of the Infiniband stack
-are in use by the QLogic VNIC service. Also, any QLogic VNIC interfaces that
-you
-created are removed (because stopping the Infiniband network stack causes the
-HCA
-driver to be unloaded which is required for the VNIC interfaces to be
-present).
-In this case, do the following:
-
- 1. Stop the QLogic VNIC service with "/sbin/service qlgc_vnic stop"
-
- 2. Stop the Infiniband stack again.
-
- 3. If you want to restart the QLogic VNIC interfaces, use
- "/sbin/service qlgc_vnic start".
-
-
-D) Assigning IP addresses etc for the QLogic VNIC interfaces
-
-This can be done with ifconfig or by setting up the ifcfg-XXX (ifcfg-veth0 for
-an interface named veth0 etc) network files for the corresponding VNIC interfaces.
-
-E) Information about the QLogic VNIC interfaces
-
-Information about VNIC interfaces on a given host can be obtained using a
-script "ib_qlgc_vnic_info" :-
-
-# ib_qlgc_vnic_info
-
-VNIC Interface : eioc0
- VNIC State : VNIC_REGISTERED
- Current Path : primary path
- Receive Checksum : true
- Transmit checksum : true
-
- Primary Path :
- VIPORT State : VIPORT_CONNECTED
- Link State : LINK_IDLING
- HCA Info. : vnic-mthca0-1
- Heartbeat : 100
- IOC String : EVIC in Chassis 0x00066a00db000010, Slot 4, Ioc 1
- IOC GUID : 66a01de000037
- DGID : fe8000000000000000066a11de000037
- P Key : ffff
-
- Secondary Path :
- VIPORT State : VIPORT_DISCONNECTED
- Link State : INVALID STATE
- HCA Info. : vnic-mthca0-2
- Heartbeat : 100
- IOC String :
- IOC GUID : 66a01de000037
- DGID : 00000000000000000000000000000000
- P Key : 0
-
-This information is collected from /sys/class/infiniband_qlgc_vnic/interfaces/
-directory under which there is a separate directory corresponding to each
-VNIC interface.
-
-F) Deleting a specific QLogic VNIC interface
-
-VNIC interfaces can be deleted by writing the name of the interface to
-the /sys/class/infiniband_qlgc_vnic/interfaces/delete_vnic file.
-
-For example to delete interface veth0
-
-echo -n veth0 > /sys/class/infiniband_qlgc_vnic/interfaces/delete_vnic
-
-G) Forced Failover feature for QLogic VNIC.
-
-VNIC interfaces, when configured with failover configuration, can be
-forced to failover to use other active path. For example, if VNIC interface
-"veth1" is configured with failover configuration, then to switch to other
-path, use command:
-
-echo -n veth1 > /sys/class/infiniband_qlgc_vnic/interfaces/force_failover
-
-This will make VNIC interface veth1 to switch to other active path, even though
-the path of VNIC interface, before the forced failover operation, is not in
-disconnected state.
-
-This feature allows the network administrator to control the path of the
-VNIC traffic at run time and reconfiguration as well as restart of VNIC
-service is not required to achieve the same.
-
-Once enabled as mentioned above, forced failover can be cleared with
-the unfailover command:
-
-echo -n veth1 > /sys/class/infiniband_qlgc_vnic/interfaces/unfailover
-
-This clears the forced failover on VNIC interface "veth1". Once cleared,
-if module parameter "default_prefer_primary" is set to 1, then VNIC
-interface switches back to primary path. If module parameter
-"default_prefer_primary" is set to 0, then VNIC interface continues to
-use its current active path.
-
-Forced failover, thus, takes priority over default_prefer_primary and the
-default_prefer_primary feature will not be active unless the forced
-failover is cleared through "unfailover".
-
-Besides this forced failover, QLogic VNIC service does retain its
-original failover feature which gets triggered when current active
-path gets disconnected.
-
-H) Infiniband Quality of Service for VNIC:-
-
-To enforce infiniband Quality of Service(QoS) for VNIC protocol, there
-is no configuration required on host side. The service level for the
-VNIC protocol can be configured using service ID or target port guid
-in the "qos-ulps" section of /etc/opensm/qos-policy.conf on the host
-running OpenSM.
-
-Service IDs for the EVIC IO controllers can be obtained from the output
-of ib_qlgc_vnic_query:
-
-HCA No = 1, HCA = mlx4_1, Port = 2, Port GUID = 0x0002c90300000786, State = Active
-
- IO Unit Info:
- port LID: 0008
- port GID: fe8000000000000000066a11de000070
- change ID: 0003
- max controllers: 0x02
-
-
- controller[ 1]
- GUID: 00066a01de000070
- vendor ID: 00066a
- device ID: 000030
- IO class : 2000
- ID: EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1
- service entries: 2
-------> service[ 0]: 1000066a00000001 / InfiniNIC.InfiniConSys.Control:01
-------> service[ 1]: 1000066a00000101 / InfiniNIC.InfiniConSys.Data:01
-
- IO Unit Info:
- port LID: 0009
- port GID: fe8000000000000000066a21de000070
- change ID: 0003
- max controllers: 0x02
-
-
- controller[ 2]
- GUID: 00066a02de000070
- vendor ID: 00066a
- device ID: 000030
- IO class : 2000
- ID: EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 2
- service entries: 2
-------> service[ 0]: 1000066a00000002 / InfiniNIC.InfiniConSys.Control:02
-------> service[ 1]: 1000066a00000102 / InfiniNIC.InfiniConSys.Data:02
-
-Numbers 1000066a00000002, 1000066a00000102 are the required service IDs.
-
-Finer control on quality of service for the VNIC protocol can be achieved by
-configuring the service level using target port guid values of the EVIC IO
-controllers. Target port guid values for the EVIC IO controllers can be
-obtained using "saquery" command supplied by OFED package.
-
-I) QLogic VNIC Dynamic Update Daemon Tool and Hot Swap support:-
-
-This tool is started and stopped as part of the QLogic VNIC service
-(refer to C above) and provides the following features:
-
-1. Dynamic update of disconnected interfaces (which have been configured
-WITHOUT using the DGID option in the configuration file) :
-
-At the start up of VNIC driver, if the HCA port through which a particular VNIC
-interface path (primary or secondary) connects to target is down or the
-EVIC/VEx IOC is not available then all the required parameters (DGID etc) for connecting
-with the EVIC/VEx cannot be determined. Hence the corresponding VNIC interface
-path is not available at the start of the VNIC service. This daemon constantly
-monitors the configured VNIC interfaces to check if any of them are disconnected.
-If any of the interfaces are disconnected, it scans for available EVIC/VEx targets using
-"ib_qlgc_vnic_query" tool. When daemon sees that for a given path of a VNIC interface,
-the configured EVIC/VEx IOC has become available, it dynamically updates the
-VNIC kernel driver with the required information to establish connection for
-that path of the interface. In this way, the interface gets connected with
-the configured EVIC/VEx whenever it becomes available without any manual
-intervention.
-
-2. Hot Swap support :
-
-Hot swap is an operation in which an existing EVIC/VEx is replaced by another
-EVIC/VEx (in the same slot of the switch chassis as the older one). In such a
-case, the current connection for the corresponding VNIC interface will have to
-be re-established. The daemon detects this hot swap case and re-establishes
-the connection automatically. To make use of this feature of the daemon, it is
-recommended that IOCSTRING be used in the configuration file to configure the
-VNIC interfaces.
-
-This is because, after a hot swap though all other parameters like DGID, IOCGUID etc
-of the EVIC/VEx change, the IOCSTRING remains the same. Thus the daemon monitors
-for changes in IOCGUID and DGID of disconnected interfaces based on the IOCSTRING.
-If these values have changed it updates the kernel driver so that the VNIC
-interface can start using the new EVIC/VEx.
-
-If in addition to IOCSTRING, DGID and IOCGUID have been used to configure
-a VNIC interface, then on a hotswap the daemon will update the parameters as required.
-But to have that VNIC interface available immediately on the next restart of the
-QLogic VNIC service, please make sure to update the configuration file with the
-new DGID and IOCGUID values. Otherwise, the creation of such interfaces will be
-delayed till the daemon runs and updates the parameters.
-
-J) Information about creating VLAN interfaces
-
-The EVIC/VEx supports VLAN tagging without having to explicitly create VLAN
-interfaces for the VNIC interface on the host. This is done by enabling
-Egress/Ingress tagging on the EVIC/VEx and setting the "Host ignores VLAN"
-option for the VNIC interface. The "Host ignores VLAN" option is enabled
-by default due to which VLAN tags are ignored on the host by the QLogic
-VNIC driver. Thus explicitly created VLAN interfaces (using vconfig command)
-for a given VNIC interface will not be operational.
-
-If you want to explicitly create a VLAN interface for a given VNIC interface,
-then you will have to disable the "Host ignores VLAN" option for the
-VNIC interface on the EVIC/VEx. The qlgc_vnic service must be restarted
-on the host after disabling (or enabling) the "Host ignores VLAN" option.
-
-Please refer to the EVIC/VEx documentation for more information on Egress/Ingress
-port tagging feature and disabling the "Host ignores VLAN" option.
-
-K) Information about enabling IB Multicast for QLogic VNIC interface
-
-QLogic VNIC driver has been upgraded to support the IB Multicasting feature of
-EVIC/VEx. This feature enables the QLogic VNIC host driver to support the IP
-multicasting more efficiently. With this feature enabled, infiniband multicast
-group acts as a carrier of IP multicast traffic. EVIC will make use of such IB
-multicast groups for forwarding IP multicast traffic to VNIC interfaces which
-are member of given IP multicast group. In the older QLogic VNIC host driver,
-IB multicasting was not being used to carry IP multicast traffic.
-
-By default, IB multicasting is disabled on EVIC/VEx; but it is enabled by
-default at the QLogic VNIC host driver.
-
-To disable IB multicast feature on the host driver, VNIC configuration file
-needs to be modified by setting the parameter IB_MULTICAST=FALSE in the
-interface configuration. Please refer to the qlgc_vnic.cfg.sample for more
-details on configuration of VNIC interfaces for IB multicasting.
-IB multicasting also needs to be enabled over EVIC/VEx. Please refer to the
-EVIC/VEx documentation for more information on enabling IB multicast
-feature over EVIC/VEx.
-
-L) Basic Troubleshooting
-
-1. In case of any problems, make sure that:
-
- a) The HCA ports you are trying to use have IB cables connected and are in an
- active state. You can use the "ibv_devinfo" tool to check the state of
- your HCA ports.
-
- b) If your HCA ports are not active, check if an SM is running on the fabric
- where the HCA ports are connected. If you have done a full install of
- OFED, you can use the "sminfo" command ("sminfo -P 2" for port 2) to
- check SM information.
-
- c) Make sure that the EVIC/VEx is powered up and its Ethernet cables are connected
- properly.
-
- d) Check /var/log/messages for any error messages.
-
-2. If some of your VNIC interfaces are not available:
-
- a) Use "ifconfig" tool with -a option to see if all interfaces are created.
- It is possible that the interfaces are created but do not have an
- IP address. Make sure that you have setup a correct ifcfg-XXX file for your
- VNIC interfaces for automatic assignment of IP addresses.
-
- If the VNIC interface is created and the ifcfg file is also correct
- but the VNIC interface is not UP, make sure that the target EVIC/VEx
- IOC has an Ethernet cable properly connected.
-
- b) Make sure that the VNIC configuration file has been setup properly
- with correct EVIC/VEx target DGID/IOCGUID/IOCSTRING information and
- instance numbers.
-
- c) Make sure that the EVIC/VEx target IOC specified for that interface is
- available. You can use the "ib_qlgc_vnic_query" tool to verify this. If it is not
- available when you started the service, but it becomes available later
- on, then the QLogic VNIC dynamic update daemon will bring up the
- interface when the target becomes available. You will see messages in
- /var/log/messages when the corresponding interface is created.
-
- d) Make sure that you have not exceeded the total number of Virtual interfaces
- supported by the EVIC/VEx. You can check the total number of Virtual interfaces
- currently in use on the HTTP interface of the EVIC/VEx.
-
+++ /dev/null
-RDS(7) RDS(7)
-
-
-
-NAME
- RDS - Reliable Datagram Sockets
-
-SYNOPSIS
- #include <sys/socket.h>
- #include <netinet/in.h>
-
-DESCRIPTION
- This is an implementation of the RDS socket API. It provides reliable,
- in-order datagram delivery between sockets over a variety of trans‐
- ports.
-
- Currently, RDS can be transported over Infiniband, and loopback.
- iWARP bcopy is supported, but not RDMA operations.
-
- RDS uses standard AF_INET addresses as described in ip(7) to identify
- end points.
-
- Socket Creation
- RDS is still in development and as such does not have a reserved proto‐
- col family constant. Applications must read the string representation
- of the protocol family value from the pf_rds sysctl parameter file
- described below.
-
- rds_socket = socket(pf_rds, SOCK_SEQPACKET, 0);
-
-
- Socket Options
- RDS sockets support a number of socket options through the setsock‐
- opt(2) and getsockopt(2) calls. The following generic options (with
- socket level SOL_SOCKET) are of specific importance:
-
- SO_RCVBUF
- Specifies the size of the receive buffer. See section on "Con‐
- gestion Control" below.
-
- SO_SNDBUF
- Specifies the size of the send buffer. See "Message Transmis‐
- sion" below.
-
- SO_SNDTIMEO
- Specifies the send timeout when trying to enqueue a message on a
- socket with a full queue in blocking mode.
-
- In addition to these, RDS supports a number of protocol specific
- options (with socket level SOL_RDS). Just as with the RDS protocol
- family, an official value has not been assigned yet, so the kernel will
- assign a value dynamically. The assigned value can be retrieved from
- the sol_rds sysctl parameter file.
-
- RDS specific socket options will be described in a separate section
- below.
-
- Binding
- A new RDS socket has no local address when it is first returned from
- socket(2). It must be bound to a local address by calling bind(2)
- before any messages can be sent or received. This will also attach the
- socket to a specific transport, based on the type of interface the
- local address is attached to. From that point on, the socket can only
- reach destinations which are available through this transport.
-
- For instance, when binding to the address of an Infiniband interface
- such as ib0, the socket will use the Infiniband transport. If RDS is
- not able to associate a transport with the given address, it will
- return EADDRNOTAVAIL.
-
- An RDS socket can only be bound to one address and only one socket can
- be bound to a given address/port pair. If no port is specified in the
- binding address then an unbound port is selected at random.
-
- RDS does not allow the application to bind a previously bound socket to
- another address. Binding to the wildcard address INADDR_ANY is not per‐
- mitted either.
-
- Connecting
- The default mode of operation for RDS is to use unconnected socket, and
- specify a destination address as an argument to sendmsg. However, RDS
- allows sockets to be connected to a remote end point using connect(2).
- If a socket is connected, calling sendmsg without specifying a destina‐
- tion address will use the previously given remote address.
-
- Congestion Control
- RDS does not have explicit congestion control like common streaming
- protocols such as TCP. However, sockets have two queue limits associ‐
- ated with them; the send queue size and the receive queue size. Mes‐
- sages are accounted based on the number of bytes of payload.
-
- The send queue size limits how much data local processes can queue on a
- local socket (see the following section). If that limit is exceeded,
- the kernel will not accept further messages until the queue is drained
- and messages have been delivered to and acknowledged by the remote
- host.
-
- The receive queue size limits how much data RDS will put on the receive
- queue of a socket before marking the socket as congested. When a
- socket becomes congested, RDS will send a congestion map update to the
- other participating hosts, who are then expected to stop sending more
- messages to this port.
-
- There is a timing window during which a remote host can still continue
- to send messages to a congested port; RDS solves this by accepting
- these messages even if the socket's receive queue is already over the
- limit.
-
- As the application pulls incoming messages off the receive queue using
- recvmsg(2), the number of bytes on the receive queue will eventually
- drop below the receive queue size, at which point the port is then
- marked uncongested, and another congestion update is sent to all par‐
- ticipating hosts. This tells them to allow applications to send addi‐
- tional messages to this port.
-
- The default values for the send and receive buffer size are controlled
- by the A given RDS socket has limited transmit buffer space. It
- defaults to the system wide socket send buffer size set in the
- wmem_default and rmem_default sysctls, respectively. They can be tuned
- by the application through the SO_SNDBUF and SO_RCVBUF socket options.
-
-
- Blocking Behavior
- The sendmsg(2) and recvmsg(2) calls can block in a variety of situa‐
- tions. Whether a call blocks or returns with an error depends on the
- non-blocking setting of the file descriptor and the MSG_DONTWAIT mes‐
- sage flag. If the file descriptor is set to blocking mode (which is the
- default), and the MSG_DONTWAIT flag is not given, the call will block.
-
- In addition, the SO_SNDTIMEO and SO_RCVTIMEO socket options can be used
- to specify a timeout (in seconds) after which the call will abort wait‐
- ing, and return an error. The default timeout is 0, which tells RDS to
- block indefinitely.
-
- Message Transmission
- Messages may be sent using sendmsg(2) once the RDS socket is bound.
- Message length cannot exceed 4 gigabytes as the wire protocol uses an
- unsigned 32 bit integer to express the message length.
-
- RDS does not support out of band data. Applications are allowed to send
- to unicast addresses only; broadcast or multicast are not supported.
-
- A successful sendmsg(2) call puts the message in the socket's transmit
- queue where it will remain until either the destination acknowledges
- that the message is no longer in the network or the application removes
- the message from the send queue.
-
- Messages can be removed from the send queue with the RDS_CANCEL_SENT_TO
- socket option described below.
-
- While a message is in the transmit queue its payload bytes are
- accounted for. If an attempt is made to send a message while there is
- not sufficient room on the transmit queue, the call will either block
- or return EAGAIN.
-
- Trying to send to a destination that is marked congested (see above),
- the call will either block or return ENOBUFS.
-
- A message sent with no payload bytes will not consume any space in the
- destination's send buffer but will result in a message receipt on the
- destination. The receiver will not get any payload data but will be
- able to see the sender's address.
-
- Messages sent to a port to which no socket is bound will be silently
- discarded by the destination host. No error messages are reported to
- the sender.
-
- Message Receipt
- Messages may be received with recvmsg(2) on an RDS socket once it is
- bound to a source address. RDS will return messages in-order, i.e. mes‐
- sages from the same sender will arrive in the same order in which they
- were be sent.
-
- The address of the sender will be returned in the sockaddr_in structure
- pointed to by the msg_name field, if set.
-
- If the MSG_PEEK flag is given, the first message on the receive is
- returned without removing it from the queue.
-
- The memory consumed by messages waiting for delivery does not limit the
- number of messages that can be queued for receive. RDS does attempt to
- perform congestion control as described in the section above.
-
- If the length of the message exceeds the size of the buffer provided to
- recvmsg(2), then the remainder of the bytes in the message are dis‐
- carded and the MSG_TRUNC flag is set in the msg_flags field. In this
- truncating case recvmsg(2) will still return the number of bytes
- copied, not the length of entire messge. If MSG_TRUNC is set in the
- flags argument to recvmsg(2), then it will return the number of bytes
- in the entire message. Thus one can examine the size of the next mes‐
- sage in the receive queue without incurring a copying overhead by pro‐
- viding a zero length buffer and setting MSG_PEEK and MSG_TRUNC in the
- flags argument.
-
- The sending address of a zero-length message will still be provided in
- the msg_name field.
-
- Control Messages
- RDS uses control messages (a.k.a. ancillary data) through the msg_con‐
- trol and msg_controllen fields in sendmsg(2) and recvmsg(2). Control
- messages generated by RDS have a cmsg_level value of sol_rds. Most
- control messages are related to the zerocopy interface added in RDS
- version 3, and are described in rds-rdma(7).
-
- The only exception is the RDS_CMSG_CONG_UPDATE message, which is
- described in the following section.
-
- Polling
- RDS supports the poll(2) interface in a limited fashion. POLLIN is
- returned when there is a message (either a proper RDS message, or a
- control message) waiting in the socket's receive queue. POLLOUT is
- always returned while there is room on the socket's send queue.
-
- Sending to congested ports requires special handling. When an applica‐
- tion tries to send to a congested destination, the system call will
- return ENOBUFS. However, it cannot poll for POLLOUT, as there is prob‐
- ably still room on the transmit queue, so the call to poll(2) would
- return immediately, even though the destination is still congested.
-
- There are two ways of dealing with this situation. The first is to sim‐
- ply poll for POLLIN. By default, a process sleeping in poll(2) is
- always woken up when the congestion map is updated, and thus the appli‐
- cation can retry any previously congested sends.
-
- The second option is explicit congestion monitoring, which gives the
- application more fine-grained control.
-
- With explicit monitoring, the application polls for POLLIN as before,
- and additionally uses the RDS_CONG_MONITOR socket option to install a
- 64bit mask value in the socket, where each bit corresponds to a group
- of ports. When a congestion update arrives, RDS checks the set of ports
- that became uncongested against the bit mask installed in the socket.
- If they overlap, a control messages is enqueued on the socket, and the
- application is woken up. When it calls recvmsg(2), it will be given the
- control message containing the bitmap. on the socket.
-
- The congestion monitor bitmask can be set and queried using setsock‐
- opt(2) with RDS_CONG_MONITOR, and a pointer to the 64bit mask variable.
-
- Congestion updates are delivered to the application via
- RDS_CMSG_CONG_UPDATE control messages. These control messages are
- always delivered by themselves (or possibly additional control mes‐
- sages), but never along with a RDS data message. The cmsg_data field of
- the control message is an 8 byte datum containing the 64bit mask value.
-
- Applications can use the following macros to test for and set bits in
- the bitmask:
-
- #define RDS_CONG_MONITOR_SIZE 64
- #define RDS_CONG_MONITOR_BIT(port) (((unsigned int) port) % RDS_CONG_MONITOR_SIZE)
- #define RDS_CONG_MONITOR_MASK(port) (1 << RDS_CONG_MONITOR_BIT(port))
-
-
- Canceling Messages
- An application can cancel (flush) messages from the send queue using
- the RDS_CANCEL_SENT_TO socket option with setsockopt(2). This call
- takes an optional sockaddr_in address structure as argument. If given,
- only messages to the destination specified by this address are dis‐
- carded. If no address is given, all pending messages are discarded.
-
- Note that this affects messages that have not yet been transmitted as
- well as messages that have been transmitted, but for which no acknowl‐
- edgment from the remote host has been received yet.
-
- Reliability
- If sendmsg(2) succeeds, RDS guarantees that the message will be vis‐
- ible to recvmsg(2) on a socket bound to the destination address as
- long as that destination socket remains open.
-
- If there is no socket bound on the destination, the message is
- silently dropped. If the sending RDS can't be sure that there is no
- socket bound then it will try to send the message indefinitely until it
- can be sure or the sent message is canceled.
-
- If a socket is closed then all pending sent messages on the socket are
- canceled and may or may not be seen by the receiver.
-
- The RDS_CANCEL_SENT_TO socket option can be used to cancel all pending
- messages to a given destination.
-
- If a receiving socket is closed with pending messages then the sender
- considers those messages as having left the network and will not
- retransmit them.
-
- A message will only be seen by recvmsg(2) once, unless MSG_PEEK was
- specified. Once the message has been delivered it is removed from the
- sending socket's transmit queue.
-
- All messages sent from the same socket to the same destination will be
- delivered in the order they're sent. Messages sent from different sock‐
- ets, or to different destinations, may be delivered in any order.
-
-SYSCTL VALUES
- These parameteres may only be accessed through their files in
- /proc/sys/net/rds. Access through sysctl(2) is not supported.
-
- pf_rds This file contains the string representation of the protocol
- family constant passed to socket(2) to create a new RDS socket.
-
- sol_rds
- This file contains the string representation of the socket level
- parameter that is passed to getsockopt(2) and setsockopt(2) to
- manipulate RDS socket options.
-
- max_unacked_bytes and max_unacked_packets
- These parameters are used to tune the generation of acknowledge‐
- ments. By default, the system receiving RDS messages does not
- send back explicit acknowledgements unless it transmits a mes‐
- sage of its own (in which case the ACK is piggybacked onto the
- outgoing message), or when the sending system requests an ACK.
-
- However, the sender needs to see an ACK from time to time so
- that it can purge old messages from the send queue. The unacked
- bytes and packet counters are used to keep track of how much
- data has been sent without requesting an ACK. The default is to
- request an acknowledgement every 16 packets, or every 16 MB,
- whichever comes first.
-
- reconnect_delay_min_ms and reconnect_delay_max_ms
- RDS uses host-to-host connections to transport RDS messages
- (both for the TCP and the Infiniband transport). If this connec‐
- tion breaks, RDS will try to re-establish the connection.
- Because this reconnect may be triggered by both hosts at the
- same time and fail, RDS uses a random backoff before attempting
- a reconnect. These two parameters specify the minimum and maxi‐
- mum delay in milliseconds. The default values are 1 and 1000,
- respectively.
-
-SEE ALSO
- rds-rdma(7), socket(2), bind(2), sendmsg(2), recvmsg(2), getsockopt(2),
- setsockopt(2).
-
-
-
- RDS(7)
+++ /dev/null
-===============================================================================
- OFED-1.5.3 RoCEE Support README
- March 2011
-===============================================================================
-
-Contents:
-=========
-1. Overview
-2. Software Dependencies
-3. User Guidelines
-4. Ported Applications
-5. Gid tables
-6. Using VLANs
-7. Statistic counters
-8. Firmware Requirements
-9. Supported hardware
-10. Added fearues
-11. Known Issues
-
-
-1. Overview
-===========
-RDMA over Converged Enhanced Ethernet (RoCEE) allows InfiniBand (IB) transport
-over Ethernet networks. It encapsulates IB transport and GRH headers in
-Ethernet packets bearing a dedicated ether type.
-While the use of GRH is optional within IB subnets, it is mandatory when using
-RoCEE. Verbs applications written over IB verbs should work seamlessly, but
-they require provisioning of GRH information when creating address vectors. The
-library and driver are modified to provide for mapping from GID to MAC
-addresses required by the hardware.
-
-2. Software Dependencies
-========================
-In order to use RoCEE over Mellanox ConnectX(R) hardware, the mlx4_en driver
-must be loaded. Please refer to MLNX_EN_README.txt for further details.
-
-
-3. User Guidelines
-==================
-Since RoCEE encapsulates InfiniBand traffic in Ethernet frames, the
-corresponding net device must be up and running. In case of Mellanox
-hardware, mlx4_en must be loaded and the corresponding interface configured.
-- Make sure mlx4_en.ko is loaded
-- Make sure an IP address has been configured to this interface
-- Run "ibv_devinfo". There is a new field named "link_layer" which can be
- either "Ethernet" or "IB". If the value is IB, then you need to use
- connectx_port_config to change the ConnectX ports designation to eth (see
- mlx4_release_notes.txt for details)
-- Configure the IP address of the interface so that the link will become
- active
-- All IB verbs applications which run over IB verbs should work on RoCEE
- links as long as they use GRH headers (that is, as long as they specify use
- of GRH in their address vector)
-- rdma_cm applications working over RoCEE will have the TOS field set to a
- default value of 3. The default value is given as a module paramter to
- rdma_cm:
- def_prec2sl:Default value for SL priority with RoCE. Valid values 0 - 7 (int).
-
-
-4. Ported Applications
-======================
-- ibv_*_pingpong examples have been ported too. The user must specify the GID
- of the remote peer using the new '-g' option. The GID has the same format as
- that in /sys/class/infiniband/mlx4_0/ports/1/gids/0
-
-- Note: Care should be taken when using ibv_ud_pingpong. The default message
- size is 2K, which is likely to exceed the MTU of the RoCEE link. Use
- ibv_devinfo to inspect the link MTU and specify an appropriate message size
-
-- All rdma_cm applications should work seamlessly without any change
-
-- libsdp works without any change
-
-- Performance tests have been ported
-
-
-5. Gid tables
-=============
-With RoCEE, there may be several entries in a port's GID table. The first entry
-always contains the IPv6 link local address of the corresponding ethernet
-interface. The link local address is formed in the following way:
-
-gid[0..7] = fe80000000000000
-gid[8] = mac[0] ^ 2
-gid[9] = mac[1]
-gid[10] = mac[2]
-gid[11] = ff
-gid[12] = fe
-gid[13] = mac[3]
-gid[14] = mac[4]
-gid[15] = mac[5]
-
-If VLAN is supported by the kernel, and there are VLAN interfaces on the main
-ethernet interface (the interface that the IB port is tied to), each such VLAN
-will appear as a new GID in the port's GID table. The format of the GID entry
-will be identical to the one decribed above with the following change:
-
-gid[11] = VLAN ID high byte (4 MS bits).
-gid[12] = VLAN ID low byte
-
-Please note that VLAN ID is 12 bits.
-
-Priority pause frames
----------------------
-Tagged ethernet frames carry a 3 bit priority field. The value of this field is
-derived from the IB SL field by taking the 3 LS bits of the SL field.
-
-
-6. Using VLANs
-==============
-In order for RoCEE traffic to used VLAN tagged frames, the user has to specify
-GID table entries that are derived from VLAN devices, when creating address
-vectors. Consider the example bellow:
-
-6.1 Make sure VLAN support is enabled by the kernel. Usually this requires
-loading the 8021q module.
-- modprobe 8021q
-
-6.2 Add a VLAN device
-- vconfig add eth2 7
-
-6.3 Assign IP address to the VLAN interface
-- ifconfig eth2.7 7.10.11.12
-suppose this created a new entry in the GID table in index 1.
-
-6.4 verbs test:
-server: ibv_rc_pingpong -g 1
-client: ibv_rc_pingpongs -g 1 server
-
-6.5 For rdma_cm applications, the user only needs to specify an IP address of a
-VLAN device for the traffic to go with that VLAN tagged frames.
-
-7. Statistic counters
-=====================
-RoCEE traffic is counted and can be read from the sysfs counters in the same
-manner as it is done for regular Infiniband devices. Only the following
-counters are supported:
-- port_xmit_packets
-- port_rcv_packets
-- port_rcv_data
-- port_xmit_data
-
-For example, to read the number of transmitted packets on port 2 of device
-mlx4_1, one needs to read the file:
-/sys/class/infiniband/mlx4_1/ports/2/counters/port_xmit_packets
-
-Note: RoCEE traffic will not show in the associated Etherent device's counters
-since it is offloaded by the hardware and does not go through Ethernet network
-driver.
-
-
-8. Firmware Requirements
-========================
-RoCEE has limited support with firmware 2.7.700 and is fully supported
-with firmware 2.8.000 and above.
-
-
-9. Supported hardware
-=====================
-Currently, ConnectX B0 hardware is supported. A0 hardware may have issues.
-
-
-10. Added fearues
-=================
-ibdev2netdev is a utility that displays the association between an HCA's port
-and the network interface bound to it. Example run:
-
-# ibdev2netdev
-mlx4_0 port 1 ==> ib0 (Down)
-mlx4_0 port 2 ==> ib1 (Down)
-mlx4_1 port 1 ==> eth2 (Up)
-mlx4_1 port 2 ==> eth3 (Up)
-
-
-
-11. Known Issues
-===============
-- PowerPC and ia64 architectures are not supported. x32 architectures were
- not tested.
-
-- SRP is not supported.
-
-- UD QPs that send traffic with VLAN tags (e.g. 802.1q tagged frames) do not
- work. This will be fixed in a subsequent release.
+++ /dev/null
-SCSI RDMA Protocol (SRP) Target driver for Linux
-=================================================
-
-SRP Target driver is designed to work directly on top of OpenFabrics
-OFED-1.x software stack (http://www.openfabrics.org) or Infiniband
-drivers in Linux kernel tree (kernel.org). It also interfaces with
-Generic SCSI target mid-level driver - SCST (http://scst.sourceforge.net)
-
-By interfacing with SCST driver we are able to work and support a lot IO
-modes on real or virtual devices in the backend
-
-1. scst_disk -- interfacing with scsi sub-system to claim and export real
- scsi devices ie. disks, hardware raid volumes, tape library as SRP's luns
-
-2. scst_vdisk -- fileio and blockio modes. This allows you to turn software
- raid volumes, LVM volumes, IDE disks, block devices and normal files into
- SRP's luns
-
-3. NULLIO mode will allow you to measure the performance without sending IOs
- to *real* devices
-
-
-Prerequisites
--------------
-0. Supported distributions: RHEL 5.2/5.3/5.4, SLES 10 sp2/sp3, SLES 11
-
-NOTES: On distribution default kernels, you can run scst_vdisk blockio mode
- to have good performance.
-
- It is required to patch and recompile the kernel to run scst_disk
- ie. scsi pass-thru mode
- OR
- You have to compile scst with -DSTRICT_SERIALIZING enabled and this
- does not yield good performance.
-
-1. Download and install SCST driver (supported version 1.0.1.1)
-
-1a. Download scst-1.0.1.1.tar.gz from this URL
- http://scst.sourceforge.net/downloads.html
-
-1b. untar and install scst-1.0.1.1
-
- $ tar zxvf scst-1.0.1.1.tar.gz
- $ cd scst-1.0.1.1
-
- THIS STEP IS SPECIFIC FOR SLES 10 sp2/sp3 distributions:
-
- $ patch -p1 -i <path to OFED>/docs/scst/scst_sles10_sp2.patch
-
- For all distributions:
-
- $ make && make install
-
-NOTES: FOR SLES 11 distribution, skip next step (step 1c) and go directly to
- step (2)
-
-1c. patch scst.h header file with scst.patch
-
- $ cd /usr/local/include/scst
- $ patch -p1 -i <path to OFED>/docs/scst/scst.patch
-
-
-2. Download/install OFED-1.5.1 package - SRP target is part of OFED package
-
-NOTES: if your system already have OFED stack installed, you need to remove
- the previous built of kernel-ib RPMs and reinstall
-
- $ cd ~/OFED-1.5.1
- $ rm RPMS/*/*/kernel-ib*
- $ ./install.pl -c ofed.conf
-
- Make sure that srpt=y in the ofed.conf
-
-2a. download OFED packages from this URL
- http://www.openfabrics.org/downloads/OFED/OFED-1.5.1/
-
-2b. install OFED - remember to choose srpt=y
-
- $ cd ~/OFED-1.5.1
- $ ./install.pl
-
-
-How-to run
------------
-
-A. On srp target machine
-
-A1. Please refer to SCST's README for loading scst driver and its dev_handlers
- drivers (scst_disk, scst_vdisk block or file IO mode, nullio, ...)
- SCST's README locates in ~/scst-1.0.1.1/ directory
-
-NOTES: In any mode you always need to have lun 0 in any group's device list
- Then you can have any lun number following lun 0 (it does not required
- have lun number in order except that the first lun is always 0)
-
- Setting SRPT_LOAD=yes in /etc/infiniband/openib.conf is not good enough
- It only load ib_srpt module and does not load scst and its dev_handlers
-
- SCST's scst_disk module (pass-thru mode) does not run on default
- distribution kernels (kernels come with RHEL 5.2/5.3/5.4 & SLES 11)
- because it requires to patch and recompile the kernel. It can only
- run with vanilla kernels.
-
-Example 1: working with VDISK BLOCKIO mode
- (using md0 device, sda, and cciss/c1d0)
-a. modprobe scst
-b. modprobe scst_vdisk
-c. echo "open vdisk0 /dev/md0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
-d. echo "open vdisk1 /dev/sda BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
-e. echo "open vdisk2 /dev/cciss/c1d0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
-f. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices
-g. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices
-h. echo "add vdisk2 2" >/proc/scsi_tgt/groups/Default/devices
-
-Example 2: working with real back-end scsi disks in scsi pass-thru mode
-a. modprobe scst
-b. modprobe scst_disk
-c. cat /proc/scsi_tgt/scsi_tgt
-ibstor00:~ # cat /proc/scsi_tgt/scsi_tgt
-Device (host:ch:id:lun or name) Device handler
-0:0:0:0 dev_disk
-4:0:0:0 dev_disk
-5:0:0:0 dev_disk
-6:0:0:0 dev_disk
-7:0:0:0 dev_disk
-
-Now you want to exclude the first scsi disk and expose the last 4 scsi disks
-as IB/SRP luns for I/O
-
-echo "add 4:0:0:0 0" >/proc/scsi_tgt/groups/Default/devices
-echo "add 5:0:0:0 1" >/proc/scsi_tgt/groups/Default/devices
-echo "add 6:0:0:0 2" >/proc/scsi_tgt/groups/Default/devices
-echo "add 7:0:0:0 3" >/proc/scsi_tgt/groups/Default/devices
-
-Example 3: working with scst_vdisk FILEIO mode
- (using md0 device and file 10G-file)
-a. modprobe scst
-b. modprobe scst_vdisk
-c. echo "open vdisk0 /dev/md0" > /proc/scsi_tgt/vdisk/vdisk
-d. echo "open vdisk1 /10G-file" > /proc/scsi_tgt/vdisk/vdisk
-e. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices
-f. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices
-
-A2. modprobe ib_srpt
-
-
-B. On initiator machines you can manualy do the following steps:
-
-B1. modprobe ib_srp
-B2. ipsrpdm -c -d /dev/infiniband/umadX
- (to discover new SRP target)
- umad0: port 1 of the first HCA
- umad1: port 2 of the first HCA
- umad2: port 1 of the second HCA
-B3. echo {new target info} > /sys/class/infiniband_srp/srp-mthca0-1/add_target
-B4. fdisk -l (will show new discovered scsi disks)
-
-Example:
-Assume that you use port 1 of first HCA in the system ie. mthca0
-
-[root@lab104 ~]# ibsrpdm -c -d /dev/infiniband/umad0
-id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4,
-dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4
-[root@lab104 ~]# echo id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4,
-dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 >
-/sys/class/infiniband_srp/srp-mthca0-1/add_target
-
-OR
-
-+ You can edit /etc/infiniband/openib.conf to load srp driver and srp HA daemon
-automatically ie. set SRP_LOAD=yes, SRP_DAEMON_ENABLE=yes, and SRPHA_ENABLE=yes
-+ To set up and use high availability feature you need dm-multipath driver
-and multipath tool
-+ Please refer to OFED-1.5.1 SRP's user manual for more in-details instructions
-on how-to enable/use HA feature (OFED-1.5.1/docs/srp_release_notes.txt)
-
-
-Here is an example of srp target setup file
---------------------------------------------
-
-*********************** srpt.sh *****************************************
-#!/bin/sh
-modprobe scst scst_threads=1
-modprobe scst_vdisk scst_vdisk_ID=100
-
-echo "open vdisk0 /dev/cciss/c1d0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
-echo "open vdisk1 /dev/sdb BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
-echo "open vdisk2 /dev/sdc BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
-echo "open vdisk3 /dev/sdd BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
-echo "add vdisk0 0" > /proc/scsi_tgt/groups/Default/devices
-echo "add vdisk1 1" > /proc/scsi_tgt/groups/Default/devices
-echo "add vdisk2 2" > /proc/scsi_tgt/groups/Default/devices
-echo "add vdisk3 3" > /proc/scsi_tgt/groups/Default/devices
-
-modprobe ib_srpt
-
-echo "add "mgmt"" > /proc/scsi_tgt/trace_level
-echo "add "mgmt_dbg"" > /proc/scsi_tgt/trace_level
-echo "add "out_of_mem"" > /proc/scsi_tgt/trace_level
-
-*********************** End srpt.sh **************************************
-
-
-How-to unload/shutdown
------------------------
-
-1. Unload ib_srpt
- $ modprobe -r ib_srpt
-2. Unload scst and its dev_handlers
- $ modprobe -r scst_vdisk scst
-3. Unload ofed
- $ /etc/rc.d/openibd stop
-
-===========================================================================
-Known Issues
-===========================================================================
-
-- With active connections/sesssions and active I/Os, unload ib_srpt driver
- will randomly fail and got stuck.
-
-- With active connections/sessions with active I/Os, reboot system will
- randomly get stuck.
-
+++ /dev/null
-IB Bonding
-===============================================================================
-
-1. Introduction
-2. How to work with interface configuration scripts
-2.1 Configuration with initscripts support
-2.1.1 Writing network scripts under Redhat-AS4 (Update 6, 7 or 8)
-2.1.2 Writing network scripts under Redhhat-EL5
-2.2 Configuration with sysconfig support
-2.2.1 Writing network scripts under SLES-10
-2.2.2 Writing network scripts under SLES-11 SP1
-2.3 Configuring Ethernet slaves
-
-1. Introduction
--------------------------------------------------------------------------------
-ib-bonding is a High Availability solution for IPoIB interfaces. It is based
-on the Linux Ethernet Bonding Driver and was adopted to work with IPoIB.
-However, the support for for IPoIB interfaces is only for the active-backup
-mode, other modes should not be used.
-
-2. How to work with interface configuration scripts
--------------------------------------------------------------------------------
-To create an interface configuration script for the ibX and bondX interfaces,
-you should use the standard syntax (depending on your OS).
-
-2.1 Configuration with initscripts support
-------------------------------------------
-Note: This feature is available only for Redhat-AS4 (Update 4, Update 5,
-Update 6 or Update 7) and for Redhat-EL5 and above.
-
-2.1.1 Writing network scripts under Redhat-AS4 (Update 4, 5, 6 or 7)
------------------------------------------------------------------
-* In the master (bond) interface script add the line:
-TYPE=Bonding
-MTU=<according to the slave's MTU>
-
-Exmaple: for bond0 (master) the file is named /etc/sysconfig/network-scripts/ifcfg-bond0
-with the following text in the file:
-
-DEVICE=bond0
-IPADDR=192.168.1.1
-NETMASK=255.255.255.0
-NETWORK=192.168.1.0
-BROADCAST=192.168.1.255
-ONBOOT=yes
-BOOTPROTO=none
-USERCTL=no
-TYPE=Bonding
-MTU=65520
-
-Note: 65520 is a valid mtu value only if all IPoIB slaves operate in connected
-mode and are configured with the same value. For IPoIB slaves that work in
-datagram modee, use MTU=2044. If you don't set correct mtu or don't set mtu at
-all (and letting it to be set to the default value), performance of the
-interface might decrease.
-
-* In the slave (ib) interface script put the following lines:
-SLAVE=yes
-MASTER=<bond name>
-TYPE=InfiniBand
-PRIMARY=<yes|no>
-
-Example: the script for ib0 (slave) would be named /etc/sysconfig/network-scripts/ifcfg-ib0
-with the following text in the file:
-
-DEVICE=ib0
-USERCTL=no
-ONBOOT=yes
-MASTER=bond0
-SLAVE=yes
-BOOTPROTO=none
-TYPE=InfiniBand
-PRIMARY=yes
-
-Note: If the slave interface is not primary then the line PRIMARY= is not
-required and can be omitted.
-
-After the configuration is saved, restart the network service by running:
-/etc/init.d/network restart
-
-2.1.2 Writing network scripts under Redhhat-EL5
------------------------------------------------
-Follow the instructions in 3.1.1 (Writing network scripts under Redhat-AS4)
-with the following changes:
-* In the bondX (master) script - the line TYPE=Bonding is not needed.
-* In the bondX (master) script - you may add to the configuration more options
-with the following line
-BONDING_OPTS=" primary=ib0 updelay=0 downdelay=0"
-* in the ibX (slave) script - the line TYPE=InfiniBand necessary when using
- bonding over devices configured with partitions ( p_key)
-Example:
- ifcfg-ibX.8003 and ifcfg-ibY.8003 must include TYPE=InfiniBand line in
- their configuration files, when using as slaves for bondX device
-* in /etc/modprobe.conf add the following lines
-alias bond0 bonding
-options bond0 miimon=100 mode=1 max_bonds=1
-
-If you want more than one bonding interface, name them bond1, bond2... and
-just add the necessary lines in /etc/modprobe.conf and change max_bonds=1 to
-max_bonds=N where N=number_of_bonding_interfaces
-
-Note: restarting OFED doesn't keep the bonding configuration via initscripts.
-You have to restart the network service in order to recreate the bonding
-interface.
-
-2.2 Configuration with sysconfig support
-----------------------------------------
-Note: This feature is available only for SLES-10 and above.
-
-2.2.1 Writing network scripts under SLES-10
------------------------------------------------
-* In the master (bond) interface script add the lines:
-
-BONDING_MASTER=yes
-BONDING_MODULE_OPTS="mode=active-backup miimon=<value>"
-BONDING_SLAVE0=slave0
-BONDING_SLAVE1=slave1
-MTU=<according to the slave's MTU>
-
-Exmaple: for bond0 (master) the file is named /etc/sysconfig/network/ifcfg-bond0
-with the following text in the file:
-
-BOOTPROTO="static"
-BROADCAST="10.0.2.255"
-IPADDR="10.0.2.10"
-NETMASK="255.255.0.0"
-NETWORK="10.0.2.0"
-REMOTE_IPADDR=""
-STARTMODE="onboot"
-BONDING_MASTER="yes"
-BONDING_MODULE_OPTS="mode=active-backup miimon=100 primary=ib0 updelay=0 downdelay=0"
-BONDING_SLAVE0=ib0
-BONDING_SLAVE1=ib1
-MTU=65520
-
-Note: 65520 is a valid mtu value only if all IPoIB slaves operate in connected
-mode and are configured with the same value. For IPoIB slaves that work in
-datagram modee, use MTU=2044. If you don't set correct mtu or don't set mtu at
-all (and letting it to be set to the default value), performance of the
-interface might decrease.
-
-Note: primary, downdelay and updelay is an optional bonding interface
-configuration. You may choose to use them, change them or delete them from the
-configuration script (by editing the line that starts with BONDING_OPTS)
-
-* The slave (ib) interace script should look like this:
-
-BOOTPROTO='none'
-STARTMODE='off'
-PRE_DOWN_SCRIPT=/etc/sysconfig/network/unenslave.sh
-
-After the configuration is saved, restart the network service by running:
-/etc/init.d/network restart
-
-2.2.2 Writing network scripts under SLES-11 SP1
------------------------------------------------
-Need to change the following parameters in ifcfg-ibX:
-BOOTPROTO=static
-STARTMODE=auto
-
-Logs:
-ifccfg-ibX
-BOOTPROTO=static
-STARTMODE="auto"
-PRE_DOWN_SCRIPT=/etc/sysconfig/network/unenslave.sh
-
-ifcfg-bondX
-BOOTPROTO="static"
-BROADCAST=195.168.80.255
-IPADDR=195.168.80.12
-NETMASK=255.255.255.0
-NETWORK=195.168.80.0
-REMOTE_IPADDR=""
-STARTMODE=auto
-BONDING_MASTER=yes
-BONDING_MODULE_OPTS="mode=active-backup miimon=100"
-BONDING_SLAVE0=ib0
-BONDING_SLAVE1=ib1
-
-2.3 Configuring Ethernet slaves
--------------------------------
-It is not possible to have a mix of Ethernt slaves and IPoIB slaves under the
-same bonding master. It is possible however that a bonding master of Ethernet
-slaves and a bonding master of IPoIB slaves will co-exist in one machne.
-To configure Ethernet slaves under a bonding master use the following
-instructios (depending on the OS)
-
-* Under Redhat-AS4
-
-Use the same instructions as for IPoIB slaves with the following exceptions
-
-- In the master configuration file add the line
-SLAVEDEV=1
-- In the slave configuration file leave the line
-TYPE=InfiniBand
-- For Ethernet, it is possible to set parameters of the bonding module in /etc/modprobe.conf
-with the following line for example
-options bonding miimon=100 mode=1 primary=eth0
-Note that alias names for the bonding module (such as bond0) may not work.
-
-* Under Redhat-AS5
-
-No special instructions are required.
-
-* Under SLES10
-
-When using both type of bonding under, it is neccessary to update the
-MANDATORY_DEVICES environment variable in /etc/sysconfig/network/config with the names
-of the InfiniBand devices ( ib0, ib1, etc. ). Otherwise, bonding devices will be created
-before InfiniBand devices at boot time.
-
-Note: If there is more than one Ethernet NIC installed then there might be a
-race for the interface name eth0, eth1 etc. This may lead to unexpected
-relation between logical and physical devices which may lead to wrong bonding
-configuration. This issue may be solved by binding a logical device name (e.g.
-eth0) to a physical (hardware) device by specifying the MAC address in the
-ethN configuration file.
+++ /dev/null
-
- Open Fabrics Enterprise Distribution (OFED)
- ehca in OFED 1.5.3 Release Notes
-
- February 2011
-
-
-Overview
---------
-ehca is the low level driver implementation for all IBM GX-based HCAs.
-
-Supported HCAs
---------------
-- GX Dual-port SDR 4x IB HCA
-- GX Dual-port SDR 12x IB HCA
-- GX Dual-port DDR 4x IB HCA
-- GX Dual-port DDR 12x IB HCA
-
-Available Parameters
---------------------
-In order to set ehca parameters, add the following line(s) to /etc/modprobe.conf:
-
- options ib_ehca <parameter>=<value>
-
-whereby <parameter> is one of the following items:
-- debug_level debug level (0: no debug traces (default), 1: with debug traces)
-- port_act_time time to wait for port activation (default: 30 sec)
-- scaling_code scaling code (0: disable (default), 1: enable)
-- open_aqp1 Open AQP1 on startup (default: no) (bool)
-- hw_level Hardware level (0: autosensing (default), 0x10..0x14: eHCA, 0x20..0x23: eHCA2) (int)
-- nr_ports number of connected ports (-1: autodetect (default), 1: port one only, 2: two ports) (int)
-- use_hp_mr Use high performance MRs (default: no) (bool)
-- poll_all_eqs Poll all event queues periodically (default: yes) (bool)
-- static_rate Set permanent static rate (default: no static rate) (int)
-- lock_hcalls Serialize all hCalls made by the driver (default: autodetect) (bool)
-- number_of_cqs Max number of CQs which can be allocated (default: autodetect) (int)
-- number_of_qps Max number of QPs which can be allocated (default: autodetect) (int)
-
-New Features
-------------
-- None
-
-Fixed Bugs ofed-1.5.3
----------------------
-- Fixed initialization of IRQ tasklet
-- Catch fails in ioremap()
-
-Fixed Bugs ofed-1.5.2
----------------------
-- Fixed automatic detection if hcall locks should be enabled or not
-
-Fixed Bugs ofed-1.5.1
----------------------
-- Fixed crash when reading sysfs performance counters
-- Do not disable IRQs when processing EQs
-- Allow query of max_dest_rd_atomic and max_qp_rd_atomic values
-
-Fixed Bugs ofed-1.5
----------------------
-- SRQ overflow prevention
-- Performance improvements for QP creation
-- MAD redirection fix
-
-Fixed Bugs ofed-1.4.1
----------------------
-- none
-
-Fixed Bugs ofed-1.4
----------------------
-- Reject send work requests only for RESET, INIT and RTR state
-- Reject receive work requests if QP is in RESET state
-- In case of lost interrupts, trigger EOI to reenable interrupts
-- Filter PATH_MIG events if QP was never armed
-- Release mutex in error path of alloc_small_queue_page()
-- Check idr_find() return value
-- Discard double CQE for one WR
-- Generate flush status CQ entries
-- Don't allow creating UC QP with SRQ
-- Fix reported max number of QPs and CQs in systems with >1 adapter
-- Reject dynamic memory add/remove when ehca adapter is present
-- Remove reference to special QP in case of port activation failure
-- Fix locking for shca_list_lock
-
-Fixed Bugs ofed-1.3.1
----------------------
-- Support all ibv_devinfo values in query_device() and query_port()
-- Prevent posting of SQ WQEs if QP not in RTS
-- Remove mr_largepage parameter, ie always enable large page support
-- Allocate event queue size depending on max number of CQs and QPs
-- Protect QP against destroying until all async events for it are handled
-
-Fixed Bugs ofed-1.3
--------------------
-- Serialize HCA-related hCalls if necessary
-- Fix static rate if path faster than link
-- Return physical link information in query_port()
-- Fix clipping of device limits to INT_MAX
-- Fix issues related to path migration support
-- Support more than 4k QPs for userspace and kernelspace
-- Prevent sending UD packets to QP0
-- Prevent RDMA-related connection failures on some eHCA2 hardware
-
-Available backports
--------------------
-- RedHat EL5 up5: 2.6.18-194.ELsmp
-- RedHat EL6: 2.6.32-71.el6
-- SLES11SP1: 2.6.32.12-0.7-default
-- SLES10SP3: 2.6.16.60-0.54.5
-- kernel.org: 2.6.30-36
-
-Known Issues
-------------
-1. The port(s) needs to be connected to an active switch port while
-loading the ehca device driver.
-
-2. Dynamic memory operations are tolerated by ehca, but are prevented by
-the driver while it is loaded.
+++ /dev/null
-
- Open Fabrics Enterprise Distribution (OFED)
- iSER initiator in OFED 1.5.x Release Notes
-
- March 2010
-
-
-* Background
-
- iSER allows iSCSI to be layered over RDMA transports (including
- InfiniBand and iWARP (RNIC)).
-
- The OpenFabrics iSER initiator implementation is inter-operable with
- open-iscsi (http://www.open-iscsi.org/). It provides an alternative
- transport to iscsi_tcp in the open-iscsi framework. The iSER transport
- exposes a transport API to scsi_transport_iscsi, and a SCSI LLD API to
- the Linux SCSI mid-layer (scsi_mod). Currently, the OpenFabrics iSER
- initiator can be layered over InfiniBand (no iWARP support yet).
-
-* Supported platforms
-
- - kernel.org: 2.6.30 and higher
- - RHEL 5.4
-
- Except for these platforms, OFED-1.5.x will not install iSER on top of
- the kernel and the original iSER module coming with Linux Distribution
- will stop working because of mismatch in symbols version.
-
-* Fixed Bugs and Enhancements since OFED 1.3
- iSER:
- - Add logical unit reset support
- - Update URLs of iSER docs
- - Add change_queue_depth method
- - Fix list iteration bug
- - Handle iser_device allocation error gracefully
- - Don't change ITT endianess
- - Move high-volume debug output to higher debug level
- - Count FMR alignment violations per session
- Open-iSCSI:
- - Update open-iscsi rpm versions from
- 2.0-754 to 2.0-754.1 and from 2.0-865.15 to 2.0-869.2
- - Change open-iscsi defaults
- - iscsi_discovery: fixed printing debug information
- - iscsi_discovery: check if iscsid is running
- - Set open-iscsi for auto-startup when installing OFED
- - iscsiadm: bail out if daemon isn't running
-
-* Known Issues
- Open-iSCSI:
- - modifying node transport_name while session is active
- will create stale session. It will be deleted only after reboot.
-
-* Installation/upgrade of open-iscsi
- If iSER is selected to be installed with OFED, open-iscsi will be also
- installed (or upgraded if another version of open-iscsi is already
- installed). Installing/upgrading open-iscsi is required for iSER to
- work properly. Before installing OFED, please make sure that no version
- of open-iscsi is installed or add the following key to your ofed.conf
- file: upgrade_open_iscsi=yes. Using this key will remove any old version
- of open-iscsi.
-
- If an older version of open-iscsi was installed, it is recommended to
- delete its records before running open-iscsi. This can easily be done by
- running the following command (while open-iscsi is stopped):
-
- rm -rf /etc/iscsi/nodes/* /etc/iscsi/send_targets/*
-
- Then, open-iscsi may be started, and targets may be discovered by running
- 'iscsi_discovery <target_ip>'.
-
-* iSER links
-
- Wiki pages
-
- Information on building/configuring/running the open iscsi initiator over
- iSER: https://wiki.openfabrics.org/tiki-index.php?page=iSER
-
- IETF pages
-
- iSCSI and iSER specifications come out of the IETF IP storage (IPS) work
- group.
-
- iSCSI specification: http://www.ietf.org/rfc/rfc3720.txt
- iSER specification: http://www.ietf.org/rfc/rfc5046.txt
-
- "About" page
-
- general and detailed information on iSCSI and iSER
- http://www.voltaire.com/Products/Server_Products/iSER_iSCSI_RDMA
-
+++ /dev/null
- Open Fabrics Enterprise Distribution (OFED)
- STGT/iSER target in OFED 1.5 Release Notes
-
- December 2009
-
-
-* Background
-
- iSER allows iSCSI to be layered over RDMA transports (including InfiniBand
- and iWARP (RNIC)). Linux target framework (tgt) aims to simplify various SCSI
- target driver (iSCSI, Fibre Channel, SRP, etc) creation and maintenance.
-
- tgt supports the following target drivers (among othets)
-
- - iSCSI software (tcp) target driver for Ethernet/IPoIB NICs
- - iSER software target driver for Infiniband and RDMA NICs
-
- For iSCSI and iSER tgt consists of user-space daemon, and user-space
- tools. That is, no special kernel support is needed other than the
- kernel (and user space) RDMA stacks.
-
- The code is under the GNU General Public License version 2.
-
- This package is based on a snapshot (clone) of the tgt git tree taken
- on August 28th, 2008
-
-* Supported platforms
-
- RHEL 5 and its updates
- SLES 10 and its service-packs
-
- The release has been tested against the Linux open iscsi initiator
-
-* STGT/iSER links
-
- STGT home page
- http://stgt.berlios.de
-
- STGT git
- git://git.kernel.org/pub/scm/linux/kernel/git/tomo/tgt.git
-
- the STGT sources have some embedded documentation, specifically
- the README and REDMA.iscsi files would be usefull
-
- Wiki pages
-
- Information on building/configuring/running the stgt/iser target
- https://wiki.openfabrics.org/tiki-index.php?page=iSER-target
-
- general and detailed information on iSCSI and iSER
- http://www.voltaire.com/Products/Server_Products/iSER_iSCSI_RDMA
+++ /dev/null
- Open Fabrics Enterprise Distribution (OFED)
- mthca in OFED 1.5 Release Notes
-
- December 2009
-
-
-===============================================================================
-Table of Contents
-===============================================================================
-1. Overview
-2. Fixed Bugs since OFED 1.3.1
-3. Bug fixes and enhancements since OFED 1.4
-4. Known Issues
-
-===============================================================================
-1. Overview
-===============================================================================
-mthca is the low level driver implementation for the following Mellanox
-Technologies HCAs: InfiniHost, InfiniHost III Ex and InfiniHost III Lx.
-
-mthca Available Parameters
---------------------------
-In order to set mthca parameters, add the following line to /etc/modpobe.conf:
-
- options ib_mthca parameter=<value>
-
-mthca parameters:
- catas_reset_disable: disable reset on catastrophic event if nonzero
- (int)
- fw_cmd_doorbell: post FW commands through doorbell page if
- nonzero (and supported by FW) (int)
- debug_level: Enable debug tracing if > 0 (int)
- msi_x: attempt to use MSI-X if nonzero (int)
- tune_pci: increase PCI burst from the default set by BIOS if nonzero (int)
- num_qp: maximum number of QPs per HCA (int)
- rdb_per_qp: number of RDB buffers per QP (int)
- num_cq: maximum number of CQs per HCA (int)
- num_mcg: maximum number of multicast groups per HCA (int)
- num_mpt: maximum number of memory protection table entries per HCA (int)
- num_mtt: maximum number of memory translation table segments per HCA (int)
- num_udav: maximum number of UD address vectors per HCA (int)
- fmr_reserved_mtts: number of memory translation table segments reserved for
- FMR (int)
- log_mtts_per_seg: Log2 number of MTT entries per segment (1-5) (int)
-
-===============================================================================
-2. Fixed Bugs
-===============================================================================
-- Fix access to freed memory in catastrophic processing
- catas_reset() uses pointer to mthca_dev, but mthca_dev is not valid after
- call __mthca_restart_one().
-
-
-===============================================================================
-3. Bug fixes and enhancements since OFED 1.4
-===============================================================================
-- Added a module parameter (log_mtts_per_seg) for number of MTTs per segment.
- This enable to register more memory with the same number of segments.
-- Bring INIT_HCA and other commands timeout into consistency with PRM. This
- solve an issue when had more than 2^18 max qp's configured.
-
-===============================================================================
-4. Known Issues
-===============================================================================
-1. A UAR size other than 8MB prevents mthca driver loading. The default UAR
- size is 8MB. If the size is changed, the following error message will be
- logged to /var/log/messages upon attempting to load the mthca driver:
- ib_mthca 0000:04:00.0: Missing UAR, aborting.
-
-2. If a user level application using multicast receives a control signal
- in the process of detaching from a multicast group, its QP may remain a
- member of the multicast group (in HCA).
- Workaround: Destroy the multicast group after detaching the QP from it.
-
-3. In mem-free devices, RC QPs can be created with a maximum of (max_sge - 1)
- entries only; UD QPs can be created with a maximum of (max_sge - 3) entries.
-
-4. Performance can be degraded due to a wrong BIOS configuration:
- The PCI Express specification requires the BIOS to set the MaxReadReq
- register for each HCA card for maximum performance and stability.
-
- If you experience bandwidth performance degradation, try forcing the card to
- behave not according to the PCI Express specification by setting the
- tune_pci=1 module parameter. This tune_pci=1 assignment was the default
- setting in OFED 1.0; therefore, it may have masked performance degradation
- on some systems.
-
- If tune_pci=1 improves bandwidth, please report the issue to your BIOS
- vendor. Please note that Mellanox Technologies does not recommend using
- tune_pci=1 in production systems: working with tune_pci=1 set is untested
- and is known to trigger instability issues on some platforms.
-
Version: OpenSM 3.3.x
Repo: git://git.openfabrics.org/~alexnetes/opensm.git
-Date: Nov 2011
+Date: Aug 2012
1 Overview
----------
This document describes the contents of the OpenSM 3.3 release.
OpenSM is an InfiniBand compliant Subnet Manager and Administration,
and runs on top of OpenIB. The OpenSM version for this release
-is opensm-3.3.12.
+is opensm-3.3.15.
This document includes the following sections:
1 This Overview section (describing new features and software
fabrics which have some CA nodes attached closer to the roots than some
switch nodes.
+* SSSP Unicast routing algorithm - SSSP unicast routing algorithm - a
+ single-source-shortest-path routing algorithm, which globally balances the
+ number of routes per link to optimize link utilization. This routing
+ algorithm has no restrictions in terms of the underlying topology.
+
+* DFSSSP unicast routing algorithm - a deadlock-free single-source-
+ shortest-path routing, which uses the SSSP algorithm as the base to optimize
+ link utilization and uses Infiniband virtual lanes (SL) to provide deadlock-
+ freedom.
+
+* SRIOV (Alias GUID) Support - In order to support virtualized environments,
+ alias GUID support is added to OpenSM. This support allows an SA client to
+ add and remove additional port GUIDs based on SubAdmSet/SubAdmDelete of
+ GUIDInfoRecord.
+
* Extended speed support
This provides support for FDR and EDR speeds.
+* Congestion control support
+
* Many code improvements, optimizations and cleanups.
* Windows support (early stage).
1.2 Minor New Features:
+7e6bdef opensm/man/opensm.8.in: Add section for MKey support
+aebe678 opensm: Add support for multicast service records
+5509234 opensm/scripts/sldd.sh: Update to support guid2mkey/neighbors
+2ae1477 opensm: Ensure sweep interval/mkey lease are sensibly set
+cefe79b opensm: Check for valid mkey protection level in config file
+8fa0d2c opensm: Add neighboring link cache file
+5088d08 opensm: Log errors on SubnGet timeouts
+9eed9c6 opensm: Add support for setting mkey protection levels
+5c4157d opensm: Add locking where necessary around osm_req_*
+2f74f34 opensm: Allow recovery of subnets with misset mkeys
+e5dc557 opensm: Add guid2mkey cache file support
+3659b37 opensm/osm_sa_class_port_info.c: Indicate support for PortInfo CapMask2
+ matching in SA ClassPortInfo:CapabilityMask2
+8016d3b opensm/osm_base.h: Add some SA ClassPortInfo CapabilityMask2 bits
+1a31c44 opensm/osm_perfmgr.c: Use non conflicting error codes in log messages
+03a75d0 opensm/osm_sa_path_record.c: Restore osm_get_path_params functionality
+a9340cf opensm: Support (null) being specified for per_module_logging_file
+ option
+00375aa opensm/osm_perfmgr.c: Eliminate compile warning
+7868c98 opensm: Remove unused per_module_logging option
+49c460b Call drop manager before checking for other Master SM in the fabric
+c0604f3 Increase p_port->discovery_count only when received PortInfo for port 0
+ of the switch
+24b30d2 opensm/osm_node_info_rcv.c: Handle non-compliant SMA gracefully
+a4f2689 opensm/osm_vendor_ibumad: Add management class into match criteria
+4be6375 opensm/osm_sa*.c: Log requester port GUID at DEBUG level
+4cca51d opensm/osm_sa_mcmember_record.c: Log requester port GUID at DEBUG level
+15b3eae opensm/osm_sa_path_record.c: Log requester port GUID at DEBUG level
+0b580ca opensm/osm_sa_path_record.c: Add debug logging to
+ pr_match_mgrp_attributes
+ed4b7fb opensm/osm_sa_mcmember_record.c: In mcmr_rcv_join_mgrp, add MGID to log
+ message
+872dae4 opensm/osm_sa_mcmember_record.c: Dump MCMemberRecord in mcmr_query_mgrp
+71f2ce7 Sending SL2VL and VLARB SET MADs in distributed manner
+f07bcc1 opensm/osm_subnet.c: Cosmetic formatting change
+c823a5b opensm/osm_link_mgr.c: Set PortInfo:PortState to LinkDown when remote
+ port isn't accessible
+ab88df6 Add support to reread configuration file when stacked in rediscovery
+ loop.
+67c9bae opensm: Move per_mod_log_tbl array from subn to log structure
+2cbd9f5 opensm/cl_atomic_osd.h: Cosmetic formatting change
+cd63dec opensm/osm_helper.c: Add CapabilityMask2 to notice dump for trap 144
+9205812 opensm/ib_types.h: Add CapabilityMask2 to notice for trap 144
+25de706 opensm/osm_sa_path_record.c: Add missing end-of-line in the log message
+3551530 opensm/osm_trap_rcv.c: Remove vestigial comment
+067d217 opensm/osm_inform.c: Make log message format consistent for error
+ messages
+759b82a opensm/osm_trap_rcv.c: Add better logging for traps 257 and 258
+b806657 opensm/osm_sa_mad_ctrl.c: Eliminate commented out code line
+a851693 opensm/perfmgr: add logging of error counters
+52fa659 opensm/console: add perfmgr "print_errors" (pe) console command.
+a832ce2 opensm/console: Add human readable output for perfmgr data counters
+77a1756 opensm/console: add abreviations for perfmgr commands
+aca9a07 opensm/console; add port option to perfmgr print_counters
+00628cc opensm/console: add "print all" to print_counters console command
+f5de9b5 opensm: perfmgr mark inactive nodes in perfmgr db
+ff06340 opensm: perfmgr delete "inactive" nodes from the DB
+059d8f4 opensm/console: protect against 0 entered for the perfmgr sweep_time
+cc86607 opensm/perfmgr: Add config option to ignore Channel Adapters.
+91f0c00 opensm/osm_node_info_rcv.c: In ni_rcv_process_existing_ca_or_router,
+ handle error
+c930a23 opensm/osm_subnet.c: Indicate lmc and lmc_esp0 are not changeable
+ "on the fly"
+7825e67 opensm/libopensm.map: Removed unimplemented routine
+7de7b04 opensm/main.c: Handle daemon mode with guid specified as 0 more
+ gracefully
+9478fbf opensm/osm_subnet.c: Support MLNX ExtendedPortInfo for ConnectIB device
+f8fc334 opensm/osm_node_info_rec.c: Also handle non compliant SMA in
+ ni_rcv_process_existing
+b4a481d opensm: Eliminate unneeded field in DR path structure
+01bc8c9 opensm/osm_state_mgr.c: Force subn->need_update when coming out of
+ STANDBY
+276be8b opensm: Dump info functions update for per module logging
+64b512a opensm/osm_vendor_ibumad.c: Make binding log message clearer
+e70c8c1 opensm: Add enum for FILE_ID for per module logging
+2a2db8c opensm: Add per module logging support
+e21b106 opensm: Cosmetic changes
+3ddb2e3 opensm: Add partition manager configuration doc to docs
+9586649 opensm/opensm_release_notes-3.3.txt: Update Unsupported IB Compliance
+ Statements
+826b5c4 opensm/osm_ucast_dfsssp.c: Use osm_log_is_active
+b2cad9d opensm/complib/cl_fleximap.h: Cosmetic changes
+322a310 opensm/osm_ucast_ftree.c: Add a couple of asserts
e3a946d opensm: Add FDR10 support
6cea3df opensm/osm_sa_portinfo_record.c: Add SA PortInfoRecord support
for CapabilityMask2 matching
cb1484d opensm/osm_sa_mcmember_record.c: Add rate validation if supplied
485d068 opensm/osm_sa_path_record.c: Add rate validation if supplied
300f4d9 opensm/osm_helper: Add ib_rate_is_valid
+9b50961 opensm: Change osm_routing_engine struct to not use C++ reserved word
+c9c0aa6 opensm/man/torus-2QoS.conf.5.in: Update portgroup_max_ports section
+3c97f06 opensm: Add the precreation of multicast groups
+a9b9f09 opensm/osm_sminfo_rcv.c: Handle SMP status
+1190c15 opensm/osm_switch.c: In osm_switch_set_hops, return, error when port_num is invalid
+3d149db Changed sl_path API to include slid and dlid only
+6cfb0eb Optimized and deadlock-free routing algorithm for InfiniBand
+45f93ec opensm: Add additional IBM vendor ID/OUI
+c386eb2 opensm/osm_state_mgr.c: Cosmetic change to log message
+703e596 opensm: Add support for partition enforcement types to accomodate IBA extended link speeds
+a2a03a8 Check block_num validity in set_guidinfo() and del_guidinfo() requests
+e6ec61f end error resoponse to invalid LID in GUIDInfo request
+7fce500 opensm/Makefile.am: Add doc/opensm-sriov.txt to docs
+264aeb1 opensm: Add documentation for SRIOV support
+c639832 opensm: Enhance osm_physp_share_this_pkey for allow_both_pkeys policy
+b17b63c opensm: When allowing both pkeys, on a switch external, (peer) port eliminate limited pkey when full pkey with same base is present
+a758da2 opensm: Add command line option for allow_both_pkeys
+f412de3 opensm: Update partition documentation and man page for (allowing) both (limited and full) memberships in the same partition
+726ce6a Support allowing both full and limited members of same partition
+4ccf32f opensm/PKeyMgr: Support pkey index reuse when there are no longer any previously unused indices available
+eb375a6 opensm/osm_pkey_mgr.c: Detect pkey table overflow in pkey_mgr_update_port
+411e742 opensm/PkeyMgr: Don't change end port pkey index when simultaneously adding and removing partitions
+15e7223 opensm/osm_sa_guidinfo_record.c: In set_guidinfo, better SM reassigned guid handing
+e79b725 opensm: Handle SubnSet GUIDInfo asynchronously from GUIDInfoRecord handling
+96c741d opensm: Some cosmetic formatting changes
+1d5e370 opensm/osm_sa_guidinfo_record.c: Better status for SA response
+efd3ba2 opensm/osm_sa.c: Change log level of message
+db8b7da opensm/osm_sa_service_record.c: Alias GUID support
+5330986 opensm/osm_sa_multipath_record.c: Add support for alias GUIDs
+44168c9 opensm/osm_sa_guidinfo_record.c: In del_guidinfo, validate guid not in use
+63eb65b opensm: Add multicast support for alias GUIDs
+700d15f opensm/osm_sa_path_record.c: Add support for alias GUIDs
+f818387 opensm/osm_sa_guidinfo_record.c: Use OSM_VENDOR_ID_OPENIB define rather than IB_OPENIB_OUI
+97e360e opensm: Dump/load SA GUIDInfoRecords
+fe74f1d opensm: Make SA assigned guids persistent across port down/up events
+eb8f1d9 opensm: Add support for alias GUIDs
+b3b1861 opensm: osm_subnet.c: Updated patch to add error-reporting to the parsing of opensm.conf
+cd8a708 opensm/man/opensm.8.in: Add description for OSM_LOG_SYS logging flag
+1308e5c opensm/osm_console.c: Add display of FDR10 ports to portstatus_parse
+f4722b0 opensm: Reset client reregistration when receiving handover
1.3 Library API Changes
If no permission to forward, the subscription should be removed and
no further forwarding should occur.
-* C14-24.1.1.5 and C14-62.1.1.22 (Initialization):
- GUIDInfo - SM should enable assigning Port GUIDInfo.
-
* C14-44 (Initialization):
If the SM discovers that it is missing an M_Key to update CA/RT/SW,
it should notify the higher level.
4.1 Major Bug Fixes
+a322f51 Skip TID 0 on 32 bit wraparound for SMP, SA and PerfMgt queries
+cded9af Fix transaction id casting
+3585f8b opensm: Fix crash found with ucast cache
+74e12d9 opensm: fix part_enforce parameter parsing crash
+647a98e Fixed crash in sm_state_mgr_send_master_sm_info_req() during fabric
+ merge
+63ebd0d Fix crash in ucast cache when chain of switches connected back at once
+ to the fabric
+3b21d6f Fix crash in ucast cache when ucast cache invalidates after updating
+ one of the switches
5654e22 Fix invalid error check, which lead to segfault
7bf7482 fix segfault corner case w/ updn routing and LMC > 0
18990fa opensm: set IS_SM bit during opensm init
4c822b0 opensm/osm_prtn.c: removing TopSpin hack
8214e2a opensm: Add support for SwitchInfo:MulticastFDBTop
1e544ba opensm: fixed memory leak in multicast spanning tree calculation
+0456b3f Fixed multicast groups reconfiguration during heawy sweep
+9ad844f Fix ucast cache crash, when switch doesn't have valid phys ports
+07aa9fa opensm: fix crash in osm_ucast_mgr
+076bd38 opensm/osm_ucast_cache.c: fix crash in ucast cache when switch with lid 0 dropped
4.2 Other Bug Fixes
+740c22b opensm/include/opensm/osm_log.h: Fix commentary cut 'n paste error
+13ebee4 opensm/osm_subnet.c: Fixed ftree/updn configuration failure when
+ root_guid_file points to non-existing file
+8a9d267 opensm: fix locking in osm_guid_mgr_process
+4d682bb opensm: Fix pthread_create() return value checks
+63c6609 opensm/osm_port.h: Fix commentary typo
+3e4e00b opensm/osmtest: fix osmtest ignores timeout parameter
+68b1d92 opensm: perfmgr fix dump_counters
+9d16039 opensm/perfmgr: fix endian conversion of PortCounters
+324f269 opensm/osm_sa_inform_info.c: Fix some error log messages
+1d5213a opensm/osm_madw.h: Fix a couple of cut 'n paste commentary errors
+3fc662d opensm/torus-2QoS: Fix some typos in documentation
+048c66e Fixed Multicast precreation parsing
+086d611 Fixes in SL2VL table distribution algorithm
+7d9f0c9 Fix deadlock between sminfo_set_req() and osm_sm_state_mgr()
+69741e6 Fix base port0 sl2vl mapping optimization
+bcda38e Fix SL2VL configuration
+7e39542 Fix pre-creation of MC group with MGID containing P_Key
+20e1a46 opensm/osm_congestion_control.c: Fix initialization hex string
+a59072d opensm/osm_congestion_control.c: Skip TID 0 on 32 bit wraparound
+7d18662 osmtest/osmtest.c: Fix permission
+e7d4574 opensm/perfmgr: update new error codes to '54' prefix
e17dae6 opensm/osm_sa_portinfo_record.c: In pir_rcv_new_pir, fix switch port 0 physp
93b2f56 libvendor/osm_vendor_ibumad.c: fix unused-but-set warning
cda58af fixed unsused-but-set warning for DEBUG variables
78e1e4e osmtest/osmt_multicast.c: Fix check of BAD RATE when connecting to existing MGID (o15.0.1.13)
e94e972 osmtest/osmt_multicast.c: Fix first MGID=0 MC group creation case
daedad7 opensm/osmtest/osmt_multicast.c: Fix an unrealistic rate case
+110ae10 opensm: fixed segfault in osm_destroy
+d666205 opensm/osm_prtn.c: Fix typo in log message
+f3ccb45 opensm: Remove duplicate definition of IB_MAD_STATUS_CLASS_MASK
+917070c Move no_fallback_routing_engine from osm_subn_opt_t to osm_opensm_t.
+d71a924 Free memory from osm_subn_opt_t when osm_subn_t destroyed
+1b75fa4 Remove duplicate initialization of scatter_ports
+b5f4570 Do not load configs from the default config file and specified config file
+f24a089 Fix memleak and segfault
+247d0d8 Fix IPoIB broadcast group creation on non-default Pkey
+78d86bd opensm/osm_subnet.c: Trivial optimization to code flow in subn_verify_sl2vl
+6bb41e3 opensm/complib/cl_atomic.h: Commentary changes
+28ee7b9 opensm/osm_sa_class_port_info.c: Conditionalize setting of OSM_CAP2_IS_MCAST_TOP_SUPPORTED
+29e59a2 Removed unused parameter "ib_mad_addr" from umad_reciever()
+63ad0bb opensm/osmeventplugin/src/osmeventplugin.c: Output LIDs in decimal
+17967c2 opensm/osm_switch.h: Fix commentary typo
+6d3e223 opensm/osm_perfmgr.c: Enhance send error log message
+08abcd4 opensm/osm_sa_mad_ctrl.c: Enhance send error log message
+13f3e0f opensm/libvendor/osm_vendor_ibumad.c: Fix DR path printing on send timeouts
+0dfd760 Fix suggest parentheses around operand warning
+ad2dbf8 Support source-target-port-guid QoS policy configuration with ULP 'any'
+080e3ad Support source-port-guid QoS policy configuration with ULP 'any'
+49777a9 Fix typo in qos-ulps parsing comment
+bcfe1b9 opensm/osm_torus.c: Use "OpenSM standard" error codes
+683397d Fix use of GNU old-style field designator extension
+719fcd4 Fix use of logical && with constant operand; switch to bitwise &
+6bc87bd opensm: fix search common pkeys
+a6ac5e3 opensm/osm_drop_mgr.c: GID out trap fix
+7717505 opensm/osm_pkey_mgr.c: fix segfault when trying to access not allocated block
+eb90efd include/opensm/osm_subnet.h: fix comment typos
+0a315e3 opensm/include/iba/ib_types.h: fix comment typos and errors
+125baa0 opensm/opensm.8.in: Fix cut 'n paste error
+f0d14d2 opensm/osm_ucast_ftree.c: Fix some typos
+841096f opensm: Fix opensm handover/relinquish corner case
+bf420ac opensm/osm_helper.c: Fix commentary typo
+2be999c opensm: fixed description in osm_routing_engine
+cf3d185 gen_chlog.sh: fixed version ordering
+8d49c5d opensm/osm_sa_guidinfo_record.c: Fix locking
+8d764b8 Fix continous looping when clearing accum_pkeys table
+32500a6 opensm/osm_link_mgr.c: Fix sending PortInfo Set for ports supporting extended speed
+e24ff39 Fix Pkey enforcement configuration
+041e47a Fix PathRecord reply to be the same for allow_both_pkeys ON and OFF
+3c9b81d OpenSM/osm_prtn_config.c: Fix non-initialized pointer usage
+28a40a2 Fix logging messages about op_vls and mtu mismatch
+5e672a6 Fix memory leak on dfsssp_context_destroy()
* Other less critical or visible bugs were also fixed.
+++ /dev/null
-
- Open Fabrics Enterprise Distribution (OFED)
- SRP in OFED 1.5.2 Release Notes
-
- December 2010
-
-
-==============================================================================
-Table of contents
-==============================================================================
-
- 1. Overview
- 2. Changes and Bug Fixes since OFED 1.5
- 3. Software Dependencies
- 4. Major Features
- 5. Loading SRP Initiator
- 6. Manually Establishing an SRP Connection
- 7. SRP Tools - ibsrpdm and srp_daemon
- 8. Automatic Discovery and Connecting to Targets
- 9. Multiple Connections from Initiator IB Port to the Target
- 10. High Availability
- 11. Shutting Down SRP
- 12. Known Issues
- 13. Vendor Specific Notes
-
-
-==============================================================================
-1. Overview
-==============================================================================
-
-The SRP standard describes the message format and protocol definitions required
-for transferring commands and data between a SCSI initiator port and a SCSI
-target port using RDMA communication service.
-
-
-==============================================================================
-2. Changes and Bug Fixes since OFED 1.5
-==============================================================================
-* Check for scsi_id in scmnd to prevent scan/rescan keep adding new scsi devices
- ie. echo "- - -" > /sys/class/scsi_host/hostXX/scan
-* Bug fixing
-
-==============================================================================
-4. Software Dependencies
-==============================================================================
-
-The SRP Initiator depends on the installation of the OFED Distribution stack
-with OpenSM running.
-
-==============================================================================
-5. Major Features
-==============================================================================
-
-This SRP Initiator is based on source taken from openib.org gen2 implementing
-the SCSI RDMA Protocol-2 (SRP-2), Doc. no. T10/1524-D. See:
-www.t10.org/ftp/t10/drafts/srp2/srp2r00a.pdf
-
-The SRP Initiator supports:
-- Basic SCSI Primary Commands -3 (SPC-3)
- (www.t10.org/ftp/t10/drafts/spc3/spc3r21b.pdf)
-- Basic SCSI Block Commands -2 (SBC-2)
- (www.t10.org/ftp/t10/drafts/sbc2/sbc2r16.pdf)
-- Basic functionality, task management and limited error handling
-
-==============================================================================
-6. Loading SRP Initiator
-==============================================================================
-
-To load the SRP module, either execute the "modprobe ib_srp" command after the
-OFED driver is up, or change the value of SRP_LOAD in
-/etc/infiniband/openib.conf to "yes" (causing the srp module to be loaded
-at driver boot).
-
-NOTE: When loading the ib_srp module, it is possible to set the module
- parameter srp_sg_tablesize. This is the maximum number of
- gather/scatter entries per I/O (default: 12).
-
- a. modprobe ib_srp srp_sg_tablesize=32
- or
- b. edit /etc/modprobe.conf and add the following line:
- options ib_srp srp_sg_tablesize=32
-
-Module paramters:
-For the list of ib_srp module parameters
- $ modinfo ib_srp
-
- + srp_sg_tablesze: Max number of scatter/gather entries per I/O
- + srp_dev_loss_tmo: Number of seconds that srp driver will not return
- DID_NO_CONNECT status when it loss connection to target.
- During this period, it will try to re-establish
- the connection to target, and return DID_RESET,
- DID_ABORT statuses for outstanding scsi command to
- prevent DM Multipath driver to failover to next paths.
- Default value is 60 seconds.
-
-==============================================================================
-7. Manually Establishing an SRP Connection
-==============================================================================
-
-The following steps describe how to manually load an SRP connection between
-the Initiator and an SRP Target. Section 8 explains how to do this
-automatically.
-
-- Make sure that the ib_srp module is loaded, the SRP Initiator is reachable
- by the SRP Target, and that an SM is running.
-
-- To establish a connection with an SRP Target and create SRP (SCSI) device(s)
- for that target under /dev, use the following command:
-
- echo -n id_ext=[GUID value],ioc_guid=[GUID value],dgid=[port GID value],\
- pkey=ffff,service_id=[service[0] value] > \
- /sys/class/infiniband_srp/srp-mthca[hca number]-[port number]/add_target
-
- a. Execution of the above "echo" command may take some time
- b. The SM must be running while the command executes
- c. It is possible to include additional parameters in the echo command:
- > max_cmd_per_lun - Default: 63
- > max_sect (short for max_sectors) - sets the request size of a command
- > io_class - Default: 0x100 as in rev 16A of the specification
- Note: In rev 10 the default was 0xff00
- > initiator_ext - Please refer to Section 9 (Multiple Connections...)
- d. See SRP Tools below for instructions on how the parameters in the
- echo command above may be obtained.
-
-NOTES:
-
-- Using the same *echo -n <same paramters>* more than one, the srp target
- will terminate the previous connection and re-establish the new
- connection. To have more than two connections to srp target, please use
- different inititiator_ext values in echo command.
-
-- To list the new SCSI devices that have been added by the echo command, you
- may use either of the following two methods:
- a. Execute "fdisk -l". This command lists all devices; the new devices are
- included in this listing.
- b. Execute *dmesg* or look at /var/log/messages to find messages with the
- names of the new devices.
-
-
-==============================================================================
-8. SRP Tools - ibsrpdm and srp_daemon
-==============================================================================
-
-To assist in performing the steps in Section 6, the OFED 1.3.1 distribution
-provides two utilities which:
-- Detect targets on the fabric reachable by the Initiator (for step 1)
-- Output target attributes in a format suitable for use in the above
- "echo" command (step 2)
-
-These utilities are: ibsrpdm and srp_daemon.
-
-The utilities can be found under /usr/local/ofed/sbin/ (or <prefix>/sbin/),
-and are part of the srptools RPM that may be installed using the
-OFED custom installation. Detailed information regarding the various
-options for these utilities are provided by their man pages.
-
-Below, several usage scenarios for these utilities are presented.
-
-ibsrpdm usage
--------------
-1. Detecting reachable targets
-
- a. To detect all targets reachable by the SRP initiator via the default
- umad device (/dev/infiniband/umad0), execute the following command:
- $ ibsrpdm
-
- This command will output information on each SRP target detected, in
- human-readable form.
-
- Sample output:
- IO Unit Info:
- port LID: 0103
- port GID: fe800000000000000002c90200402bd5
- change ID: 0002
- max controllers: 0x10
-
- controller[ 1]
- GUID: 0002c90200402bd4
- vendor ID: 0002c9
- device ID: 005a44
- IO class : 0100
- ID: LSI Storage Systems SRP Driver 200400a0b81146a1
- service entries: 1
- service[ 0]: 200400a0b81146a1 / SRP.T10:200400A0B81146A1
-
- b. To detect all the SRP Targets reachable by the SRP Initiator via
- another umad device, use the following command:
-
- $ ibsrpdm -d <umad device>
-
-2. Assistance in creating an SRP connection
-
- a. To generate output suitable for utilization in the "echo" command of
- section 5, add the "-c" option to ibsrpdm:
-
- $ ibsrpdm -c
-
- Sample output:
- id_ext=200400A0B81146A1,ioc_guid=0002c90200402bd4,
- dgid=fe800000000000000002c90200402bd5,pkey=ffff,
- service_id=200400a0b81146a1
-
- b. To establish a connection with an SRP Target (Section 6) using the output
- from the "libsrpdm -c" example above, execute the following command:
-
- $ echo -n id_ext=200400A0B81146A1,ioc_guid=0002c90200402bd4,
- dgid=fe800000000000000002c90200402bd5,pkey=ffff,
- service_id=200400a0b81146a1
- > /sys/class/infiniband_srp/srp-mlnx_0-1/add_target
-
- The SRP connection should now be up; the newly created SCSI devices should
- appear in the listing obtained from the "fdisk -l" command.
-
-
-srp_daemon
-----------
-The srp_daemon utility is based on ibsrpdm and extends its functionality.
-In addition to the ibsrpdm functionality described above, srp_daemon can also
-- Establish an SRP connection by itself (without the need to issue the "echo"
- command described in Section 6)
-- Continue running in background, detecting new targets and establishing SRP
- connections with them (daemon mode)
-- Discover reachable SRP targets given an infiniband HCA name and port, rather
- than just by /dev/umad<N> where <N> is a digit
-- Enable High Availability operation (together with Device-Mapper Multipath)
-- Have a configuration file that determines the targets to connect to
-
-a. srp_daemon commands equivalent to ibsrpdm:
-
- "srp_daemon -a -o" is equivalent to "ibsrpdm"
- "srp_daemon -c -a -o" is equivalent to "ibsrpdm -c"
-
-Note: These srp_daemon commands can behave differently than the equivalent
- ibsrpdm command when /etc/srp_daemon.conf is not empty.
-
-b. srp_daemon extensions to ibsrpdm
-
- - To discover SRP Targets reachable from HCA device <infiniband HCA name>,
- port <port num>, (and generate output suitable for 'echo') you may execute
-
- srp_daemon -c -a -o -i <infiniband HCA name> -p <port number>
-
- - To both discover the SRP Targets and establish connections with them, just
- add the -e option to the above command.
-
- - Executing srp_daemon over a port without the -a option will only display
- the reachable targets via the port and to which the initiator is not
- connected. If executing with the -e option it is better to omit -a.
-
- - It is recommended to use the -n option. This option adds the initiator_ext
- to the connecting string. (See Section 9 for more details).
-
- - srp_daemon has a configuration file that can be set, where the default is
- /etc/srp_daemon.conf. Use the -f to supply a different configuration file
- that configures the targets srp_daemon is allowed to connect to. The
- configuration file can also be used to set values for additional
- parameters (e.g., max_cmd_per_lun, max_sect).
-
- - A continuous background (daemon) operation, providing an automatic ongoing
- detection and connection capability. See Section 8.
-
-==============================================================================
-9. Automatic Discovery and Connecting to Targets
-==============================================================================
-
-- Make sure that the ib_srp module is loaded, the SRP Initiator can reach an
- SRP Target, and that an SM is running.
-
-- To connect to all the existing Targets in the fabric, execute
- srp_daemon -e -o. This utility will scan the fabric once, connect to
- every Target it detects, and then exit.
-
-NOTE: srp_daemon will follow the configuration it finds in
- /etc/srp_daemon.conf. Thus, it will ignore a target that is disallowed in
- the configuration file.
-
-- To connect to all the existing Targets in the fabric and to connect
- to new targets that will join the fabric, execute srp_daemon -e. This utility
- continues to execute until it is either killed by the user or encounters
- connection errors (such as no SM in the fabric).
-
-- To execute SRP daemon as a daemon you may execute run_srp_daemon
- (found under /usr/local/ofed/sbin/ or <prefix>/sbin/), providing it with
- the same options used for running srp_daemon.
-
- Note: Make sure only one instance of run_srp_daemon runs per port.
-
-- To execute SRP daemon as a daemon on all the ports, execute srp_daemon.sh
- (found under /usr/local/ofed/sbin/ or <prefix>/sbin/).
- srp_daemon.sh sends its log to /var/log/srp_daemon.log.
-
-- It is possible to configure this script to execute automatically when the
- InfiniBand driver starts by changing the value of SRP_DAEMON_ENABLE in
- /etc/infiniband/openib.conf to "yes" and SRP_LOAD to yes as well.
-
- Another option to to configure this script to execute automatically when the
- InfiniBand driver starts is by changing the value of SRPHA_ENABLE in
- /etc/infiniband/openib.conf to "yes". However, this option also enables
- SRP High Availability that has some more features. (Please read the High
- Availability section).
-
-==============================================================================
-10. Multiple Connections from Initiator IB Port to the Target
-==============================================================================
-
-Some system configurations may need multiple SRP connections from
-the SRP Initiator to the same SRP Target: to the same Target IB port,
-or to different IB ports on the same Target HCA.
-
-In case of a single Target IB port, i.e., SRP connections use the same path,
-the configuration is enabled using a different initiator_ext value for each
-SRP connection. The initiator_ext value is a 16-hexadecimal-digit value
-specified in the connection command.
-
-Also in case of two physical connections (i.e., network paths) from a single
-initiator IB port to two different IB ports on the same Target HCA, there is
-need for a different initiator_ext value on each path. The conventions is to
-use the Target port GUID as the initiator_ext value for the relevant path.
-
-If you use srp_daemon with -n flag, it automatically assigns initiator_ext
-values according to this convention. For example:
-
- id_ext=200500A0B81146A1,ioc_guid=0002c90200402bec,
- dgid=fe800000000000000002c90200402bed,
- pkey=ffff,service_id=200500a0b81146a1,initiator_ext=ed2b400002c90200
-
- Notes:
- a. It is recommended to use the -n flag for all srp_daemon invocations.
- b. ibsrpdm does not have a corresponding option.
- c. srp_daemon.sh always uses the -n option (whether invoked manually by
- the user, or automatically at startup by setting SRPHA_ENABLE or
- SRP_DAEMON_ENABLE to yes).
-
-==============================================================================
-11. High Availability (HA)
-==============================================================================
-
-High Availability Overview
---------------------------
-
-High Availability works using the Device-Mapper (DM) multipath and the
-SRP daemon.
-
-Each initiator is connected to the same target from several ports/HCAs.
-The DM multipath is responsible for joining together different paths to the
-same target and for fail-over between paths when one of them goes offline.
-Multipath will be execute on newly joined SCSI devices.
-
-Each initiator should execute several instances of the SRP daemon, one for each
-port. At startup, each SRP daemon detects the SRP targets in the fabric and
-sends requests to the ib_srp module to connect to each of them. These
-SRP daemons also detect targets that subsequently join the fabric, and send the
-ib_srp module requests to connect to them as well.
-
-High Availability Operation
----------------------------
-
-When a path (from port1) to a target fails, the ib_srp module starts an error
-recovery process. If this process gets to the reset_host stage and there is no
-path to the target from this port, ib_srp will remove this scsi_host. After
-the scsi_host is removed, multipath switches to another path to this target
-(from another port/HCA).
-
-When the failed path recovers, it will be detected by the SRP daemon. The SRP
-daemon will then request ib_srp to connect to this target. Once the connection
-is up, there will be a new scsi_host for this target. Multipath will be
-executed on the devices of this host, returning to the original state (prior to
-the failed path).
-
-High Availability Prerequisites
--------------------------------
-
-Installation for RHEL4 and RHEL5: (Execute once)
- - Verify that the standard device-mapper-multipath rpm is installed. If not,
- install it from the RHEL distribution.
-
-Installation for SLES10: (Execute once)
- - Verify that multipath is installed. If not, install it from the
- installation (You can use yast).
-
- - Update udev: (Execute once - for manual activation of High Availability only)
-
- - Add a file to /etc/udev/rules.d/ (you can call it 91-srp.rules)
- This file should have one line:
- ACTION=="add", KERNEL=="sd*[!0-9]", RUN+="/sbin/multipath %M:%m"
-
- Note: When SRPHA_ENABLE is set to "yes" (see Automatic Activation of High
- Availability below), this file is created upon each boot of the driver and
- is deleted when the driver is unloaded.
-
-
-Manual Activation of High Availability
---------------------------------------
-
-Initialization: (Execute after each boot of the driver)
- 1) Execute modprobe dm-multipath
- 2) Execute modprobe ib-srp
- 3) Make sure you have created file /etc/udev/rules.d/91-srp.rules
- as described above
- 4) Execute for each port and each HCA:
- srp_daemon -c -e -R 300 -i <InfiniBand HCA name> -p <port number>
- (You can use another value for -R. See under the Known Issues section
- the workaround for the rare race condition.)
-
- This step can be performed by executing srp_daemon.sh, which sends
- its log to /var/log/srp_daemon.log.
-
- Now it is possible to access the SRP LUNs on /dev/mapper/.
-
- NOTE: It is possible for regular (non-SRP) LUNs to also be present;
- the SRP LUNs may be identified by their names. You can configure the
- /etc/multipath.conf file to change multipath behavior.
-
-
-Automatic Activation of High Availability
------------------------------------------
-- Set the value of SRPHA_ENABLE in /etc/infiniband/openib.conf to "yes".
- Also make sure SRP_LOAD=yes and SRP_DAEMON_ENABLE=yes.
-
-- From the next loading of the driver it will be possible to access the SRP
- LUNs on /dev/mapper/
- NOTE: It is possible that regular (not SRP) LUNs may also be present;
- the SRP LUNs may be identified by their name.
-
-- It is possible to see the output of the SRP daemon in /var/log/srp_daemon.log
-
-
-==============================================================================
-12. Shutting Down SRP
-==============================================================================
-
-SRP can be shutdown by using "modprobe -r ib_srp", or by stopping the OFED
-("/etc/init.d/openibd stop"), or as a by-product of a complete system shutdown.
-
-Prior to shutting down SRP, it is REQUIRED to remove all references to it.
-The actions you need to take depend on the way SRP was loaded. There are
-three cases.
-
-a. Without High Availability
-------------------------------------
-When working without High Availability, you should unmount all SRP
-partitions that were mounted prior to shutting down SRP.
-For example, /dev/sdd1 is srp partition and mount to /mnt/test
-$ umount /mnt/test
-$ modprobe -r ib_srp
-
-NOTES: the umount may get stuck ~90 seconds per connection to target if the
- target is down. This is due to the srp_dev_loss_tmo=60 seconds which
- srp driver waits for the target coming back before returning error
- status.
- If you have shutdown/remove srp target and the host have 4 connections
- to the SRP target, you should wait ~4-5 minutes for the unmount to exit.
- Do not ctrl+c to kill umount process.
-
-b. After Manual Activation of High Availability
------------------------------------------------
-If you manually activated SRP High Availability, perform the following steps:
-- Unmount all SRP partitions that were mounted
-- Kill all SRP daemon instances.
-- Make sure there are no multipath instances running. If there are multiple
- instances, wait for them to end or kill them.
-- Execute multipath -F
-
-Example:
-$ umount /mnt/test1 /mnt/test2 (wait for it to exit, do not ctrl+c)
-$ ps -ax and kill all srp_daemon processes.
-$ multipath -ll (wait for it to exit, do not ctrl+c)
-$ multipath -F
-$ modprobe -r ib_srp
-
-c. After Automatic Activation of High Availability
---------------------------------------------------
-If SRP High Availability was automatically activated, SRP shutdown must be
-part of the driver shutdown ("/etc/init.d/openibd stop") which performs
-steps 2-5 of case (b) above. However, you still have to unmount all SRP
-partitions that were mounted before driver shutdown.
-
-
-HAL Issue
----------
-The HAL (Hardware Abstraction Layer) system includes a daemon that examines
-all devices in the system. In this process, it frequently holds a reference
-to the ib_srp module. If you attempt to shutdown SRP while this daemon is
-holding a reference to ib_srp, the shutdown will fail. Therefore, you
-should make sure this will not occur. One solution may be to stop "haldaemon"
-(/etc/init.d/haldaemon stop) prior to SRP shutdown.
-
-
-==============================================================================
-13. Known Issues
-==============================================================================
-
-- There is a very rare race condition which can cause the SRP daemon to miss a
- target that joins the fabric. The race can occur if a target joins and leaves
- the fabric several times in a short time (e.g., if the cable is not connected
- well). In such a case, the SM may ignore this quick change of state and may
- not send an InformInfo to the srp_daemon.
-
- Workaround: Execute the srp_daemon command with the -R <sec> option. This
- option causes the SRP daemon to perform a full rescan of the fabric every
- <sec> seconds.
-
-- The srp_daemon does not support different pkeys other than the default
- pkey=ffff
-
-- It is recommended to use an SM that supports the enhanced capability mask
- matching feature (errata MGTWG8372). With SMs which support this feature, the
- SRP daemon generates significantly less communication traffic.
-
-- When booting OFED with SRP High Availability enabled, executing multipath for
- all LUNs on all connections may take some time (several minutes). However, it
- is possible to start working while this process is in progress.
-
-- Stopping the driver while SRP High Availability is enabled kills all
- multipath processes. Consider appropriate actions in case multipath is used
- for other purposes.
-
-- AS High Availability is based on Device Mapper multipath, it embodies
- multipath limitations and also its configuration and tuning options.
- See http://christophe.varoqui.free.fr/wiki/wakka.php?wiki=Home
- for information on multipath.
- To modify and tune multipath configuration, edit the file /etc/multipath.conf
- according to instructions and tips listed in
- /usr/share/doc/packages/multipath-tools/multipath.conf.*
-
-- In case your topology has two physical connections (i.e., network paths) from
- a single initiator IB port to two different IB ports on the same Target HCA,
- and you wish to have an SRP connection on the one path coexist with an SRP
- connection on the second path, you must set a different initiator_ext value
- on each path. See Section 9, "Multiple Connections from Initiator IB Port
- to the Target" for details.
-
-- The srp_daemon tool reads by default the configuration file
- /etc/srp_daemon.conf. In case this configuration file disallows connecting
- to a certain target, srp_daemon will ignore the target. If you find out
- that srp_daemon ignores a target, please check the /etc/srp_daemon.conf file.
-
-- Rebooting the system with unclean mounted filesystem and dead connection
- to SRP target, the system may get stuck.
-
-- After establish the connection with srp target and rebooting the system,
- initiator will fail to connect to target @ first manual *echo -n* command
- (target reject with stale connection). You need to do *echo -n* one more
- time.
- You do not see this problem with srp_daemon mode since srp_daemon will
- retry to connect.
-
-- The combination of "weak" single lun srp target, I/O with big block size,
- default max_command_per_lun=63 while using /dev/urandom to create file with
- ext3 fs on srp lun, may cause ext3 remount with "read-only" flag
- ie.
- Example:
- sdb1 is first partition of srp lun sdb, ext3 fs is created
- $ mount /dev/sdb1 /mnt/sdb1; cd /mnt/sdb1
- $ dd if=/dev/urandom of=10G-file bs=1G count=10
- --> ext3 fs may remount with read-only flag
-
- Workarounds:
- ------------
- a. Log into the target with small max_command_per_lun (3,4,8)
- $ echo id_ext=0002c9030008fc0c,ioc_guid=0002c9030008fc0c,
- dgid=fe800000000000000002c90300084417,max_cmd_per_lun=4,pkey=ffff,
- service_id=0002c9030008fc0c > /sys/class/infiniband_srp/srp-mlx4_0-1/add_target
-
- ----------------OR-------------------
-
- b. Run dd with /dev/zero instead of /dev/urandom
- $ dd if=/dev/zero of=10G-file bs=1G count=10
-
- ----------------OR-------------------
-
- c. Run dd with smaller block size
- $ dd if=/dev/urandom of=10G-file bs=128K count=40000
-
- ----------------OR-------------------
-
- d. Combine the a,b,c steps (This is the recommended workaround)
-
-==============================================================================
-14. Vendor Specific Notes
-==============================================================================
-
-Hosts connected to Qlogic SRP Targets must perform one of the following
-steps after upgrading to OFED 1.3.1 to continue accessing their storage
-successfully:
-
-1. When issuing the "echo" command to add a new SRP Target, the host
- must append the string ",initiator_ext=0000000000000001" to the original
- echo string.
- Example:
- 'ibsrpdm -c' output is as follows:
-
- id_ext=0000000000000001,ioc_guid=00066a0138000165,dgid=fe8000000000000
- 000066a0260000165,pkey=ffff,service_id=0000494353535250,io_class=ff00
-
- id_ext=0000000000000001,ioc_guid=00066a0238000165,dgid=fe8000000000000
- 000066a0260000165,pkey=ffff,service_id=0000494353535250,io_class=ff00
-
- To connect to the first target, the echo command must be:
-
- echo -n \
- id_ext=0000000000000001,ioc_guid=00066a0138000165,\
- dgid=fe8000000000000000066a0260000165,pkey=ffff,\
- service_id=0000494353535250,io_class=ff00,\
- initiator_ext=0000000000000001 > \
- /sys/class/inifiniband_srp/srp-mthca0-1/add_target
-
-
-2. Change the SRP map on the Qlogic SRP Target to set the expected initiator
- extension to 0. For details on how to change the SRP map on a Qlogic SRP
- Target, please refer to product documentation.
-
-
+++ /dev/null
-diff -Naur scst/scst.h scst.wk/scst.h
---- scst/scst.h 2009-08-14 17:09:47.000000000 -0700
-+++ scst.wk/scst.h 2009-08-14 17:01:28.000000000 -0700
-@@ -2581,7 +2581,7 @@
- void scst_aen_done(struct scst_aen *aen);
-
- #if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 24)
--
-+#ifndef RHEL_RELEASE_CODE
- static inline struct page *sg_page(struct scatterlist *sg)
- {
- return sg->page;
-@@ -2610,6 +2610,7 @@
- sg->length = len;
- }
-
-+#endif
- #endif /* LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 24) */
-
- static inline void sg_clear(struct scatterlist *sg)
+++ /dev/null
-diff -Naur scst-1.0.1.1-orig/src/scst_lib.c scst-1.0.1.1/src/scst_lib.c
---- scst-1.0.1.1-orig/src/scst_lib.c 2010-03-05 18:17:22.000000000 -0800
-+++ scst-1.0.1.1/src/scst_lib.c 2010-03-05 18:22:50.000000000 -0800
-@@ -1680,7 +1680,7 @@
- return res;
- }
-
--#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 18)
-+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 16)
- static void scst_req_done(struct scsi_cmnd *scsi_cmd)
- {
- struct scsi_request *req;
-@@ -1743,7 +1743,7 @@
- TRACE_EXIT();
- return;
- }
--#else /* LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 18) */
-+#else /* LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 16) */
- static void scst_send_release(struct scst_device *dev)
- {
- struct scsi_device *scsi_dev;
-@@ -1790,7 +1790,7 @@
- TRACE_EXIT();
- return;
- }
--#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 18) */
-+#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 16) */
-
- /* scst_mutex supposed to be held */
- static void scst_clear_reservation(struct scst_tgt_dev *tgt_dev)
-@@ -2038,7 +2038,7 @@
- sBUG_ON(cmd->inc_blocking || cmd->needs_unblocking ||
- cmd->dec_on_dev_needed);
-
--#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 18)
-+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 16)
- #if defined(CONFIG_SCST_EXTRACHECKS)
- if (cmd->scsi_req) {
- PRINT_ERROR("%s: %s", __func__, "Cmd with unfreed "
-@@ -2212,7 +2212,7 @@
- return;
- }
-
--#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 18)
-+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 16)
- int scst_alloc_request(struct scst_cmd *cmd)
- {
- int res = 0;
-diff -Naur scst-1.0.1.1-orig/src/scst_main.c scst-1.0.1.1/src/scst_main.c
---- scst-1.0.1.1-orig/src/scst_main.c 2010-03-05 18:17:22.000000000 -0800
-+++ scst-1.0.1.1/src/scst_main.c 2010-03-05 18:34:48.000000000 -0800
-@@ -1676,7 +1676,7 @@
-
- TRACE_ENTRY();
-
--#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 18)
-+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 16)
- {
- struct scsi_request *req;
- BUILD_BUG_ON(SCST_SENSE_BUFFERSIZE !=
-diff -Naur scst-1.0.1.1-orig/src/scst_priv.h scst-1.0.1.1/src/scst_priv.h
---- scst-1.0.1.1-orig/src/scst_priv.h 2010-03-05 18:17:22.000000000 -0800
-+++ scst-1.0.1.1/src/scst_priv.h 2010-03-05 18:35:07.000000000 -0800
-@@ -27,7 +27,7 @@
- #include <scsi/scsi_device.h>
- #include <scsi/scsi_host.h>
-
--#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 18)
-+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 16)
- #include <scsi/scsi_request.h>
- #endif
-
-@@ -322,7 +322,7 @@
- void scst_check_retries(struct scst_tgt *tgt);
- void scst_tgt_retry_timer_fn(unsigned long arg);
-
--#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 18)
-+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 16)
- int scst_alloc_request(struct scst_cmd *cmd);
- void scst_release_request(struct scst_cmd *cmd);
-
-diff -Naur scst-1.0.1.1-orig/src/scst_targ.c scst-1.0.1.1/src/scst_targ.c
---- scst-1.0.1.1-orig/src/scst_targ.c 2010-03-05 18:17:22.000000000 -0800
-+++ scst-1.0.1.1/src/scst_targ.c 2010-03-05 18:36:03.000000000 -0800
-@@ -1273,7 +1273,7 @@
- return context;
- }
-
--#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 18)
-+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 16)
- static inline struct scst_cmd *scst_get_cmd(struct scsi_cmnd *scsi_cmd,
- struct scsi_request **req)
- {
-@@ -1325,7 +1325,7 @@
- TRACE_EXIT();
- return;
- }
--#else /* LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 18) */
-+#else /* LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 16) */
- static void scst_cmd_done(void *data, char *sense, int result, int resid)
- {
- struct scst_cmd *cmd;
-@@ -1347,7 +1347,7 @@
- TRACE_EXIT();
- return;
- }
--#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 18) */
-+#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 16) */
-
- static void scst_cmd_done_local(struct scst_cmd *cmd, int next_state,
- enum scst_exec_context pref_context)
-@@ -1881,7 +1881,7 @@
- static int scst_do_real_exec(struct scst_cmd *cmd)
- {
- int res = SCST_EXEC_NOT_COMPLETED;
--#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 18)
-+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 16)
- int rc;
- #endif
- bool atomic = scst_cmd_atomic(cmd);
-@@ -1948,7 +1948,7 @@
- }
- #endif
-
--#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 18)
-+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 16)
- if (unlikely(scst_alloc_request(cmd) != 0)) {
- if (atomic) {
- res = SCST_EXEC_NEED_THREAD;
-@@ -1999,7 +1999,7 @@
- scst_set_cmd_error(cmd, SCST_LOAD_SENSE(scst_sense_hardw_error));
- goto out_done;
-
--#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 18)
-+#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 16)
- out_busy:
- scst_set_busy(cmd);
- /* go through */
-
projects / compat-rdma / docs.git / commitdiff