Open Fabrics Enterprise Distribution (OFED)
- Version 4.8-2 Release Notes
- March 2018
+ Version 4.17 RC1 Release Notes
+ September 2018
===============================================================================
Table of Contents
===============================================================================
1. Overview, which includes:
- - OFED Distribution Rev 4.8-1 Contents
+ - OFED Distribution Rev 4.17 Contents
- Supported Platforms and Operating Systems
- Supported HCA and RNIC Adapter Cards and Firmware Versions
- Tested Switch Platforms
1. Overview
===============================================================================
These are the release notes of OpenFabrics Enterprise Distribution (OFED)
-release 4.8-2. The OFED software package is composed of several software modules,
+release 4.17. The OFED software package is composed of several software modules,
and is intended for use on a computer cluster constructed as an InfiniBand
Fabric, an iWARP Network or a RoCE Fabric.
with a mismatch message, or running with inbox drivers instead of the drivers
that are part of OFED.
-1.1 OFED 4.8-2 Contents
+1.1 OFED 4.17 Contents
-----------------------
The OFED package contains the following components:
- OpenFabrics core and ULPs:
- IB HCA drivers (mthca, mlx4, mlx5, qib, ehca)
- - iWARP RNIC driver (cxgb3, cxgb4, nes, i40iw, qedr)
- - RoCE drivers (mlx4, mlx5, ocrdma, qedr, bnxt_re)
+ - iWARP RNIC driver (cxgb3, cxgb4, i40iw, qedr)
+ - RoCE drivers (mlx4, mlx5, qedr, bnxt_re)
- IB core
- Upper Layer Protocols:
- - IPoIB, SRP Initiator, iSER, uDAPL, qlgc_vnic
- - NVMe-oF (kernel 4.8 only)
- - NFS-RDMA (SLES12.2 RH7.3 only).
+ - IPoIB, SRP Initiator, iSER, uDAPL
+ - NVMe-oF
- OpenFabrics utilities:
- OpenSM (OSM): InfiniBand Subnet Manager
- Diagnostic tools
- Performance tests
- Extra packages:
- - infinipath-psm: Performance-Scaled Messaging API, an accelerated
- interface to Intel(R) HCAs
- libfabric - library that exports interfaces for fabric services to applications
- - Sources of all software modules (under conditions mentioned in the modules'
- LICENSE files)
+ - Sources of all software modules (under conditions mentioned in the modules LICENSE files)
- Documentation
- ppc64
o Linux Operating Systems:
- - RedHat EL7.0 3.10.0-123.el7
- - RedHat EL7.1 3.10.0-210.el7
- - RedHat EL7.2 3.10.0-327.el7
- - RedHat EL7.3 3.10.0-514.el7
- RedHat EL7.4 3.10.0-693.el7
- - SLES12 3.12.28-4
- - SLES12.1 3.12.49-11.1
- - SLES12.2 4.4.21-69-default
+ - RedHat EL7.5 3.10.0-862.el7
- SLES12.3 4.4.73-5-default
- - kernel.org 4.8 *
+ - SLES15 4.12.14-23.1-default
+ - kernel.org 4.17 *
* Minimal QA for these versions.
1.3 HCAs and RNICs Supported
----------------------------
-This release supports IB HCAs by IBM, Intel and Mellanox Technologies, iWARP RNICs by Chelsio Communications and Intel and RoCE adapters by Emulex, IBM, Mellanox, Cavium, and Broadcom.
+This release supports IB HCAs by IBM, Intel and Mellanox Technologies, iWARP RNICs by Chelsio Communications and Intel and RoCE adapters by IBM, Mellanox, Cavium, and Broadcom.
InfiniBand Adapters
o IBM HCAs:
o Mellanox Technologies HCAs (EDR, FDR, FDR10 Modes are Supported):
- ConnectX-4 (Rev 12.12.x and above)
+ - ConnectX-5 (Rev xx.xx.x and above)
+
+ o Mellanox Technologies HCAs (HDR, EDR, FDR, FDR10 Modes are Supported):
+ - ConnectX-6 (Rev xx.xx.x and above)
For official firmware versions please see:
http://www.mellanox.com/content/pages.php?pg=firmware_download
- T6: T62100-CR, T62100-LP-CR, T6225-CR
o Intel RNICs:
- - NE020 10Gb iWARP Adapter
- - i40iw 10Gb iWARP Adapter (kernel 4.8 only)
+ - i40iw 10Gb iWARP Adapter (kernel 4.17 only)
-RoCE Adapters
- o Emulex - using ocrdma
- - Emulex OCe14102 2-port 10 GbE RoCE
- - Emulex OCe14401 1-port 40 GbE RoCE
+ o Cavium
+ - QL4xxxx series of converged network adapters (iWARP)
+RoCE Adapters
o IBM
- IBM Flex System EN4132 2-port 10 GbE RoCE
- IBM EL27 PCIe LP 2-Port 10GbE RoCE SFP+ adapter
o Mellanox
- ConnectX-2 EN (Rev 2.9.1200 and above)
- ConnectX-3 EN (Rev 2.31.5050 and above)
+ - ConnectX-4 EN (Rev x.xx.xxxx and above)
+ - ConnectX-5 EN (Rev x.xx.xxxx and above)
+ - ConnectX-6 EN (Rev x.xx.xxxx and above)
o Cavium
- QL4xxxx series of converged network adapters (RoCEv2)
Other Drivers
- VMware Paravirtual RDMA Driver (vmw_pvrdma)
- Software RDMA over Ethernet (RoCEv2)
- - NVMe-oF Host and Target (kernel 4.8 only)
+ - NVMe-oF Host and Target
1.4 Switches Supported
----------------------
-This release was tested with switches and gateways provided by the following
-companies:
+This release was tested with switches and gateways as follow:
+
InfiniBand Switches
o Flextronics
- F-X430044
1.5 Third Party Packages
------------------------
-The following third party packages have been tested with OFED 4.8-1:
-- Open MPI - 1.8
-- Intel MPI Library 2017
-- MVAPICH2
+The following third party packages have been tested with OFED 4.17:
+- Open MPI - 3.x
+- Intel MPI Library 2018
1.6 OFED Sources
----------------
Linux:
------
-URL: git://git.openfabrics.org/compat-rdma/linux-4.8.git
+URL: git://git.openfabrics.org/compat-rdma/linux-4.17.git
Branch: master
- Linux kernel sub-tree that includes files relevant for the OFED project only.
- Based on v4.8. Used to shorten git clone time.
+ Based on v4.17. Used to shorten git clone time.
Note: the regular Linux git tree can be used as well.
compat:
===============================================================================
2. Change log
===============================================================================
+
+-------------------------------------------------------------------------------
+OFED-4.17 Main Changes from OFED-4.8-2
+-------------------------------------------------------------------------------
+1. Add support for RHEL7.5 and SLES15
+2. Remove support for RHEL7.0,7.1,7.2,7.3, and SLES12.0,12.1,12.2
+
+3. compat-rdma
+
+4. Updated packages
+
+5. ofed_scripts
+
+-------------------------------------------------------------------------------
OFED-4.8-2 Main Changes from OFED-4.8-1
-------------------------------------------------------------------------------
1. compat-rdma
Open Fabrics Enterprise Distribution (OFED)
- Tips for Working with OFED 4.8
- May 2014
+ Tips for Working with OFED 4.17
+ May 2018
===============================================================================
Table of Contents
1. OFED Utilities
2. Debug HOWTOs
3. Pinning (Locking) User Memory Pages
-4. External Module Compilation Over OFED-4.8
+4. External Module Compilation Over OFED-4.17
5. Adding vendor specific actions to the installation of OFED
6. How to compile OFED sources manually
-------------------------------------------
The text file BUILD_ID provides data on all OFED components (whether installed
or not). This file is a part of the ofed-docs RPM and installed under
-/usr/share/doc/ofed-docs-4.8 on RedHat, and under
-/usr/share/doc/packages/ofed-docs-4.8 on SuSE.
+/usr/share/doc/ofed-docs-4.17 on RedHat, and under
+/usr/share/doc/packages/ofed-docs-4.17 on SuSE.
The same information can be obtained by executing the 'ofed_info' command. For
example:
# ofed_info -s
-OFED-4.8:
+OFED-4.17:
# ofed_info
-OFED-4.8:
+OFED-4.17:
compat-rdma:
git://git.openfabrics.org/compat-rdma/compat-rdma.git master
compat-rdma:
linux:
-git://git.openfabrics.org/compat-rdma/linux-4.8.git master
+git://git.openfabrics.org/compat-rdma/linux-4.17.git master
commit 4b9e97b1c859c8fe54a596b554215f39d22bc761
compat-rdma:
===============================================================================
-4. External Module Compilation Over OFED-4.8
+4. External Module Compilation Over OFED-4.17
===============================================================================
To build kernel modules depending on OFED's modules, take the Modules.symvers
MODULES_DIR=/lib/modules/`uname -r`/updates ./create_Module.symvers.sh
See "Module versioning & Module.symvers" in the modules.txt from kernel
-documentation (e.g. linux-4.8/Documentation/kbuild/modules.txt).
+documentation (e.g. linux-4.17/Documentation/kbuild/modules.txt).
Example of the Makefile:
6.1 Compiling the kernel modules
--------------------------------
-1. tar xzf OFED-4.8.tgz
-2. rpm -ihv OFED-4.8/SRPMS/compat-rdma-4.8-OFED.4.8.src.rpm
+1. tar xzf OFED-4.17.tgz
+2. rpm -ihv OFED-4.17/SRPMS/compat-rdma-4.17-OFED.4.17.src.rpm
3. cd /usr/src/redhat/SOURCES
-4. tar xzvf compat-rdma-4.8.tgz
-5. cd compat-rdma-4.8
+4. tar xzvf compat-rdma-4.17.tgz
+5. cd compat-rdma-4.17
6. configure:
run ./configure --help for a list of options.
basic invocation is:
6.2 Compiling the user space libraries
--------------------------------------
-To install user space library from the source RPM provided by OFED-4.8 manually,
+To install user space library from the source RPM provided by OFED-4.17 manually,
do the following:
Example for libibverbs:
-1. tar xzf OFED-4.8.tgz
+1. tar xzf OFED-4.17.tgz
2. rpm -ihv SRPMS/libibverbs-1.1.7-1.src.rpm
3. cd ~<user>/rpmbuild/SOURCES (for RedHat)
or
Open Fabrics Enterprise Distribution (OFED)
- Version 3.12 README
- May 2014
+ Version 4.17 README
+ May 2018
==============================================================================
Table of contents
1. Overview
==============================================================================
-This is the OpenFabrics Enterprise Distribution (OFED) version 3.12
+This is the OpenFabrics Enterprise Distribution (OFED) version 4.17
software package supporting InfiniBand, iWARP and RoCE fabrics. It is composed
of several software modules intended for use on a computer cluster
constructed as an InfiniBand subnet or an iWARP network.
See examples of ofed.conf and ofed_net.conf under OFED-X.X.X/docs.
Run './install.pl -p' to get ofed.conf with all available packages included.
- To enable selection of the technology preview for XEON PHI support, the
- parameter --with-xeon-phi must be provided. Both kernel packages ibp_server
- and ibscif needs to be selected as well as the user-mode packages ibpd and
- libibscif. The interactive installation and --all will include these,
- neither --hpc nor --basic will. This requires the linux version of
- Intel(R) Manycore Platform Software Stack (Intel(R) MPSS) version 3.4 or
- later to be installed and can be downloaded from
- http://software.intel.com/mic-developer.
Install Process Results:
------------------------
-ofed-scripts=y
-core=y
-mthca=y
-mlx4=y
-mlx4_en=y
-mlx5=y
-cxgb3=y
-cxgb4=y
-nes=y
-qib=y
-ocrdma=y
-ipoib=y
-srp=y
-nfsrdma=y
compat-rdma=y
compat-rdma-devel=y
+compat-rdma-firmware=y
+rdma-core=y
+rdma-core-devel=y
libibverbs=y
-libibverbs-devel=y
-libibverbs-devel-static=y
libibverbs-utils=y
-libmthca=y
-libmthca-devel-static=y
-libmlx4=y
-libmlx4-devel=y
-libmlx5=y
-libmlx5-devel=y
-libcxgb3=y
-libcxgb3-devel=y
-libcxgb4=y
-libcxgb4-devel=y
-libnes=y
-libnes-devel-static=y
-libocrdma=y
-libocrdma-devel=y
-libipathverbs=y
-libipathverbs-devel=y
-libibcm=y
-libibcm-devel=y
-libibumad=y
-libibumad-devel=y
-libibumad-static=y
-libibmad=y
-libibmad-devel=y
-libibmad-static=y
-ibsim=y
-ibacm=y
librdmacm=y
librdmacm-utils=y
-librdmacm-devel=y
-opensm-libs=y
+libibumad=y
+ibacm=y
+iwpmd=y
+srp_daemon=y
opensm=y
+opensm-libs=y
opensm-devel=y
+opensm-debuginfo=y
opensm-static=y
dapl=y
dapl-devel=y
dapl-devel-static=y
dapl-utils=y
+dapl-debuginfo=y
perftest=y
mstflint=y
-srptools=y
rds-tools=y
rds-devel=y
ibutils=y
infiniband-diags=y
qperf=y
+qperf-debuginfo=y
ofed-docs=y
-infinipath-psm=y
-infinipath-psm-devel=y
+ofed-scripts=y
+libfabric=y
+libfabric-devel=y
+libfabric-debuginfo=y
+fabtests=y
+fabtests-debuginfo=y
+core=y
+mlx4=y
+mlx4_en=y
+mlx5=y
+cxgb3=y
+cxgb4=y
+qib=y
+ipoib=y
+nvme_host=y
+qedr=y
+bnxt_re=y
+vmw_pvrdma=y
+srp=y
+iser=y
+
+++ /dev/null
- Open Fabrics Enterprise Distribution (OFED)
- ipath in OFED 1.5 Release Notes
-
- December 2009
-
-======================================================================
-1. Overview
-======================================================================
-ipath is the low level driver implementation for the
-QLogic HyperTransport HCA only (model QHT7140).
-
-The qib driver is the currently supported driver for all
-PCI-Express based Infiniband HCAs.
+++ /dev/null
- Open Fabrics Enterprise Distribution (OFED)
- NetEffect Ethernet Cluster Server Adapter Release Notes
- June 2015
-
-
-The iw_nes module and libnes user library provide RDMA and L2IF
-support for the NetEffect Ethernet Cluster Server Adapters.
-
-=============================================
-Supported Architectures and Operating Systems
-=============================================
-
-* CPU architectures:
- - x86_64
- - x86
-
-* Linux Operating Systems:
- - RedHat EL6.5 2.6.32-431.el6
- - RedHat EL6.6 2.6.32-504.el6
- - RedHat EL7.0 3.10.0-123.el7
- - RedHat EL7.1 3.10.0-229.el7
- - SLES11 SP3 3.0.76-0.9.1
- - SLES12 3.12.28-4-default
- - kernel.org 3.18 (minimal QA for this version)
-
-==========
-What's New
-==========
-
-OFED 3.18 contains the following new feature for iw_nes driver:
-
- - iWARP Port Mapper: The iWARP drivers use the port mapper API to request
- mappings of tcp ports. The port mapper consists of kernel space functionaliy,
- which is part of core/iw_cm and a user space service, which comes
- with the libiwpm-1.0.1 package, included in OFED-3.18
-
-OFED 3.12 contains the following fixes for iw_nes driver:
-
- - Fixes for MPA V2 implementaion with respect to the ird/ord peer negotiation
- - Fixed an incorrect QP pointer passed to user space after Flush error
- - NES_ABI versions have changed for nes driver and libnes (nes user space library)
- - Released new libnes-1.1.4 package with updated NES_ABI version
- - Added backports for Linux 2.6 and 3.0 kernels
-
-OFED 3.5 contains several important bug fixes for iw_nes driver:
-
- - Removed an incorrect macro routine from the loopback code.
- - Fixed a couple of problems when unlinking skbs (interoperability with Chelsio).
- - Added a lock, so that FPDUs are forwarded in order to iw_nes hardware (interoperability with Chelsio).
- - Fixed an incorrect setting of a multicast address in the perfect filter table.
- - Fixed a problem when adding timer and a timer is already pending.
- - Fixed a problem when initializing terminate timer more than once.
- - Fixed a problem when establishing iWarp connection and tx checksum offload is disabled.
- - Added backports for Linux 2.6 and 3.0 kernels
- - iw_nes driver version is changed to 1.5.0.1
-
-========================================
-Required Setting - Power Management Mode
-========================================
-If possible, disable Active State Power Management in the BIOS, e.g.:
-
- PCIe ASPM L0s - Advanced State Power Management: DISABLED
-
-
-=======================
-Loadable Module Options
-=======================
-The following options can be used when loading the iw_nes module by modifying
-/etc/modprobe.conf.local file in the SLES hosts and /etc/modprobe.d/rnic.conf file in the RHEL hosts.
-
-wide_ppm_offset=0
- Set to 1 will increase CX4 interface clock ppm offset to 300ppm.
- Default setting 0 is 100ppm.
-
-mpa_version=1
- MPA version to be used int MPA Req/Resp (1 or 2).
-
-disable_mpa_crc=0
- Disable checking of MPA CRC.
- Set to 1 to enable MPA CRC.
-
-send_first=0
- Send RDMA Message First on Active Connection.
-
-nes_drv_opt=0x00000100
- Following options are supported:
-
- 0x00000010 - Enable MSI
- 0x00000080 - No Inline Data
- 0x00000100 - Disable Interrupt Moderation
- 0x00000200 - Disable Virtual Work Queue
- 0x00001000 - Disable extra doorbell read after write
-
-debug_level=0
- Specify debug output level.
-
-wqm_quanta=65536
- Set size of data to be transmitted at a time.
-
-limit_maxrdreqsz=0
- Limit PCI read request size to 256 bytes.
-
-nes_lro_max_aggr=64
- Max number of LRO packets to be aggregated.
-
-===============
-Runtime Options
-===============
-The following options can be used to alter the behavior of the iw_nes module:
-NOTE: Assuming NetEffect Ethernet Cluster Server Adapter is assigned eth2.
-
- ifconfig eth2 mtu 9000 - largest mtu supported
-
- ethtool -K eth2 tso on - enables TSO
- ethtool -K eth2 tso off - disables TSO
-
- ethtool -C eth2 rx-usecs-irq 128 - set static interrupt moderation
-
- ethtool -C eth2 adaptive-rx on - enable dynamic interrupt moderation
- ethtool -C eth2 adaptive-rx off - disable dynamic interrupt moderation
- ethtool -C eth2 rx-frames-low 16 - low watermark of rx queue for dynamic interrupt moderation
- ethtool -C eth2 rx-frames-high 256 - high watermark of rx queue for dynamic interrupt moderation
- ethtool -C eth2 rx-usecs-low 40 - smallest interrupt moderation timer for dynamic interrupt moderation
- ethtool -C eth2 rx-usecs-high 1000 - largest interrupt moderation timer for dynamic interrupt moderation
-
-===================
-uDAPL Configuration
-===================
-Rest of the document assumes the following uDAPL settings in /etc/dat.conf:
-
- ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 0" ""
-
-========================
-Chelsio Interoperability
-========================
-The firmware version supporting interoperability with Chelsio is 3.23 or greater.
-The load time Chelsio parameter peer2peer must be set to 1.
-
-==============
-mpd.hosts file
-==============
-mpd.hosts is a text file with a list of nodes, one per line, in the MPI ring.
-Use either fully qualified hostname or IP address.
-
-===========================
-100% CPU Utilization remark
-===========================
-Most of the RDMA applications use CQ Polling mode to decrease latency.
-This operational mode can cause 100% CPU utilization.
-
-To switch to Event Driven mode and lower CPU utilization please refer to README or
-Release Notes for specific application.
-
-========================================
-Recommended Settings for Intel MPI 4.0.x
-========================================
-Add the following to mpiexec command:
-
- -genv I_MPI_FALLBACK_DEVICE 0
- -genv I_MPI_DEVICE rdma:ofa-v2-iwarp
-
-Example mpiexec command line for uDAPL-2.0:
-
- mpiexec -genv I_MPI_FALLBACK_DEVICE 0
- -genv I_MPI_DEVICE rdma:ofa-v2-iwarp
- -ppn 1 -n 2
- /opt/intel/impi/4.0.3.008/bin64/IMB-MPI1
-
-Intel MPI use CQ Polling mode as a default.
-To switch to wait mode add the following to mpiexec command:
- -genv I_MPI_WAIT_MODE 1
-
-NOTE: Wait mode supports the sock device only.
-
-========================================
-Recommended Setting for MVAPICH2 and OFA
-========================================
-From OFED-3.5 package MVAPICH2 is not a part of the OFED
-
-Download mvapich2-1.8.1 package from following location:
-
-http://mvapich.cse.ohio-state.edu/download/mvapich2/
-
-Example mpirun_rsh command line:
-
- mpirun_rsh -ssh -np 2 -hostfile /root/mpd.hosts
- /usr/mpi/gcc/mvapich2-1.8.1/tests/osu_benchmarks-3.1.1/osu_latency
-
-MVAPICH2 use CQ Polling mode as a default.
-To switch to Blocking mode add the following to mpirun_rsh command:
- MV2_USE_BLOCKING=1
-
-==========================================
-Recommended Setting for MVAPICH2 and uDAPL
-==========================================
-Add the following to the mpirun_rsh command for 64 or more processes:
-
- -env MV2_ON_DEMAND_THRESHOLD <number of processes>
-
-Example mpirun_rsh command with uDAPL-2.0:
-
- mpirun_rsh -ssh -np 64 -hostfile /root/mpd.hosts
- MV2_DAPL_PROVIDER=ofa-v2-iwarp
- MV2_ON_DEMAND_THRESHOLD=64
- /usr/mpi/gcc/mvapich2-1.8.1/tests/IMB-3.2/IMB-MPI1
-
-MVAPICH2 use CQ Polling mode as a default.
-To switch to Blocking mode add the following to mpirun_rsh command:
- MV2_USE_BLOCKING=1
-
-=======================================
-Recommended Settings for Open MPI 1.6.1
-=======================================
-
-From OFED-3.5 package OpenMPI is not a part of the OFED
-
-Download openmpi-1.6.1-1 package from following location:
-
-http://www.open-mpi.org/software/ompi/v1.6/
-
-There is more than one way to specify MCA parameters in
-Open MPI. Please visit this link and use the best method
-for your environment:
-
-http://www.open-mpi.org/faq/?category=tuning#setting-mca-params
-
-Example mpirun command line:
-
- mpirun -np 2 -hostfile /opt/mpd.hosts
- -mca btl openib,self,sm
- -mca btl_mpi_leave_pinned 0
- /usr/mpi/gcc/openmpi-1.6.1/tests/IMB-3.2/IMB-MPI1
-
-OpenMPI use CQ Polling mode as a default.
-No command parameter available to switch to Event Driven mode.
-
-
-==================================
-Intel Multicast Acceleration (IMA)
-==================================
-
-OFED-3.5 doesn't support Intel Multicast Acceleration (IMA) functionality.
-
-
-NetEffect is a trademark of Intel Corporation in the U.S. and other countries.
-
+++ /dev/null
- Open Fabrics Enterprise Distribution (OFED)
-
- Emulex OCe14000 Adapter RELEASE NOTES
-
- June 2015
-
-
-
-
-
-The ocrdma and be2net modules provide RDMA and NIC support for the
-
-Emulex OCe14000 family of adapters. Make sure you choose the 'ocrdma'
-
-and 'libocrdma' options while building OFED rpms.
-
-
-
-============================================
-
-New for OFED-3.18
-
-============================================
-
-- Bug Fixes in ocrdma and libocrdma packages
-
-===========================
-
-Supported Operating Systems
-
-===========================
-
- o CPU architectures:
-
- - x86_64
-
- - x86
-
- o Linux Operating Systems:
-
-
- - RedHat EL6.5 2.6.32-431.el6
-
- - RedHat EL6.6 2.6.32-504.el6
-
- - RedHat EL7.0 3.10.0-123.el7
-
- - RedHat EL7.1 3.10.0-229.el7
-
- - SLES11 SP3 3.0.76-0.9.1
-
- - SLES12 3.12.28-4-default
-
- - kernel.org 3.18 *
-
-
-
- * Minimal QA for these versions.
-
-================
-
-Supported Cards
-
-================
-
-The OCe14000 family of adapters are supported.
-
-=========================================
-
-Priority Flow Control recommendations
-
-=========================================
-
-- If the HCA port is connected to a PFC enabled switch, the OCe14000 adapter
- is programmed to use priority 5. Configure the priority-groups
- and switch ports accordingly.
-
-- If the HCA port is connected to a PFC enabled switch, it is
- highly recommended that you use a VLAN interface for traffic. On a non-VLAN
- interface driver would insert VID 0 for all outgoing packets.
-
-- In back-to-back configurations of OCe14000 series, PFC is enabled by default.
-
-
-
-====================
-
-Errata (Open Issues)
-
-====================
-
-
-- For a 40Gb adapter, ibv_devinfo -v will display:
-
- active_width: 4X (2)
-
- active_speed: 10.0 Gbps (4)
-
-
- 4x * 10 = 40G
-
- link-width is not applicable to RoCE. ibv_devinfo needs a fix.
-
-
-=================
-
-Updating Firmware
-
-=================
-
-
-Firmware version 10.2.x or greater includes new features that required new
-
-flash regions to support them. Firmware versions earlier than 10.0.803.37
-
-did not have the ability to configure the flash regions to support these
-
-new features.
-
-
-
-Note 1: Do not use any inbox Linux drivers to update from any version of
-
- 10.0.x firmware to any version of 10.2.x or greater firmware.
-
-
-Note 2: If you are using out-of-box Linux drivers and you are updating
-
- from a firmware version earlier than 10.0.803.37, you must first
-
- update the out-of-box Linux driver to version 10.2.x, and then
-
- perform the firmware update procedure twice.
-
-
-Note 3: If you are using out-of-box Linux drivers and you are updating
-
- from firmware version 10.0.803.37 or later, you must first update
-
- the out-of-box Linux driver to version 10.2.x. It is not necessary
-
- to perform the firmware update procedure twice.
-
-
-
-Note 4: After you have updated the firmware, you must not downgrade it to
-
- a version earlier than 10.0.803.37.
-
-
-
-- Get oc14-10.2.XX.X.ufi or later firmware from www.emulex.com.
-
-- Copy the firmware file into /lib/firmware
-
-- Run: ethtool -f ethX <fw-image-filename>
-
-- Reboot the machine.
-
-
-
-The driver in this distribution has been tested with versions 10.6.88.0 and
-10.6.103.8
-
-eg:
-
-# cp <path-to>/oc14-10.6.88.0.ufi /lib/firmware/
-
-# ethtool -f eth2 oc14-10.6.88.0.ufi
-
-# reboot
-
-
-=============
-
-OpenMPI:
-
-=============
-
-From the OFED-3.5 package onwards, OpenMPI is not a part of the OFED
-
-package. You must manually download and install it.
-
-
-
-- Download the latest OpenMPI released SRPM from the following location.
-
- http://www.open-mpi.org/software/ompi
-
-
-
-- Build OpenMPI using the following steps:
-
- ./configure --with-openib
-
- make
-
- make install
-
-
-
-- Use the following parameters while running an MPI application:
-
- --mca btl openib,self,sm --mca pml ob1 --mca btl_openib_flags 306 --mca
-
- btl_openib_receive_queues P,65536,120,64,32 --mca btl_openib_cpc_include
-
- rdmacm
-
-
-
- eg:
-
- mpirun -np 8 -hostfile <path to hostfile> --mca btl openib,self,sm --mca pml ob1 \
-
- --mca btl_openib_flags 306 --mca btl_openib_receive_queues P,65536,120,64,32 \
-
- --mca btl_openib_cpc_include rdmacm <path to MPI application>
-
-
-
-- For further details refer to the OpenMPI documentation.
-
-
-
-=============
-
-MVAPICH2:
-
-=============
-
-From the OFED-3.5 package onwards, MVAPICH2 is not a part of the OFED
-
-package. You must manually download and install it.
-
-
-
-- Download the latest MVAPICH2 released SRPM from the following location:
-
- http://mvapich.cse.ohio-state.edu/download/mvapich2/
-
-
-
-- Build MVAPICH2 with the following configure options
-
- ./configure --prefix=/usr --with-device=ch3:mrail --with-rdma=gen2 --sysconfdir=/etc --libdir=/usr/lib64
-
- make
-
- make install
-
-- Use the following -env options to run the MPI job in the mpirun command:
-
- "-env MV2_USE_RDMAOE 1 -env MV2_USE_RDMA_CM 1 -env MV2_USE_SHARED_MEM 1
-
- -env MV2_USE_BLOCKING 0 -env MV2_IBA_HCA ocrdma0"
-
- eg:
-
- mpirun -np 8 -f <Path to hostfile> -env MV2_USE_RDMAOE 1 -env MV2_USE_RDMA_CM 1 \
-
- -env MV2_DEFAULT_MAX_CQ_SIZE 128 -env MV2_USE_SRQ 0 -env MV2_MX_SEND_WR 64 \
-
- -env MV2_MAX_RECV_WR 64 -env MV2_USE_SHARED_MEM 1 -env MV2_USE_BLOCKING 0 \
-
- -env MV2_IBA_HCA ocrdma0 <path to MPI application>
-
-
-
-- For further details refer to the MVAPICH2 documentation.
-
-
-
-============================================
-
-Testing connectivity with ping and rping:
-
-============================================
-
-
-
-- modprobe be2net
-
-- Configure the Ethernet interfaces for the Emulex device.
-
-- Configure the IP address to the interfaces.
-
-- modprobe ocrdma
-
-- ibv_devinfo should display the ocrdma device in an active state.
-
-- ping <IP of peer Emulex device> should succeed.
-
-
-
-Test RDMA connectivity, using the rping command. It is included in the
-
-librdmacm-utils rpm:
-
-
-
-On the server machine:
-
-
-
-# rping -s -p 9999
-
-
-
-On the client machine:
-
-
-
-# rping -c -VvC10 -a server_ip_addr -p 9999
-
-
-
-You should see ping data like this on the client:
-
-
-
-ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
-
-ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
-
-ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
-
-ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
-
-ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv
-
-ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw
-
-ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx
-
-ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy
-
-ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
-
-ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA
-
-client DISCONNECT EVENT...
-
-
-
-
-
+++ /dev/null
- Open Fabrics Enterprise Distribution (OFED)
- PSM in OFED 3.12 Release Notes
-
- May 2014
-
-======================================================================
-1. Overview
-======================================================================
-
-The Performance Scaled Messaging (PSM) API is Intel's low-level user-level
-communications interface for the Intel(R) True Scale Fabric family of products.
-
-The PSM libraries are included in the infinipath-psm-3.2-2_ga8c3e3e_open.src.rpm
-source RPM and get built and installed as part of a default OFED
-install process.
-
-The primary way to use PSM is by compiling applications with an MPI that has
-been built to use the PSM layer as its interface to Intel HCAs.
-PSM is the high-performance interface to the Intel(R) True Scale HCAs.
-
-Minimal instructions* for building two MPIs tested with OFED
-with PSM support are as follows:
-
-
-Open MPI:
-
-- Download a recent Open MPI tar ball from
- http://www.open-mpi.org/software/ompi/v1.8/ .
- Version 1.8.1 has been tested with PSM from this OFED release.
-- Untar the file and cd to the Open MPI directory. Then
- ./configure --with-psm --prefix=<install directory>
- make
- make install
-
-MVAPICH2:
-
-- Download a recent MVAPICH2 tar ball from
- http://mvapich.cse.ohio-state.edu/download/mvapich2/
- Version 1.9 has been tested with PSM from this OFED release.
-- Untar the file and go to the mvapich2-1.x directory
-- Execute the configure and make commands as follows:
- ./configure --prefix=<install directory> --with-device=ch3:psm \
- --enable-shared
- make
- make install
-
-Once these MPIs are built, using mpirun/mpiexec will result in the use
-of PSM by default -- no additional options needed. Also if you have two
-Intel True Scale HCAs per node, both of them will be used by default
-when you run with 2 or more ranks per node. No option is necessary on
-the mpirun/mpiexec command.
-
-If you want to use just one of the HCAs, set the IPATH_UNIT variable to:
-export IPATH_UNIT=0 (or =1)
-If you have two Intel True Scale HCAs per node, and want to get additional
-bandwidth performance even when only one MPI rank is active per node, set:
-export PSM_MULTIRAIL=0
-For most applications, it is better not to use this PSM_MULTIRAIL variable
-since it increases MPI latency somewhat, but it is helpful for a minority
-of applications that require additional bandwidth from single ranks.
-
-
-* To configure with a different compiler suite than the native GCC suite on your
- Linux machine, set the configure or script variables: CC, CXX, F77, F90 to be
- assigned to the appropriate compiler names for your suite, such as for the
- Intel Compiler suite:
- CC=icc
- CXX=icpc
- F77=ifort
- F90=ifort
-
-
- README/Release Notes
- OFED 3.18-2 DAPL Release 2.1.9-2
- April 2016
+ README/Release Notes
+ OFED 4.17 DAPL Release 2.1.10-1
+ November 2018
User space libraries/utilities for Direct Access Transport (DAT) v2.0. DAT is
a transport-independent, platform-independent Application Programming
Interface that supports RDMA (remote direct memory access) devices.
Note: v1.2 is no longer supported and will not be included with OFED releases
- MIC support is provided with the new MCM provider and MPXYD service, since dapl-2.1.0.
- MCM requires the Intel(R) MPSS 3.x (YOCTO) release for Linux to be installed on your system.
- MPSS 3.x for Linux can be downloaded from: http://software.intel.com/mic-developer
-
For latest documentation and packages: //www.openfabrics.org/downloads/dapl/
=================
1.0 Release Notes
=================
+ dapl-2.1.10-1 CCL Proxy optimization for post_send, dtest/dtest_suite improvements, bug fixes, UCM log for timers
dapl-2.1.9-2 changes include bug fixes, MFO fixes, GID query attributes
dapl-2.1.9-1 changes include dtestcm improvement, bug fixes for MIC, Fix SCM interop issue with MTU
projects / ~aditr / compat-rdma-docs.git / commitdiff