Since SDP is a transparent TCP replacement, it can sometimes be difficult to
know that it is working correctly.
-To know if the traffic was passing through SDP or TCP, check
-/proc/net/sdpstats and see the the counters are running.
+To figure out whether traffic is passing through SDP or TCP, check
+/proc/net/sdpstats and monitor which counters are running.
sdpnetstat:
-----------
-----------
For smaller buffers, the overhead of preparing a user buffer to be RDMA'ed is
-too big - therefore it is more efficient to use BCopy.
-Large buffers could be sent by using RDMA, and thus lowering to CPU
-utilization. This mode is called "ZCopy combined mode". The sendmsg syscall is
-blocked till the buffer is transfered to the socket's peer, and the data is
-copied directly from the user buffer to the user buffer at the sink side.
+too big; therefore, it is more efficient to use BCopy. (Large buffers can also
+be sent using RDMA, but they lower CPU utilization.) This mode is called
+"ZCopy combined mode". The sendmsg syscall is blocked until the buffer is
+transfered to the socket's peer, and the data is copied directly from the user
+buffer at the source side to the user buffer at the sink side.
+
+To set the threshold, use the module parameter sdp_zcopy_thresh. This parameter
+can be accessed through sysfs (/sys/module/ib_sdp/parameters/sdp_zcopy_thresh).
+Setting it to 0, disables ZCopy.
-To set the threshold, use the module parameter sdp_zcopy_thresh. This paramter
-could be accessed through sysfs
-(/sys/module/ib_sdp/parameters/sdp_zcopy_thresh). Setting it to 0, disable
-ZCopy.
==============================================================================
9. Uninstalling OFED
- SDP configuration file: /etc/libsdp.conf
- OpenSM configuration file: /etc/ofa/opensm.conf (for RedHat)
/etc/sysconfig/opensm (for SuSE) - should be
- created manually if required.
+ created manually if required.
- DAPL configuration file: /etc/dat.conf
See packages Release Notes for more details.
installation parameters can be found by running
/etc/infiniband/info.
+
==============================================================================
12. Related Documentation
==============================================================================
/usr/share/doc/ofed-docs-x.x.x for RedHat
/usr/share/doc/packages/ofed-docs-x.x.x for SuSE
-Document list:
+Documents list:
o README.txt
o OFED_Installation_Guide.txt
o All release notes and README files
For more information, please visit the OpenFabrics web site:
-
- http://www.openfabrics.org/
+ http://www.openfabrics.org
open-iscsi documentation is located at:
- RedHat: /usr/share/doc/iscsi-initiator-utils-<version number>
- SuSE: /usr/share/doc/packages/open-iscsi
For more information, please visit the open-iscsi web site:
-
- http://www.open-iscsi.org/
+ http://www.open-iscsi.org
Open Fabrics Enterprise Distribution (OFED)
- Version 1.5.1
- Release Notes
- March 2010
+ Version 1.5.1
+ Release Notes
+ March 2010
===============================================================================
Table of Contents
===============================================================================
1. Overview, which includes:
- - OFED Distribution Rev 1.5 Contents
+ - OFED Distribution Rev 1.5.1 Contents
- Supported Platforms and Operating Systems
- Supported HCA and RNIC Adapter Cards and Firmware Versions
- Tested Switch Platforms
1. Overview
===============================================================================
These are the release notes of OpenFabrics Enterprise Distribution (OFED)
-release 1.5. The OFED software package is composed of several software modules,
+release 1.5.1. The OFED software package is composed of several software modules,
and is intended for use on a computer cluster constructed as an InfiniBand
subnet or iWARP network.
all of its nodes to this new version.
-1.1 OFED 1.5 Contents
+1.1 OFED 1.5.1 Contents
-----------------------
The OFED package contains the following components:
- OpenFabrics core and ULPs:
- Documentation
Notes:
-1. iSER target and NFS-RDMA is of Beta quality.
+1. iSER Target and NFS-RDMA are of Beta quality.
2. All other OFED components are of production quality.
3. See release notes for each package in the docs directory.
4. Any Topspin copyright belongs to Cisco Systems, Inc.
o Kernel code based on 2.6.30
o libraries location - all userspace libraries can be downloaded from
http://www.openfabrics.org/downloads/
- See BUILD_ID for exact location
- o Qlogic replaced low level driver from ipath to qib.
+ See BUILD_ID for exact locations
+ o Qlogic moved the low level driver from ipath to qib.
2.2 SDP
o Performance improvements
o Zero copy in beta level
2.3 uDAPL
- o New UCM provider (ofa-v2-mlx4_0-1u) with IB UD based CM per process. More
- scalable then rdma_cm (cma) or socket cm (scm).
- o Common code base with WinOF 2.1.
- o Bug fixes.
+ o New UCM provider (ofa-v2-mlx4_0-1u) with IB UD-based CM per process.
+ It is more scalable than rdma_cm (cma) or socket cm (scm).
+ o Common code base with WinOF 2.1
+ o Bug fixes
2.4 perftest
2.5 Management
o OpenSM
- - Mesh Analysis for LASH routing algorithm.
- - Reloadable OpenSM configuration (preliminary implemented)
- - Routing paths sorted balancing (for UpDown and MinHops)
- - Weighted Lid Matrices calculation (for UpDown, MinHop and DOR).
- - I/O nodes connectivity (for FatTree).
+ - Support for Mesh Analysis for LASH routing algorithm
+ - Reloadable OpenSM configuration (preliminary implementation)
+ - Routing paths sorted balancing (for UpDown and MinHops)
+ - Weighted LID matrices calculation (for UpDown, MinHop and DOR)
+ - I/O nodes connectivity (for FatTree)
o Diagnostic tools:
+ TBD
2.6 MPI:
a. OSU MVAPICH 1.2.0
2.8 NFS-RDMA
o Added support for RHEL5.4, SLES10 SP3, kernel.org 2.6.25 and 2.6.30.
- kernels 2.6.26 and 2.6.27 are not supported
- o NFS-RDMA is in beta level.
+ Kernels 2.6.26 and 2.6.27 are not supported
+ o NFS-RDMA is in Beta level
===============================================================================
3. Main Changes from OFED 1.5
===============================================================================
-1. Added RoCEE support. See RoCEE_README.txt.
+1. Added RoCEE support - see RoCEE_README.txt
2. Added enhanced atomic operations to ConnectX (kernel only).
See mlx4_release_notes.txt.
3. Updated Open MPI to rev 1.4.1-2ofed
7. Updated librdmacm to rev 1.0.11
8. Removed tvflash RPM
9. NFS-RDMA is not supported on SLES10 SP3
-10. Fixup IPv6 support and IPv4 routing corner cases for RDMA CM
+10. Fixed IPv6 support and IPv4 routing corner cases for RDMA CM
11. Bug fixes
The following is a list of general limitations and known issues of the various
components of the OFED 1.5.1 release.
-1. When upgrading from an earlier OFED version, the installation script does not
- stop the earlier OFED version prior to uninstalling it.
+1. When upgrading from an earlier OFED version, the installation script does
+ not stop the earlier OFED version prior to uninstalling it.
Workaround: Stop the old OFED stack (/etc/init.d/openibd stop) before
upgrading to OFED 1.5.1 or reboot the server after OFED installation.
2. Memory registration by the user is limited according to administrator
setting. See "Pinning (Locking) User Memory Pages" in OFED_tips.txt for
system configuration.
3. Fork support from kernel 2.6.12 and above is available provided
- that applications do not use threads. The fork() is supported as long
- as the parent process does not run before the child exits or calls exec().
+ that applications do not use threads. fork() is supported as long as the
+ parent process does not run before the child exits or calls exec().
The former can be achieved by calling wait(childpid), and the latter can be
achieved by application specific means. The Posix system() call is
supported.
4. The qib driver is supported only on 64-bit platforms.
-5. When installing OFED on OpenSuse or Ubuntu one should use the
- --without-depcheck option of the install.pl script
+5. When installing OFED on OpenSuse or Ubuntu, use the --without-depcheck
+ option of the install.pl script.
6. IPoIB: brctl utilities do not work on IPoIB interfaces. The reason for that
is that these utilities support devices of type Ethernet only.
7. "openibd stop" can sometime fail with the error:
ERROR: Module ib_cm is in use by ib_ipoib
Workaround: run "openibd stop" again or remove ib_ipoib aliases from
/etc/modprobe.conf.
-8. When working with ISCSI over IPoIB or mlx4_en, you must disable LRO (even
- if IPoIB is set to connected mode). This is because there is a bug in older
- kernels which causes a kernel panic.
-9. On SLES11 in case that uninstall is failing, need to look at the error log
- and remove the RPMs manually using 'rpm -e <rpms list>'
-10. On SLES11 one should set allow_unsupported_modules parameter to 1 in file:
+8. When working with ISCSI over IPoIB or mlx4_en, you must disable LRO (even
+ if IPoIB is set to connected mode). This is due to a bug in older kernels
+ which causes a kernel panic.
+9. On SLES11, and in case uninstall is failing, check the error log and remove
+ the remaining RPMs manually using 'rpm -e <rpms list>'.
+10. On SLES11, set allow_unsupported_modules parameter to 1 in file:
/etc/modprobe.d/unsupported-modules. Without this the modules will not
load.
-11. iSER is supported on kernel.org: 2.6.30, 2.6.31 and 2.6.32 only
- OFED-1.5 will not install iSER on other kernels and original iSER
- module coming with Linux Distribution will stop working because of
- mismatch in symbols version.
-12. On SLES10 SP3 kernel 2.6.16.60-0.54.5 should be updated to 2.6.16.60-0.59.1
- or later. As original kernel may cause kernel panic during
- '/etc/init.d/openibd restart'.
+11. iSER is supported on kernel.org 2.6.30, 2.6.31 and 2.6.32 only.
+ OFED-1.5 will not install iSER on other kernels, and the original iSER
+ module that comes with the Linux distribution will stop working due to a
+ mismatch in the symbols version.
+12. On SLES10 SP3, the kernel 2.6.16.60-0.54.5 should be updated to
+ 2.6.16.60-0.59.1 or later. The original kernel may cause kernel panic
+ during '/etc/init.d/openibd restart'.
Note: See the release notes of each component for additional issues.
+
Open Fabrics Enterprise Distribution (OFED)
Version 1.5.1
- README
+ README
- March 2010
+ March 2010
This is the OpenFabrics Enterprise Distribution (OFED) version 1.5.1
software package supporting InfiniBand and iWARP fabrics. It is composed
sysctl -w net.ipv4.conf.all.arp_ignore=1
- To learn more about the arp_ignore parameter, see Documentation/networking/ip-sysctl.txt.
+ To learn more about the arp_ignore parameter, see
+ Documentation/networking/ip-sysctl.txt.
Note that distributions have the means to make kernel parameters persistent.
2. There are IPoIB alias lines in modprobe.conf which prevent stopping/
To warn about this, a message is produced in the system log each time MTU is
set to a value higher than 2K.
-7. IPoIB IPv6 support is broken for between systems with kernels < 2.6.12 and
+7. IPoIB IPv6 support is broken for systems with kernels < 2.6.12 and
kernels >= 2.6.12. The reason for that is that kernel 2.6.12 puts the link
layer address at an offset of two bytes with respect to older kernels. This
causes the other host to misinterpret the hardware address resulting in failure
(~5%) than in datagram mode. As a workaround, use datagram mode.
9. Single-socket TCP bandwidth for kernels < 2.6.18 is lower than with
- newer kernels. We recommend kernels from 2.6.18 and up for
+ newer kernels. It is recommended to use kernel 2.6.18 or up for
best IPoIB performance.
10. Connectivity issues encountered when using IPv6 on ia64 systems.
11. The IPoIB module uses a Linux implementation for Large Receive Offload
(LRO) in kernel 2.6.24 and later. These kernels require installing the
- "inet_lro" module.
+ "inet_lro" module.
-12. ConnectX only: If you have a port configured as ETH, and are running IPoIB
- in connected mode -- and then change the port type to IB, the IPoIB mode
- changes to datagram mode.
+12. ConnectX only: If you have a port configured as ETH and IPoIB is running
+ in connected mode, and then you change the port type to IB, the IPoIB mode
+ will change to datagram mode.
-13. When working with ISCSI, you must disable LRO (even if you are working in
+13. When working with iSCSI, you must disable LRO (even if you are working in
connected mode). This is because there is a bug in older kernels which causes
a kernel panic.
14. IPoIB datagram mode initial packet loss (bug #1287): When the datagram test
- gets to packet size 8192 and larger it always loose the first packet in the
+ gets to packet size 8192 or larger, it always loses the first packet in the
sequence.
Workaround: Increase the number of pending skb's before a neighbor is
resolved (default is 3). This value can be changed with:
sysctl net.ipv4.neigh.ib0.unres_qlen.
15. IPoIB multicast support is broken in RH4.x kernels. This is because
- ndisc_mc_map() does not handle IPOIB hardware addresses.
+ ndisc_mc_map() does not handle IPoIB hardware addresses.
-16. If bonding uses IPoIB slave then unenslaving all slaves (or downing them
- with ifdown) followed by unloading the module ib_ipoib might crash the
+16. If bonding uses an IPoIB slave, then un-enslaving all slaves (or downing
+ them with ifdown) followed by unloading the module ib_ipoib might crash the
kernel. To avoid this leave the IPoIB interfaces enslaved when unloading
ib_ipoib.
datagram on each call to ifup, ifdown, etc. To avoid this, add the line
IPOIB_MODE=connected
to the interface configuration file (e.g. ifcfg-ib0)
-18. When installing OFED on a machine that runs kernel 2.6.30 (or another kernel
- from kernel.org that OFED supports), the installation scripts blocks the
- installation of ib-bonding since the bonding module that comes with the
+
+18. When installing OFED on a machine that runs kernel 2.6.30 (or another
+ kernel from kernel.org that OFED supports), the installation script blocks
+ the installation of ib-bonding since the bonding module that comes with the
kernel has all the functionality to support IPoIB slaves. This approach
- however doesn't patch the sysconfig (SuSE) or initscripts (RedHat) package
- so the network configuration scrips may not work properly.
+ however doesn't patch the sysconfig (SuSE) or initscripts (RedHat) package,
+ so the network configuration script may not work properly.
For example, if you install OFED on RHEL5.2 that runs kernel 2.6.30 and
you try to configure and run bonding, you won't be able to restart the
network and see bond0 up and running with IPoIB slaves.
- A workaround to this problem would be this
- a. Compile ib-bonding source rpm (under SRPMS directory) separately on a
- machine with RHEL5.2 and kernel 2.6.18-92.el5 (default for this OS)
- 2. Install the binary rpm while machine runs kernel 2.6.18-92.el5.
- This will patch the OS configuration scripts and install the bonding module.
- 3. Switch to kernel 2.6.30. The module that was compiled in (2) will not be
- loaded since it was compiled and installed for a different kernel.
- 4. Configure bonding and restart the network. The bonding interface
+ A workaround to this problem would be as follows:
+ a. Compile ib-bonding source rpm (under SRPMS directory) separately on
+ a machine with RHEL5.2 and kernel 2.6.18-92.el5 (default for this OS).
+ b. Install the binary RPM while the machine runs kernel 2.6.18-92.el5.
+ This will patch the OS configuration scripts and install the bonding
+ module.
+ c. Switch to kernel 2.6.30. The module that was compiled in (a) will
+ not be loaded since it was compiled and installed for a different
+ kernel.
+ d. Configure bonding and restart the network. The bonding interface
should be up and running afterwards.
+
19. On RHEL5.X, '/etc/init.d/openibd start' prints the following messages while
bringing up IPoIB interfaces:
This does not affect IPoIB configuration and interfaces are configured as
expected.
-20. In IPoIB connected mode packages larger than 2016 bytes are not sent.
+
+20. In IPoIB connected mode, packages larger than 2016 bytes are not sent.
https://bugs.openfabrics.org/show_bug.cgi?id=1839
21. Under SLES11, if an IP configuration exists for an IPoIB interface
that later becomes a slave of a bonding master, a network restart
- doesn't erase the IP configuration from the slave and it appears to has
- an IP adddress even tough the new configuration doesn't set one. This
- may cause problems when tring to use the bonded network interface. To
+ doesn't erase the IP configuration from the slave and it appears to have
+ an IP address even though the new configuration doesn't set one. This
+ may cause problems when trying to use the bonded network interface. To
avoid this, restart the IB stack (openib restart) once you change the
configuration.
This issue is described in
https://bugs.openfabrics.org/show_bug.cgi?id=1975
-22. IPoIB LRO, currently, is not supported on ConnectX2 devices
+22. Currently, IPoIB LRO is not supported on ConnectX-2 devices
===============================================================================
3. IPoIB Configuration Based on DHCP
===============================================================================
-Setting an IPoIB interface configuration based on DHCP (v3.1.2 which is available
-via www.isc.org) is performed similarly to the configuration of Ethernet
-interfaces. In other words, you need to make sure that IPoIB configuration files
-include the following line:
+Setting an IPoIB interface configuration based on DHCP (v3.1.3 which is
+available via www.isc.org) is performed similarly to the configuration of
+Ethernet interfaces. In other words, you need to make sure that IPoIB
+configuration files include the following line:
For RedHat:
BOOTPROTO=dhcp
For SLES:
/etc/sysconfig/network/ on a SuSE machine
Note: A patch for DHCP is required for supporting IPoIB. The patch file for
-DHCP v3.1.2, dhcp.patch, is available under the docs/ directory.
+DHCP v3.1.3, dhcp.patch, is available under the docs/ directory.
Standard DHCP fields holding MAC addresses are not large enough to contain an
IPoIB hardware address. To overcome this problem, DHCP over InfiniBand messages
4.1 DHCP Server
In order for the DHCP server to provide configuration records for clients, an
appropriate configuration file needs to be created. By default, the DHCP server
-looks for a configuration file called dhcpd.conf under /etc. You can either edit
-this file or create a new one and provide its full path to the DHCP server using
-the -cf flag. See a file example at docs/dhcpd.conf of this package.
+looks for a configuration file called dhcpd.conf under /etc. You can either
+edit this file or create a new one and provide its full path to the DHCP server
+using the -cf flag. See a file example at docs/dhcpd.conf of this package.
The DHCP server must run on a machine which has loaded the IPoIB module.
To run the DHCP server from the command line, enter:
4.2 DHCP Client (Optional)
Note: A DHCP client can be used if you need to prepare a diskless machine with
-an IB driver.
+an IB driver.
+
In order to use a DHCP client identifier, you need to first create a
configuration file that defines the DHCP client identifier. Then run the DHCP
client with this file using the following command:
dhclient cf <client conf file> <IB network interface name>
-Example of a configuration file for the ConnectX (PCI Device ID 25418), called
+Example of a configuration file for the ConnectX (PCI Device ID 26428), called
dhclient.conf:
# The value indicates a hexadecimal number
interface "ib1" {
-send dhcp-client-identifier 00:02:c9:03:00:00:10:39;
+send dhcp-client-identifier ff:00:00:00:00:00:02:00:00:02:c9:00:00:02:c9:03:00:00:10:39;
}
-Example of a configuration file for InfiniHost III Ex (PCI Device ID 25218), called
-dhclient.conf:
+Example of a configuration file for InfiniHost III Ex (PCI Device ID 25218),
+called dhclient.conf:
# The value indicates a hexadecimal number
interface "ib1" {
send dhcp-client-identifier 20:00:55:04:01:fe:80:00:00:00:00:00:00:00:02:c9:02:00:23:13:92;
}
In order to use the configuration file, run:
-host1# dhclient cf dhclient.conf ib1
+host1# dhclient -cf dhclient.conf ib1
===============================================================================
* Using /etc/infiniband/openib.conf to create a persistent configuration is
no longer supported
* On RHEL4_U7, cannot set a slave interface as primary.
-* ib-bonding will not be compiled and installed with OFED on OS with kernel
- that is >= 2.6.27 (e.g. SLES11). The bonding driver that comes with those kernels
- already supports enslaving of IPoIB interfaces. In addition, an OS can come
- with an older kernel but with a patched bonding driver that also doesn't
- require modification (e.g. RHEL5.4). OFED will not replace the bonding
- module in such cases also.
- However, there still might be a issue of OS configuration tools (like sysconfig or
- initscripts) that needs a fix but such issues were not observed yet.
+* ib-bonding will not be compiled and installed with OFED on an OS with kernel
+ that is >= 2.6.27 (e.g., SLES11). The bonding driver that comes with those
+ kernels already supports enslaving IPoIB interfaces. In addition, an OS
+ can come with an older kernel but with a patched bonding driver that also
+ doesn't require modification (e.g., RHEL5.4). OFED will not replace the
+ bonding module in such cases either.
+ However, there might still be issues with OS configuration tools (like
+ sysconfig or initscripts) that may need fixing, but such issues have not
+ been observed yet.
===============================================================================
===============================================================================
5.1 Subinterfaces
-
+-----------------
You can create subinterfaces for a primary IPoIB interface to provide traffic
-isolation. Each such subinterface (also called a child interface) has a
+isolation. Each such subinterface (also called a child interface) has
different IP and network addresses from the primary (parent) interface. The
default Partition Key (PKey), ff:ff, applies to the primary (parent) interface.
5.1.1 Creating a Subinterface
-
+-----------------------------
To create a child interface (subinterface), follow this procedure:
Note: In the following procedure, ib0 is used as an example of an IB
subinterface.
Step 4. As can be seen, the interface does not have IP or network addresses so
it needs to be configured.
-Step 5. To be able to use this interface, a configuration of the Subnet Manager is needed so that
-the PKey chosen, which defines a broadcast address, be recognized.
+Step 5. To be able to use this interface, a configuration of the Subnet Manager
+is needed so that the PKey chosen, which defines a broadcast address, can be
+recognized.
5.1.2 Removing a Subinterface
To remove a child interface (subinterface), run:
===============================================================================
- Performance tuning is enabled by default for IPOIB CM.
- Clear IPOIB_FLAG_ADMIN_UP if ipoib_open fails
-- disable napi while cq is being drained (bugzilla #1587)
-- rdma_cm: Use rate from ipoib broadcast when joining ipoib multicast
- When joining IPoIB multicast group, use the same rate as in the broadcast
- group. Otherwise, if rdma_cm creates this group before IPoIB does, it might get
- a different rate. This will cause IPoIB to fail joining to the same group later
- on, because IPoIB has a strict rate selection.
-- fix unprotected use of priv->broadcast in ipoib_mcast_join_task.
+- Disable napi while cq is being drained (bugzilla #1587)
+- rdma_cm: Use the rate from the ipoib broadcast when joining an ipoib
+ multicast. When joining an IPoIB multicast group, use the same rate as in the
+ broadcast group. Otherwise, if rdma_cm creates this group before IPoIB does,
+ it might get a different rate. This will cause IPoIB to fail joining the same
+ group later on, because IPoIB has a strict rate selection.
+- Fixed unprotected use of priv->broadcast in ipoib_mcast_join_task.
- Do not join broadcast group if interface is brought down
10. Bug Fixes and Enhancements Since OFED 1.5.0
===============================================================================
-- Fix lockup of the tx queue on mixed CM/UD traffic
- When there is high rate of send traffic on both CM and UD QPs. the trasmitter
- could be stopped by CM path but not re-enabled.
+- Fixed lockup of the TX queue on mixed CM/UD traffic
+ When there is a high rate of send traffic on both CM and UD QPs, the
+ transmitter can be stopped by the CM path but not re-enabled.
===============================================================================
-11. Performance tuning
+11. Performance Tuning
===============================================================================
When IPoIB is configured to run in connected mode, tcp parameter tuning is
-performed at driver startup -- to improve the throughput of medium and large
+performed at driver startup to improve the throughput of medium and large
messages.
The driver startup scripts set the following TCP parameters as follows:
net.ipv4.tcp_rmem="4096 87380 16777216"
net.ipv4.tcp_wmem="4096 65536 16777216"
-This tuning is effective only for connected mode. If you run in datagram mode,
+This tuning is effective only for connected mode. If you run in datagram mode,
it actually reduces performance.
If you change the IPoIB run mode to "datagram" while the driver is running,
Table of Contents
===============================================================================
1. Overview
-2. Supported Firmware Versions
+2. Supported firmware versions
3. VPI (Virtual Process Interconnect)
-4. Infiniband new features and bug fixes since OFED 1.3.1
-5. Infiniband (mlx4_ib) new features and bug fixes since OFED 1.4
+4. InfiniBand new features and bug fixes since OFED 1.3.1
+5. InfiniBand (mlx4_ib) new features and bug fixes since OFED 1.4
6. Eth (mlx4_en) new features and bug fixes since OFED 1.4
7. New features and bug fixes since OFED 1.4.1
8. New features and bug fixes since OFED 1.4.2
9. New features and bug fixes since OFED 1.5
10. Known Issues
-11. mlx4 Available Parameters
+11. mlx4 available parameters
===============================================================================
1. Overview
mlx4 is the low level driver implementation for the ConnectX adapters designed
by Mellanox Technologies. The ConnectX can operate as an InfiniBand adapter,
as an Ethernet NIC, or as a Fibre Channel HBA. The driver in OFED 1.4 supports
-Infiniband and Ethernet NIC configurations. To accommodate the supported
+InfiniBand and Ethernet NIC configurations. To accommodate the supported
configurations, the driver is split into three modules:
- mlx4_core
Handles Ethernet specific functions and plugs into the netdev mid-layer.
===============================================================================
-2. Supported Firmware Versions
+2. Supported firmware versions
===============================================================================
- This release was tested with FW 2.7.000.
- The minimal version to use is 2.3.000.
===============================================================================
3. VPI (Virtual Protocol Interconnect)
===============================================================================
-VPI enables ConnectX to be configured as an Ethernet NIC and/or an Infiniband
+VPI enables ConnectX to be configured as an Ethernet NIC and/or an InfiniBand
adapter.
o Overview:
The VPI driver is a combination of the Mellanox ConnectX HCA Ethernet and
- Infiniband drivers.
- It supplies the user with the ability to run Infiniband and Ethernet
+ InfiniBand drivers.
+ It supplies the user with the ability to run InfiniBand and Ethernet
protocols on the same HCA (separately or at the same time).
For more details on the Ethernet driver see MLNX_EN_README.txt.
o Firmware:
The VPI driver works with FW 25408 version 2.6.000 or higher.
One needs to use INI files that allow different protocols over same HCA.
o Port type management:
- By default both ConnectX ports are initialized as Infiniband ports.
+ By default both ConnectX ports are initialized as InfiniBand ports.
If you wish to change the port type use the connectx_port_config script after
the driver is loaded.
Running "/sbin/connectx_port_config -s" will show current port configuration
Possible port types are:
"eth" - Always Ethernet.
- "ib" - Always Infiniband.
+ "ib" - Always InfiniBand.
"auto" - Link sensing mode - detect port type based on the attached
network type. If no link is detected, the driver retries link
sensing every few seconds.
===============================================================================
-4. Infiniband new features and bug fixes since OFED 1.3.1
+4. InfiniBand new features and bug fixes since OFED 1.3.1
===============================================================================
-Features that are enabled with FW 2.5.0 only:
+Features that are enabled with ConnectX firmware 2.5.0 only:
- Send with invalidate and Local invalidate send queue work requests.
- Resize CQ support.
-Features that are enabled with FW 2.6.0 only:
+Features that are enabled with ConnectX firmware 2.6.0 only:
- Fast register MR send queue work requests.
- Local DMA L_Key.
- Raw Ethertype QP support (one QP per port) -- receive only.
-Non FW dependent features:
-- Allow 4K messages for UD QPs.
-- Allocate/free fast register MR page lists.
-- More efficient MTT allocator.
-- RESET->ERR QP state transition no longer supported (IB Spec 1.2.1).
-- Pass congestion management class MADs to the HCA.
-- Enable firmware diagnostic counters available via sysfs.
-- Enable LSO support for IPOIB.
-- IB_EVENT_LID_CHANGE is generated more appropriately.
+Non-firmware dependent features:
+- Allow 4K messages for UD QPs
+- Allocate/free fast register MR page lists
+- More efficient MTT allocator
+- RESET->ERR QP state transition no longer supported (IB Spec 1.2.1)
+- Pass congestion management class MADs to the HCA
+- Enable firmware diagnostic counters available via sysfs
+- Enable LSO support for IPOIB
+- IB_EVENT_LID_CHANGE is generated more appropriately
- Fixed race condition between create QP and destroy QP (bugzilla 1389)
===============================================================================
-5. Infiniband new features and bug fixes since OFED 1.4
+5. InfiniBand new features and bug fixes since OFED 1.4
===============================================================================
- Enable setting via module param (set_4k_mtu) 4K MTU for ConnectX ports.
- Support optimized registration of huge pages backed memory.
- Masked Compare and Swap (MskCmpSwap)
The MskCmpSwap atomic operation is an extension to the CmpSwap operation
defined in the IB spec. MskCmpSwap allows the user to select a portion of the
- 64 bit target data for the “compare” check as well as to restrict the swap to a
- (possibly different) portion.
+ 64 bit target data for the "compare" check as well as to restrict the swap to
+ a (possibly different) portion.
- Masked Fetch and Add (MFetchAdd)
The MFetchAdd Atomic operation extends the functionality of the standard IB
FetchAdd by allowing the user to split the target into multiple fields of
selectable length. The atomic add is done independently on each one of this
fields. A bit set in the field_boundary parameter specifies the field
boundaries.
-- Improved vlan tagging performance for the mlx4_en driver.
+- Improved VLAN tagging performance for the mlx4_en driver.
- RSS support for Ethernet UDP traffic on ConnectX-2 cards with firmware
2.7.700 and higher.
-------------
- Bonding stops functioning when one of the Ethernet ports is closed.
- "Scheduling while atomic" errors in /var/log/messages when working with
- bonding and mlx4_en drivers in several OS's.
+ bonding and mlx4_en drivers in several operating systems.
===============================================================================
10. Known Issues
===============================================================================
-- The SQD feature is not supported:
-- To load the driver on machines with 64KB default page size UAR bar must be
- enlarged. 64KB page size is the default of PPC with RHEL5 and Itanium with
- SLES 11 or when 64KB page size enabled.
+- The SQD feature is not supported
+- To load the driver on machines with a 64KB default page size, the UAR bar
+ must be enlarged. 64KB page size is the default of PPC with RHEL5 and Itanium
+ with SLES 11 or when 64KB page size enabled.
Perform the following three steps:
1. Add the following line in the firmware configuration (INI) file under the
[HCA] section:
3. Reboot the system.
-===============================================================================
-11. mlx4 Available Parameters
-===============================================================================
+================================================================================
+11. mlx4 available parameters
+================================================================================
In order to set mlx4 parameters, add the following line(s) to /etc/modpobe.conf:
options mlx4_core parameter=<value>
and/or
options mlx4_en parameter=<value>
mlx4_core parameters:
- set_4k_mtu: attempt to set 4K MTU to all ConnectX ports (int)
- debug_level: Enable debug tracing if > 0 (int)
- block_loopback: Block multicast loopback packets if > 0 (int)
- msi_x: attempt to use MSI-X if nonzero (int)
- log_num_mac: Log2 max number of MACs per ETH port (1-7) (int)
- use_prio: Enable steering by VLAN priority on ETH ports (0/1, default 0)
- (bool)
- log_num_qp: log maximum number of QPs per HCA (int)
- log_num_srq: log maximum number of SRQs per HCA (int)
- log_rdmarc_per_qp: log number of RDMARC buffers per QP (int)
- log_num_cq: log maximum number of CQs per HCA (int)
- log_num_mcg: log maximum number of multicast groups per HCA (int)
- log_num_mpt: log maximum number of memory protection table entries per HCA
- (int)
- log_num_mtt: log maximum number of memory translation table segments per HCA
- (int)
- log_mtts_per_seg: Log2 number of MTT entries per segment (1-5) (int)
- enable_qos: Enable Quality of Service support in the HCA (default: off)
- (bool)
- internal_err_reset: Reset device on internal errors if non-zero
- (default 1) (int)
+ set_4k_mtu: try to set 4K MTU to all ConnectX ports (int)
+ debug_level: enable debug tracing if > 0 (int)
+ block_loopback: block multicast loopback packets if > 0 (int)
+ msi_x: attempt to use MSI-X if nonzero (int)
+ log_num_mac: log2 max number of MACs per ETH port (1-7, int)
+ use_prio: enable steering by VLAN priority on ETH ports
+ (0/1, default 0) (bool)
+ log_num_qp: log maximum number of QPs per HCA (int)
+ log_num_srq: log maximum number of SRQs per HCA (int)
+ log_rdmarc_per_qp: log number of RDMARC buffers per QP (int)
+ log_num_cq: log maximum number of CQs per HCA (int)
+ log_num_mcg: log maximum number of multicast groups per HCA
+ (int)
+ log_num_mpt: log maximum number of memory protection table
+ entries per HCA (int)
+ log_num_mtt: log maximum number of memory translation table
+ segments per HCA (int)
+ log_mtts_per_seg: log2 number of MTT entries per segment (1-5)
+ (int)
+ enable_qos: enable Quality of Service support in the HCA
+ (default: off) (bool)
+internal_err_reset: reset device on internal errors if non-zero
+ (default 1) (int)
mlx4_ib parameters:
- debug_level: Enable debug tracing if > 0 (default 0)
+ debug_level: enable debug tracing if > 0 (default 0)
mlx4_en parameters:
- udp_rss: Enable RSS for incoming UDP traffic or disabled (0)
- tcp_rss: Enable RSS for incoming TCP traffic or disabled (0)
- num_lro: Number of LRO sessions per ring or disabled (0) (default is 32)
- ip_reasm: Allow reassembly of fragmented IP packets (default is enabled)
- pfctx: Priority based Flow Control policy on TX[7:0].
- Per priority bit mask (default is 0)
- pfcrx: Priority based Flow Control policy on RX[7:0].
- Per priority bit mask (default is 0)
- inline_thold: threshold for using inline data (default is 128)
+ udp_rss: enable RSS for incoming UDP traffic or disabled (0)
+ tcp_rss: enable RSS for incoming TCP traffic or disabled (0)
+ num_lro: number of LRO sessions per ring or disabled (0)
+ (default is 32)
+ ip_reasm: allow reassembly of fragmented IP packets (default
+ is enabled)
+ pfctx: priority based Flow Control policy on TX[7:0]
+ per priority bit mask (default is 0)
+ pfcrx: priority based Flow Control policy on RX[7:0]
+ per priority bit mask (default is 0)
+ inline_thold: threshold for using inline data (default is 128)
projects / ~aditr / compat-rdma-docs.git / commitdiff