From 2b02493c3bae90e4215f7cbf64b8380d7a817424 Mon Sep 17 00:00:00 2001 From: Tatyana Nikolova Date: Tue, 11 Dec 2012 11:17:00 -0600 Subject: [PATCH] RDMA/nes: Updated release notes Signed-off-by: Tatyana Nikolova --- release_notes/nes_release_notes.txt | 216 ++++++++++------------------ 1 file changed, 74 insertions(+), 142 deletions(-) diff --git a/release_notes/nes_release_notes.txt b/release_notes/nes_release_notes.txt index bb750ff..e6f1ff6 100644 --- a/release_notes/nes_release_notes.txt +++ b/release_notes/nes_release_notes.txt @@ -1,43 +1,39 @@ Open Fabrics Enterprise Distribution (OFED) NetEffect Ethernet Cluster Server Adapter Release Notes - January 2012 - + December 2012 The iw_nes module and libnes user library provide RDMA and L2IF support for the NetEffect Ethernet Cluster Server Adapters. +============================================= +Supported Architectures and Operating Systems +============================================= + +* CPU architectures: + - x86_64 + - x86 + +* Linux Operating Systems: + - RedHat EL6.2 2.6.32-220.el6 + - RedHat EL6.3 2.6.32-279.el6 + - SLES11 SP2 3.0.13-0.27-default + - kernel.org 3.5 (minimal QA for this version) + ========== What's New ========== -OFED 1.5.4.1 contains bug fixes for iw_nes driver. - -* Fixed a problem with QP destroy timer and improved AE handling. -* Fixed a problem with sending MPA reject message. -* Fixed fast memory registration issues. - -OFED 1.5.4 contains several enhancements and bug fixes to iw_nes driver. - -* Added backports for 2.6.35 to 3.0 kernels. -* Fixed a couple of problems which caused IMA to crash. -* Fixed a problem with VLAN flag for IMA. -* Enabled bonding with iw_nes. -* Fixed a couple of IB_EVENT issues. -* Fixed an SFP+ link status issue. -* Added support for Chelsio Interoperability. -* Added support for MPA version 2. - - -============================================ -Required Setting - RDMA Unify TCP port space -============================================ -RDMA connections use the same TCP port space as the host stack. To avoid -conflicts, set rdma_cm module option unify_tcp_port_space to 1 by adding -the following to /etc/modprobe.conf: - - options rdma_cm unify_tcp_port_space=1 +OFED 3.5 contains several important bug fixes for iw_nes driver: + - Removed an incorrect macro routine from the loopback code. + - Fixed a couple of problems when unlinking skbs (interoperability with Chelsio). + - Added a lock, so that FPDUs are forwarded in order to iw_nes hardware (interoperability with Chelsio). + - Fixed an incorrect setting of a multicast address in the perfect filter table. + - Fixed a problem when adding timer and a timer is already pending. + - Fixed a problem when initializing terminate timer more than once. + - Added backports for Linux 2.6 and 3.0 kernels + - iw_nes driver version is changed to 1.5.0.1 ======================================== Required Setting - Power Management Mode @@ -51,7 +47,7 @@ If possible, disable Active State Power Management in the BIOS, e.g.: Loadable Module Options ======================= The following options can be used when loading the iw_nes module by modifying -/etc/modprobe.conf file. +/etc/modprobe.conf.local file in the SLES hosts and /etc/modprobe.d/rnic.conf file in the RHEL hosts. wide_ppm_offset=0 Set to 1 will increase CX4 interface clock ppm offset to 300ppm. @@ -76,7 +72,7 @@ nes_drv_opt=0x00000100 0x00000200 - Disable Virtual Work Queue 0x00001000 - Disable extra doorbell read after write -nes_debug_level=0 +debug_level=0 Specify debug output level. wqm_quanta=65536 @@ -85,6 +81,8 @@ wqm_quanta=65536 limit_maxrdreqsz=0 Limit PCI read request size to 256 bytes. +nes_lro_max_aggr=64 + Max number of LRO packets to be aggregated. =============== Runtime Options @@ -92,30 +90,25 @@ Runtime Options The following options can be used to alter the behavior of the iw_nes module: NOTE: Assuming NetEffect Ethernet Cluster Server Adapter is assigned eth2. - ifconfig eth2 mtu 9000 - largest mtu supported + ifconfig eth2 mtu 9000 - largest mtu supported - ethtool -K eth2 tso on - enables TSO - ethtool -K eth2 tso off - disables TSO + ethtool -K eth2 tso on - enables TSO (Operation not supported for rhel6.2/rhel6.3) + ethtool -K eth2 tso off - disables TSO (Operation not supported for rhel6.2/rhel6.3) - ethtool -C eth2 rx-usecs-irq 128 - set static interrupt moderation + ethtool -C eth2 rx-usecs-irq 128 - set static interrupt moderation - ethtool -C eth2 adaptive-rx on - enable dynamic interrupt moderation - ethtool -C eth2 adaptive-rx off - disable dynamic interrupt moderation - ethtool -C eth2 rx-frames-low 16 - low watermark of rx queue for dynamic - interrupt moderation - ethtool -C eth2 rx-frames-high 256 - high watermark of rx queue for - dynamic interrupt moderation - ethtool -C eth2 rx-usecs-low 40 - smallest interrupt moderation timer - for dynamic interrupt moderation - ethtool -C eth2 rx-usecs-high 1000 - largest interrupt moderation timer - for dynamic interrupt moderation + ethtool -C eth2 adaptive-rx on - enable dynamic interrupt moderation + ethtool -C eth2 adaptive-rx off - disable dynamic interrupt moderation + ethtool -C eth2 rx-frames-low 16 - low watermark of rx queue for dynamic interrupt moderation + ethtool -C eth2 rx-frames-high 256 - high watermark of rx queue for dynamic interrupt moderation + ethtool -C eth2 rx-usecs-low 40 - smallest interrupt moderation timer for dynamic interrupt moderation + ethtool -C eth2 rx-usecs-high 1000 - largest interrupt moderation timer for dynamic interrupt moderation =================== uDAPL Configuration =================== Rest of the document assumes the following uDAPL settings in /etc/dat.conf: - OpenIB-iwarp u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "eth2 0" "" ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 0" "" ======================== @@ -139,29 +132,20 @@ This operational mode can cause 100% CPU utilization. To switch to Event Driven mode and lower CPU utilization please refer to README or Release Notes for specific application. -============================================== +======================================== Recommended Settings for Intel MPI 4.0.x -============================================== +======================================== Add the following to mpiexec command: -genv I_MPI_FALLBACK_DEVICE 0 -genv I_MPI_DEVICE rdma:ofa-v2-iwarp - -genv I_MPI_USE_RENDEZVOUS_RDMA_WRITE 1 Example mpiexec command line for uDAPL-2.0: mpiexec -genv I_MPI_FALLBACK_DEVICE 0 -genv I_MPI_DEVICE rdma:ofa-v2-iwarp - -genv I_MPI_USE_RENDEZVOUS_RDMA_WRITE 1 - -ppn 1 -n 2 - /opt/intel/impi/4.0.0.025/bin64/IMB-MPI1 - -Example mpiexec command line for uDAPL-1.2: - mpiexec -genv I_MPI_FALLBACK_DEVICE 0 - -genv I_MPI_DEVICE rdma:OpenIB-iwarp - -genv I_MPI_USE_RENDEZVOUS_RDMA_WRITE 1 -ppn 1 -n 2 - /opt/intel/impi/4.0.0.025/bin64/IMB-MPI1 + /opt/intel/impi/4.0.3.008/bin64/IMB-MPI1 Intel MPI use CQ Polling mode as a default. To switch to wait mode add the following to mpiexec command: @@ -172,124 +156,72 @@ NOTE: Wait mode supports the sock device only. ======================================== Recommended Setting for MVAPICH2 and OFA ======================================== -Example mpirun_rsh command line: +From OFED-3.5 package MVAPICH2 is not a part of the OFED - mpirun_rsh -ssh -np 2 -hostfile /root/mpd.hosts - /usr/mpi/gcc/mvapich2-1.7/tests/osu_benchmarks-3.1.1/osu_latency +Download mvapich2-1.8.1 package from following location: + +http://mvapich.cse.ohio-state.edu/download/mvapich2/ +Example mpirun_rsh command line: + + mpirun_rsh -ssh -np 2 -hostfile /root/mpd.hosts + /usr/mpi/gcc/mvapich2-1.8.1/tests/osu_benchmarks-3.1.1/osu_latency + MVAPICH2 use CQ Polling mode as a default. To switch to Blocking mode add the following to mpirun_rsh command: MV2_USE_BLOCKING=1 - + ========================================== Recommended Setting for MVAPICH2 and uDAPL ========================================== Add the following to the mpirun_rsh command for 64 or more processes: - + -env MV2_ON_DEMAND_THRESHOLD - + Example mpirun_rsh command with uDAPL-2.0: - + mpirun_rsh -ssh -np 64 -hostfile /root/mpd.hosts MV2_DAPL_PROVIDER=ofa-v2-iwarp MV2_ON_DEMAND_THRESHOLD=64 - /usr/mpi/gcc/mvapich2-1.7/tests/IMB-3.2/IMB-MPI1 - -Example mpirun_rsh command with uDAPL-1.2: - - mpirun_rsh -ssh -np 64 -hostfile /root/mpd.hosts - MV2_DAPL_PROVIDER=OpenIB-iwarp - MV2_ON_DEMAND_THRESHOLD=64 - /usr/mpi/gcc/mvapich2-1.7/tests/IMB-3.2/IMB-MPI1 - + /usr/mpi/gcc/mvapich2-1.8.1/tests/IMB-3.2/IMB-MPI1 + MVAPICH2 use CQ Polling mode as a default. To switch to Blocking mode add the following to mpirun_rsh command: MV2_USE_BLOCKING=1 + +======================================= +Recommended Settings for Open MPI 1.6.1 +======================================= + +From OFED-3.5 package OpenMPI is not a part of the OFED + +Download openmpi-1.6.1-1 package from following location: + +http://www.open-mpi.org/software/ompi/v1.6/ -=========================== -Modify Settings in Open MPI -=========================== There is more than one way to specify MCA parameters in Open MPI. Please visit this link and use the best method for your environment: http://www.open-mpi.org/faq/?category=tuning#setting-mca-params -======================================= -Recommended Settings for Open MPI 1.4.3 -======================================= -Allow the sender to use RDMA Writes: - - -mca btl_openib_flags 2 - Example mpirun command line: mpirun -np 2 -hostfile /opt/mpd.hosts -mca btl openib,self,sm -mca btl_mpi_leave_pinned 0 - -mca btl_openib_flags 2 - /usr/mpi/gcc/openmpi-1.4.3/tests/IMB-3.2/IMB-MPI1 - + /usr/mpi/gcc/openmpi-1.6.1/tests/IMB-3.2/IMB-MPI1 + OpenMPI use CQ Polling mode as a default. -No command parameter available to swith to Event Driven mode. - -=================================== -iWARP Multicast Acceleration (IMA) -=================================== - -iWARP multicast acceleration enables raw L2 multicast traffic kernel -bypass using user-space verbs API using the new defined QP type -IBV_QPT_RAW_ETH. - -The L2 RAW_ETH acceleration assumes that user application transmits and -receives a whole L2 frame including MAC/IP/UDP/TCP headers. - -ETH RAW QP usage: -First the application creates IBV_QPT_RAW_ETH QP with associated CQ, PD, -completion channels as it is performed for RDMA connection. - -Next step is enabling L2 MAC address RX filters for directing received -multicasts to the RAW_ETH QPs using ibv_attach_multicast() verb. - -From this point the application is ready to receive and transmit multicast -traffic. +No command parameter available to switch to Event Driven mode. -In multicast acceleration the user application passes to ibv_post_send() -whole IGMP frame including MAC header, IP header, UDP header and UDP payload. -It is a user responsibility to make IP fragmentation when required payload -is larger than MTU. Every fragment is a separate L2 frame to transmit. -The ibv_poll_cq() provides an information about the status of transmit buffer. -On receive path, ibv_poll_cq() returns information about received L2 -packet, the Rx buffer (previously posted by ibv_post_recv() ) contains -whole L2 frame including MAC header, IP header and UDP header. -It is a user application responsibility to check if received packet is -a valid UDP frame so the fragments must be checked and checksums must be -computed. - -IMA API description (NE020 specific): -User application must create separate CQs for RX and TX path. -Only single SGE on tranmit is supported. -User application must post at least 65 rx buffers to keep RX path working. - -IMA device: -IMA requires creation of the /dev/infiniband/nes_ud_sksq device to get -access to optimized IMA transmit path. The best method for creation of this -device is manual addition following line to /etc/udev/rules.d/90-ib.rules -file after OFED distribution installation and rebooting machine. - -KERNEL=="nes_ud_sksq", NAME="infiniband/%k", MODE="0644" - -As a result the 90-ib.rules should look like: - -KERNEL=="umad*", NAME="infiniband/%k" -KERNEL=="issm*", NAME="infiniband/%k" -KERNEL=="ucm*", NAME="infiniband/%k", MODE="0666" -KERNEL=="uverbs*", NAME="infiniband/%k", MODE="0666" -KERNEL=="ucma", NAME="infiniband/%k", MODE="0666" -KERNEL=="rdma_cm", NAME="infiniband/%k", MODE="0666" -KERNEL=="nes_ud_sksq", NAME="infiniband/%k", MODE="0644" +================================== +Intel Multicast Acceleration (IMA) +================================== +OFED-3.5 doesn't support Intel Multicast Acceleration (IMA) functionality. NetEffect is a trademark of Intel Corporation in the U.S. and other countries. + -- 2.41.0