From 84cb328fd87ad6e10d2096030ed10c56e97af3d9 Mon Sep 17 00:00:00 2001 From: Vipul Pandya Date: Mon, 21 Nov 2011 19:49:58 +0530 Subject: [PATCH] Add cxgb4 Release notes Signed-off-by: Vipul Pandya --- release_notes/cxgb4_release_notes.txt | 273 ++++++++++++++++++++++++++ 1 file changed, 273 insertions(+) create mode 100644 release_notes/cxgb4_release_notes.txt diff --git a/release_notes/cxgb4_release_notes.txt b/release_notes/cxgb4_release_notes.txt new file mode 100644 index 0000000..802a530 --- /dev/null +++ b/release_notes/cxgb4_release_notes.txt @@ -0,0 +1,273 @@ + Open Fabrics Enterprise Distribution (OFED) + CHELSIO T4 RNIC RELEASE NOTES + November 2011 + + +The iw_cxgb4 and cxgb4 modules provide RDMA and NIC support for the +Chelsio T4 series adapters. Make sure you choose the 'cxgb4' and +'libcxgb4' options when generating your OFED rpms. + +============================================ +New for OFED-1.5.4 +============================================ + +- Added Chelsio T4 iWARP driver and library. + +========================== +Supported Operating System +========================== + + o CPU architectures: + - x86_64 + - x86 + + o Linux Operating Systems: + - RedHat EL5 up6 2.6.18-238.el5 + - RedHat EL5 up7 2.6.18-274.el5 + - RedHat EL6 2.6.32-71.el6 + - RedHat EL6.1 2.6.32-131.0.15.el6 + - SLES10 SP4 2.6.16.60-0.85.1-smp + - SLES11 SP1 2.6.32.12-0.7-default + - kernel.org 2.6.30 - 3.0 * + + * Minimal QA for these versions + +=============== +Supported Cards +================ + +- T420-CR +- T440-CR +- T404-BT +- T422-CR + +================= +Updating Firmware +================= +This driver requires released firmware for T4 card. This driver has been tested +with t4fw-1.3.10.0 and later firmware. This firmware is available at +service.chelsio.com inside the driver package. + +If your distro/kernel cxgb4 driver supports firmware loading, you can place +the chelsio firmware image in /lib/firmware/cxgb4, then unload and reload the +cxgb4 module to get the new images loaded. If this does not work, then you can +load the firmware images manually as follows: + +Obtain the cxgbtool tool from service.chelsio.com. This tool is available +inside driver package. + +To build cxgbtool: + +# cd +# make && make install + +Then build and load the released cxgb4 driver available at service.chelsio.com. + +# cd +# make && make install +# modprobe cxgb4 + +Now note the ethernet interface name for the T4 device. This can be +done by typing 'ifconfig -a' and noting the interface name for the +interface with a HW address that begins with "00:07:43". + +# cxgbtool ethxx loadfw +# reboot + +NOTE: Driver package on the service.chelsio.com is getting updated +periodically. Along with driver package T4 firmware will also get updated. +It is recommended to use latest available firmware at service.chelsio.com. +This driver should work with latest available firmware. In case of any issue +please contact Chelsio support at support@chelsio.com. + +============================== +Setting shell for Remote Login +============================== +User needs to set up authentication on the user account on all systems in the +cluster to allow user to remotely logon or executing commands without +password. + +Quick steps to set up user authentication: +- Change to user home directory +# cd +- Generate authentication key +# ssh-keygen -t rsa +- Press enter upon prompting to accept default setup +and empty password phrase. +- Create authorization file +# cd .ssh +# cat *.pub > authorized_keys +# chmod 600 authorized_keys +- Copy directory .ssh to all systems in the cluster +# cd +# scp -r .ssh remotehostname-or-ipaddress: + +====================== +Enabling MPA version 2 +====================== +We can enable MPA version 2 by setting iw_cxgb4 module parameter as shown below +# modprobe iw_cxgb4 mpa_rev=2 + +MPA v2 is an enahanced RDMA connection establishment. More details are available at +http://tools.ietf.org/html/draft-ietf-storm-mpa-peer-connect-08 location. + +============================================ +Enabling Intel and Platform MPI +============================================ + +For Intel MPI and Platform MPI: you must set the iw_cxgb4 +module option peer2peer=1 on all systems. This can be done by writing +to the /sys/module file system during boot. EG: + +# echo 1 > /sys/module/iw_cxgb4/parameters/peer2peer + +Or you can add the following line to /etc/modprobe.conf to set the option +at module load time: + +options iw_cxgb4 peer2peer=1 + +For Intel MPI and Platform MPI: Enable the chelsio device by adding +an entry to /etc/dat.conf for the chelsio interface. For instance, +if your chelsio interface name is eth2, then the following line adds +a DAT version 1.2 and 2.0 devices named "chelsio" and "chelsio2" for +that interface: + +chelsio u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "eth2 0" "" +chelsio2 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 0" "" + +============= +Intel MPI: +============= +Download latest Intel MPI from the Intel website + +Copy COM_L___CF8J-98P6MBWL.lic into l_mpi_p_x.y.z directory + +Create machines.LINUX (list of node names) in l_mpi_p_x.y.z + +Install software on every node. +#./install.sh + +Register and set IntelMPI with mpi-selector (do this on all nodes). +#mpi-selector --register intelmpi --source-dir /opt/intel/impi/3.1/bin/ +#mpi-selector --set intelmpi + +Edit .bashrc and add these lines: + +export RSH=ssh +export DAPL_MAX_INLINE=64 +export I_MPI_DEVICE=rdssm:chelsio +export MPIEXEC_TIMEOUT=180 +export MPI_BIT_MODE=64 + +Logout & log back in. + +Populate mpd.hosts with node names. +Note: The hosts in this file should be Chelsio interface IP addresses. + +NOTE: I_MPI_DEVICE=rdssm:chelsio assumes you have an entry in +/etc/dat.conf named "chelsio". + +NOTE: MPIEXEC_TIMEOUT value might be required to increase if heavy traffic +is going across the systems. + +Contact Intel for obtaining their MPI with DAPL support. + +To run Intel MPI applications: + +#mpdboot -n -r ssh --ncpus= +#mpiexec -ppn -n + +============= +OpenMPI: +============= + +OpenMPI iWARP support is only available in OpenMPI version 1.3 or greater. + +Open MPI will work without any specific configuration via the openib btl. +Users wishing to performance tune the configurable options may wish to +inspect the receive queue values. Those can be found in the "Chelsio T4" +section of mca-btl-openib-hca-params.ini. + +To run OpenMPI applications: + + mpirun --host , -mca btl openib,sm,self + +============= +MVAPICH2: +============= + +The following env vars enable MVAPICH2 version 1.4-2. Place these +in your user env after installing and setting up MVAPICH2 MPI: + +export MVAPICH2_HOME=/usr/mpi/gcc/mvapich2-1.4/ +export MV2_USE_IWARP_MODE=1 +export MV2_USE_RDMA_CM=1 + +On each node, add this to the end of /etc/profile. + + ulimit -l 999999 + +On each node, add this to the end of /etc/init.d/sshd and restart sshd. + + ulimit -l 999999 + % service sshd restart + +Verify the ulimit changes worked. These should show '999999': + + % ulimit -l + % ssh ulimit -l + +Note: You may have to restart sshd a few times to get it to work. + +Create mpd.hosts with list of hostname or ipaddrs in the cluster. They +should be names/addresses that you can ssh to without passwords. (See +Passwordless SSH Setup). + +On each node, create /etc/mv2.conf with a single line containing the +IP address of the local T4 interface. This is how MVAPICH2 picks which +interface to use for RDMA traffic. + +On each node, edit /etc/hosts file. Comment the entry if there is an +entry with 127.0.0.1 IP Address and local host name. Add an entry for +corporate IP address and local host name (name that you have given in +mpd.hosts file) in /etc/hosts file. + +To run MVAPICH2 application: + + mpirun_rsh -ssh -np 8 -hostfile mpd.hosts + +============================================ +Testing connectivity with ping and rping: +============================================ + +Configure the ethernet interfaces for your T4 device. After you +modprobe iw_cxgb4 you will see ethernet interfaces for the T4 device. +Configure them with an appropriate ip address, netmask, etc. +You can use the Linux ping command to test basic connectivity via the +T4 interface. + +To test RDMA, use the rping command that is included in the librdmacm-utils +rpm: + +On the server machine: + +# rping -s -a 0.0.0.0 -p 9999 + +On the client machine: + +# rping -c -VvC10 -a server_ip_addr -p 9999 + +You should see ping data like this on the client: + +ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr +ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs +ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst +ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu +ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv +ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw +ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx +ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy +ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz +ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA +client DISCONNECT EVENT... + -- 2.41.0