--- /dev/null
+ Open Fabrics Enterprise Distribution (OFED)
+ CHELSIO T4 RNIC RELEASE NOTES
+ November 2011
+
+
+The iw_cxgb4 and cxgb4 modules provide RDMA and NIC support for the
+Chelsio T4 series adapters. Make sure you choose the 'cxgb4' and
+'libcxgb4' options when generating your OFED rpms.
+
+============================================
+New for OFED-1.5.4
+============================================
+
+- Added Chelsio T4 iWARP driver and library.
+
+==========================
+Supported Operating System
+==========================
+
+ o CPU architectures:
+ - x86_64
+ - x86
+
+ o Linux Operating Systems:
+ - RedHat EL5 up6 2.6.18-238.el5
+ - RedHat EL5 up7 2.6.18-274.el5
+ - RedHat EL6 2.6.32-71.el6
+ - RedHat EL6.1 2.6.32-131.0.15.el6
+ - SLES10 SP4 2.6.16.60-0.85.1-smp
+ - SLES11 SP1 2.6.32.12-0.7-default
+ - kernel.org 2.6.30 - 3.0 *
+
+ * Minimal QA for these versions
+
+===============
+Supported Cards
+================
+
+- T420-CR
+- T440-CR
+- T404-BT
+- T422-CR
+
+=================
+Updating Firmware
+=================
+This driver requires released firmware for T4 card. This driver has been tested
+with t4fw-1.3.10.0 and later firmware. This firmware is available at
+service.chelsio.com inside the driver package.
+
+If your distro/kernel cxgb4 driver supports firmware loading, you can place
+the chelsio firmware image in /lib/firmware/cxgb4, then unload and reload the
+cxgb4 module to get the new images loaded. If this does not work, then you can
+load the firmware images manually as follows:
+
+Obtain the cxgbtool tool from service.chelsio.com. This tool is available
+inside driver package.
+
+To build cxgbtool:
+
+# cd <path-to-cxgbtool>
+# make && make install
+
+Then build and load the released cxgb4 driver available at service.chelsio.com.
+
+# cd <path-to-driver-package>
+# make && make install
+# modprobe cxgb4
+
+Now note the ethernet interface name for the T4 device. This can be
+done by typing 'ifconfig -a' and noting the interface name for the
+interface with a HW address that begins with "00:07:43".
+
+# cxgbtool ethxx loadfw <firmware_file>
+# reboot
+
+NOTE: Driver package on the service.chelsio.com is getting updated
+periodically. Along with driver package T4 firmware will also get updated.
+It is recommended to use latest available firmware at service.chelsio.com.
+This driver should work with latest available firmware. In case of any issue
+please contact Chelsio support at support@chelsio.com.
+
+==============================
+Setting shell for Remote Login
+==============================
+User needs to set up authentication on the user account on all systems in the
+cluster to allow user to remotely logon or executing commands without
+password.
+
+Quick steps to set up user authentication:
+- Change to user home directory
+# cd
+- Generate authentication key
+# ssh-keygen -t rsa
+- Press enter upon prompting to accept default setup
+and empty password phrase.
+- Create authorization file
+# cd .ssh
+# cat *.pub > authorized_keys
+# chmod 600 authorized_keys
+- Copy directory .ssh to all systems in the cluster
+# cd
+# scp -r .ssh remotehostname-or-ipaddress:
+
+======================
+Enabling MPA version 2
+======================
+We can enable MPA version 2 by setting iw_cxgb4 module parameter as shown below
+# modprobe iw_cxgb4 mpa_rev=2
+
+MPA v2 is an enahanced RDMA connection establishment. More details are available at
+http://tools.ietf.org/html/draft-ietf-storm-mpa-peer-connect-08 location.
+
+============================================
+Enabling Intel and Platform MPI
+============================================
+
+For Intel MPI and Platform MPI: you must set the iw_cxgb4
+module option peer2peer=1 on all systems. This can be done by writing
+to the /sys/module file system during boot. EG:
+
+# echo 1 > /sys/module/iw_cxgb4/parameters/peer2peer
+
+Or you can add the following line to /etc/modprobe.conf to set the option
+at module load time:
+
+options iw_cxgb4 peer2peer=1
+
+For Intel MPI and Platform MPI: Enable the chelsio device by adding
+an entry to /etc/dat.conf for the chelsio interface. For instance,
+if your chelsio interface name is eth2, then the following line adds
+a DAT version 1.2 and 2.0 devices named "chelsio" and "chelsio2" for
+that interface:
+
+chelsio u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "eth2 0" ""
+chelsio2 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 0" ""
+
+=============
+Intel MPI:
+=============
+Download latest Intel MPI from the Intel website
+
+Copy COM_L___CF8J-98P6MBWL.lic into l_mpi_p_x.y.z directory
+
+Create machines.LINUX (list of node names) in l_mpi_p_x.y.z
+
+Install software on every node.
+#./install.sh
+
+Register and set IntelMPI with mpi-selector (do this on all nodes).
+#mpi-selector --register intelmpi --source-dir /opt/intel/impi/3.1/bin/
+#mpi-selector --set intelmpi
+
+Edit .bashrc and add these lines:
+
+export RSH=ssh
+export DAPL_MAX_INLINE=64
+export I_MPI_DEVICE=rdssm:chelsio
+export MPIEXEC_TIMEOUT=180
+export MPI_BIT_MODE=64
+
+Logout & log back in.
+
+Populate mpd.hosts with node names.
+Note: The hosts in this file should be Chelsio interface IP addresses.
+
+NOTE: I_MPI_DEVICE=rdssm:chelsio assumes you have an entry in
+/etc/dat.conf named "chelsio".
+
+NOTE: MPIEXEC_TIMEOUT value might be required to increase if heavy traffic
+is going across the systems.
+
+Contact Intel for obtaining their MPI with DAPL support.
+
+To run Intel MPI applications:
+
+#mpdboot -n <num nodes> -r ssh --ncpus=<num cpus>
+#mpiexec -ppn <process per node> -n <num nodes> <MPI Application Path>
+
+=============
+OpenMPI:
+=============
+
+OpenMPI iWARP support is only available in OpenMPI version 1.3 or greater.
+
+Open MPI will work without any specific configuration via the openib btl.
+Users wishing to performance tune the configurable options may wish to
+inspect the receive queue values. Those can be found in the "Chelsio T4"
+section of mca-btl-openib-hca-params.ini.
+
+To run OpenMPI applications:
+
+ mpirun --host <node1>,<node2> -mca btl openib,sm,self <OpenMPI Application Path>
+
+=============
+MVAPICH2:
+=============
+
+The following env vars enable MVAPICH2 version 1.4-2. Place these
+in your user env after installing and setting up MVAPICH2 MPI:
+
+export MVAPICH2_HOME=/usr/mpi/gcc/mvapich2-1.4/
+export MV2_USE_IWARP_MODE=1
+export MV2_USE_RDMA_CM=1
+
+On each node, add this to the end of /etc/profile.
+
+ ulimit -l 999999
+
+On each node, add this to the end of /etc/init.d/sshd and restart sshd.
+
+ ulimit -l 999999
+ % service sshd restart
+
+Verify the ulimit changes worked. These should show '999999':
+
+ % ulimit -l
+ % ssh <peer> ulimit -l
+
+Note: You may have to restart sshd a few times to get it to work.
+
+Create mpd.hosts with list of hostname or ipaddrs in the cluster. They
+should be names/addresses that you can ssh to without passwords. (See
+Passwordless SSH Setup).
+
+On each node, create /etc/mv2.conf with a single line containing the
+IP address of the local T4 interface. This is how MVAPICH2 picks which
+interface to use for RDMA traffic.
+
+On each node, edit /etc/hosts file. Comment the entry if there is an
+entry with 127.0.0.1 IP Address and local host name. Add an entry for
+corporate IP address and local host name (name that you have given in
+mpd.hosts file) in /etc/hosts file.
+
+To run MVAPICH2 application:
+
+ mpirun_rsh -ssh -np 8 -hostfile mpd.hosts <MVAPICH2 Application Path>
+
+============================================
+Testing connectivity with ping and rping:
+============================================
+
+Configure the ethernet interfaces for your T4 device. After you
+modprobe iw_cxgb4 you will see ethernet interfaces for the T4 device.
+Configure them with an appropriate ip address, netmask, etc.
+You can use the Linux ping command to test basic connectivity via the
+T4 interface.
+
+To test RDMA, use the rping command that is included in the librdmacm-utils
+rpm:
+
+On the server machine:
+
+# rping -s -a 0.0.0.0 -p 9999
+
+On the client machine:
+
+# rping -c -VvC10 -a server_ip_addr -p 9999
+
+You should see ping data like this on the client:
+
+ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
+ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
+ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
+ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
+ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv
+ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw
+ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx
+ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy
+ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
+ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA
+client DISCONNECT EVENT...
+
projects / compat-rdma / docs.git / commitdiff