Open Fabrics Enterprise Distribution (OFED)
- Tips for Working with OFED 1.5.x
+ Tips for Working with OFED 3.5
- December 2011
+ August 2012
===============================================================================
Table of Contents
1. OFED Utilities
2. Debug HOWTOs
3. Pinning (Locking) User Memory Pages
-4. External Module Compilation Over OFED-1.5.x
+4. External Module Compilation Over OFED-3.5
5. Adding/Deleting a patch to OFED package
6. Adding vendor specific actions to the installation of OFED
7. How to compile OFED sources manually
Examples:
ibstat -l # list all IB devices
- ibstat mthca0 2 # stat port 2 of mthca0
+ ibstat mlx4_0 2 # stat port 2 of mlx4_0
c. Using sysfs file system
The driver supports the sysfs file system under: /sys/class/infiniband
Examples:
- > ls /sys/class/infiniband/mthca0/
+ > ls /sys/class/infiniband/mlx4_0/
board_id device fw_ver hca_type hw_rev node_desc node_guid node_type
ports sys_image_guid
- > cat /sys/class/infiniband/mthca0/board_id
- MT_0200000001
+ > cat /sys/class/infiniband/mlx4_0/board_id
+ MT_1090110019
- > ls /sys/class/infiniband/mthca0/ports/1/
+ > ls /sys/class/infiniband/mlx4_0/ports/1/
cap_mask counters gids lid lid_mask_count phys_state pkeys rate sm_lid
sm_sl state
- > cat /sys/class/infiniband/mthca0/ports/1/state
+ > cat /sys/class/infiniband/mlx4_0/ports/1/state
4: ACTIVE
1.2 Performance Tests
The same information can be obtained by executing the 'ofed_info' command. For
example:
-> ofed_info
-OFED-1.5.1:
+# ofed_info -s
+OFED-3.5-rc1:
-compat-dapl:
-http://www.openfabrics.org/downloads/dapl/compat-dapl-1.2.16.tar.gz
+# ofed_info
+OFED-3.5-rc1:
+
+ofed-scripts:
+git://git.openfabrics.org/~vlad/ofed_scripts.git compat-rdma
+ar.gz
+
+compat-rdma:
+git://git.openfabrics.org/compat-rdma/compat-rdma.git master
+commit 2036874cf4c072c04b96b69e5b965a788fc77a6d
dapl:
-http://www.openfabrics.org/downloads/dapl/dapl-2.0.27.tar.gz
+http://www.openfabrics.org/downloads/dapl/dapl-2.0.36.tar.gz
ib-bonding:
-http://www.openfabrics.org/~monis/ofed_1_5/ib-bonding-0.9.0-42.src.rpm
+http://www.openfabrics.org/downloads/ib-bonding/ib-bonding-0.9.0-43.src.rpm
+
+ibacm:
+http://www.openfabrics.org/downloads/rdmacm/ofed/ibacm-1.0.7.tar.gz
ibsim:
http://www.openfabrics.org/downloads/ibsim/ibsim-0.5-0.1.g327c3d8.tar.gz
ibutils:
-http://www.openfabrics.org/downloads/ibutils/ibutils-1.5.4-0.1.g0464fe6.tar.gz
+http://www.openfabrics.org/downloads/ibutils/ibutils-1.5.7-0.1.g05a9d1a.tar.gz
infiniband-diags:
-http://www.openfabrics.org/downloads/management/infiniband-diags-1.5.5.tar.gz
+http://www.openfabrics.org/downloads/management/infiniband-diags-1.6.1.tar.gz
+
+infinipath-psm:
+http://www.openfabrics.org/downloads/infinipath-psm/infinipath-psm-2.9-926.1005_open.tar.gz
libcxgb3:
-http://www.openfabrics.org/downloads/cxgb3/libcxgb3-1.2.5.tar.gz
+http://www.openfabrics.org/downloads/cxgb3/libcxgb3-1.3.1.tar.gz
+
+libcxgb4:
+http://www.openfabrics.org/downloads/cxgb4/libcxgb4-1.2.0.tar.gz
libehca:
-http://www.openfabrics.org/downloads/libehca/libehca-1.2.1-0.1.g0a82a52.tar.gz
+http://www.openfabrics.org/downloads/libehca/libehca-1.2.2-0.1.g69e1a88.tar.gz
libibcm:
http://www.openfabrics.org/downloads/rdmacm/libibcm-1.0.5.tar.gz
libibmad:
-http://www.openfabrics.org/downloads/management/libibmad-1.3.4.tar.gz
+http://www.openfabrics.org/downloads/management/libibmad-1.3.9.tar.gz
libibumad:
-http://www.openfabrics.org/downloads/management/libibumad-1.3.4.tar.gz
+http://www.openfabrics.org/downloads/management/libibumad-1.3.8.tar.gz
libibverbs:
-http://www.openfabrics.org/downloads/libibverbs/libibverbs-1.1.3-0.6.g932f1a2.tar.gz
+http://www.openfabrics.org/downloads/verbs/libibverbs-1.1.6.tar.gz
libipathverbs:
http://www.openfabrics.org/downloads/libipathverbs/libipathverbs-1.2.tar.gz
libmlx4:
-http://www.openfabrics.org/downloads/libmlx4/libmlx4-1.0-0.7.g2432360.tar.gz
+http://www.openfabrics.org/downloads/mlx4/libmlx4-1.0.4.tar.gz
libmthca:
http://www.openfabrics.org/downloads/libmthca/libmthca-1.0.5-0.1.gbe5eef3.tar.gz
libnes:
-http://www.openfabrics.org/downloads/nes/libnes-1.0.1.tar.gz
+http://www.openfabrics.org/downloads/nes/libnes-1.1.3.tar.gz
librdmacm:
-http://www.openfabrics.org/downloads/rdmacm/librdmacm-1.0.11.tar.gz
-
-libsdp:
-http://www.openfabrics.org/downloads/libsdp/libsdp-1.1.100-0.1.g920ea31.tar.gz
-
-mpi-selector:
-http://www.openfabrics.org/downloads/mpi-selector/mpi-selector-1.0.3-1.src.rpm
-
-mpitests:
-http://www.openfabrics.org/~pasha/ofed_1_5/mpitests/mpitests-3.2-916.src.rpm
+http://www.openfabrics.org/downloads/rdmacm/librdmacm-1.0.16.tar.gz
mstflint:
-http://www.openfabrics.org/downloads/mstflint/mstflint-1.4-0.2.g0fccc0f.tar.gz
-
-mvapich:
-http://www.openfabrics.org/~pasha/ofed_1_5_1/mvapich/mvapich-1.2.0-3635.src.rpm
-
-mvapich2:
-http://www.openfabrics.org/~perkinjo/ofed_1_5/mvapich2-1.4.1-1.src.rpm
-
-ofa_kernel:
-git://git.openfabrics.org/ofed_1_5/linux-2.6.git ofed_kernel_1_5
-commit 17badd753b40fb1046dc2e5474739357a921fb86
+http://www.openfabrics.org/downloads/mstflint/mstflint-1.4-1.18.g1adcfbf.tar.gz
ofed-docs:
-git://git.openfabrics.org/~tziporet/docs.git ofed_1_5
-commit 137b089f71570fed2d5b9080410fa0b8f35d7fd5
-
-open-iscsi-generic1:
-http://www.openfabrics.org/downloads/iscsi/open-iscsi-generic-2.0-754.1.src.rpm
-
-open-iscsi-generic2:
-http://www.openfabrics.org/downloads/iscsi/open-iscsi-generic-2.0-869.2.src.rpm
-
-openmpi:
-http://www.openfabrics.org/~jsquyres/ofed_1_5/openmpi-1.4.1-2ofed.src.rpm
+git://git.openfabrics.org/compat-rdma/docs.git ofed_3_2
+commit 6dff336cfb197351c9fa0b50c0715b59a4d822cf
opensm:
-http://www.openfabrics.org/downloads/management/opensm-3.3.5.tar.gz
+http://www.openfabrics.org/downloads/management/opensm-3.3.15.tar.gz
perftest:
-http://www.openfabrics.org/downloads/perftest/perftest-1.2.3-0.10.g90b10d8.tar.gz
+http://www.openfabrics.org/downloads/perftest/perftest-1.4.0-0.66.g9074687.tar.gz
qlvnictools:
http://www.openfabrics.org/downloads/qlvnictools/qlvnictools-0.0.1-0.1.ge27eef7.tar.gz
http://www.openfabrics.org/downloads/qperf/qperf-0.4.6-0.1.gb81434e.tar.gz
rds-tools:
-http://www.openfabrics.org/~vlad/ofed_1_5/rds-tools/rds-tools-1.5-1.src.rpm
-
-rnfs-utils:
-http://www.openfabrics.org/~swise/ofed_1_5/rnfs-utils/rnfs-utils-1.1.5-10.OFED.src.rpm
-
-sdpnetstat:
-http://www.openfabrics.org/downloads/sdpnetstat/sdpnetstat-1.60-0.2.g8844f04.tar.gz
+http://www.openfabrics.org/downloads/rds-tools/rds-tools-2.0.4.tar.gz
srptools:
http://www.openfabrics.org/downloads/srptools/srptools-0.0.4-0.1.gce1f64c.tar.gz
-tgt-generic:
-http://www.openfabrics.org/downloads/iscsi/tgt-generic-0.1-20080828.src.rpm
-
-ofed-scripts:
-git://git.openfabrics.org/~vlad/ofed_scripts.git ofed_1_5
-commit f3359b78fbe6cd9e3f78d2c39498546270b809ad
2.2 Installed OFED Components
-------------------------------
For example:
- > /etc/infiniband/info
+# /etc/infiniband/info
prefix=/usr
-Kernel=2.6.18-164.el5
-
-Configure options: --with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mthca-mod --with-mlx4-mod --with-mlx4_en-mod --with-ipoib-mod --with-sdp-mod --with-rds-mod
+Kernel=2.6.32-220.el6.x86_64
+Configure options: --with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mthca-mod --with-mlx4-mod --with-mlx4_en-mod --with-cxgb3-mod --with-cxgb4-mod --with-nes-mod --with-qib-mod --with-ipoib-mod --with-srp-mod
2.3 Building/Installing InfiniBand (IB) Modules With Debug Information
----------------------------------------------------------------------
7.1 Compiling the kernel modules
--------------------------------
-1. tar xzf OFED-1.5.1.tgz
-2. rpm -ihv OFED-1.5.1/SRPMS/ofa_kernel-1.5.1-ofed1.5.1.src.rpm
+1. tar xzf OFED-3.5.tgz
+2. rpm -ihv OFED-3.5/SRPMS/compat-rdma-3.5-OFED.3.5.src.rpm
3. cd /usr/src/redhat/SOURCES
-4. tar xzvf ofa_kernel-1.5.1.tgz
-5. cd ofa_kernel-1.5.1
+4. tar xzvf compat-rdma-3.5.tgz
+5. cd compat-rdma-3.5
6. configure:
run ./configure --help for a list of options.
basic invocation is:
- ./configure --with-core-mod --with-ipoib-mod --with-mthca-mod --with-mlx4_core-mod --with-mlx4_inf-mod
+ ./configure --with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mlx4-mod --with-mlx4_en-mod --with-cxgb3-mod --with-cxgb4-mod --with-nes-mod --with-qib-mod --with-ipoib-mod --with-srp-mod
7. make
make install
NOTES:
- 1. configure applies the patches to the source code according to the current
- kernel. If you wish to rerun configure it is recommend to untar the source
- code tree from the beginning and start with a clean state.
- An alternative is to pass the option: --without-patch to the configure invocation.
-
- 2. The modules select for install are written to configure.mk.kernel
+ 1. The modules select for install are written to configure.mk.kernel
+ 2. INSTALL_MOD_DIR = "extra/ofa_kernel" for RedHat base Distros
+ INSTALL_MOD_DIR = "updates" for other Distros
7.2 Compiling the user space libraries
--------------------------------------
Example for libibverbs:
-1. tar xzf OFED-1.5.1.tgz
-2. rpm -ihv SRPMS/libibverbs-1.1.3-0.6.g932f1a2.src.rpm
+1. tar xzf OFED-3.5.tgz
+2. rpm -ihv SRPMS/libibverbs-1.1.6-1.src.rpm
3. cd /usr/src/redhat/SOURCES (for RedHat)
or
cd /usr/src/packages/SOURCES (for SuSE)
-4. tar xzf libibverbs-1.1.3.tgz
-5. cd libibverbs-1.1.3
+4. tar xzf libibverbs-1.1.6.tgz
+5. cd libibverbs-1.1.6
6. ./configure (specify parameters, if required)
7. make
8. make install