From 22d9d1b485f2d980f0a50fccca4e5ff942515dae Mon Sep 17 00:00:00 2001 From: Rupert Dance Date: Tue, 14 Jul 2015 04:24:41 -0700 Subject: [PATCH] Updates to OFED Release notes and uDAPL after updating git tree Signed-off-by: Rupert Dance --- OFED_release_notes.txt | 24 +++--- release_notes/uDAPL_release_notes.txt | 112 +++++++++++++------------- 2 files changed, 73 insertions(+), 63 deletions(-) diff --git a/OFED_release_notes.txt b/OFED_release_notes.txt index b2c2b77..1d075bb 100644 --- a/OFED_release_notes.txt +++ b/OFED_release_notes.txt @@ -1,6 +1,6 @@ Open Fabrics Enterprise Distribution (OFED) - Version 3.18-rc3 Release Notes - June 2015 + Version 3.18 Release Notes + July 2015 =============================================================================== Table of Contents @@ -207,6 +207,7 @@ OFED-3.18-rc3 Main Changes from OFED 3.18-rc2 ------------------------------------------------------------------------------- 1. Updated packages: - dapl-2.1.5 + - fabtests-1.0.0 - libfabric-1.0.0 - librdmacm-1.0.21 - mstflint-4.0.1-1.42.g0d3b372 @@ -298,21 +299,26 @@ components of the OFED 3.18 release. 07. On SLES11.x, set allow_unsupported_modules parameter to 1 in file: /etc/modprobe.d/unsupported-modules. Without this the modules will not load. -08. There are a few known issues with NFSoRDMA that are documented in bugs 2489 - and 2507. We believe the issues have been resolved in kernel 3.18 but the - backports have not been applied to OFED 3.12-1. -09. Bug 2515: when an Intel HCA is attached directly to a Mellanox ConnectX3 - and the OpenSM is started on the Inetl HCA, the link will not go to active. +08. RDS is not supported. +09. Bugs 2489 & 2507: There are a few known issues with NFSoRDMA that are documented + in bugs 2489 and 2507. We believe the issues have been resolved in kernel 3.18 + but the backports have not been applied to OFED 3.12-1. +10. Bug 2515: when an Intel HCA is attached directly to a Mellanox ConnectX3 + and the OpenSM is started on the Intel HCA, the link will not go to active. The workaround is to start the OpenSM on the Mellanox HCA. -10. RDS is not supported. 11. Bug 2544: Libfabric depends on infinipath-psm. When OFED is compiled --with-xeon-phi, it will fail to compile/install because infinipath-psm is renamed when intel-mic-psm RPMs are now built/installed. This will be resolved in OFED 3.18-1 12. Bug 2545: ipath will not compile on SLES11 SP4 and RHEL6.7 because these releases are not supported in OFED 3.18. They will be added in 3.18-1 -13. Bug 2551: OFED 3.18 will not compile on ppc64 with libfabric and fabtests enabled. +13. Bug 2547: HCA driver seems to load, but does not seem to configure a device +14. Bug 2551: OFED 3.18 will not compile on ppc64 with libfabric and fabtests enabled. A worked around is to install OFED 3.18 with the following flags: ./install.pl --all --without-libfabric --without-fabtests --without-libiwpm +15. Bug 2555: ib_send_bw –run_infinitely fails to terminate on the server after client + disconnects +16. Bug 2559: unexpected CM event 8 error seen on running iwarp traffic after the + restart of iwpmd service Note: See the release notes of each component for additional issues. diff --git a/release_notes/uDAPL_release_notes.txt b/release_notes/uDAPL_release_notes.txt index 65f1bb9..1520684 100644 --- a/release_notes/uDAPL_release_notes.txt +++ b/release_notes/uDAPL_release_notes.txt @@ -1,4 +1,4 @@ - Release Notes for + Release Notes for OFED 3.18 DAPL Release 2.1.5 June 2015 @@ -160,23 +160,23 @@ Note: example for user /home/user1, (assumes /home/user1 is exported) and MLX4 adapter, port 1 - Download latest 2.x package: http://www.openfabrics.org/downloads/dapl/dapl-2.0.42.tar.gz + Download latest 2.1.x package: http://www.openfabrics.org/downloads/dapl/dapl-2.1.5.tar.gz untar in /home/user1 - cd /home/user1/dapl-2.0.42 + cd /home/user1/dapl-2.1.5 ./configure LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include" make Create /home/user1/dat.conf with following 3 lines. (entries with path to new libraries): - ofa-v2-mlx4_0-1u u2.0 nonthreadsafe default /home/user1/dapl-2.0.42/dapl/udapl/.libs/libdaploucm.so.2 dapl.2.0 "mlx4_0 1" "" - ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default /home/user1/dapl-2.0.42/dapl/udapl/.libs/libdaploscm.so.2 dapl.2.0 "mlx4_0 1" "" - ofa-v2-ib0 u2.0 nonthreadsafe default /home/user1/dapl-2.0.42/dapl/udapl/.libs/libdaplcma.so.1 dapl.2.0 "ib0 0" "" + ofa-v2-mlx4_0-1u u2.0 nonthreadsafe default /home/user1/dapl-2.1.5/dapl/udapl/.libs/libdaploucm.so.2 dapl.2.0 "mlx4_0 1" "" + ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default /home/user1/dapl-2.1.5/dapl/udapl/.libs/libdaploscm.so.2 dapl.2.0 "mlx4_0 1" "" + ofa-v2-ib0 u2.0 nonthreadsafe default /home/user1/dapl-2.1.5/dapl/udapl/.libs/libdaplcma.so.1 dapl.2.0 "ib0 0" "" Run uDAPL application or Intel MPI that uses uDAPL, with (assuming mlx4_0 adapters) following: setenv DAT_OVERRIDE=/home/user1/dat.conf - setenv LD_LIBRARY_PATH=/home/user1/dapl-2.0.42/dapl/udapl/.libs:$LD_LIBRARY_PATH + setenv LD_LIBRARY_PATH=/home/user1/dapl-2.1.5/dapl/udapl/.libs:$LD_LIBRARY_PATH If running Intel MPI and uDAPL socket cm, set the following: @@ -203,67 +203,71 @@ Current release package: dapl-2.1.5.tar.gz - * Sample host build from source package (ofed must installed) + * Sample host build from source package (ofed must be installed) - ./autogen.sh - ./configure \ - --enable-mcm \ - --prefix=/usr \ - --libdir=/usr/lib64 \ - --sysconfdir=/etc + ./configure --enable-mcm --prefix=/usr --libdir=/usr/lib64 --sysconfdir=/etc make - sudo make install + make install - * Sample /home/user1 MIC build from source package for MPSS 3.x (MPSS must be installed) + * Sample host rpmbuild/update from release tarball, /root: + + rpmbuild -ta dapl-2.1.5.tar.gz + rpm -U /root/rpmbuild/RPMS/x86_64/dapl* + + * Sample MIC build from source package for MPSS 3.x (MPSS must be installed) + * Assume /opt is nfs mounted across cluster source /opt/mpss/3.x/environment-setup-k1om-mpss-linux - ./autogen.sh - ./configure \ - --enable-mcm - --host=x86_64-k1om-linux \ - --prefix=/home/user1/dapl-mic-install \ - CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc \ - CFLAGS="-I/opt/mpss/3.x/sysroots/k1om-mpss-linux/usr/include - LDFLAGS="-L/opt/mpss/3.x/sysroots/k1om-mpss-linux/usr/lib64" - make - sudo make install + ./configure --enable-mcm --prefix /opt/dapl/mic --host=x86_64-k1om-linux + make + make install + + copy /opt/dapl/mic/* files out to all MIC cards - * Sample /home/user1 MIC build from source package for MPSS 2.x (MPSS must be installed) - - export PATH=$PATH:/usr/linux-k1om-4.7/bin - ./autogen.sh - ./configure \ - --enable-mcm \ - --prefix=/home/user1/dapl-mic-install \ - --libdir=/opt/intel/mic/ofed/card/usr/lib64 \ - --sysconfdir=/opt/intel/mic/ofed/card/etc \ - --host=x86_64-k1om-linux \ - CFLAGS="-I/opt/intel/mic/ofed/card/usr/include" \ - LDFLAGS="-L/opt/intel/mic/ofed/card/usr/lib64" - make - sudo make install * Cluster deployment - (1) Build once on the head or on one of the nodes as described in the above steps. - - (2) Replicate these files on all the nodes: + (1) Build once on the head or on one of the nodes (with MPSS) as described in the above steps. - /etc/dat.conf - /etc/mpxyd.conf - /usr/sbin/mpxyd - /usr/lib64/libdaplomcm.so.2 - /opt/intel/mic/ofed/card/etc/dat.conf - /opt/intel/mic/ofed/card/usr/lib64/libdaplomcm.so.2 - /opt/intel/mic/ofed/card/ofed.filelist + (2) HOST: Install dapl libraries and mpxyd service, "rpm -U" all dapl RPM files on host nodes: - (3) Unload and then restart MPSS on all the nodes. + (3) MIC: Setup dapl overlay for new package (/opt/intel/dapl): + + Create /etc/mpss/conf.d/dapl.conf with following entry: + + Overlay Filelist /opt/dapl /opt/dapl/dapl.filelist on + + Create /opt/dapl/dapl.filelist with following entries: + + file /etc/dat.conf mic/etc/dat.conf 755 0 0 + file /usr/bin/dtest mic/bin/dtest 755 0 0 + file /usr/bin/dtestx mic/bin/dtestx 755 0 0 + file /usr/bin/dtestcm mic/bin/dtestcm 755 0 0 + file /usr/bin/dapltest mic/bin/dapltest 755 0 0 + file /usr/lib64/libdat.so.2.0.0 mic/lib/libdat.so.2.0.0 755 0 0 + file /usr/lib64/libdaplofa.so.2.0.0 mic/lib/libdaplofa.so.2.0.0 755 0 0 + file /usr/lib64/libdaplomcm.so.2.0.0 mic/lib/libdaplomcm.so.2.0.0 755 0 0 + file /usr/lib64/libdaploscm.so.2.0.0 mic/lib/libdaploscm.so.2.0.0 755 0 0 + file /usr/lib64/libdaploucm.so.2.0.0 mic/lib/libdaploucm.so.2.0.0 755 0 0 + + slink /usr/lib64/libdat.so libdat.so.2.0.0 777 0 0 + slink /usr/lib64/libdat.so.2 libdat.so.2.0.0 777 0 0 + slink /usr/lib64/libdaplofa.so libdaplofa.so.2.0.0 777 0 0 + slink /usr/lib64/libdaplofa.so.2 libdaplofa.so.2.0.0 777 0 0 + slink /usr/lib64/libdaplomcm.so libdaplomcm.so.2.0.0 777 0 0 + slink /usr/lib64/libdaplomcm.so.2 libdaplomcm.so.2.0.0 777 0 0 + slink /usr/lib64/libdaploscm.so libdaploscm.so.2.0.0 777 0 0 + slink /usr/lib64/libdaploscm.so.2 libdaploscm.so.2.0.0 777 0 0 + slink /usr/lib64/libdaploucm.so libdaploucm.so.2.0.0 777 0 0 + slink /usr/lib64/libdaploucm.so.2 libdaploucm.so.2.0.0 777 0 0 + + Reboot or restart MPSS and ofed-mic services - * Start the proxy daemon on all the nodes (host only) + Check for dapl overlay + micctrl --config - sudo /usr/sbin/mpxyd - * Use the MCM provider with Intel MPI 4.1.3 or greater for best out of box experiences. + * Use the MCM provider with Intel MPI 5.0 greater for best out of box experiences. (1) Recommended settings: -- 2.41.0