./configure --enable-debug
make
-Build example with OFED 1.2+ prefix (x86_64)
+Build example with OFED prefix (x86_64)
---------------------------------------------
./autogen.sh
./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
----------
make install
-Note: The development package installs DAT 2.0 include files under /usr/include/dat2 to co-exist with DAT 1.2 /usr/include/dat
-
NOTE: to link these libraries you must either use libtool and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
sample /etc/dat.conf
-#
-# DAT 1.2 and 2.0 configuration file
-#
+# DAT v2.0 configuration file
+#
# Each entry should have the following fields:
#
# <ia_name> <api_version> <threadsafety> <default> <lib_path> \
# <provider_version> <ia_params> <platform_params>
#
-# For the uDAPL cma provder, specify <ia_params> as one of the following:
+# For uDAPL cma provder, <ia_params> is one of the following:
# network address, network hostname, or netdev name and 0 for port
#
-# Simple (OpenIB-cma) default with netdev name provided first on list
-# to enable use of same dat.conf version on all nodes
-#
-# Add examples for multiple interfaces and IPoIB HA fail over, and bonding
-#
-OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "ib0 0" ""
-OpenIB-cma-1 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "ib1 0" ""
-OpenIB-cma-2 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "ib2 0" ""
-OpenIB-cma-3 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "ib3 0" ""
-OpenIB-bond u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "bond0 0" ""
-OpenIB-2-cma u2.0 nonthreadsafe default libdaplcma.so.2 dapl.2.0 "ib0 0" ""
-OpenIB-2-cma-1 u2.0 nonthreadsafe default libdaplcma.so.2 dapl.2.0 "ib1 0" ""
-OpenIB-2-cma-2 u2.0 nonthreadsafe default libdaplcma.so.2 dapl.2.0 "ib2 0" ""
-OpenIB-2-cma-3 u2.0 nonthreadsafe default libdaplcma.so.2 dapl.2.0 "ib3 0" ""
-OpenIB-2-bond u2.0 nonthreadsafe default libdaplcma.so.2 dapl.2.0 "bond0 0" ""
+# For uDAPL scm provider, <ia_params> is device name and port
+# For uDAPL ucm provider, <ia_params> is device name and port
+# For uDAPL iWARP provider, <ia_params> is netdev device name and 0
+# For uDAPL iWARP provider, <ia_params> is netdev device name and 0
+# For uDAPL RoCE provider, <ia_params> is device name and 0
+#
+ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
+ofa-v2-mlx4_0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 2" ""
+ofa-v2-ib0 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "ib0 0" ""
+ofa-v2-ib1 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "ib1 0" ""
+ofa-v2-mthca0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mthca0 1" ""
+ofa-v2-mthca0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mthca0 2" ""
+ofa-v2-ipath0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "ipath0 1" ""
+ofa-v2-ipath0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "ipath0 2" ""
+ofa-v2-ehca0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "ehca0 1" ""
+ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 0" ""
+ofa-v2-mlx4_0-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx4_0 1" ""
+ofa-v2-mlx4_0-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx4_0 2" ""
+ofa-v2-mthca0-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mthca0 1" ""
+ofa-v2-mthca0-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mthca0 2" ""
+ofa-v2-cma-roe-eth2 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 0" ""
+ofa-v2-cma-roe-eth3 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth3 0" ""
+ofa-v2-scm-roe-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
+ofa-v2-scm-roe-mlx4_0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 2" ""
+ofa-v2-mcm-1 u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx4_0 1" ""
+ofa-v2-mcm-2 u2.0 nonthreadsafe default libdaplomcm.so.2 dapl.2.0 "mlx4_0 2" ""
+ofa-v2-scif0 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "scif0 1" ""
=============================
- DAPL MCM Provider (CCL-proxy)
+ uDAPL MCM Provider and MPXYD Daemon (CCL-proxy)
+ intel-mic-ofed-dapl-2.0.36.3
+ Technology Preview
+ January 2013
- August 2012
+MCM is a new uDAPL provider that is an extension to standard DAT 2.0 libraries. The purpose of this service
+is to proxy RDMA writes from the MIC to the HOST to improve large IO performance. The provider will support
+MIC to MIC, HOST to HOST, and MIC to HOST environments. The mcm client will NOT use MPXYD when running on the host.
+It requires a new MPXYD daemon service when clients are running on a MIC KNC adapter. This package installs all the
+host side libraries and daemon service. The MIC libraries must be built and moved over to MIC adapter. This verion
+is currently included with MPSS and all libraries and services will be installed by default.
-* Build for host
+Current release package for MPSS: intel-mic-ofed-dapl-2.0.36.3.tar.gz
+
+
+* Sample host build from source package (ofed must installed)
./autogen.sh
./configure \
- --enable-debug \
--enable-mcm \
--prefix=/usr \
--libdir=/usr/lib64 \
make
sudo make install
-* Build for MIC (make sure MPSS is already installed)
+* Sample MIC build from source package (MPSS must be installed)
export PATH=$PATH:/usr/linux-k1om-4.7/bin
./autogen.sh
./configure \
- --enable-debug \
--enable-mcm \
--prefix=/opt/intel/mic/ofed/card/usr \
--libdir=/opt/intel/mic/ofed/card/usr/lib64 \
(1) Modify "/etc/mpxyd.conf" to change the settings for the proxy. Especially, try different values
of "buffer_segment_size" for performance tuning. Use a smaller value for "buffer_pool_mb"
to reduce the memory foorprint of mpxyd. Use a larger value for "scif_listen_qlen" to run
- more MPI ranks per card.
+ more MPI ranks per card. Also modify mcm_affinity_base to the desired CPU_id to insure
+ socket to adapter affinity. Best performance when HCA, MIC, and CPU are on same socket.
+ Default settings are on CPU socket 0.
(2) By default, only writes originated from MIC is proxied. However, it is also possible to proxy
host-originated writes (e.g. for debugging purpose). To do this, set the environment variable
"DAPL_MCM_ALWAYS_PROXY=1". This variable applies to the provider, not the proxy.
+
+