--- /dev/null
+ uDAPL MCM Provider and MPXYD Daemon (CCL-proxy)
+ dapl-2.1.x
+ Aug 2014
+
+MCM is a new uDAPL provider that is an extension to standard DAT 2.0 libraries. The purpose of this service
+is to proxy RDMA writes from the MIC to the HOST to improve large IO performance. The provider will support
+MIC to MIC, HOST to HOST, and MIC to HOST environments. The mcm client will NOT use MPXYD when running on the host.
+It requires a new MPXYD daemon service when clients are running on a MIC KNC adapter. This package installs all the
+host side libraries and daemon service. The MIC libraries must be built and moved over to MIC adapter. This verion
+is currently included with MPSS and all libraries and services will be installed by default.
+
+Current release package: dapl-2.1.0.tar.gz
+
+* Sample host build from source package (ofed must installed)
+
+ ./autogen.sh
+ ./configure \
+ --enable-mcm \
+ --prefix=/usr \
+ --libdir=/usr/lib64 \
+ --sysconfdir=/etc
+ make
+ sudo make install
+
+* Sample /home/user1 MIC build from source package for MPSS 3.x (MPSS must be installed)
+
+ source /opt/mpss/3.x/environment-setup-k1om-mpss-linux
+ ./autogen.sh
+ ./configure \
+ --enable-mcm
+ --host=x86_64-k1om-linux \
+ --prefix=/home/user1/dapl-mic-install \
+ CC=/usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc \
+ CFLAGS="-I/opt/mpss/3.x/sysroots/k1om-mpss-linux/usr/include
+ LDFLAGS="-L/opt/mpss/3.x/sysroots/k1om-mpss-linux/usr/lib64"
+ make
+ sudo make install
+
+* Sample /home/user1 MIC build from source package for MPSS 2.x (MPSS must be installed)
+
+ export PATH=$PATH:/usr/linux-k1om-4.7/bin
+ ./autogen.sh
+ ./configure \
+ --enable-mcm \
+ --prefix=/home/user1/dapl-mic-install \
+ --libdir=/opt/intel/mic/ofed/card/usr/lib64 \
+ --sysconfdir=/opt/intel/mic/ofed/card/etc \
+ --host=x86_64-k1om-linux \
+ CFLAGS="-I/opt/intel/mic/ofed/card/usr/include" \
+ LDFLAGS="-L/opt/intel/mic/ofed/card/usr/lib64"
+ make
+ sudo make install
+
+* Cluster deployment
+
+ (1) Build once on the head or on one of the nodes as described in the above steps.
+
+ (2) Replicate these files on all the nodes:
+
+ /etc/dat.conf
+ /etc/mpxyd.conf
+ /usr/sbin/mpxyd
+ /usr/lib64/libdaplomcm.so.2
+ /opt/intel/mic/ofed/card/etc/dat.conf
+ /opt/intel/mic/ofed/card/usr/lib64/libdaplomcm.so.2
+ /opt/intel/mic/ofed/card/ofed.filelist
+
+ (3) Unload and then restart MPSS on all the nodes.
+
+* Start the proxy daemon on all the nodes (host only)
+
+ sudo /usr/sbin/mpxyd
+
+* Use the MCM provider with Intel MPI 4.1.3 or greater for best out of box experiences.
+
+ (1) Recommended settings:
+
+ export I_MPI_MIC=1
+ export I_MPI_DEBUG=2
+ export I_MPI_FALLBACK=0
+ export I_MPI_MIC_DAPL_DIRECT_COPY_THRESHOLD=8192,262144
+
+ With these settings on MIC, messages less than 8192 bytes will be sent via pre-registered buffers; messages
+ between 8192 and 262144 bytes will be sent via the Rendezvous protocol throught the first provider; and
+ larger messages will be sent via the Rendezvous protocol through the second provider. Fine tune these
+ two sizes for the best performance.
+
+* Setup for non-root CCL Proxy testing, MPXYD running as process with different service port from your /home directory:
+
+ Using build instructions above, change prefix as follow and "make install":
+
+ Build MIC:
+ --prefix=/home/username/ccl-proxy-mic
+
+ Build host:
+ --prefix=/home/username/ccl-proxy-host
+
+ edit /home/username/ccl-proxy-host/etc/mpxyd.conf and change the following entries:
+
+ log_file /var/log/mpxyd.log to log_file /tmp/username/mpxyd.log
+ lock_file /var/log/mpxyd.pid to lock_file /tmp/username/mpxyd.log
+ scif_port_id 68 to scif_port_id 1068
+
+ start the mpxyd process on each node
+
+ ssh node1-hostname /home/username/ccl-proxy-host/sbin/mpxyd -P -O /home/username/ccl-proxy-host/etc/mpxyd.conf&
+
+ Note: override default port id using following environment variable:
+
+ export DAPL_MCM_PORT_ID=1068
+
+* Notes
+
+ (1) Modify "/etc/mpxyd.conf" to change the settings for the proxy. Especially, try different values
+ of "buffer_segment_size" for performance tuning. Use a smaller value for "buffer_pool_mb"
+ to reduce the memory foorprint of mpxyd. Use a larger value for "scif_listen_qlen" to run
+ more MPI ranks per card. Also modify mcm_affinity_base to the desired CPU_id to insure
+ socket to adapter affinity. Best performance when HCA, MIC, and CPU are on same socket.
+ Default settings are on CPU socket 0.
+
+ (2) By default, only writes originated from MIC is proxied. However, it is also possible to proxy
+ host-originated writes (e.g. for debugging purpose). To do this, set the environment variable
+ "DAPL_MCM_ALWAYS_PROXY=1". This variable applies to the provider, not the proxy.
+
+ChangeLog:
+
+Release dapl-2.0.42.1-1 (more bug fixing for MPSS 3.3, merge with master branch 2.0.42)
+
+dapltest: increase DTO evd size to prevent CQ overflow on limit_rpost test
+Creation of reserved SP moves EP state to DAT_EP_STATE_RESERVED even in failure cases. Reserv
+dapl: fix string bug in dapls_dto_op_str
+mpxyd: change affinity to avoid overlapping cores with 2 MICs in same socket
+mpxyd: add mpxyd.conf option to disable proxy-in service
+mpxyd: set default seg size to 128KB, down from 256KB
+install: base package, without mpss, incorrectly tries to install /usr/sbin as file
+mpxyd: scale-up with MPI dapl:dapl hits low mem issue with 1 byte traffic patterns
+mpxyd: add MIC client and device id to logging
+add new po-pi rdma write perf profile
+change default wrc entries from 1024 to 512
+mpxyd: MIC scale-up issue with MPI gather workloads, I_MPI_FABRICS=dapl:dapl
+mpxyd: remove unnecessary logging
+mcm: mpxyd error event of m_pi_prep_rcv_q: ERR: ib_qp == 0
+
+Release dapl-2.0.41.2-1 (bug fixing for MPSS 3.3)
+
+mpxyd,mcm: increase default CM timers for better out of box scaling
+mpxyd,mcm: changes for backward compatibility with older v4 MIC clients
+mpxyd: allow proxy_out WR stalls instead of immediate error
+mpxyd: increase max open files limit for proxy service
+
+Release dapl-2.0.41.1-1 (added proxy-in support)
+
+dapltest: change server port, from 45278 to 62000, out of registered IANA range
+dapltest: set default limit max to 1000
+dapltest: update scripts for regression testing purposes
+dapltest: Add final send/recv "sync" for transaction tests.
+mpxyd: append _free to dqconn/dqlisten for readability, improve logging
+mcm: check for shared CQs in PI mode \96 MXS-HST scaling issues
+mcm: add dev_id to all mpxyd commands
+mpxyd: scale-up improvements to support 200-300 processes per MIC
+mpxyd: serialize MD cm port space usage, add swap to rej call, resend dropped rej
+mcm: serialize CM cmds on ev_ep, add dev_id, increase dev_open listen backlog
+openib: return open failure if port not active
+mpxyd: disable inside box support via scif only, use IB
+mcm: add host to mic cross socket support to proxy-in service to improve performance
+dat: reduce log level on library load failures
+mic: take cm_msg rsvd byte for segment size exchange, power of 2
+mpxyd: init proxy WC buffer queues for new queue management
+mpxyd: simplify WR and WC queue management and fix inline post send
+mpxyd: 64KB segment sizes hang with MPI IMB pingpong cross socket
+mpxyd: add new M_READ_FROM_DONE state for send WR's and add more profiling options
+mpxyd: purge posted send data only if inline
+mpxyd: move to CONN state immediately on RTU_IN
+dtest: fix options on query times printf
+mpxyd: fix ibctx leak with device open
+dtest: add new -Q option to get provider list and query and time each
+openib: cleanup init/fini and device close on all providers
+dat: dat_ia_open should close provider after failure
+mpxyd: sync PI WC trigger to PO MP_SIG
+mpxyd: improve QP destruction to manage QP1 and QP2 variations
+mpxyd: proxy out doesn't release proxy buffer as quickly as necessary
+dtest: add times for open_query, remove sleep
+common: add provider name and len to DTO error logging
+new lightweight open_query/close_query IB extension for fast attribute query
+mpxyd: need CM to QP linking with CM references
+dist: ib collective and MIC extension include files missing
+dapltest: the quit command is missing changes for -n option.
+NULL undefined on Fedora, incorrectly using kernel stddef.h
+Merge branch 'proxy' of ssh://beany.openfabrics.org/home/ardavis/scm/dapl into proxy
+ucm: fix CM service, initial rcv msg posts incorrect
+ucm: add/cleanup debug log information
+scm: add/cleanup debug log information
+makefile: update for MCM proxy-in changes
+dtest: update for ep_mode on MCM providers
+mpxyd.conf: updated for proxy-in parameters
+mpxyd: proxy-in added to proxy-out service to increase cross socket performance
+mcm: add proxy in support to MCM provider and MPXYD interface
+open_ib common: qp, cq, and post_recv changes for proxy-in
+commom: add lmr support for proxy in service
+new definitions and states for CCL Proxy-in support
+
+
+
+
+
+
projects / ~ardavis / dapl.git / commitdiff