uDAPL MCM Provider and MPXYD Daemon (CCL-proxy)
dapl-2.1.x
- Aug 2014
+ May 2015
MCM is a new uDAPL provider that is an extension to standard DAT 2.0 libraries. The purpose of this service
is to proxy RDMA writes from the MIC to the HOST to improve large IO performance. The provider will support
host side libraries and daemon service. The MIC libraries must be built and moved over to MIC adapter. This verion
is currently included with MPSS and all libraries and services will be installed by default.
-Current release package: dapl-2.1.0.tar.gz
+Current release package: dapl-2.1.5.tar.gz
* Sample host build from source package (ofed must installed)
ChangeLog:
-Release dapl-2.0.42.1-1 (more bug fixing for MPSS 3.3, merge with master branch 2.0.42)
-
-dapltest: increase DTO evd size to prevent CQ overflow on limit_rpost test
-Creation of reserved SP moves EP state to DAT_EP_STATE_RESERVED even in failure cases. Reserv
-dapl: fix string bug in dapls_dto_op_str
-mpxyd: change affinity to avoid overlapping cores with 2 MICs in same socket
-mpxyd: add mpxyd.conf option to disable proxy-in service
-mpxyd: set default seg size to 128KB, down from 256KB
-install: base package, without mpss, incorrectly tries to install /usr/sbin as file
-mpxyd: scale-up with MPI dapl:dapl hits low mem issue with 1 byte traffic patterns
-mpxyd: add MIC client and device id to logging
-add new po-pi rdma write perf profile
-change default wrc entries from 1024 to 512
-mpxyd: MIC scale-up issue with MPI gather workloads, I_MPI_FABRICS=dapl:dapl
-mpxyd: remove unnecessary logging
-mcm: mpxyd error event of m_pi_prep_rcv_q: ERR: ib_qp == 0
-
-Release dapl-2.0.41.2-1 (bug fixing for MPSS 3.3)
-
-mpxyd,mcm: increase default CM timers for better out of box scaling
-mpxyd,mcm: changes for backward compatibility with older v4 MIC clients
-mpxyd: allow proxy_out WR stalls instead of immediate error
-mpxyd: increase max open files limit for proxy service
-
-Release dapl-2.0.41.1-1 (added proxy-in support)
-
-dapltest: change server port, from 45278 to 62000, out of registered IANA range
-dapltest: set default limit max to 1000
-dapltest: update scripts for regression testing purposes
-dapltest: Add final send/recv "sync" for transaction tests.
-mpxyd: append _free to dqconn/dqlisten for readability, improve logging
-mcm: check for shared CQs in PI mode \96 MXS-HST scaling issues
-mcm: add dev_id to all mpxyd commands
-mpxyd: scale-up improvements to support 200-300 processes per MIC
-mpxyd: serialize MD cm port space usage, add swap to rej call, resend dropped rej
-mcm: serialize CM cmds on ev_ep, add dev_id, increase dev_open listen backlog
-openib: return open failure if port not active
-mpxyd: disable inside box support via scif only, use IB
-mcm: add host to mic cross socket support to proxy-in service to improve performance
-dat: reduce log level on library load failures
-mic: take cm_msg rsvd byte for segment size exchange, power of 2
-mpxyd: init proxy WC buffer queues for new queue management
-mpxyd: simplify WR and WC queue management and fix inline post send
-mpxyd: 64KB segment sizes hang with MPI IMB pingpong cross socket
-mpxyd: add new M_READ_FROM_DONE state for send WR's and add more profiling options
-mpxyd: purge posted send data only if inline
-mpxyd: move to CONN state immediately on RTU_IN
-dtest: fix options on query times printf
-mpxyd: fix ibctx leak with device open
-dtest: add new -Q option to get provider list and query and time each
-openib: cleanup init/fini and device close on all providers
-dat: dat_ia_open should close provider after failure
-mpxyd: sync PI WC trigger to PO MP_SIG
-mpxyd: improve QP destruction to manage QP1 and QP2 variations
-mpxyd: proxy out doesn't release proxy buffer as quickly as necessary
-dtest: add times for open_query, remove sleep
-common: add provider name and len to DTO error logging
+Release 2.1.5 (OFED 3.18 RC3)
+
+ dat.conf: update comments regarding versions
+ dtest: add logging of provider private data size with -v
+ scm: remove use of msg.resv field for process id logging
+ cma: report correct CM req private data size on query
+ mpxyd: memset ib_wr structure before post_send on WC and WR requests
+ mcm: add HST side provider support for device without inline data capability
+ ucm: CM changes for UD extended port space and indexer
+ ucm: add device support for new port space hash table
+ ucm: allocate/free AH hash table for UD endpoint types
+ ucm: check for AH caching when destroying via UD extension
+ ucm: optimizations for large scale UD communication management
+ mpxyd: use wr opcode instead of wc opcode to support logging on error cases
+ mcm: HST->MXS mode, using RDMA_WRITE_WITH_IMM, fails with dtest -w
+ dapl: aarch64 support for linux
+ dapltest: add scripts to dist, set default device to IPoIB
+ mpxyd: add wc_flags to proxy work completions
+
+Release 2.1.4 (OFED 3.18 RC1)
+
+mpxyd: fix typo in configuration file
+cma: RR attributes moved to common ib_cm struct
+mpxyd: tx thread incorrectly sleeps with negative pi_rw_cnt value
+dat.conf: add entries for True Scale qib device
+mpxyd: add support for devices without inline data support
+ucm: long disconnect times with many-to-one applications
+openib: add inline data support check during device open
+cleanup ib/cm attribute management across openib providers
+dapltest: fix -Werror=format-security issue with printf
+Release 2.1.3 (targeting OFED 3.18)
+dapl: mpxyd service changes to support multi-thread single-core option
+dapl: add rdma_write_imm and write only option to dtest
+ucm: add time wait override capability for CM services
+common: dapl_ep_free must serialize CM object destroy
+dtestx: allow scale up to 1000 EP's
+ucm: RTU not retransmitted in TIMEWAIT state
+mpxyd: increase max open files for service
+mpxyd: DTO completion ERR: status 12, op RDMA_WRITE running MPI alltoall test
+mcm: HST->MXS mode incorrectly signals multiple fragments per WR
+mcm: add segmentation to HST->MXS mode for improved performance
+mpxyd: set global seg_sz to 128KB for proxy data service
+openib: add port_num to provider named attributes
+mcm: provide CPU family/model attribute on both host and mic sides
+dtestx: update IB extension example test with new v2.0.9 features
+dtest: add dtestsrq for SRQ example and provider testing
+common: add srq support for openib verbs providers
+openib: add IB UD cm_free/ah_free extension support in UCM provider
+openib: add new TIMEWAIT state for CM
+extension: add IB UD extensions to reduce provider CM and AH memory footprint
+mpxyd/mcm: add provider specific attribute DAT_IB_PROXY_VERSION
+mpxyd: log warning if running in COMPAT mode
+add provider and proxy support for GUID across platform
+common: return appropriate handles with affiliated EP and EVD async events
+
+Release 2.1.2 (OFED 3.12-1)
+mpxyd: add global routing support for proxy connections
+mcm: only call mix_get_attr if running on MIC
+openib: modify check for link_layer to handle unspecified
+dapl: add support for the s390x platform
+dtest server exchange connection info with client
+mpxyd: 2 MICs in same numa_node will overlap CPU affinity, don't reset base
+mcm: implement proxy mix_prov_attr function, add fields CPU model and family
+mpxyd: tx thread may not be signaled on small segment writes
+
+Release 2.1.1 (OFED 3.12-1 RC1)
+common: add provider name to log messages
+mpxyd: log warning message if numa_node invalid include debuginfo with build
+build: include debuginfo with build
+mpxyd: tx thread doesn't sleep during no pending IO state
+mpxyd: change MIC cpu_mask to per numa node instead of adapter
+mpxyd: set to MXS mode if device numa_node is invalid (-1)
+mpxyd: MXS based alltoall benchmark hangs or returns post_send timeout
+mpxyd: add IO profile capabilities to help debug alltoall stall cases
+mpxyd: retry stalled inline post_send, init m_idx only when signaled
+
+Release 2.1.0 (OFED 3.12-1, MIC support added)
+build: add missing NEWS file
+update autogen.sh
+add MCM provider and MPXYD service to build
+mpxyd: service startup script and configuration file
+add readme for MCM provider and MPXYD service
+update Copyright dates
+add new MIC RDMA proxy service daemon (MPXYD)
+add new dapl MIC provider (MCM) to support MIC RDMA proxy services
+MCM: new MIC provider and proxy service definitions
+cleanup build warnings
+common: add CQ,QP,MR abstractions for new MIC provider and data proxy service
+openib: cleanup, use inet_ntop for GIDs, remove some logs, destroy pipes on release
+common: new dapls_evd_cqe_to_event call, cqe to event
+common: init ring_buffer, assign hd/tl pos in range
+allow log level changes during device open
+ucm: fix cm rbuf setup, include grh pad on initialization
+ucm: remove duplicate async_event code, use common async event call
new lightweight open_query/close_query IB extension for fast attribute query
-mpxyd: need CM to QP linking with CM references
-dist: ib collective and MIC extension include files missing
-dapltest: the quit command is missing changes for -n option.
-NULL undefined on Fedora, incorrectly using kernel stddef.h
-Merge branch 'proxy' of ssh://beany.openfabrics.org/home/ardavis/scm/dapl into proxy
-ucm: fix CM service, initial rcv msg posts incorrect
-ucm: add/cleanup debug log information
-scm: add/cleanup debug log information
-makefile: update for MCM proxy-in changes
-dtest: update for ep_mode on MCM providers
-mpxyd.conf: updated for proxy-in parameters
-mpxyd: proxy-in added to proxy-out service to increase cross socket performance
-mcm: add proxy in support to MCM provider and MPXYD interface
-open_ib common: qp, cq, and post_recv changes for proxy-in
-commom: add lmr support for proxy in service
-new definitions and states for CCL Proxy-in support
+dtestcm: add more detailed debug during disconnect phase
+cma: long delays when opening cma provider with no IPoIB configured
+common: new debug levels for low system memory, IA stats, and package info
+build: remove library check for mverbs with --enable-fca
+IB extension: segfault in create collective group with non-vector type IA handle"
+build: change configure help to correctly state collective default=none
+
+ Release Notes for
+ OFED 3.18 DAPL Release 2.1.5
+ May 2015
+
+ User space libraries/utilities for Direct Access Transport (DAT) v2.0. DAT is
+ a transport-independent, platform-independent Application Programming
+ Interface that supports RDMA (remote direct memory access) devices.
+ Note: v1.2 is no longer supported and will not be included with OFED releases
+
+ MIC support has been added in dapl-2.1.0, see README.mcm for build and install details.
- Release Notes for
- Gamma 3.3 and OFED 1.2 DAPL Release
- June 2007
+ MIC support is provided with the new MCM provider and MPXYD service.
+ MCM requires the Intel(R) MPSS 3.x (YOCTO) release for Linux to be installed on your system.
+ MPSS 3.x for Linux can be downloaded from: http://software.intel.com/mic-developer
+ For latest documentation and packages: //www.openfabrics.org/downloads/dapl/
- DAPL GAMMA 3.3/OFED 1.2 RELEASE NOTES
+ uDAPL v2 (dapl-2.1.5)
- This release of the DAPL reference implementation
- is timed to coincide with OFED release 1.2 of the
- Open Fabrics (www.openfabrics.org) software stack.
+ Build Notes:
+ ------------
+
+ # NON_DEBUG build/install example for x86_64, OFED targets
+ ./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
+ make install
+
+ # DEBUG build/install example for x86_64, using OFED targets
+ ./configure --enable-debug --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
+ make install
+
+ # COUNTERS build/install example for x86_64, using OFED targets
+ ./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include -DDAPL_COUNTERS"
+ make install
+
+ Provider descriptions and PROS/CONS (cma, scm, ucm)
+ ---------------------------------------------------
+
+ 1. CMA - uses OFA rdma_cm to setup QP's. IPoIB, ARP, and SA queries required.
+
+ Provider name: ofa-v2-ib0
+ PROs: OFA rdma_cm has the most testing across many applications.
+ Supports both iWARP and IB.
+
+ CONs: Serialization of conn processing with kernel based CM service
+ Requires IPoIB ARP for name resolution, storms
+ Requires SA for path record queries for IB fabrics.
+ Conn Request private data limited to 52 bytes.
+
+ Settings for larger clusters (512+ cores):
+
+ setenv DAPL_CM_ROUTE_TIMEOUT_MS 20000
+ setenv DAPL_CM_ARP_TIMEOUT_MS 10000
+
+ 2. SCM - uses sockets to exchange QP information. IPoIB, ARP, and SA queries NOT required.
+
+ Provider name (connectx): ofa-v2-mlx4_0-1
+ PROs: Each rank has own instance of socket cm. More private data with requests.
+ Doesn't require path-record lookup.
+
+ CONs: Socket resources grow with scale-out, serialization of
+ connections with kernel based tcp sockets,
+ Competes for MPI socket resources/port space and other TCP applications.
+ Sockets remain in TIMEWAIT state for minutes after closure.
+ Requires ARP for name resolution.
+ Doesn't support iWARP devices.
+
+ Settings for larger clusters (512+ cores):
+
+ setenv DAPL_ACK_RETRY 7 /* IB RC Ack retry count */
+ setenv DAPL_ACK_TIMER 20 /* IB RC Ack retry timer */
+
+ 3. UCM - use's IB UD QP to exchange QP info. Sockets, ARP, IPoIB, and SA queries NOT required.
+
+ Provider name (connectx): ofa-v2-mlx4_0-1u
+ PROs: Each rank has own instance of CM in user process
+ Resources fixed per rank regardless of scale-out size
+ No serialization of user or kernel resources establishing connections,
+ Simple 3-way msg handsake, CM messages fit in inline data for lowest message latency,
+ Supports alternate paths
+ No address resolution required.
+ No path resolution required.
+
+ CONs: New provider with limited testing, a little tougher to debug.
+ Doesn't support iWARP
+
+ Settings for larger clusters (512+ cores):
+
+ setenv DAPL_UCM_REP_TIME 2000 /* REQUEST timer, waiting for REPLY in millisecs */
+ setenv DAPL_UCM_RTU_TIME 2000 /* REPLY timer, waiting for RTU in millisecs */
+ setenv DAPL_UCM_CQ_SIZE 2000 /* CM completion queue */
+ setenv DAPL_UCM_QP_SIZE 2000 /* CM message queue */
+ setenv DAPL_UCM_RETRY 7 /* REQUEST and REPLY retries */
+ setenv DAPL_ACK_RETRY 7 /* IB RC Ack retry count */
+ setenv DAPL_ACK_TIMER 20 /* IB RC Ack retry timer */
+
+ CM Performance: CPS profile for cma, scm, and ucm v2 uDAPL providers:
+ -----------------------------------------------------------------------
+ Intel SR1600 Servers with Xeon(R) CPU X5570 @ 2.93GHz
+ Urbanna Platform - 2 node, 8 cores per node, Mellanox MLX4 IB QDR, no switch.
+
+ dtestcm (server/client):
+
+ cma: Connections: 183.21 usec, CPS 5458.31 Total 0.18 secs, poll_cnt=3403, Num=1000
+ scm: Connections: 178.80 usec, CPS 5592.93 Total 0.18 secs, poll_cnt=2344, Num=1000
+ ucm: Connections: 122.43 usec, CPS 8167.93 Total 0.12 secs, poll_cnt=2609, Num=1000
+
+ dapl_cm_bw: MPI uDAPL/CM profiling application (all-to-all connections, all ranks)
+
+ CMA
+ 2 Connect times (10): Total 0.0020 per 0.0002 CPS=4997.98
+ 4 Connect times (40): Total 0.0077 per 0.0002 CPS=5224.59
+ 8 Connect times (240): Total 0.0276 per 0.0001 CPS=8710.76
+ 16 Connect times (1120): Total 0.1194 per 0.0001 CPS=9379.37
+ 32 Connect times (4800): Total 6.1949 per 0.0013 CPS=774.83
+
+ SCM
+ 2 Connect times (10): Total 0.0024 per 0.0002 CPS=4103.61
+ 4 Connect times (40): Total 0.0060 per 0.0002 CPS=6622.41
+ 8 Connect times (240): Total 0.0206 per 0.0001 CPS=11634.15
+ 16 Connect times (1120): Total 9.0118 per 0.0080 CPS=124.28
+ 32 Connect times (4800): Total 21.0198 per 0.0044 CPS=228.36
+
+ UCM
+ 2 Connect times (10): Total 0.0014 per 0.0001 CPS=7353.27
+ 4 Connect times (40): Total 0.0045 per 0.0001 CPS=8816.19
+ 8 Connect times (240): Total 0.0191 per 0.0001 CPS=12582.44
+ 16 Connect times (1120): Total 0.0799 per 0.0001 CPS=14017.68
+ 32 Connect times (4800): Total 0.3337 per 0.0001 CPS=14385.21
+
+
+ BKM for build and running new DAPL library on your cluster without any impact on existing OFED install:
+ -------------------------------------------------------------------------------------------------------
+
+ Note: example for user /home/user1, (assumes /home/user1 is exported) and MLX4 adapter, port 1
+
+ Download latest 2.x package: http://www.openfabrics.org/downloads/dapl/dapl-2.0.42.tar.gz
+
+ untar in /home/user1
+ cd /home/user1/dapl-2.0.42
+ ./configure LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
+ make
+
+ Create /home/user1/dat.conf with following 3 lines. (entries with path to new libraries):
+
+ ofa-v2-mlx4_0-1u u2.0 nonthreadsafe default /home/user1/dapl-2.0.42/dapl/udapl/.libs/libdaploucm.so.2 dapl.2.0 "mlx4_0 1" ""
+ ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default /home/user1/dapl-2.0.42/dapl/udapl/.libs/libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
+ ofa-v2-ib0 u2.0 nonthreadsafe default /home/user1/dapl-2.0.42/dapl/udapl/.libs/libdaplcma.so.1 dapl.2.0 "ib0 0" ""
+
+ Run uDAPL application or Intel MPI that uses uDAPL, with (assuming mlx4_0 adapters) following:
+
+ setenv DAT_OVERRIDE=/home/user1/dat.conf
+ setenv LD_LIBRARY_PATH=/home/user1/dapl-2.0.42/dapl/udapl/.libs:$LD_LIBRARY_PATH
+
+ If running Intel MPI and uDAPL socket cm, set the following:
+
+ setenv I_MPI_DAPL_PROVIDER=ofa-v2-mlx4_0-1
+
+ or if running Intel MPI and uDAPL IB UD cm, set the following (recommended):
+
+ setenv I_MPI_DAPL_PROVIDER=ofa-v2-mlx4_0-1u
+
+ or if running Intel MPI and uDAPL rdma_cm, set the following:
+
+ setenv I_MPI_DAPL_PROVIDER=ofa-v2-ib0
+
+
+ Summary of Fixes/Changes:
+ -------------------------
+
+ Release 2.1.5 (OFED 3.18 RC3)
+ update release notes, readme
+ dat.conf: update comments regarding versions
+ dtest: add logging of provider private data size with -v
+ scm: remove use of msg.resv field for process id logging
+ cma: report correct CM req private data size on query
+ mpxyd: memset ib_wr structure before post_send on WC and WR requests
+ mcm: add HST side provider support for device without inline data capability
+ ucm: CM changes for UD extended port space and indexer
+ ucm: add device support for new port space hash table
+ ucm: allocate/free AH hash table for UD endpoint types
+ ucm: check for AH caching when destroying via UD extension
+ ucm: optimizations for large scale UD communication management
+ mpxyd: use wr opcode instead of wc opcode to support logging on error cases
+ mcm: HST->MXS mode, using RDMA_WRITE_WITH_IMM, fails with dtest -w
+ dapl: aarch64 support for linux
+ dapltest: add scripts to dist, set default device to IPoIB
+ mpxyd: add wc_flags to proxy work completions
+
+ Release 2.1.4 (OFED 3.18 RC1)
+ mpxyd: fix typo in configuration file
+ cma: RR attributes moved to common ib_cm struct
+ mpxyd: tx thread incorrectly sleeps with negative pi_rw_cnt value
+ dat.conf: add entries for True Scale qib device
+ mpxyd: add support for devices without inline data support
+ ucm: long disconnect times with many-to-one applications
+ openib: add inline data support check during device open
+ cleanup ib/cm attribute management across openib providers
+ dapltest: fix -Werror=format-security issue with printf
+ Release 2.1.3 (targeting OFED 3.18)
+ dapl: mpxyd service changes to support multi-thread single-core option
+ dapl: add rdma_write_imm and write only option to dtest
+ ucm: add time wait override capability for CM services
+ common: dapl_ep_free must serialize CM object destroy
+ dtestx: allow scale up to 1000 EP's
+ ucm: RTU not retransmitted in TIMEWAIT state
+ mpxyd: increase max open files for service
+ mpxyd: DTO completion ERR: status 12, op RDMA_WRITE running MPI alltoall test
+ mcm: HST->MXS mode incorrectly signals multiple fragments per WR
+ mcm: add segmentation to HST->MXS mode for improved performance
+ mpxyd: set global seg_sz to 128KB for proxy data service
+ openib: add port_num to provider named attributes
+ mcm: provide CPU family/model attribute on both host and mic sides
+ dtestx: update IB extension example test with new v2.0.9 features
+ dtest: add dtestsrq for SRQ example and provider testing
+ common: add srq support for openib verbs providers
+ openib: add IB UD cm_free/ah_free extension support in UCM provider
+ openib: add new TIMEWAIT state for CM
+ extension: add IB UD extensions to reduce provider CM and AH memory footprint
+ mpxyd/mcm: add provider specific attribute DAT_IB_PROXY_VERSION
+ mpxyd: log warning if running in COMPAT mode
+ add provider and proxy support for GUID across platform
+ common: return appropriate handles with affiliated EP and EVD async events
+
+ Release 2.1.2 (OFED 3.12-1)
+ mpxyd: add global routing support for proxy connections
+ mcm: only call mix_get_attr if running on MIC
+ openib: modify check for link_layer to handle unspecified
+ dapl: add support for the s390x platform
+ dtest server exchange connection info with client
+ mpxyd: 2 MICs in same numa_node will overlap CPU affinity, don't reset base
+ mcm: implement proxy mix_prov_attr function, add fields CPU model and family
+ mpxyd: tx thread may not be signaled on small segment writes
+
+ Release 2.1.1 (OFED 3.12-1 RC1)
+ common: add provider name to log messages
+ mpxyd: log warning message if numa_node invalid include debuginfo with build
+ build: include debuginfo with build
+ mpxyd: tx thread doesn't sleep during no pending IO state
+ mpxyd: change MIC cpu_mask to per numa node instead of adapter
+ mpxyd: set to MXS mode if device numa_node is invalid (-1)
+ mpxyd: MXS based alltoall benchmark hangs or returns post_send timeout
+ mpxyd: add IO profile capabilities to help debug alltoall stall cases
+ mpxyd: retry stalled inline post_send, init m_idx only when signaled
+
+ Release 2.1.0 (OFED 3.12-1, MIC support added)
+ build: add missing NEWS file
+ update autogen.sh
+ add MCM provider and MPXYD service to build
+ mpxyd: service startup script and configuration file
+ add readme for MCM provider and MPXYD service
+ update Copyright dates
+ add new MIC RDMA proxy service daemon (MPXYD)
+ add new dapl MIC provider (MCM) to support MIC RDMA proxy services
+ MCM: new MIC provider and proxy service definitions
+ cleanup build warnings
+ common: add CQ,QP,MR abstractions for new MIC provider and data proxy service
+ openib: cleanup, use inet_ntop for GIDs, remove some logs, destroy pipes on release
+ common: new dapls_evd_cqe_to_event call, cqe to event
+ common: init ring_buffer, assign hd/tl pos in range
+ allow log level changes during device open
+ ucm: fix cm rbuf setup, include grh pad on initialization
+ ucm: remove duplicate async_event code, use common async event call
+ new lightweight open_query/close_query IB extension for fast attribute query
+ dtestcm: add more detailed debug during disconnect phase
+ cma: long delays when opening cma provider with no IPoIB configured
+ common: new debug levels for low system memory, IA stats, and package info
+ build: remove library check for mverbs with --enable-fca
+ IB extension: segfault in create collective group with non-vector type IA handle"
+ build: change configure help to correctly state collective default=none
+
+ Release 2.0.42 fixes (OFED 3.12 GA)
+ dapltest: increase DTO evd size to prevent CQ overflow on limit_rpost test
+ dapltest: RSP limit test fails. Creation of reserved SP moves EP state to DAT_EP_STATE_RESERVED in error cases.
+ dapl: fix string bug in dapls_dto_op_str
+
+ Release 2.0.41 fixes (OFED 3.12 RC1)
+ dapltest: change server port, from 45278 to 62000, out of registered IANA range
+ dat: lower log level on load errors of provider library
+ dat: dat_ia_open should close provider after failure
+ dapltest: set default limit max to 1000
+ openib: add new provider specific attributes
+ dapltest: update scripts for regression testing purposes
+ dapltest: Add final send/recv "sync" for transaction tests.
+
+ Release 2.0.40 fixes (OFED 3.12)
+ dist: ib collective extension include files missing
+ dapltest: the quit command is missing changes for -n option
+ dat.conf: remove v1, add Mellanox Connect-IB and Intel Xeon Phi MIC
+ NULL undefined on Fedora, incorrectly using kernel stddef.h
+
+ Release 2.0.39 fixes (OFED 3.5-2 GA)
+ dapltest: fix endian swap issue with performance test
+ scm: getifaddrs modfications for better out of the box experience
+ ucm, scm: UD mode triggers list_head assert with large scale alltoall test
+
+ Release 2.0.38
+ dapltest: add -n parameter to override default server port number (45278)
+ ucm,scm: UD mode creates many CR objects per EP that needs cleaned up
+ cma: add DAPL_CM_TOS environment variable to enable passing a TOS to the RDMA CM
+
+ Release 2.0.37
+ common: add support for ia name during dat_ia_query
+ common: dapl_os_atomic_inc/dec() not working as expected on ppc64 machines.
+ dapltest: ppc64 endian issue with exchanged mem handle and address
+
+ Release 2.0.36
+ scm: increase ACK timeout to 20 for a default value to match other providers.
+ common: allow qp modify in init state
+ common: check for valid states during ep posting
+ dat.conf: keep list of providers in order for backward compatibility
+ ucm: record and silently drop a duplicate reject CM message
+ windows: new version of getlocalipaddr not portable
+ dapltest: DFLT_QLEN is defined in multiple tests
+
+ Release 2.0.35
+ config/build: remove post/postun hacking used to modify dat.conf
+ config: clean up help option displays with ext-type options
+ windows: Provide auto-detect between RoCE and Infiniband for Windows.
+ ucm: update UD cm provider to support new CM stat and error counters
+ scm: update socket cm provider to support new CM stat and error counters
+ commom: add cm, link, and diag event counters in IB extended builds
+ scm: use ioctl SIOCIFCONF to get complete list of configured netdev interfaces
+ ucm: UD send failures at scale, ucm_send ERR: get_smsg(hd=149,tl=150)
+ scm: fix retry count on connection pending timeout
+ ucm: cleanup debug message, ntohl on p_size is incorrect
+ cma, scm, ucm: allow EP (QP) creation without EVD (CQ)
+ common: add DAPL_DBG_TYPE_CM_STATS (0x40000) to debug log options
+ common: dapls_ep_flush_cq will segfault when no CQ is attached to EP
+ common: ep_create should allow max_request_iov attribute setting of zero
+ common: add check for NULL handle on ext calls, SRQ free, and helper functions
+ common: add missing sub-types to dat_strerror()
+ common: extended CR event processing missing rejects on errors
+ ucm: incorrectly sends user reject during CR callback errors
+ common: change dbg level on CR callback if not listening on SP
+ scm: incorrectly sends user reject during CR callback errors
+ dat: add check for NULL handle on IA calls
+ cma,scm,ucm: extra reference on EP, with RSP, causes dat_ep_free() to hang
+ common: RSP service points incorrectly freed during CR callback
+ common: clean up dat_rsp_create log message
+ common: cleanup debug message on EVD overflows
+ scm: return correct event error code when remote host refuses requests
+ dapltest: server CR EVD is too small for multi-client configurations.
+ Common: CR EVD overflow causes segfault.
+
+ Release 2.0.34
+ scm: change debug message level for listen/bind errors
+ common: increase default IB ack timer from 16 to 20
+ common: remote ia address null pointer creates seg fault
+ common: posting events on full queue returns wrong error code
+ common: dat_ep_modify seg faults with null ep_param ptr
+ common: dat_evd_free seg faults with resized software EVD
+ common: remove assert for incorrect events during cm_request
+ dat: dat_cno_query with NULL cno_handle causes segmentation fault
+ scm: dat_psp_create returns wrong error code on bind/listen failure
+ scm: socket connect request count is reset improperly on retry
+ scm: when hostname has loopback addr assigned, default to eth0 instead of failing
+ scm: add port number to error log during hca_open failures
+ common: query calls return incorrect IA handle to consumer
+ common: srq create asserts with !dapl_llist_is_empty(head) failed
+
+ Release 2.0.33
+ scm,ucm: fix compatibility issues and set minimum protocol support
+ build: link librdmacm dependency to ib_acm usage for ucm and scm providers
+ build: add selective enable/disable-xxx build switch for each provider
+ build: add extended header files to EXTRA_DIST and fix missing backslash
+ build: set IB extended coll-type to none by default
+ common: change errno mapping of EINVAL to DAT_INVALID_PARAMETER
+ build: add IB collective and FCA provider to dapl build package as an option
+ common: add new dapls_evd_post_event_ext call for extended events
+ ucm: add support for IB collective providers
+ scm: add support for IB collective providers
+ cma: add support for IB collective providers
+ common: add supported collective types in named attributes for query
+ common: add collective call mappings via standard dapli_post_ext()
+ common: new debug bitmask definition for extension logging
+ common: new IB collective provider for Mellanox Fabric Collective Agent
+ dat: add definitions for MPI offloaded collectives in IB transport extensions
+ common: cleanup debug messages when building with ibacm feature
+
+ Release 2.0.32 fixes (OFED 1.5.3 GA):
+
+ cma: reduce output log level in disconnect from WARN to CM_WARN
+ ucm: delay freeing of active side UD cm object in case RTU is dropped
+ ucm: cm object needs to be on work queue before req sent on wire
+ ucm,scm: remove use of usec_sleep delays and use events for disc and destroy
+ common: reduce default max inline data size because of performance anomaly
+ common: dapls_evd_dto_wait() dbg message should print status and not errno
+ ucm, scm: exchange max_qp_rd_atom and limit outstanding requests
+ scm: retry socket connect on ECONNREFUSED under heavy load
+ common: qp modify RTR using wrong ep attribute parameter for dest_rd_atomic
+
+ Release 2.0.31 fixes (OFED 1.5.3 RC1):
+
+ common: clean up build warning for unused variable event_ptr
+ scm, ucm: set RAI_NOROUTE flag with rdma_getaddrinfo() call to avoid blocking.
+ cma: definition for dapl_sp_remove_ep() is missing in cm.c
+ libdat: static provider entries created for local SR database not freed
+ libdat: memory leak in static registration during parsing
+ common: increase default IB inline send threshold to 400
+ common cq: a mixup of errno and the -1 return from poll in dapls_wait_comp_channel
+ ucm: release UD cm objects after AH is exchanged to avoid duplicate request drops
+ ucm: decrease timeout retry count for disconnect requests
+ ucm: hold lock when sending cm_msgs to sync timer start with packet send
+ ucm: add debugging to include process id for better scale up debug aids
+ cma: disconnect can block for excessive times waiting for rdma_cm DREP timeout
+ ucm: configure the recv channel FD to non-blocking
+ windows: Missing librdmacm include path for build
+ debug build: only timestamp if sending to stdout to avoid performance hit
+ common: print out errors on free build and not just debug builds
+ cma: fix debug build issue
+ scm, ucm: MPI spawn test on oversubcribed server taking excessive time to complete
+ common: add high resolution time stamps and thread id to sdtout debug logs
+ common: modify debug in dat_evd_dequeue to reduce noise, only output on non-empty
+ cma: rdma_destroy_id called twice during device open bind error
+ common: dat_evd_dequeue (poll_cq) fails with invalid parameter after EP (qp) free
+ ucm: allow configuration of CM burst (signal) threshold on posting
+ cma: fix debug build
+ windows: debug version of windows does not build.
+ Allow DAPL out of band connection models to use ibacm to obtain path record data.
+ ucm: add missing map file for UCM provider
+ ibal: delay QP transition during disconnect phase
+ Revert "ibal: delay QP transition during disconnect phase"
+ ibal: delay QP transition during disconnect phase
+ common: restructure EVD processing to handle EP destruction phase
+ ibal: sync QP destruction and device close
+ ucm: remove unnecessary debug warning in async callback
+
+ v1.2 Package:
+
+ Release 1.2.19 fixes (OFED 1.5.2 GA):
+
+ common, cma: disconnect and cleanup CR linkings after DTO error on EP
+ common: race conditions with DTO error, disconnect and dapl_reset_ep
+ common: add new dapl_os_sleep_usec() function
+ configure: need a false conditional for verbs attr.link_layer member check
+ config: add conditional check for new verbs port_attr.link_layer
+ cma, scm: new provider entries for Mellanox RDMA over Ethernet device for uDAPL v1.2
+ cma: memory leak of verbs CQ and completion channels created during dat_ia_open
+ cma: memory leak of FD's (pipe) created during dat_evd_create
+
+
+--- HISTORY -----------
+
+ OFED 1.5.1 RELEASE NOTES
+ uDAPL v1 (1.2.16-1) and v2 (2.0.27-1)
+
+ ----------------
+
+ * New Features (v2 only) - UCM provider with IB UD based CM per process.
+ More scalable then rdma_cm (cma) or socket cm (scm).
+ ----------------
+
+ * Bug Fixes
+
+ V2.0 Package
+
+ Release 2.0.27
+ windows: add scm makefile
+ windows does not require rdma_cma_abi.h, move the include from common code
+ windows patch to fix IB_INVALID_HANDLE name collision
+ scm: dat_ep_connect fails on 32bit servers
+ undefined symbol: dapls_print_cm_list
+ cleanup CM object lock before freeing CM object memory
+ destroy verbs completion channels created via ia_open or ep_create.
+ package: update Copyright file and include the 3 license files in distribution
+ common: when copying private_data out of rdma_cm events, use the
+ cma: fix referencing freed address
+ dapl: move close device after async thread is done
+
+ Release 2.0.26
+ openib_common: add check for both gid and global routing in RTR
+ openib_common: remote memory read privilege set multi times
+ ucm, scm: DAPL_GLOBAL_ROUTING enabled causes segv
+
+ Release 2.0.25
+ winof scm: initialize opt for NODELAY setsockopt
+ winof cma: windows definition for EADDRNOTAVAIL missing
+ scm: client side setsockopt NODELAY fails if data arrives before setting
+ cma: setup_listener Cannot assign requested address
+ common: seg fault in dapl_evd_wait with multi-thread application using CNO's.
+ ucm: inbound DREQ/DREP handshake should transition QP.
+ winof: Remove duplicate include of comp_channel.cpp from cm.c as it is
+ included in opensm_ucb/device.c.
+
+ Release 2.0.24
+ winof: Utilize WinOF version of inet_ntop() for Windows OSes which do not
+ support inet_ntop().
+ ucm: windows build issue with new CQ completion channel
+ winof: add ucm provider to windows build
+ winof: add missing build files for ibal, scm
+ scm: connection peer resets under heavy load, incorrect event on error
+ ucm: increase default reply and rtu timeout values.
+ ucm: change some debug message levels and add check for valid UD REPLY during retries.
+ ucm: increase timers during subsequent retries
+ ucm, scm: address handles need destroyed when freeing Endpoints with UD QP's.
+ openib_common: ignore pd free errors, clear pd_handle and return.
+ ucm: using UD type QP's, ucm reports wrong reject event when user rejects AH resolution request.
+ ucm, scm, cma: Fix CNO support on DTO type EVD's
+ ucm: fix lock init bug in ucm_cm_find
+ ucm: fix build problem with latest windows ucm changes
+ ucm: The HCA should not be closed until all resources have been released.
+ ucm: Fix build warning when compiling on 32-bit systems.
+ ucm: Trying to deregister the same memory region twice leads to an
+ dat: reduce debug message level when parsing for location of dat.conf
+ ucm: update ucm provider for windows environment
+ ucm: add timer/retry CM logic to the ucm provider
+
+ Release 2.0.23
+ cma: cannot reuse the cm_id and qp for new connection, must reallocate a new one.
+ scm, cma: update DAPL cm protocol revision with latest address/port changes
+ ucm: modify IB address format to align better with sockaddr_in6
+ Add definition for getpid similar to that used by the other dtest apps.
+ WinOF provides a common implementation of gettimeofday that should
+ The completion manager was updated to provide an abstraction that
+ dtestcm: remove IB verb definitions
+ dtest, dtestx: remove IB verb definitions
+ scm: tighten up socket options to insure similiar behavior on Windows and Linux.
+ cma: improve serialization of destroy and event processing
+ scm: improve serialization of destroy and state changes
+ common: no cleanup/release code for timer thread
+ scm, cma: dapli_thread doesn't always get teminated on library close.
+ ucm: tighten up locking with CM processing, state changes
+ ucm: For UD type QP's, return CR p_data with CONN_EST event on passive side.
+ ucm: cleanup extra cr/lf
+ ucm: fix issues with UD QP's.
+ winof: Convert windows version of dapl and dat libaries to use private heaps.
+ dtest, dtestx: modifications for UD QP testing with ucm provider.
+ scm, ucm: UD QP support was broken when porting to common openib code base.
+ cma: cleanup warning with unused local variable, ret, in disconnect
+ cma: remove debug message after rdma_disconnect failure
+ scm: socket errno check needs O/S dependent wrapper
+ dapltest: update script files for WinOF
+ cma: conditional check for new rdma_cm definition.
+
+ Release 2.0.22
+ dapltest: add mdep processor yield and use with dapltest
+ ucm: Add new provider using a DAPL based IB-UD cm mechanism for MPI implementations.
+
+ Release 2.0.21
+ scm: Fix disconnect. QP's need to move to ERROR state in
+ modify dtest.c to cleanup CNO wait code and consolidate into
+ CNO events, once triggered will not be returned during the cno wait.
+ CNO support broken in both CMA and SCM providers.
+ common osd: include winsock2.h for IPv6 definitions.
+ common osd: include w2tcpip.h for sockaddr_in6 definitions.
+ DAPL introduced the concept of directly waiting on the CQ for
+ dapltest: Implement a malloc() threshold for the completion reaping.
+ scm: handle connected state when freeing CM objects
+ scm, dtest: changes for winof gettimeofday and FD_SETSIZE settings.
+ scm: set TCP_NODELAY sockopt on the server side for sends.
+ remove obsolete files in dapl/udapl source tree
+ dtestcm: add UD type QP option to test
+ scm: destroy QP called before disconnect
+ cma: add support for rdma_cm TIME_WAIT event.
+ scm: remove old udapl_scm code replaced by openib_scm.
+ winof: fix issues after consolidating cma, scm code base.
+ cma: lock held when exiting as a result of a rdma_create_event_channel failure.
+ windows: all dlist functions have been moved to the header file.
+ dtestcm windows: add build infrastructure for new dtestcm test suite
+ openib_common: reorganize code base to share common mem, cq, qp, dto functions
+ scm: fixes and optimizations for connection scaling
+ scm: double the default fd_set_size
+ scm: EP reference in CR should be cleared during ep_destroy
+ dtestx: fix conn establishment event checking
+ dtestcm: new test to measure dapl connection rates.
+
+ Release 2.0.20
+ common,scm: add debug capabilities to print in-process CM lists
+ scm: disconnect EP before cleaning up orphaned CR's during dat_ep_free
+ dapltest: windows scripts updated
+ scm: private data is not handled properly via CR rejects.
+ scm: cleanup orphaned UD CR's when destroying the EP
+ scm: provider specific query for default UD MTU is wrong.
+ scm: update CM code to shutdown before closing socket
+ dapltest: windows script dt-cli.bat updated
+ dapl/windows cma provider: add support for network devices based on index
+ openib: remove 1st gen provider, replaced with openib_cma and openib_scm
+ dapltest: update windows script files
+ dapltest: windows batch files in sripts directory
+ windows_osd/linux_osd: new dapl_os_gettid macro to return thread id
+ windows: missing build files for common and udapl sub-directories
+ windows: add build files for openib_scm, remove /Wp64 build option.
+ scm: multi-hca CM processing broken. Need cr thread wakeup mechanism per HCA.
+ dtest: add connection timers on client side
+ linux_osd: use pthread_self instead of getpid for debug messages
+ windows ibal-scm: dapl/dirs file needs updated to remove ibal-scm
+
+ v1.2 Package:
+
+ Release 1.2.16
+ package: update Copyright file and include the 3 license files in distribution
+ cma: max sge incorrectly decremented during ibv_device_query
+
+ Release 1.2.15
+ dtest, dapltest: conflict with dapl-2 utils package, change to dapl1, dapltest1
+ scm: fix compiler warning, unused variable
+
+ ----------------
+
+ * BKM for running new DAPL library on your cluster without any impact on existing OFED installation:
+
+ Note: example for user /home/user1, (assumes /home/user1 is exported) and MLX4 adapter, port 1
+
+ Download latest 2.x package: http://www.openfabrics.org/downloads/dapl/dapl-2.0.25.tar.gz
+
+ untar in /home/user1
+ cd /home/user1/dapl-2.0.25
+ ./configure && make (build on node with OFED 1.3 or higher installed, dependency on verb/rdma_cm libraries)
+
+ create /home/user1/dat.conf with following 3 lines. (entries with path to new libraries):
+
+ ofa-v2-ib0 u2.0 nonthreadsafe default /home/user1/dapl-2.0.19/dapl/udapl/.libs/libdaplcma.so.1 dapl.2.0 "ib0 0" ""
+ ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default /home/user1/dapl-2.0.19/dapl/udapl/.libs/libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
+ ofa-v2-mlx4_0-1u u2.0 nonthreadsafe default /home/user1/dapl-2.0.19/dapl/udapl/.libs/libdaploucm.so.2 dapl.2.0 "mlx4_0 1" ""
+
+ Run uDAPL application or an MPI that uses uDAPL, with (assuming MLX4 connectx adapters) following:
+
+ setenv DAT_OVERRIDE=/home/user1/dat.conf
+
+ If running Intel MPI and uDAPL socket cm, set the following:
+
+ setenv I_MPI_DEVICE=rdssm:ofa-v2-mlx4_0-1
+
+ or if running Intel MPI and uDAPL IB UD cm, set the following:
+
+ setenv I_MPI_DEVICE=rdssm:ofa-v2-mlx4_0-1u
+
+ or if running Intel MPI and uDAPL rdma_cm, set the following:
+
+ setenv I_MPI_DEVICE=rdssm:ofa-v2-ib0
+
+-------------------------
+
+ OFED 1.4.1 RELEASE NOTES
+
+ NEW SINCE OFED 1.4 - new versions of uDAPL v1 (1.2.14-1) and v2 (2.0.19-1)
+
+ * New Features - optional counters, must be configured/built with -DDAPL_COUNTERS
+
+ * Bug Fixes
+
+ v2 - scm, cma: dat max_lmr_block_size is 32 bit, verbs max_mr_size is 64 bit
+ v2 - scm, cma: use direct SGE mappings from dat_lmr_triplet to ibv_sge
+ v2 - dtest: add flush EVD call after data transfer errors
+ v2 - scm: increase default MTU size from 1024 to 2048
+ v2 - dapltest: reset server listen ports to avoid collisions during long runs
+ v2 - dapltest: avoid duplicating ports, increment based on ep/thread count
+ v2 - dapltest: fix assumptions that multiple EP's will connect in order
+ v2 - common: sync missing with when removing items off of EVD pending queue
+ v2 - scm: reduce open time with thread start up
+ v2 - scm: getsockopt optlen needs initialized to size of optval
+ v2 - scm: cr_thread cleanup
+ v2 - OFED and WinOF code sync
+ v2 - scm: remove unnecessary query gid/lid from connection phase code.
+ v2 - scm: add optional 64-bit counters, build with -DDAPL_COUNTERS.
+ v1,v2 - spec files missing Requires(post) statements for sed/coreutils
+ v1,v2 - dtest/dapltest: use $(top_builddir) for .la files during test builds
+ v1,v2 - scm: remove unecessary thread when using direct objects
+ v1,v2 - Fix SuSE 11 build issues, asm/atomic.h no longer exists
+
+ * Build Notes:
+
+ # NON_DEBUG build/install example for x86_64, OFED targets
+ ./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
+ make install
+
+ # DEBUG build/install example for x86_64, using OFED targets
+ ./configure --enable-debug --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
+ make install
+
+ # COUNTERS build/install example for x86_64, using OFED targets
+ ./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include -DDAPL_COUNTERS"
+ make install
+
+ * BKM for running new DAPL library on your cluster without any impact on existing OFED installation:
+
+ Note: example for user /home/user1, (assumes /home/user1 is exported) and MLX4 adapter, port 1
+
+ Download latest 2.x package: http://www.openfabrics.org/downloads/dapl/dapl-2.0.19.tar.gz
+
+ untar in /home/user1
+ cd /home/user1/dapl-2.0.19
+ ./configure && make (build on node with OFED 1.3 or higher installed, dependency on verb/rdma_cm libraries)
+
+ create /home/user1/dat.conf with following 2 lines. (entries with path to new libraries):
+
+ ofa-v2-ib0 u2.0 nonthreadsafe default /home/user1/dapl-2.0.19/dapl/udapl/.libs/libdaplcma.so.1 dapl.2.0 "ib0 0" ""
+ ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default /home/user1/dapl-2.0.19/dapl/udapl/.libs/libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
+
+ Run uDAPL application or an MPI that uses uDAPL, with (assuming MLX4 connectx adapters) following:
+
+ setenv DAT_OVERRIDE=/home/user1/dat.conf
+
+ If running Intel MPI and uDAPL socket cm, set the following:
+
+ setenv I_MPI_DEVICE=rdssm:ofa-v2-mlx4_0-1
+
+ if running Intel MPI and uDAPL rdma_cm, set the following:
+
+ setenv I_MPI_DEVICE=rdssm:ofa-v2-ib0
+
+-------------------------
+
+ OFED 1.4 RELEASE NOTES
+
+ NEW SINCE OFED 1.3.1 - new versions of uDAPL v1 (1.2.12-1) and v2 (2.0.15-1)
+
+ * New Features
+
+ 1. The new socket CM provider, introduced in 1.2.8 and 2.0.11 packages,
+ assumes homogeneous cluster and will setup the QP's based on local HCA port
+ attributes and exchanges QP information via socket's using the hostname of
+ each node. IPoIB and rdma_cm are NOT required for this provider. QP attributes
+ can be adjusted via the following environment parameters:
+
+ DAPL_ACK_TIMER (default=16 5 bits, 4.096us*2^ack_timer. 16 == 268ms)
+ DAPL_ACK_RETRY (default=7 3 bits, 7 * 268ms = 1.8 seconds)
+ DAPL_RNR_TIMER (default=12 5 bits, 12 == 64ms, 28 == 163ms, 31 == 491ms)
+ DAPL_RNR_RETRY (default=7 3 bits, 7 == infinite)
+ DAPL_IB_MTU (default=1024 limited to active MTU max)
+
+ The new socket cm entries in /etc/dat.conf provide a link to the actual HCA
+ device and port. Example v1 and v2 entries for a Mellanox connectx device, port 1:
+
+ OpenIB-mlx4_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mlx4_0 1" ""
+ ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
+
+ This new socket cm provider, was successfully tested on the TATA CRL cluster
+ (#8 on Top500) with Intel MPI, achieving a HPLinpack score of 132.8TFlops on
+ 1798 nodes, 14384 cores at ~76.9% of peak. DAPL_ACK_TIMER was increased to 21
+ for this scale.
+
+ 2. New v2 definitions for IB unreliable datagram extension (only supported in
+ scm provider, libdaploscm.so.2)
+
+ Extended EP dat_service_type, with DAT_IB_SERVICE_TYPE_UD
+ Add IB extension call dat_ib_post_send_ud().
+ Add address handle definition for UD calls.
+ Add IB event definitions to provide remote AH via connect and connect requests
+ See dtestx (-d) source for example usage model
+
+ * Bug Fixes
+
+ v1,v2 - dapltest: trans test moves to cleanup stage before rdma_read processing is complete
+ v1,v2 - Fix static registration (dat.conf) to include sysconfdir override
+ v1,v2 - dat.conf: add default iwarp entry for eth2
+ v1,v2 - dapl: adjust max_rdma_read_iov to 1 for iWARP devices
+ v1,v2 - dtest: reduce default IOV's for ep_create to support iWARP
+ v1,v2 - dtest: fix 32-bit build issues
+ v1,v2 - build: $(DESTDIR) prepend needed on install hooks for dat.conf
+ v2 - scm: UD shares EP;s which requires serialization
+ v2 - dapl: fixes for IB UD extensions in common code and socket cm provider.
+ v2 - dapl: add provider specific attribute query option for IB UD MTU size
+ v2 - dapl build: add correct CFLAGS, set non-debug build by default for v2
+ v2 - dtestx: fix stack corruption problem with hostname strcpy
+ v2 - dapl extension: dapli_post_ext should always allocate cookie for requests.
+ v2 - dapltest: manpage - rdma write example incorrect
+ v1,v2 - dat, dapl, dtest, dapltest, providers: fix compiler warnings in dat common code
+ v1,v2 - dapl cma: debug message during query needs definition for inet_ntoa
+ v1,v2 - dapl scm: fix corner case that delivers duplicate disconnect events
+ v1,v2 - dat: include stddef.h for NULL definition in dat_platform_specific.h
+ v1,v2 - dapl: add debug messages during async and overflow events
+ v1,v2 - dapltest: add check for duplicate disconnect events in transaction test
+ v1,v2 - dapl scm: use correct device attribute for max_rdma_read_out, max_qp_init_rd_atom
+ v1,v2 - dapl scm: change IB RC qp inline and timer defaults.
+ v1,v2 - dapl scm: add mtu adjustments via environment, default = 1024.
+ v1,v2 - dapl scm: change connect and accept to non-blocking to avoid blocking user thread.
+ v1,v2 - dapl scm: update max_rdma_read_iov, max_rdma_write_iov EP attributes during query
+ v1,v2 - dat: allow TYPE_ERR messages to be turned off with DAT_DBG_TYPE
+ v1,v2 - dapl: remove needless terminating 0 in dto_op_str functions.
+ v1,v2 - dat: remove reference to doc/dat.conf in makefile.am
+ v1,v2 - dapl scm: fix ibv_destroy_cq busy error condition during dat_evd_free.
+ v1,v2 - dapl scm: add stdout logging for uname and gethostbyname errors during open.
+ v1,v2 - dapl scm: support global routing and set mtu based on active_mtu
+ v1,v2 - dapl: add opcode to string function to report opcode during failures.
+ v1,v2 - dapl: remove unused iov buffer allocation on the endpoint
+ v1,v2 - dapl: endpoint pending request count is wrong
+
+-------------------------
+
+ OFED 1.3.1 RELEASE NOTES
+
+ NEW SINCE OFED 1.3 - new versions of uDAPL v1 (1.2.7-1) and v2 (2.0.9-1)
+
+ * New Features - None
+
+ * Bug Fixes
+ v2 - add private data exchange with reject
+ v1,v2 - better error reporting in non-debug builds
+ v1,v2 - update only OFA entries in dat.conf, cooperate with non-ofa providers
+ v1,v2 - support for zero byte operations, iov==NULL
+ v1,v2 - multi-transport support for inline data and private data differences
+ v1,v2 - fix memory leaks and other reported bugs since OFED 1.3
+ v1,v2 - dtest,dtestx,dapltest build issues on RHEL5.1
+ v1,v2 - long delay during dat_ia_open when DNS not configured
+ v1,v2 - use rdma_read_in/out from ep_attr per consumer instead of HCA max
+
+-------------------------
+
+ OFED 1.3 RELEASE NOTES
+
+ NEW SINCE OFED 1.2
+
+ * New Features
+
+ 1. Add v2.0 library support for new 2.0 API Specification
+ 2. Separate v1.2 library release to co-exist with v2.0 libraries.
+ 3. New dat.conf with both 1.2 and 2.0 support
+ 4. New v2.0 dtestx utilities to test IB extensions
+
+ * Bug Fixes
+
+ v1.2 and v2.0
+ - uDAT: static/dynamic registry parsing fixes
+ - uDAPL: provider fixes for dat_psp_create_any
+ - dtest/dapltest: change default provider names to sync with dat.conf
+ - openib_cma: issues with destroy_cm_id and init/resp exchange
+ - dapltest: use gettimeofday instead of get_cycles for better portability
+ - dapltest: endian issue with mem_handle, mem_address
+ - dapltest fix to include inet_ntoa definitions
+ - fix build problems on 32-bit and 64-bit PowerPC
+ - cleanup packaging
+
+ v2.0
+ - set default config options to match spec file, --enable-debug --enable-ext-type=ib
+ - use unique devel target names, libdat2.so, /usr/include/dat2
+ - dtestx fix memory leak, freeaddrinfo after getaddrinfo
+ - Fix for IB extended DTO cookie deallocation on inbound rdma_Write_immed
+ - WinOF: Update OFED code base to include WinOF changes, work from same code base
+ - WinOF: add DAT_API definition, __stdcall for windows, nothing for linux
+ - dtest: add dat_evd_query to check correct size
+ - openib_cma: add macro to convert SID to PORT
+ - dtest: endian support for exchanging RMR info
+ - openib_cma: lower default settings, inline and RDMA init/resp
+ - openib_cma: missing ia_query for max_iov_segments_per_rdma_write
+
+ v1.2
+ - openib_cma: turn down dbg noise level on rejects
+ - dtest: typo in memset
+
+
+ BUILD: v1 and v2 uDAPL source install/build instructions (redhat example):
+
+ # cd to distribution SRPMS directory
+ cd /tmp/OFED-1.3/SRPMS
+ rpm -i dapl-1.2*.rpm
+ rpm -i dapl-2.0*.rpm
+ cd /usr/src/redhat/SOURCES
+ tar zxf dapl-1.2*.tgz
+ tar zxf dapl-2.0*.tgz
+
+ # NON_DEBUG build example for x86_64, using OFED targets
+
+ ./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64
+ LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
+
+ # build and install
+
+ make
+ make install
+
+ # DEBUG build example for x86_64, using OFED targets
+
+ ./configure --enable-debug --prefix /usr --sysconf=/etc --libdir /usr/lib64
+ LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
+
+ # build and install
+
+ make
+ make install
+
+ # DEBUG messages: set environment variable DAPL_DBG_TYPE, default
+ mapping is 0x0003
+
+ DAPL_DBG_TYPE_ERR = 0x0001,
+ DAPL_DBG_TYPE_WARN = 0x0002,
+ DAPL_DBG_TYPE_EVD = 0x0004,
+ DAPL_DBG_TYPE_CM = 0x0008,
+ DAPL_DBG_TYPE_EP = 0x0010,
+ DAPL_DBG_TYPE_UTIL = 0x0020,
+ DAPL_DBG_TYPE_CALLBACK = 0x0040,
+ DAPL_DBG_TYPE_DTO_COMP_ERR= 0x0080,
+ DAPL_DBG_TYPE_API = 0x0100,
+ DAPL_DBG_TYPE_RTN = 0x0200,
+ DAPL_DBG_TYPE_EXCEPTION = 0x0400,
+ DAPL_DBG_TYPE_SRQ = 0x0800,
+ DAPL_DBG_TYPE_CNTR = 0x1000
+
+-------------------------
+
+ OFED 1.2 RELEASE NOTES
NEW SINCE Gamma 3.2 and OFED 1.1
projects / ~ardavis / dapl.git / commitdiff