Arlin Davis [Wed, 25 Jan 2012 19:50:21 +0000 (11:50 -0800)]
common: ep_create should allow max_request_iov attribute setting of zero
When creating an EP without a request EVD (cq) the max_request_iov
and max_request_sge will be 0. Allow this combination when checking
attribute settings for ARG6.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Thu, 12 Jan 2012 17:54:59 +0000 (09:54 -0800)]
common: extended CR event processing missing rejects on errors
When processing an inbound CR event callback a non-user reject should be
sent to client in the case of a non-listening SP, allocation error,
or EVD overrun. Changes made to dapls_evd_post_cr_event_ext callback.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Thu, 12 Jan 2012 17:39:46 +0000 (09:39 -0800)]
ucm: incorrectly sends user reject during CR callback errors
Add reason checking on provider rejects and set appropriate op type
in reject message. Reject can be called from cr callback during
failures. User reject will be IB_CM_REJ_REASON_CONSUMER_REJ.
Add warning message on active side.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Mon, 9 Jan 2012 23:03:21 +0000 (15:03 -0800)]
scm: incorrectly sends user reject during CR callback errors
Add reason checking on provider rejects and set appropriate op type
in reject message. Reject can be called from cr callback during
failures. User reject will be IB_CM_REJ_REASON_CONSUMER_REJ.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Thu, 8 Dec 2011 00:39:55 +0000 (16:39 -0800)]
cma,scm,ucm: extra reference on EP, with RSP, causes dat_ep_free() to hang
Need to add check for RSP or PSP provider type service points during
passive side accepts before taking CR reference on the EP. In these
cases, the EP is already linked to inbound CR.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Wed, 2 Nov 2011 17:29:34 +0000 (10:29 -0700)]
common: increase default IB ack timer from 16 to 20
For larger, more congested fabrics, a larger ACK timer is needed.
Consumers can still change default with environment variable
DAPL_ACK_TIMER if they need to increase or decrease.
This applies to SCM and UCM providers only. The CMA provider, which
uses rdma_cm, has no way to control ack timer with current API.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Mon, 24 Oct 2011 20:59:55 +0000 (13:59 -0700)]
scm: when hostname has loopback addr assigned, default to eth0 instead of failing
There are some cases where the eth0 device is configured with an IP address
but the getaddrinfo() will only return loopback address because the
hostname is configured in the /etc/hosts file with 127.0.0.1. In this case,
the provider will now retry address on eth0 before failing the open.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Mon, 24 Oct 2011 20:40:45 +0000 (13:40 -0700)]
common: query calls return incorrect IA handle to consumer
The IA handle from the consumer perspective is an IA vector and
not the provider IA address handle. Need to convert IA handle
to IA vector for consumer calls.
Modify dats_ia_get_handle call to convert both ways depending
on handle type provided so a dapl provider can convert to vector
on query calls. This fix is backward compatible with older libdat2
libraries. Function is already exported and syntax is unchanged.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Mon, 29 Aug 2011 02:10:37 +0000 (19:10 -0700)]
scm,ucm: fix compatibility issues and set minimum protocol support
allow latest version to work with previous versions to allow
compatibility back to OFED 1.5, dapl-2.0.23. If rdma_atomic_in
is not exchanged default back to original settings set by
consumer.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Fri, 19 Aug 2011 19:40:24 +0000 (12:40 -0700)]
build: add selective enable/disable-xxx build switch for each provider
The following switches have been added to configure:
--disable-cma (disables the rdma_cm dapl provider build)
--disable-scm (disables the socket cm provider build)
--disable-ucm (disables the IB UD cm provider build)
all providers are enabled by default.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Mon, 8 Aug 2011 05:59:27 +0000 (22:59 -0700)]
build: add IB collective and FCA provider to dapl build package as an option
New collective support and the FCA provider will only be built with
configure option of "--enable-coll-type=fca". Dependencies include
fca devel package, fca library, and mverbs library. This will not
be included in default builds.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Mon, 8 Aug 2011 05:06:09 +0000 (22:06 -0700)]
dat: add definitions for MPI offloaded collectives in IB transport extensions
The collective extensions are designed to support MPI and general
multicast operations over IB fabrics that support offloaded collectives.
Where feasible, they come as close to MPI semantics as possible.
Unless otherwise stated, all members participating in a data collective
operation must call the associated collective routine for the data
transfer operation to complete. Unless otherwise stated, the
root collective member of a data operation will receive its own portion
of the collective data. In most cases, the root member can prevent
sending/receiving data when such operations would be redundant. When root
data is already "in place" the root member may set the send and/or receive
buffer pointer argument to NULL.
Unlike standard DAPL movement operations that require registered
memory and LMR objects, collective data movement operations employ
pointers to user-virtual address space that do not require
pre-registration by the application. From a resource usage point
of view, the API user should consider that the provider implementation
my perform memory registrations/deregistration on behalf of the
application to accomplish a data transfer.
Most collective calls are asynchronous. Upon completion, an event
will be posted to the EVD specified when the collective was created.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Sat, 12 Feb 2011 19:46:08 +0000 (11:46 -0800)]
ucm: delay freeing of active side UD cm object in case RTU is dropped
The ucm was freeing the UD CM object to quickly so a retried REPLY
was dropped and the passive side never received the AH info via RTU.
Keep active side UD cm objects on work queue until QP is destroyed
so RTU can be resent if necessary.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Sat, 12 Feb 2011 19:36:35 +0000 (11:36 -0800)]
ucm: cm object needs to be on work queue before req sent on wire
With this delay in cm object queuing there is potential for replies
being dropped coming back with a NO MATCH. Start with INIT
state and queue it up, move to state REP_PENDING when
sending out on the wire to start request timer.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Fri, 21 Jan 2011 02:31:30 +0000 (18:31 -0800)]
common: reduce default max inline data size because of performance anomaly
Increasing max inline causes small message rates to decrease from
4M/sec when set to 64 to about 1M/sec when set to 400. This has
been observed on latest mlx4 adapters. Set default to 64 until resolved.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Mon, 17 Jan 2011 22:43:58 +0000 (14:43 -0800)]
ucm, scm: exchange max_qp_rd_atom and limit outstanding requests
exchange and add proper checking to limit outstanding
rdma reads and atomics. Use one of the reserve bytes
in CM message protocol to exchange limits and reset
EP attribute rdma_out and set QP RTS attribute properly.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Tue, 7 Dec 2010 00:06:47 +0000 (16:06 -0800)]
libdat: static provider entries created for local SR database not freed
During load (dat_sr_init) the SR database is created with all dat.conf entries
but are never cleaned up during unload. Add new functions dat_sr_remove_all()
and dat_sr_remove() calls to cleanup and deallocate SR database entries and
database via dat_sr_fini().
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Fri, 3 Dec 2010 22:56:21 +0000 (14:56 -0800)]
ucm: release UD cm objects after AH is exchanged to avoid duplicate request drops
When EP is in UD mode, AH resolution is handled with DAT connection semantics
connect and accept. Since AH info can be resolved for the same EPs you can
get false duplicate requests because a previous CR from is still on the
CM processing list. The CM object will remain on the EP free list and not
be freed until EP is destroyed given the possibilty of consumer accessing CR
private data buffer.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Fri, 3 Dec 2010 22:24:40 +0000 (14:24 -0800)]
ucm: hold lock when sending cm_msgs to sync timer start with packet send
releasing the lock after setting start timer and before
ucm_send could result in incorrect timeout on CM operations
if thread is scheduled out when releasing lock.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Fri, 3 Dec 2010 22:02:25 +0000 (14:02 -0800)]
ucm: add debugging to include process id for better scale up debug aids
use part of the resv[] area of the cm_msg to include local and
remote process ids. Add more debug messages to help isolate
problems related to many process problems.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Fri, 3 Dec 2010 18:25:46 +0000 (10:25 -0800)]
cma: disconnect can block for excessive times waiting for rdma_cm DREP timeout
rdma_cm uses the same timeout values for connect and disconnect
request/reply. Disconnect abrupt option allows DAT consumers to
specify a prompt disconnect with immediate event. If the remote
node goes down or is non-responsive a CM disconnect event could
take minutes. Add a time limit waiting for event and move EP to
disconnected state to prevent callback from issuing duplicate
disconnect event via callback. The EP to CM linking will
cleanup/cancel any pending events before destroying cm_id.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Fri, 22 Oct 2010 17:15:15 +0000 (10:15 -0700)]
scm, ucm: MPI spawn test on oversubcribed server taking excessive time to complete
Simultanious DREQ processing from user and CM thread caused some improper
state change on UCM. State change can incorrectly change from FREE back
to DISC in certain corner cases. Add checking on internal disconnect call
to prevent double callback events and improper state change.
For SCM, a remote DREQ will shutdown socket which will cause POLLERR
on the disconnected FD. This will in turn cause the cm_thread to
wakeup continuously unnecessarily. Fix thread thrashing by moving
CM object to FREE state and removing object FD from pollfd array.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
ardavis [Tue, 19 Oct 2010 16:52:45 +0000 (09:52 -0700)]
common: dat_evd_dequeue (poll_cq) fails with invalid parameter after EP (qp) free
Failure occured during Intel MPI spawn test on windows.
The QP's need to be flushed and processed via EVD's during
the EP (QP) destroy to avoid an error on poll_cq. IBAL
provider was not moving to ERR state during QP destroy.
Better flush CQ processing was added and pushed down to the provider
level via dapls_ib_qp_free() where it can move QP to ERR, flush CQ,
and then free QP after flushing. Because there is no QP_ERR_FLUSH
state on a QP the spin on poll_cq (until empty) after modify_qp
to ERR could return empty and before all WQE's are flushed. This
could result in a CQE being added to CQ with a invalid QP reference.
So, an additional check was added to flush_evds for the recv_q to
poll_cq until all recv's pending are complete. For transmit_q there
is no quarantee that the posted work is signaled and so the best
that can be done is poll_cq until empty.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
ardavis [Mon, 11 Oct 2010 19:24:31 +0000 (12:24 -0700)]
ucm: allow configuration of CM burst (signal) threshold on posting
Add new DAPL_UCM_TX_BURST environment variable, default=50.
Every 50 posted send messages will signal event which
is 10 percent blocks of default 500 message limit.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
ardavis [Thu, 7 Oct 2010 18:14:03 +0000 (11:14 -0700)]
Allow DAPL out of band connection models to use ibacm to obtain
path record data. This will enable support for a wider range of
topologies, where the SL is required from the SA to prevent
deadlock.
DAPL will obtain path record data using rdma_getaddrinfo, provided
that IB ACM support is enabled. On failure, dapl will fall back to
using its default SL value. The IB ACM can be configured to cache
path information or always query the SA to ensure that the SL that is
obtained is current.
ibal provider calls ib_cm_drep in response to receiving
a dreq. The result is that the user's QP is transitioned
through the error state, which fails any outstanding send
operations and flushes all receives. The disconnect request
is then reported to the user.
Since a user can receive errors from the QP before they are
aware of a pending disconnect request, the application may
respond to the errors as, well, actual errors. Fix this by
delaying the QP transition until the user responds to the
dreq.
This fixes an error with Intel MPI running over the ibal
dapl provider with a 'spawn' test.
Arlin Davis [Wed, 22 Sep 2010 17:35:24 +0000 (10:35 -0700)]
ibal: delay QP transition during disconnect phase
The ibal provider calls ib_cm_drep in response to receiving
a dreq. The result is that the user's QP is transitioned
through the error state, which fails any outstanding send
operations and flushes all receives. The disconnect request
is then reported to the user.
Since a user can receive errors from the QP before they are
aware of a pending disconnect request, the application may
respond to the errors as, well, actual errors. Fix this by
delaying the QP transition until the user responds to the
dreq.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
---
Arlin Davis [Mon, 20 Sep 2010 17:42:41 +0000 (10:42 -0700)]
common: restructure EVD processing to handle EP destruction phase
EVD processing in the common code will return unformated events
if EP context is invalid as a result of destruction. During
EP destruction, add changes to flush EVD and process DTO completions
before the EP freeing is called. Simplified the locking in the
EVD code to eliminate the unecessary and very confusing condition
checking of evd_producer_locking_needed.
new dapls_ep_flush_cqs() call created to syncronize flush and
event processing.
unnecessary KDAPL code removed in the EVD processing.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Arlin Davis [Mon, 13 Sep 2010 23:19:44 +0000 (16:19 -0700)]
ibal: sync QP destruction and device close
Make QP destruction synchronous to ensure that no callbacks are
in progress for a QP after dapl has destroyed it. This fixes a
use after free error accessing the dapl ep structure from a qp
callback that results in an application crash.
projects / ~ardavis / dapl.git / log