Arlin Davis [Tue, 18 Aug 2009 17:15:15 +0000 (10:15 -0700)]
ucm: Add new provider using a DAPL based IB-UD cm mechanism for MPI implementations.
New provider uses it's own CM protocol on top of IB-UD queue pairs.
During device open, this provider creates a UD queue pair and
returns local address information via dat_ia_query. This 24 byte
opaque address must be exchange out-of-band before connecting to a
server via dat_ep_connect. This provider is targeted for MPI
implementations that already exchange address information
during mpi_init phase.
Future release may provide some ARP mechanism via multicast.
dtest, dtestx, and dtestcm was modified to report the lid and qpn
information on the server side so you can provide appropriate
destination address information for the client test suite.
dapltest will not work with this provider.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Wed, 5 Aug 2009 03:49:09 +0000 (20:49 -0700)]
scm: Fix disconnect. QP's need to move to ERROR state in
order to flush work requests and notify consumer. Moving to
RESET removed all requests but did not notify consumer.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Wed, 5 Aug 2009 03:48:03 +0000 (20:48 -0700)]
modify dtest.c to cleanup CNO wait code and consolidate into
collect_event() call. After waking up from CNO wait the
consumer must check all EVD's. The EVD's under the CNO
could be dropped if already triggered or could come in any order.
DT_RetToString changed to DT_RetToStr and DT_EventToSTr
changed to DT_EventToStr for consistency.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Wed, 5 Aug 2009 03:47:17 +0000 (20:47 -0700)]
CNO events, once triggered will not be returned during the cno wait.
Check for triggered state before going to sleep in cno_wait. Reset
triggered EVD reference after reporting.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Sun, 2 Aug 2009 21:21:09 +0000 (14:21 -0700)]
CNO support broken in both CMA and SCM providers.
CQ thread/callback mechanism was removed by mistake. Still
need indirect DTO callbacks when CNO is attached to EVD's.
Add CQ event channel to cma provider's thread and add
to select for rdma_cm and async channels.
For scm provider there is not easy way to add this channel
to the select across sockets on windows. So, for portablity
reasons 2 thread is started to process the ASYNC and
CQ channels for events.
Must disable EVD (evd_endabled=FALSE) during destroy
to prevent EVD events firing for CNOs and re-arming CQ while
CQ is being destroyed.
Change dtest to check EVD after CNO timesout.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Sean Hefty [Mon, 27 Jul 2009 22:07:33 +0000 (15:07 -0700)]
DAPL introduced the concept of directly waiting on the CQ for
events by adding a compile time flag and special handling in the common
code. Rather than using the compile time flag and modifying the
common code, let the provider implement the best way to wait for
CQ events.
This simplifies the code and allows the common openib providers to
optimize for Linux and Windows platforms independently, rather than
assuming a specific implementation for signaling events.
Arlin Davis [Thu, 16 Jul 2009 19:41:22 +0000 (12:41 -0700)]
dapltest: Implement a malloc() threshold for the completion reaping.
change byte vector allocation to stack in functions:
DT_handle_send_op, DT_handle_rdma_op & DT_handle_recv_op.
When allocation size is under the threshold, use a stack local
allocation instead of malloc/free. Move redundant bzero() to
be called only in the case of using local stack allocation as
DT_Mdep_malloc() already does a bzero(). Consolidate error handling
return and free()check to a single point by using goto.
Arlin Davis [Thu, 16 Jul 2009 19:32:09 +0000 (12:32 -0700)]
scm: handle connected state when freeing CM objects
The QP could be freed before being disconnected
so the provider needs process disconnect before freeing
the CM object. The disconnect clean will finish
the destroy process during the disc callback.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Thu, 2 Jul 2009 21:11:20 +0000 (14:11 -0700)]
dtestcm: add UD type QP option to test
Add -u for UD type QP's during connection setup.
Will setup UD QPs and provide remote AH
in connect establishment event. Measures
setup/exchange rates.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Thu, 2 Jul 2009 21:07:36 +0000 (14:07 -0700)]
scm: destroy QP called before disconnect
Handle the case where QP is destroyed before
disconnect processing. Windows supports
reinit_qp during a disconnect call by
destroying the QP and recreating the
QO instead of state change from reset
to init. Call disconnect in destroy
CM code to handle this unexpected state.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Mon, 29 Jun 2009 15:57:46 +0000 (08:57 -0700)]
openib_common: reorganize provider code base to share common mem, cq, qp, dto functions
add new openib_common directory with cq, qp, util, dto, mem function calls
and definitions. This basically leaves the unique CM and Device definitions
and functions to the individual providers directory of openib_scm and openib_cma.
modifications to dapl_cr_accept required. ep->cm_handle is allocated
and managed entirely in provider so dapl common code should not update
ep_handle->cm_handle from the cr->cm_handle automatically. The provider
should determine which cm_handle is required for the accept.
openib_cma defines _OPENIB_CMA_ and openib_scm defines _OPENIB_SCM_ for provider
specific build needs in common code.
Arlin Davis [Fri, 26 Jun 2009 21:45:34 +0000 (14:45 -0700)]
scm: fixes and optimizations for connection scaling
Prioritize accepts on listen ports via FD_READ
process the accepts ahead of other work to avoid
socket half_connection (SYN_RECV) stalls.
Fix dapl_poll to return DAPL_FD_ERROR on
all event error types.
Add new state for socket released, but CR
not yet destroyed. This enables scm to release
the socket resources immediately after exchanging
all QP information. Also, add state to str call.
Only add the CR reference to the EP if it is
RC type. UD has multiple CR's per EP so when
a UD EP disconnect_clean was called, from a
timeout, it destroyed the wrong CR.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Sat, 20 Jun 2009 03:52:51 +0000 (20:52 -0700)]
common,scm: add debug capabilities to print in-process CM lists
Add a new debug bit DAPL_DBG_TYPE_CM_LIST.
If set, the pending CM requests will be
dumped when dat_print_counters is called.
Only provided when built with -DDAPL_COUNTERS
Add new dapl_cm_state_str() call for state
to string conversion for debug prints.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Wed, 10 Jun 2009 16:09:56 +0000 (09:09 -0700)]
scm: cleanup orphaned UD CR's when destroying the EP
UD CR objects are kept active because of direct private data references
from CONN events. The cr->socket is closed and marked inactive but the
object remains allocated and queued on the CR resource list. There can
be multiple CR's associated with a given EP and there is no way to
determine when consumer is finished with event until the dat_ep_free.
Schedule destruction for all CR's associated with this EP during
free call. cr_thread will complete cleanup with state of SCM_DESTROY.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Wed, 10 Jun 2009 17:06:59 +0000 (10:06 -0700)]
scm: update CM code to shutdown before closing socket
data could be lost without calling shutdown on the socket
before closing. Update to shutdown and then close. Add
definition for SHUT_RW to SD_BOTH for windows.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
---
Sean Hefty [Thu, 4 Jun 2009 15:19:12 +0000 (08:19 -0700)]
dapl/windows cma provider: add support for network devices based on index
The linux cma provider provides support for named network devices, such
as 'ib0' or 'eth0'. This allows the same dapl configuration file to
be used easily across a cluster.
To allow similar support on Windows, allow users to specify the device
name 'rdma_devN' in the dapl.conf file. The given index, N, is map to a
corresponding IP address that is associated with an RDMA device.
Arlin Davis [Mon, 18 May 2009 16:06:19 +0000 (09:06 -0700)]
windows: add build files for openib_scm, remove /Wp64 build option.
Add build files for windows socket cm and change build
option on windows providers. The new Win7 WDK issues a
depreciated compiler option warning for /Wp64
(Enable 64-bit porting warnings)
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Mon, 18 May 2009 15:50:35 +0000 (08:50 -0700)]
scm: multi-hca CM processing broken. Need cr thread wakeup mechanism per HCA.
Currently there is only one pipe across all
device opens. This results in some posted CR work
getting delayed or not processed at all. Provide
pipe for each device open and cr thread created
and manage on a per device level.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Fri, 15 May 2009 16:48:38 +0000 (09:48 -0700)]
linux_osd: use pthread_self instead of getpid for debug messages
getpid provides process ids which are not unique. Use unique thread
id's in debug messages to help isolate issues across many device
opens with multiple CM threads.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Wed, 29 Apr 2009 15:39:37 +0000 (08:39 -0700)]
dtest: add flush EVD call after data transfer errors
Flush and print entries on async, request, and receive
queues after any data transfer error. Will help
identify failing operation during operations
without completion events requested.
Fix -B0 so burst size of 0 works.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Thu, 16 Apr 2009 21:35:18 +0000 (14:35 -0700)]
dapltest: reset server listen ports to avoid collisions during long runs
If server is running continuously the port number increments
from base without reseting between tests. This will
eventually cause collisions in port space.
Signed-off-by: Arlin Davis <ardavis@ichips.intel.com>
Sean Hefty [Thu, 16 Apr 2009 17:21:51 +0000 (10:21 -0700)]
To avoid duplicating port numbers between different tests, the next port
number to use must increment based on the number of endpoints per thread *
the number of threads.
Sean Hefty [Thu, 16 Apr 2009 17:21:45 +0000 (10:21 -0700)]
dapltest assumes that events across multiple endpoints occur in a specific
order. Since this is a false assumption, avoid this by directing events to
per endpoint EVDs, rather than using shared EVDs.
Sean Hefty [Thu, 16 Apr 2009 17:21:41 +0000 (10:21 -0700)]
Synchronization is missing between removing items from an EVD and queuing
them. Since the removal thread is the user's, but the queuing thread is
not, the synchronization must be provided by DAPL. Hold the evd lock
around any calls to dapls_rbuf_*.
Sean Hefty [Thu, 16 Apr 2009 17:21:26 +0000 (10:21 -0700)]
Communication to the CR thread is done using an internal socket. When a
new connection request is ready for processing, an object is placed on
the CR list, and data is written to the internal socket. The write causes
the CR thread to wake-up and process anything on its cr list.
If multiple objects are placed on the CR list around the same time, then
the CR thread will read in a single character, but process the entire list.
This results in additional data being left on the internal socket. When
the CR does a select(), it will find more data to read, read the data, but
not have any real work to do. The result is that the thread spins in a
loop checking for changes when none have occurred until all data on the
internal socket has been read.
Avoid this overhead by reading all data off the internal socket before
processing the CR list.
Sean Hefty [Thu, 16 Apr 2009 17:21:13 +0000 (10:21 -0700)]
The dapl connect call takes as input an address (sockaddr) and a port number
as separate input parameters. It modifies the sockaddr address to set the
port number before trying to connect. This leads to a situation in
dapltest with multiple threads that reference the same buffer for their
address, but specify different port numbers, where the different threads
end up trying to connect to the same remote port.
To solve this, do not modify the caller's address buffer and instead use
a local buffer. This fixes an issue seen running multithreaded tests with
dapltest.
Sean Hefty [Fri, 10 Apr 2009 15:17:32 +0000 (08:17 -0700)]
The connection request thread adds sockets to a select list unless
the cr->socket is invalid and the cr request state is set to destroy. If the
cr->socket is invalid, but the cr->state is not destroy, then the cr->socket
is added to an FD set for select/poll. This results in select/poll
returning an error when select is called. As a result, the cr thread never
actually blocks during this state.
Fix this by only destroying a cr based on its state being set to destroy
and skip adding cr->sockets to the FD set when they are invalid.
Sean Hefty [Fri, 10 Apr 2009 15:08:03 +0000 (08:08 -0700)]
The IBAL library allocates a small number of threads for callbacks to the
user. If the user blocks all of the callback threads, no additional
callbacks can be invoked. The DAPL IBAL provider cancels listen requests
from within an IBAL callback, then waits for a second callback to confirm
that the listen has been canceled. If there is a single IBAL callback
thread, or multiple listens are canceled simultaneously, then the provider
can deadlock waiting for a cancel callback that never occurs.
This problem is seen when running dapltest with multiple threads.
Sean Hefty [Fri, 10 Apr 2009 15:07:57 +0000 (08:07 -0700)]
We need to check the return value from select for errors before checking
the FD sets. An item may be in an FD set but select could have returned
an error.
Sean Hefty [Fri, 10 Apr 2009 15:07:49 +0000 (08:07 -0700)]
Enable building with CQ_WAIT_OBJECTS support to directly wait on CQ
completion channels in the Windows version of the openib_scm provider.
Also minor fixup to use DAPL_DBG_TYPE_UTIL for debug log messages
instead of DAPL_DBG_TYPE_CM.
Sean Hefty [Fri, 10 Apr 2009 15:07:44 +0000 (08:07 -0700)]
The IBAL-SCM provider will run into an inifinite loop if the check for
cr->socket > SCM_MAX_CONN - 1 fails. The code continues back to the start
of the while loop without moving to the next connection request entry
in the list.
Sean Hefty [Fri, 10 Apr 2009 15:07:40 +0000 (08:07 -0700)]
next_cr is set just before and inside the check
if ((cr->socket == DAPL_INVALID_SOCKET && cr->state == SCM_DESTROY)
Remove setting it inside the if statement.
Sean Hefty [Fri, 10 Apr 2009 15:07:32 +0000 (08:07 -0700)]
The WinOF HCA driver cannot handle transitioning from RTS -> RESET ->
INIT -> ERROR. Simply delete the QP and re-create it to reinitialize
the endpoint until the bug is fixed.
Sean Hefty [Fri, 10 Apr 2009 15:06:53 +0000 (08:06 -0700)]
Move from using pipes to sockets for internal communication. This
avoids issues with windows only supporting select() on sockets.
Remove windows specific definition of dapl_dbg_log.
Update to latest windows libibverbs implementation using completion
channel abstraction to improve windows scalability and simplify
porting where FD's are accessed directly in Linux.
Arlin Davis [Fri, 13 Mar 2009 20:39:12 +0000 (12:39 -0800)]
uDAPL: scm provider, remove query gid/lid from connection setup phase
move lid/gid queries from the connection setup phase
and put them in the open call to avoid overhead
of more fd's during connections. No need
to query during connection setup since uDAPL
binds to specific hca/ports via dat_ia_open.
Signed-off-by: Arlin Davis <ardavis@ichips.intel.com>
Arlin Davis [Wed, 4 Mar 2009 18:04:13 +0000 (10:04 -0800)]
dapl scm: remove unecessary thread when using direct objects
A thread is created for processing events on devices without
direct event objecti support. Since all openfabrics devices support
direct events there is no need to start a thread. Move this under
Signed-off-by: Arlin Davis <ardavis@ichips.intel.com>
Arlin Davis [Tue, 3 Mar 2009 17:25:26 +0000 (09:25 -0800)]
uDAPL common: add 64 bit counters for IA, EP, and EVD's.
-DDAPL_COUNTERS to build-in counters for cma and scm providers.
New extension calls in dat_ib_extensions.h for counters
dat_print_counters, dat_query_counters
Counters for operations, async errors, and data
Update dtestx (-p) with print and query counter examples
Signed-off-by: Arlin Davis <ardavis@ichips.intel.com>
projects / ~ardavis / dapl.git / log