Doug Ledford [Wed, 18 Jun 2014 17:45:23 +0000 (10:45 -0700)]
rdma_server: handle IBV_SEND_INLINE correctly
Not all RDMA devices support IBV_SEND_INLINE. At least some of those
that don't will ignore the flag passed to rdma_post_send and attempt to
send the command by using an sge entry instead. Because we don't
register the send memory, this fails. The proper way to deal with the
fact that IBV_SEND_INLINE is not guaranteed is to check the returned
value in our cap struct to see if we have support for inline data, and
if not, fall back to non-inline sends and to register the send memory
region.
Signed-off-by: Doug Ledford <dledford@redhat.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Doug Ledford [Wed, 18 Jun 2014 17:44:49 +0000 (10:44 -0700)]
rdma_client: handle IBV_SEND_INLINE correctly
Not all RDMA devices support IBV_SEND_INLINE. At least some of those
that don't will ignore the flag passed to rdma_post_send and attempt to
send the command by using an sge entry instead. Because we don't
register the send memory, this fails. The proper way to deal with the
fact that IBV_SEND_INLINE is not guaranteed is to check the returned
value in our cap struct to see if we have support for inline data, and
if not, fall back to non-inline sends and to register the send memory
region.
Signed-off-by: Doug Ledford <dledford@redhat.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Doug Ledford [Wed, 18 Jun 2014 17:44:28 +0000 (10:44 -0700)]
rdma_server: use perror, unwind allocs on failure
Our main test function prints out errno directly, which is hard to read
as it's not decoded at all. Instead, use perror() to make failures more
readable. Also redo the failure flow so that we can do a simple unwind
at the end of the function and just jump to the right unwind spot on
error.
Signed-off-by: Doug Ledford <dledford@redhat.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Doug Ledford [Wed, 18 Jun 2014 17:44:13 +0000 (10:44 -0700)]
rdma_client: use perror, unwind allocs on failure
Our main test function prints out errno directly, which is hard to read
as it's not decoded at all. Instead, use perror() to make failures more
readable. Also redo the failure flow so that we can do a simple unwind
at the end of the function and just jump to the right unwind spot on
error.
Signed-off-by: Doug Ledford <dledford@redhat.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Doug Ledford [Wed, 18 Jun 2014 17:43:04 +0000 (10:43 -0700)]
cmtime: rework program to be multithread
When using very large numbers of connections (10,000 was in use here),
we ran into a problem where when we resolved a performance problem in
the kernel cma.c code, we suddenly developed a new problem. That new
problem turned out to be the fact that with the underlying kernel issue
resolved, 10,000 connect requests would flood the server side of the
test and the cmtime application would respond as quickly as possible.
However, the client side would not bother to check any of the returns
until after having sent all 10,000 connect requests. When the kernel
had a serializing performance problem, this was OK. When it was fixed,
this caused a general slowdown in connect operations due to overruns in
the event processing. This patch causes the client side to fire off
threads that will handle responses to connect requests as they come in
instead of allowing them to backlog uncontrollably. Times for a 10,000
connect run changed from this:
[root@rdma-dev-01 ~]# more
3.12.0-rc1.cached_gids+optimized_connect+trimmed_cache+.output
ib1:
step total ms max ms min us us / conn
create id : 46.64 0.10 1.00 4.66
bind addr : 89.61 0.04 7.00 8.96
resolve addr : 50.63 26.18 23976.00 5.06
resolve route: 565.44 538.77 26736.00 56.54
create qp : 4028.31 5.70 326.00 402.83
connect : 50077.42 49990.49 90734.00 5007.74
disconnect : 5277.25 4850.35 380017.00 527.72
destroy : 42.15 0.04 2.00 4.21
ib0:
step total ms max ms min us us / conn
create id : 34.82 0.04 1.00 3.48
bind addr : 25.94 0.02 1.00 2.59
resolve addr : 48.18 25.01 22779.00 4.82
resolve route: 501.28 476.26 25071.00 50.13
create qp : 3274.12 6.05 257.00 327.41
connect : 55549.64 55490.32 62150.00 5554.96
disconnect : 5263.64 4851.18 375628.00 526.36
destroy : 47.20 0.07 2.00 4.72
to this:
[root@rdma-dev-01 ~]# more
3.12.0-rc1.cached_gids+optimized_connect+trimmed_cache+-fixed-cmtime.output
ib1:
step total ms max ms min us us / conn
create id : 34.45 0.08 1.00 3.44
bind addr : 88.41 0.04 7.00 8.84
resolve addr : 33.59 4.65 612.00 3.36
resolve route: 618.68 0.61 97.00 61.87
create qp : 4024.03 6.30 341.00 402.40
connect : 6983.35 6886.33 8509.00 698.33
disconnect : 5066.47 230.34 831.00 506.65
destroy : 37.02 0.03 2.00 3.70
ib0:
step total ms max ms min us us / conn
create id : 42.61 0.14 1.00 4.26
bind addr : 27.05 0.03 2.00 2.70
resolve addr : 40.65 10.73 869.00 4.06
resolve route: 626.75 0.60 103.00 62.68
create qp : 3334.50 6.48 273.00 333.45
connect : 6310.29 6251.59 13298.00 631.03
disconnect : 5111.12 365.87 867.00 511.11
destroy : 36.57 0.02 2.00 3.66
with this patch.
Signed-off-by: Doug Ledford <dledford@redhat.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Sean Hefty [Thu, 22 May 2014 23:13:08 +0000 (16:13 -0700)]
indexer: Free index_map resources when cleared
Free memory allocated for index map entries when they are no
longer in use. To handle this, count the number of entries
stored by the index map item arrays and release the arrays when
no items are being tracked.
This reduces valgrind noise.
Problem reported by: Hannes Weisbach <hannes_weisbach@gmx.net>
Sean Hefty [Thu, 17 Apr 2014 05:01:51 +0000 (22:01 -0700)]
rsocket: Relax requirement for minimal inline data
Inline data support is optional. Allow rsockets to work
with devices that do not support inline data, provided
that they do support RDMA writes with immediate data.
This allows rsockets to work over Intel TrueScale HCA.
Patch derived from work by: Amir Hanania
Signed-off-by: Amir Hanania <amir.hanania@intel.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Sean Hefty [Thu, 17 Apr 2014 05:33:38 +0000 (22:33 -0700)]
rsocket: Modify when control messages are available
Rsockets currently tracks how many control messages (i.e.
entries in the send queue) that are available using a
single ctrl_avail counter. Seems simple enough.
However, control messages currently require the use of
inline data. In order to support control messages that
do not use inline data, we need to associate each
control message with a specific data buffer. This will
become easier to manage if we modify how we track when
control messages are available.
We replace the single ctrl_avail counter with two new
counters. The new counters conceptually treat control
messages as if each message had its own sequence number.
The sequence number will then be able to correspond to
a specific data buffer in a follow up patch.
ctrl_seqno will be used to indicate the current control
message being sent. ctrl_max_seqno will track the
highest control message that may be sent.
A side effect of this change is that we will be able to
see how many control messages have been sent. This also
separates the updating of the control count on the
sending side, versus the receiving side.
Sean Hefty [Thu, 17 Apr 2014 15:37:47 +0000 (08:37 -0700)]
rsocket: Dedicate a fixed number of SQEs for control messages
The number of SQEs allocated for control messages is set
to 1 of 2 constant values (either 4 or 2). A default
value is used unless the size of the SQ is below a certain
threshold (16 entries). This results in additional code
complexity, and it is highly unlikely that the SQ would
ever be allocated smaller than 16 entries.
Simplify the code to use a single constant value for the
number of SQEs allocated for control messages. This will
also help in subsequent patches that will need to deal
with HCAs that do not support inline data.
Sean Hefty [Thu, 17 Apr 2014 04:42:06 +0000 (21:42 -0700)]
rsocket: Check max inline data after creating QP
The ipath provider will ignore the max_inline_size
specified as input into ibv_create_qp and instead
return the size that it supports (which is 0) on
output.
Update the actual inline size returned from create QP,
and check that it meets the minimum requirement for
rsockets.
Sean Hefty [Wed, 9 Apr 2014 19:19:25 +0000 (12:19 -0700)]
librdmacm: Support lazy initialization
librdmacm currently opens a device context per configured HCA. This is
usually done in rdma_create_event_channel() or first time whenever
ucma_init() is called. If a process is only going to use one of the
configured HCAs/RDMA IPs then the remaining device contexts are not
used/required. Opening a device context on each device apriori limits the
maximum number of processes that can be supported on a node to the maximum
number of open context supported per HCA regardless of number of HCAs present
in the system.
Signed-off-by: Shamir Rabinovitch <shamir.rabinovitch@oracle.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Sean Hefty [Thu, 6 Mar 2014 21:42:31 +0000 (13:42 -0800)]
rsocket: Fix sbuf_bytes_avail counter 'overrun' with iwarp
Reported-by: Jonas Pfefferle1 <JPF@zurich.ibm.com>
"The problem is that on the client side sbuf_bytes_avail overflows
in rs_poll_cq. And from what I debugged so far there are 2
completions for every send and this is because I use iWarp hardware
which does not support write with immediate so there is one completion
for the write and one for the send (both go into the default case
and add the length to sbuf_bytes_avail)."
To avoid the issue, we flag send message operations that are used
in place of immediate data. Other send message operations are
not affected. The completion code can then check whether the
completion is for a send message which was paired with an RDMA
write transaction and adjust the behavior accordingly. Additionally,
such send messages only carry the opcode in their WR_ID, with the
data portion zeroed. This avoids adding the length value twice.
Sean Hefty [Mon, 27 Jan 2014 19:30:34 +0000 (11:30 -0800)]
udaddy: Remove support for port space IB
UD support for the IB port space requires that the application
use rdma_create_ep, rather than rdma_create_id. However, using
rdma_create_ep results in address and route resolution being
performed synchronously as part of the rdma_create_ep call.
Since udaddy is an example, we want to show how it can be used
with asynchronous events. So, rather than update udaddy to
use rdma_create_ep in order to support the IB port space, it
would be better to remove that support.
Sean Hefty [Tue, 26 Nov 2013 21:16:19 +0000 (13:16 -0800)]
librdmacm: Check 'init' under mutex
ucma_ib_init() does a quick check that access to ibacm has
been initialized. This check is done outside of the
acm_lock mutex. We need to check init again inside of
holding the mutex to ensure that we don't run the
initialization code twice.
Sean Hefty [Mon, 18 Nov 2013 21:12:04 +0000 (13:12 -0800)]
rping: Fix server reporting error on exit
Commit e57196c71ddd850e14f3e66355f02786e4914f72
rping: added checks to the return values functions
resulted in the rping server always reporting that
it failed. Fix this by only failing in the case of
an unexpected termination, and not the result of
the client completing.
Sean Hefty [Mon, 11 Nov 2013 18:24:54 +0000 (10:24 -0800)]
Retrieve SGID after calling rdma_bind_addr
A change was made to rdma_bind_addr when AF_IB is enabled
to only retrieve the resulting bound address. Previously,
rdma_bind_addr would retrieve the corresponding SGID as
well. This breaks some apps which were checking the
SGID after binding to an IP address. Revert to the
previous behavior of also retrieving the SGID after
calling rdma_bind_addr.
Tested-by: Christoph Lameter <cl@linux.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Sean Hefty [Fri, 16 Aug 2013 22:15:12 +0000 (15:15 -0700)]
rsockets: Handle race between rshutdown and rpoll
Multi-threaded applications which call rpoll and rshutdown
simultaneously can hang. Ceph developers reported an issue
with the rsocket implementation. Ceph calls rpoll in
one thread, and while that thread is blocked in rpoll,
a second thread may cann rshutdown on the socket. In
normal sockets, this results in the poll call unblocking
(since a call to read on the socket will no longer block).
however, rsockets does not free the thread blocked on the
rpoll call.
To fix this, we add some additional state checking to
protect against threads calling rpoll and rshutdown
simultaneously. We also have the rshutdown call
transition the QP into an error state. This causes all
posted receives to complete as flushed, which results
in unblocking the thread in rpoll (to process the flushed
receives).
Sean Hefty [Mon, 5 Aug 2013 17:57:43 +0000 (10:57 -0700)]
cmtime: Add example program that times rdma cm calls
cmtime is a new sample program that measures how long it
takes for each step in the connection process to complete.
It can be used to analyze the performance of the various
CM steps.
Sean Hefty [Fri, 26 Jul 2013 16:52:55 +0000 (09:52 -0700)]
rstream: Use rsocket option to set route directly
If we're using GID addressing, rdma_getaddrinfo can return
routing data directly. Add an option for the user to
indicate that rdma_getaddrinfo should be called in place of
getaddrinfo. And if routing data is available, call
rsetsockopt to set the route.
This helps test rsockets when ibacm and AF_IB support are
available.
Sean Hefty [Mon, 10 Jun 2013 19:33:20 +0000 (12:33 -0700)]
rsockets: Add ability to set the IB route directly
Add an RDMA specific rsocket option that allows the user
to program the RDMA route directly. This is useful
for apps that have path record data available, e.g. from
ibacm.
Sean Hefty [Thu, 18 Jul 2013 20:26:15 +0000 (13:26 -0700)]
rsockets: Support native IB addressing on connected rsockets
Update rsockets to support AF_IB addresses on connected rsockets.
Support for datagram rsockets is more difficult as a result of
using real UDP sockets for QP resolution, so that support is
deferred. For connected sockets, we need to update internal
checks to handle AF_IB.
Sean Hefty [Mon, 10 Jun 2013 17:57:56 +0000 (10:57 -0700)]
init: Remove USE_IB_ACM configuration option
When the librdmacm is configured, it sets the USE_IB_ACM option
if infininband/acm.h is found. We can remove this option with
very little overhead, which would allow a user to install
ACM after installing the librdmacm, and the librdmacm would be
able to make use of ACM.
Sean Hefty [Mon, 10 Jun 2013 18:07:12 +0000 (11:07 -0700)]
acm: Define needed ACM protocol messages
The librdmacm needs message definitions used to communicate
with the ibacm. It currently pulls these from infiniband/acm.h,
which is installed by ibacm. This creates an install order
dependency on ibacm. However, work on the scalable SA has
the ibacm using the librdmacm (via rsockets) for communication
between the different SSA components.
To resolve this issue, have the librdmacm define the message
structures that it needs to communicate with ibacm. The
librdmacm already defines some ACM messages through configuration
checks. We just expand that capability, which isolates the librdmacm
package from the ibacm package.
File opened by librdmacm are not supposed to be inherited across
exec*(), most of the files are of no use for another program, and
others cannot be used without the associated memory mapping.
This patch changes fopen() open() and socket() to always set
close on exec flag.
This patch also add checks to configure to guess if fopen() supports
"e" flag. If O_CLOEXEC and SOCK_CLOEXEC are supported, fopen() should
support "e". If not supported, its discarded according to POSIX. Many
operating systems have support for fopen("e").
You might find more information about close on exec in the following articles:
- "Excuse me son, but your code is leaking !!!" by Dan Walsh
http://danwalsh.livejournal.com/53603.html
- "Secure File Descriptor Handling" by Ulrich Drepper
http://udrepper.livejournal.com/20407.html
Note: this patch won't set close on exec flag on file descriptors
created by the kernel for completion channel and such.
This is addressed by another kernel patch.
Signed-off-by: Yann Droneaud <ydroneaud@opteya.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Following advice in "Autotool Mythbuster" [1], option subdir-objects
can be used to have Makefiles create object files in the same
directory than theirs source files.
It reduces clobbering in the build directory.
[1] "Autotool Mythbuster", by Diego Elio "Flameeyes" Petten`o
http://www.flameeyes.eu/autotools-mythbuster/automake/nonrecursive.html
Signed-off-by: Yann Droneaud <ydroneaud@opteya.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Jeff Squyres [Tue, 16 Jul 2013 21:59:47 +0000 (23:59 +0200)]
autogen.sh: Use autoreconf in autogen.sh
The old sequence of Autotools commands listed in autogen.sh is no
longer correct. Instead, just use the single "autoreconf" command,
which will invoke all the Right Autotools commands in the correct
order.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com> Signed-off-by: Yann Droneaud <ydroneaud@opteya.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Makefile.am:1: `INCLUDES' is the old name for `AM_CPPFLAGS' (or `*_CPPFLAGS')
A quote from the automake manual:
INCLUDES
This does the same job as AM_CPPFLAGS (or any per-target _CPPFLAGS variable
if it is used). It is an older name for the same functionality. This
variable is deprecated; we suggest using AM_CPPFLAGS and per-target
_CPPFLAGS instead.
Signed-off-by: Bart Van Assche <bvanassche@acm.org> Signed-off-by: Yann Droneaud <ydroneaud@opteya.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Switch to the modern form of the AM_INIT_AUTOMAKE macro and tell
automake that the librdmacm package does not follow the GNU
standards. This change makes it possible to use 'autoreconf' for the
librdmacm package.
Signed-off-by: Bart Van Assche <bvanassche@acm.org> Signed-off-by: Yann Droneaud <ydroneaud@opteya.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Sean Hefty [Thu, 11 Apr 2013 17:05:29 +0000 (10:05 -0700)]
rsocket: Add support for iWarp
iWarp does not support RDMA writes with immediate data.
Instead of sending messages using immediate data, allow
the rsocket protocol to exchange messages using sends.
The rsocket protocol remains the same. RDMA writes are
used for data transfers, with send messages used to transfer
rsocket protocol messages.
Sean Hefty [Fri, 12 Apr 2013 21:41:52 +0000 (14:41 -0700)]
rsocket: Merge usage of wr_id between stream and datagram svcs
The rsocket data streaming and datagram services use different
formats for the wr_id. Although some differences are needed,
we can make them more similar. This will be useful when the
wr_id is used for iwarp support, plus eliminates use of wr_id
bits that aren't actually needed.
Sean Hefty [Wed, 20 Feb 2013 04:03:58 +0000 (20:03 -0800)]
librdmacm/rsocket: Fix resetting O_NONBLOCK after calling shutdown
Shutdown switches an rsocket from nonblocking to blocking to
ensure that all data has been sent. After completing all
transfers, it should switch back to nonblocking; this handles
partial shutdown situations, where only half the connection
is shut down. However, the code uses the value of '1' to
set the nonblocking flag, rather than O_NONBLOCK. Fix this.
Sean Hefty [Tue, 5 Feb 2013 00:52:18 +0000 (16:52 -0800)]
librdmacm/rstream: Reduce default transfer count
1 million ping-pong transfers takes over 3 seconds to
complete, and I'm impatient. Reduce the default number of
transfers for small messsages to speed up running
performance tests, especially when running over slower
connections, like TCP sockets or over a WAN.
Older kernels have a bug where it can report an event with the
uid set to 0. The librdmacm crashes when casting the uid to
an rdma_cm_id and dereferencing the NULL pointer.
There are a limited number of events where this can occur and
in most cases it's safe to simply discard the event. (This is
what the kernel does anyway.) However, it's possible for us
to process an RDMA_CM_EVENT_ESTABLISHED event with the uid
set to 0. (See kernel commit 418edaaba96e58112b15c82b4907084e2a9caf42.)
Although it's rare for this to occur, it does in fact happen
in practice. To work-around the kernel bug, when the uid of an
established event is set to 0, we first try to locate the correct
user space id based on related data before discarding the event.
Sean Hefty [Mon, 28 Jan 2013 22:56:25 +0000 (14:56 -0800)]
librdmacm: Define ucma_ib_init when IB_ACM is disabled
ucma_ib_init is only defined if IB_ACM is enabled, which is
determined by looking for the infiniband/acm.h header file.
Define ucma_ib_init when IB_ACM is disabled.
Problem reportedy by Suresh Shelvapille <suri@baymicrosystems.com>
Sean Hefty [Wed, 9 Jan 2013 22:54:47 +0000 (14:54 -0800)]
rsockets: Add support for existing UDP apps
Support for existing UDP applications is done via the rspreload
library. However, when the preload library is loaded, socket
calls used by rsockets get intercepted and converted into
rsocket calls.
The preload library was able to handle this for TCP rsockets
by using a per thread variable and checking for recursive calls
coming from rsockets back into the preload library. The preload
library would direct such calls to the real socket calls.
The problem is more complex for UDP rsockets, which can invoke
socket calls from an internal rsocket thread. The result is that
the preload library intercepts socket calls that originate from
the rsocket library which are not recursive.
Although, this is really a problem with the preload library,
the simplest solution is for rsockets to fully initialize the
library when allocating the first rsocket, versus deferring
initialization until required. The preload library can then
detect the recursive calls.
Sean Hefty [Fri, 9 Nov 2012 18:26:38 +0000 (10:26 -0800)]
rsocket: Add datagram support
Add datagram support through the rsocket API.
Datagram support is handled through an entirely different protocol and
internal implementation than streaming sockets. Unlike connected rsockets,
datagram rsockets are not necessarily bound to a network (IP) address.
A datagram socket may use any number of network (IP) addresses, including
those which map to different RDMA devices. As a result, a single datagram
rsocket must support using multiple RDMA devices and ports, and a datagram
rsocket references a single UDP socket, plus zero or more UD QPs.
Rsockets uses headers inserted before user data sent over UDP sockets to
resolve remote UD QP numbers. When a user first attempts to send a datagram
to a remote address (IP and UDP port), rsockets will take the following steps:
1. Store the destination address into a lookup table.
2. Resolve which local network address should be used when sending
to the specified destination.
3. Allocate a UD QP on the RDMA device associated with the local address.
4. Send the user's datagram to the remote UDP socket.
A header is inserted before the user's datagram. The header specifies the
UD QP number associated with the local network address (IP and UDP port) of
the send.
A service thread is used to process messages received on the UDP socket. This
thread updates the rsocket lookup tables with the remote QPN and path record
data. The service thread forwards data received on the UDP socket to an
rsocket QP. After the remote QPN and path records have been resolved, datagram
communication between two nodes are done over the UD QP.
Sean Hefty [Mon, 5 Nov 2012 19:53:03 +0000 (11:53 -0800)]
rsocket: Remove fscanf build warnings
Cast fscanf return values to (void) to indicate that we don't
care if the call fails. In the case of a failure, we simply
fall back to using default values.
Problem reported by Or Gerlitz <ogerlitz@mellanox.com>
Sean Hefty [Wed, 24 Oct 2012 17:23:52 +0000 (10:23 -0700)]
riostream: Add example program for using iomap routines.
riostream is based on rstream, but uses the new riomap, riounmap,
and riowrite calls instead. It runs a series of latency and
bandwidth tests using remote iomapped memory.
riostream is limited to using zero copy transfers at the
receiving side only at this time.
Sean Hefty [Sun, 21 Oct 2012 21:16:03 +0000 (14:16 -0700)]
rsocket: Add APIs for direct data placement
We introduce rsocket extensions for supporting direct
data placement (also known as zero copy). Direct data
placement avoids data copies into network buffers when
sending or receiving data. This patch implements zero
copies on the receive side, but adds some basic framework for
supporting it on the sending side.
Integrating zero copy support into the existing socket APIs
is difficult to achieve when the sockets are set as
nonblocking. Any such implementation is likely to be unusable
in practice. The problem stems from the fact that socket
operations are synchronous in nature. Support for asynchronous
operations is limited to connection establishment.
Therefore we introduce new calls to handle direct data placement.
The use of the new calls is optional and does not affect the
use of the existing calls. An attempt is made to have the new
routines integrate naturally with the existing APIs. The new
functions are: riomap, riounmap, and riowrite. The basic operation
can be described as follows:
1. App A calls riomap to register a data buffer with the local
RDMA device. Riomap returns an off_t offset value that
corresponds to the registered data buffer. The app may
select the offset value.
2. Rsockets will transmit an internal message to the remote
peer with information about the registration. This exchange
is hidden from the applications.
3. App A sends a notification message to app B indicating that
the remote iomapped buffer is now available to receive data.
4. App B calls riowrite to transmit data directly into the
riomapped data buffer.
5. App B sends a notification message to app A indicating that
data is available in the mapped buffer.
6. After all transfers are complete, app A calls riounmap to
deregister its data buffer.
Riomap and riounmap are functionally equivalent to RDMA
memory registration and deregistration routines. They are loosely
based on the mmap and munmap APIs.
off_t riomap(int socket, void *buf, size_t len,
int prot, int flags, off_t offset)
Riomap registers an application buffer with the RDMA hardware
associated with an rsocket. The buffer is registered either for
local only access (PROT_NONE) or for remote write access (PROT_WRITE).
When registered for remote access, the buffer is mapped to a given
offset. The offset is either provided by the user, or if the user
selects -1 for the offset, rsockets selects one. The remote peer may
access an iomapped buffer directly by specifying the correct offset.
The mapping is not guaranteed to be available until after the remote
peer receives a data transfer initiated after riomap has completed.
int riounmap(int socket, void *buf, size_t len)
Riounmap removes the mapping between a buffer and an rsocket.
Riowrite allows an application to transfer data over an rsocket
directly into a remotely iomapped buffer. The remote buffer is specified
through an offset parameter, which corresponds to a remote iomapped buffer.
From the sender's perspective, riowrite behaves similar to rwrite. From
a receiver's view, riowrite transfers are silently redirected into a pre-
determined data buffer. Data is received automatically, and the receiver
is not informed of the transfer. However, iowrite data is still considered
part of the data stream, such that iowrite data will be written before a
subsequent transfer is received. A message sent immediately after
initiating an iowrite may be used to notify the receiver of the iowrite.
It should be noted that the current implementation primarily focused
on being functional for evaluation purposes. Some checks have been
deferred for subsequent patches, and performance is currently limited
by linear lookups.
Roland Dreier [Tue, 16 Oct 2012 19:44:39 +0000 (19:44 +0000)]
rdma_xserver/client: Fix man page formatting
Putting 'r' at the beginning of a line in the nroff source for man pages
is confusing to nroff because lines that start with a single quote
character ' or a dot character . are treated as control lines, which is
not what's intended here. Some of the man page text ends up left out of
the formatted output.
Fix this by just wrapping the text slightly differently in the source
(which doesn't matter since nroff reflows the text anyway). Also add a
missing ".TP" so that the -p and -c options are not run together in the
formatted output.
Signed-off-by: Roland Dreier <roland@purestorage.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Sean Hefty [Mon, 8 Oct 2012 17:33:21 +0000 (10:33 -0700)]
librdmacm: Disable ACM support if ibacm.port is not found
The librdmacm will try to connect port 6125 if ibacm.port is
not found. The problem is that some other service or application
could be using that port and respond with garbage. Rather
than falling back to a hard coded port number, if ibacm.port
is not found, simply disable ACM support.
This has the effect of removing support for older versions
of ibacm, unless the port file is created manually.
Patch created based on feedback from Doug Ledford and Florian
Weimer from RedHat.
Sean Hefty [Thu, 4 Oct 2012 18:49:59 +0000 (11:49 -0700)]
ucmatose: Fix number of connections to disconnect
When ucmatose aborts because of issues trying to connect
to the server, it moves to disconnecting all connections.
However, not all connections may have been established.
The result is that ucmatose will hang in disconnect_events.
Fix this by setting the number of times that we need to
disconnect to the number of times that we successfully
connect.
This problem is based on a report by Doug Ledford
<dledford@redhat.com>
Sean Hefty [Wed, 3 Oct 2012 22:05:20 +0000 (15:05 -0700)]
rping: Reduce retry_count to fit in 3-bits
retry_count is a 3 bit value on IB, reduce it from
10 to 7.
A value of 10 prevents rping from working over the Intel
IB HCA. Problem reported by Doug Ledford <dledford@redhat.com>
The retry_count is also not set when calling rdma_accept.
Rather than passing different values into rdma_accept than
what was specified by the remote side, use the values given
in the connection request.
Sean Hefty [Mon, 10 Sep 2012 21:32:45 +0000 (14:32 -0700)]
rsockets: Document rsocket protocol and design
Include a brief overview of the rsocket protocol and underlying design
with the source code to make it easier for someone trying to decipher
the actual code.
Sean Hefty [Tue, 28 Aug 2012 19:33:04 +0000 (12:33 -0700)]
librdmacm: Support using GIDs with rdma_getaddrinfo
Allow the user to specify a GID as the node parameter into
rdma_getaddrinfo.
To distinguish between the node being an IPv6 address or a GID,
we add a new flag, RAI_FAMILY, which can be set as part of the
hints to rdma_getaddrinfo. When set, this flag indicates that the
value of ai_family in the hints should be used when interpretting
the node parameter.
Sean Hefty [Fri, 7 Sep 2012 17:20:53 +0000 (10:20 -0700)]
rspreload: Fix state checks in dup2
The patch to add dup2 support was never updated to handle the fd
state. The check for the fd type == fd_fork is no longer valid.
We need to instead check the fd state before handling forking.
Problem pointed out by Alex Couvrard <acouvrard@gmail.com>
projects / ~shefty / librdmacm.git / log