Sean Hefty [Fri, 6 Feb 2015 05:17:03 +0000 (21:17 -0800)]
rsockets: Fix setting flags in rfctl
The rfcntl() call to set rsocket flags merely OR's in
the updated flags with the existing ones, rather than
replacing them. Also, it does not handle setting an
rsocket from nonblocking mode back to blocking mode.
Sean Hefty [Fri, 6 Feb 2015 05:17:03 +0000 (21:17 -0800)]
rsockets: Fix setting flags in rfctl
The rfcntl() call to set rsocket flags merely OR's in
the updated flags with the existing ones, rather than
replacing them. Also, it does not handle setting an
rsocket from nonblocking mode back to blocking mode.
Steve Wise [Mon, 12 Jan 2015 16:57:40 +0000 (10:57 -0600)]
rping: create persistent server threads in DETACHED state
Since the persistent server threads aren't joined, they must be created in
the DETACHED state or resources will not be cleaned up when they exit.
This results in pthread_create() failures after thousands of rping
instances are run against a persistent server.
Also check the return from all calls to pthread_create() so we don't
ignore a thread creation failure.
Signed-off-by: Steve Wise <swise@opengridcomputing.com>
Hariprasad S [Thu, 6 Nov 2014 09:12:56 +0000 (14:42 +0530)]
rping: Fixes race, where ibv context was getting freed before memory was deregistered
While running rping as a client without server on the other end,
rping_test_client fails and the ibv context was getting freed
before memory was deregistered. This patch fixes it.
Signed-off-by: Hariprasad Shenai <hariprasad@chelsio.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Hariprasad S [Thu, 6 Nov 2014 09:12:56 +0000 (14:42 +0530)]
[1/2] rping: Fixes race, where ibv context was getting freed before memory was deregistered
While running rping as a client without server on the other end, rping_test_client fails
and the ibv context was getting freed before memory was deregistered. This patch fixes it.
While waiting for a completion event, rsocket state is incorrectly
set to error when interrupted. Instead, the caller of get
completion event should decide what to do with it based on
errno. The fix is do not change the state to rs_error when
errno is EINTR inside get completion event.
Perform completion event acknowledgments in batches instead
of individually to minimze locking overheads. Size of the
completion queue decides the size of a batch.
Signed-off-by: Sreedhar Kodali <srkodali@linux.vnet.ibm.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
rsockets: Add fine grained interception mechanism for preload library
By default the R-Sockets pre-loading library intercepts all
the stream and datagram sockets belonging to a launched
program processes and threads.
However, distributed application and database servers may
require fine grained interception to ensure that only the
processes which are listening for remote connections on the
RDMA transport need to be enabled with RDMA while remaining
can continue to use TCP as before. This allows proper
communication happening between various server components locally.
A configuration file based mechanism is introduced to facilitate
this fine grained interception mechanism. As part of preload
initialization, the configuration file is scanned and an
in-memory record store is created with all the entries found.
When a request is made to intercept a socket, its attributes
are cross checked with stored records to see whether we
should proceed with rsocket switch over.
Note: Right now, the fine grained interception mechanism is
enabled only for newly created sockets. Going forward,
this can be extened to select connections based on the
specified host/IP addresses and ports as well.
"preload_config" is the name of the configuration file which
should exist in the default configuration location
(usually the full path to this configuration file is:
<install-root>/etc/rdma/rsocket/preload_config)
of an installed rsocket library.
The sample format for this configuration file is shown below:
# Sample config file for preloading in a program specific way
#
# Each line entry should have the following format:
#
# program domain type protocol
#
# where,
#
# program - program or command name (string without spaces)
# domain - the socket domain: AF_INET / AF_INET6 / AF_IB
# type - the socket type: SOCK_STREAM / SOCK_DGRAM
# protocol - the socket protocol: IPPROTO_TCP / IPPROTO_UDP
#
# The wildcard value of '*' is supported for any
#
# Note:
# Lines beginning with '#' character are treated as comments.
Sean Hefty [Thu, 4 Sep 2014 18:19:28 +0000 (11:19 -0700)]
rsockets: Support calling listen multiple times on same rsocket
Standard sockets allows an application to call listen() multiple
times on the same socket without error. This allows a multi-threaded
app to call listen from all threads.
rsockets will fail the second listen call. Modify the behavior to
match standard sockets.
Problem reported by: Sreedhar Kodali <srkodali@linux.vnet.ibm.com>
rsocket: Index map item is cleaned before it is used in iomapping cleanup
rs_free function clears index map item corresponding to the roscket
(in idm_clear called from rs_remove) and then uses it in
iomapping cleanup (in riounmap called from rs_free_iomappings).
Signed-off-by: Sasha Kotchubievsky <sashakot@mellanox.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
rsocket: Segmentation fault fix in case of multiple connections
In case of more than 16 rsocket connections
are established, "svc->rss" buffer is reallocated
with more memory. Index 0 is reserved for the service's
communication socket, and this is not taken in count
when data is copied from old buffer location to
new one.
Signed-off-by: Ilya Nelkenbaum <ilyan@mellanox.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Sean Hefty [Wed, 16 Jul 2014 20:44:56 +0000 (13:44 -0700)]
riostream: Only verify last data transfer
Data verification will fail when running the bandwidth
tests or the transfer count is > 1. The issue is that
subsequent writes by the initiator side will overwrite
the data in the target buffer before the receiver can
verify that it is correct.
To fix this, only verify that the data in the buffer
is correct after the last transfer has completed.
0-byte RDMA writes appears to be working correctly with
HCAs from 2 different vendors. The original problem that
was reported turned out to be a user error.
Sean Hefty [Thu, 3 Jul 2014 20:45:52 +0000 (13:45 -0700)]
rsocket: Update correct rsocket keepalive time
When the keepalive time of an rsocket is updated, the
updated information is forwarded to the keepalive service
thread. However, the thread updates the time for the
wrong service as shown:
Sean Hefty [Thu, 3 Jul 2014 20:55:39 +0000 (13:55 -0700)]
rsocket: Fix removing rsocket from service thread
When removing an rsocket from a service thread, we replace
the removed service with the one at the end of the service list.
This keeps the array tightly packed. However, rs_svc_rm_rs
decrements the rsocket count before doing the swap. The result
is that the entry at the end of the list gets dropped off.
Defer decrementing the count until the swap has been made.
In this case, the cnt value is a valid index into the array,
because we start at index 1. Index 0 is used internally by
the service thread.
Sean Hefty [Wed, 2 Jul 2014 22:37:10 +0000 (15:37 -0700)]
rsocket: Fix crash resulting from keepalive timeout
The following crash was reported by Hal Rosenstock,
<hal@mellanox.com>, with keepalive enabled. The crash
occurs in the keepalive thread attempting to send a
keepalive message.
report:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffecf08700 (LWP 6013)]
rs_post_write (rs=<value optimized out>, sgl=0x0, nsge=0, wr_data=3758096385,
flags=0, addr=0, rkey=0) at src/rsocket.c:1660
1660 return rdma_seterrno(ibv_post_send(rs->cm_id->qp, &wr, &bad));
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.107.el6.x86_64
(gdb)
(gdb) p/x rs
$1 = value has been optimized out
So I added in the following to debug:
1660 if (rs == NULL)
1661 abort();
1662 if (rs->cm_id == NULL)
1663 abort();
1664 if (rs->cm_id->qp == NULL)
1665 abort();
1666 return rdma_seterrno(ibv_post_send(rs->cm_id->qp, &wr, &bad));
1667 }
And saw in gdb:
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffecf08700 (LWP 8096)]
0x00000030d50328a5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.107.el6.x86_64
(gdb)
(gdb) bt
#0 0x00000030d50328a5 in raise () from /lib64/libc.so.6
#1 0x00000030d5034085 in abort () from /lib64/libc.so.6
#2 0x00007ffff057fe23 in rs_post_write (rs=<value optimized out>, sgl=0x1fa0,
nsge=6, wr_data=4294967295, flags=0, addr=0, rkey=0) at src/rsocket.c:1665
#3 0x00007ffff058193d in tcp_svc_send_keepalive (arg=0x7ffff0789f20)
at src/rsocket.c:4245
#4 tcp_svc_run (arg=0x7ffff0789f20) at src/rsocket.c:4279
#5 0x00000030d5807851 in start_thread () from /lib64/libpthread.so.0
#6 0x00000030d50e890d in clone () from /lib64/libc.so.6
(gdb) fr 2
#2 0x00007ffff057fe23 in rs_post_write (rs=<value optimized out>, sgl=0x1fa0,
nsge=6, wr_data=4294967295, flags=0, addr=0, rkey=0) at src/rsocket.c:1665
1665 abort();
So qp is NULL somehow...
:end report
There is an issue if an rsocket is closed without going through
the rshutdown.
int rshutdown(int socket, int how)
{
...
if (rs->opts & RS_OPT_SVC_ACTIVE)
rs_notify_svc(&tcp_svc, rs, RS_SVC_REM_KEEPALIVE);
We remove the rsocket from the keepalive thread in rshutdown.
int rclose(int socket)
{
...
if (rs->state & rs_connected)
rshutdown(socket, SHUT_RDWR);
...
rs_free(rs);
rclose will call shutdown only if we're connected. However, if the
keepalive failed, the socket will be in an error state. So,
no call to rshutdown, which will leave the freed rsocket on
the keepalive thread's list.
The fix is to to have rclose remove an rsocket from being processed
by a service thread if it is still active.