Sean Hefty [Fri, 11 May 2012 17:33:13 +0000 (10:33 -0700)]
librdmacm/rstream: Set rsocket nonblocking for base tests
The base set of rstream tests want nonblocking rsockets, but don't
actually set the rsocket to nonblocking. It instead relies on the
MSG_DONTWAIT flag. Make the code match the expected behavior and
set the rsocket to nonblocking and make nonblocking the default.
Provide a test option to switch it back to blocking mode. We keep
the existing nonblocking test option for compatibility.
Sean Hefty [Thu, 10 May 2012 18:17:32 +0000 (11:17 -0700)]
librdmacm/rsocket: Succeed setsockopt REUSEADDR on connected sockets
The RDMA CM fail calls to set REUSEADDR on an rdma_cm_id if
it is not in the idle state. As a result, this causes a failure
in NetPipe when run with socket calls intercepted by rsockets.
Fix this by returning success when REUSEADDR is set on an rsocket
that has already been connected. When running over IB, REUSEADDR
is not necessary, since the TCP/IP addresses are mapped.
Sean Hefty [Thu, 10 May 2012 18:17:32 +0000 (11:17 -0700)]
librdmacm/rsocket: Succeed setsockopt REUSEADDR on connected sockets
The RDMA CM fail calls to set REUSEADDR on an rdma_cm_id if
it is not in the idle state. As a result, this causes a failure
in NetPipe when is run with socket calls intercepted by rsockets.
Fix this by returning success when REUSEADDR is set on an rsocket
that has already been connected. When running over IB, REUSEADDR
is not necessary, since the TCP/IP addresses are mapped.
Sean Hefty [Thu, 10 May 2012 18:17:32 +0000 (11:17 -0700)]
librdmacm/rsocket: Succeed setsockopt REUSEADDR on connected sockets
The RDMA CM fail calls to set REUSEADDR on an rdma_cm_id if
it is not in the idle state. As a result, this causes a failure
in NetPipe when is run with socket calls intercepted by rsockets.
Fix this by returning success when REUSEADDR is set on an rsocket
that has already been connected. When running over IB, REUSEADDR
is not necessary, since the TCP/IP addresses are mapped.
Sean Hefty [Tue, 8 May 2012 00:16:47 +0000 (17:16 -0700)]
rsockets: Optimize synchronization to improve performance
Hotspot performance analysis using VTune showed pthread_mutex_unlock()
as the most significant hotspot when transferring small messages using
rstream. To reduce the impact of using pthread mutexes, replace it
with a custom lock built using an atomic variable and a semaphore.
When there's no contention for the lock (which is the expected case
for nonblocking sockets), the synchronization is reduced to
incrementing and decrementing an atomic variable.
A test that acquired and released a lock 2 billion times reported that
the custom lock was roughly 20% faster than using the mutex.
26.6 seconds versus 33.0 seconds.
Unfortunately, further analysis showed that using the custom lock
provided a minimal performance gain on rstream itself, and simply
moved the hotspot to the custom unlock call. The hotspot is likely
a result of some other interaction, rather than caused by slowness
in releasing a lock. However, we keep the custom lock based on
the results of the direct lock tests that were done.
Sean Hefty [Tue, 8 May 2012 00:16:47 +0000 (17:16 -0700)]
rsockets: Optimize synchronization to improve performance
Hotspot performance analysis using VTune showed pthread_mutex_unlock()
as the most significant hotspot when transferring small messages using
rstream. To reduce the impact of using pthread mutexes, replace it
with a custom lock built using an atomic variable and a semaphore.
When there's no contention for the lock (which is the expected case
for nonblocking sockets), the synchronization is reduced to
incrementing and decrementing an atomic variable.
A test that acquired and released a lock 2 billion times reported that
the custom lock was roughly 20% faster than using the mutex.
26.6 seconds versus 33.0 seconds.
Unfortunately, further analysis showed that using the custom lock
provided a minimal performance gain on rstream itself, and simply
moved the hotspot to the custom unlock call. The hotspot is likely
a result of some other interaction, rather than caused by slowness
in releasing a lock. However, we keep the custom lock based on
the results of the direct lock tests that were done.
Sean Hefty [Tue, 8 May 2012 00:16:47 +0000 (17:16 -0700)]
rsockets: Optimize synchronization to improve performance
Performance analysis using VTune showed that pthread_mutex_unlock()
is the single biggest contributor to increasing latency for 64-byte
transfers. Unlocked was followed by get_sw_cqe(), then
__pthread_mutex_lock(). Replace the use of mutexes with an atomic
and a semaphore. When there's no contention for the lock (which
would usually be the case when using nonblocking sockets), the
code simply increments and decrements an atomic varible. Semaphores
are only used when contention occurs.
Sean Hefty [Tue, 8 May 2012 00:16:47 +0000 (17:16 -0700)]
rsockets: Optimize synchronization to improve performance
Performance analysis using VTune showed that pthread_mutex_unlock()
is the single biggest contributor to increasing latency for 64-byte
transfers. Unlocked was followed by get_sw_cqe(), then
__pthread_mutex_lock(). Replace the use of mutexes with an atomic
and a semaphore. When there's no contention for the lock (which
would usually be the case when using nonblocking sockets), the
code simply increments and decrements an atomic varible. Semaphores
are only used when contention occurs.
Sean Hefty [Tue, 8 May 2012 00:16:47 +0000 (17:16 -0700)]
rsockets: Optimize synchronization to improve performance
Performance analysis using VTune showed that pthread_mutex_unlock()
is the single biggest contributor to increasing latency for 64-byte
transfers. Unlocked was followed by get_sw_cqe(), then
__pthread_mutex_lock(). Replace the use of mutexes with an atomic
and a semaphore. When there's no contention for the lock (which
would usually be the case when using nonblocking sockets), the
code simply increments and decrements an atomic varible. Semaphores
are only used when contention occurs.
Sean Hefty [Tue, 8 May 2012 00:16:47 +0000 (17:16 -0700)]
rsockets: Optimize synchronization to improve performance
Performance analysis using VTune showed that pthread_mutex_unlock()
is the single biggest contributor to increasing latency for 64-byte
transfers. Unlocked was followed by get_sw_cqe(), then
__pthread_mutex_lock(). Replace the use of mutexes with an atomic
and a semaphore. When there's no contention for the lock (which
would usually be the case when using nonblocking sockets), the
code simply increments and decrements an atomic varible. Semaphores
are only used when contention occurs.
Sean Hefty [Mon, 9 Apr 2012 19:05:39 +0000 (12:05 -0700)]
rsocket: Succeed setting SO_KEEPALIVE option
memcached sets SO_KEEPALIVE, so succeed any requests to set
that option. We don't actually implement keepalive at this time.
To implement keepalive, we would need to record the last time
that a message was sent or received on an rsocket. If no
new messages are processed within the keepalive timeout, then
we would need to issue a keepalive message. For rsockets,
this would simply mean sending a 0-byte control message that
gets ignored on the remote side.
The only real difficulty with handlng keepalive is doing it
without adding threads. This requires additional work in
poll to occasionally timeout, send keepalive messages, then
return to polling if no new data has arrived. Alternatively,
we can add a thread to handle sending keepalive messages.
Sean Hefty [Fri, 6 Apr 2012 22:40:10 +0000 (15:40 -0700)]
rsocket: Succeed SO_LINGER socket option
Succeed calls to set the SO_LINGER socket option. We don't
actually implement SO_LINGER semantics because we never place
an rsocket into a timewait state. Unlike socket behavior,
we do wait for all pending data to be transferred by the hardware.
This is done so that the disconnect message can be sent over
the rsocket connection.
Sean Hefty [Tue, 3 Apr 2012 21:25:23 +0000 (14:25 -0700)]
rsocket: Allow use of LD_PRELOAD to intercept socket calls
Intercept socket calls and convert TCP socket operation to
streaming over RDMA.
Allow falling back from rsockets to normal sockets on error
or when trying to bind/connect to a reserved port. This is
needed to handle MPI job startup, where MPI should use rsockets,
but mpiexect needs to communicate using ssh over normal sockets.
Sean Hefty [Thu, 1 Dec 2011 20:52:28 +0000 (12:52 -0800)]
rsocket: Add sample application to copy files over rsockets
rcopy will copy files from a source system to a specified remote
server. It's essentially a really dumb FTP type program that can
be used to quickly transfer files between systems, which can be
useful to verify data integrity.
(It was easier to create this program than modify an existing FTP
client and server application, which was my first choice. Fork
support is difficult.)
Sean Hefty [Thu, 8 Dec 2011 19:30:12 +0000 (11:30 -0800)]
rsocket: Add example program that uses rsocket
rstream provides an example that uses either rsocket or socket
APIs. The latter allows rstream to be used to verify rsocket
behavior compared to socket.
Sean Hefty [Tue, 3 Apr 2012 21:25:04 +0000 (14:25 -0700)]
librdmacm: Define streaming over RDMA interface (rsockets)
Introduces a new set of APIs that support a byte streaming interface
over RDMA devices. The new interface matches sockets, except that all
function calls are prefixed with an 'r'.
Client connect_events() shoudl fail if it received some error,
otherwise the program will try to reach a non-existent QP
resource resulting in a segfault. Return an error from
cma_handler() if we had a connection error.
Signed-off-by: Sagi Grimberg <sagig@mellanox.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
projects / ~shefty / librdmacm.git / log