Sean Hefty [Fri, 25 May 2012 17:48:47 +0000 (10:48 -0700)]
librdmacm: Delay ACM connection until resolving an address
Avoid creating a connection to the ACM service when
it's not needed. For example, if the user of the librdmacm
is a server application, it will not use ACM services.
Sean Hefty [Fri, 25 May 2012 01:18:43 +0000 (18:18 -0700)]
rsockets: Reduce the default inline size
Inline data consumes the same space used by the SGL. Since
we reduced the default number of SGEs per SQ entry to 1,
also reduce the default inline data size to 16 bytes.
Otherwise, the SQ size won't actually be reduced.
Although this increases the latency of small messages over
16 bytes, tests show that decreasing the inline data size
from 64 bytes to 32 or 16 bytes improves large message
bandwidth 8-10%.
Sean Hefty [Thu, 24 May 2012 21:31:12 +0000 (14:31 -0700)]
rsockets: Change the default QP size from 512 to 384
Simple bandwidth tests using rstream showed no difference in
performance between using a QP sized to 384 entries versus 512.
Reduce the overhead of a default rsocket by using 384 entries.
A user can request a larger size by calling rsetsockopt.
Sean Hefty [Thu, 24 May 2012 21:36:41 +0000 (14:36 -0700)]
rsocket: Reduce SQ from 2 SGE per WR to 1 SGE
We currently request 2 SGEs per WR when allocating a QP. The
second SGE is only used when sending data at the end of
the circular send buffer and the start. All other sends are
restricted to a single SGE.
Reduce the size of the SQ by only requesting 1 SGE per WR. The
resulting performance is basically unaffected.
Sean Hefty [Thu, 24 May 2012 21:36:41 +0000 (14:36 -0700)]
rsocket: Reduce SQ from 2 SGE per WR to 1 SGE
We currently request 2 SGEs per WR when allocating a QP. The
second SGE is only used when sending data at the end of
the circular send buffer and the start. All other sends are
restricted to a single SGE.
Reduce the size of the SQ by only requesting 1 SGE per WR. The
resulting performance is basically unaffected.
Sean Hefty [Thu, 24 May 2012 21:36:41 +0000 (14:36 -0700)]
rsocket: Reduce SQ from 2 SGE per WR to 1 SGE
We currently request 2 SGEs per WR when allocating a QP. The
second SGE is only used when sending data at the end of
the circular send buffer and the start. All other sends are
restricted to a single SGE.
Reduce the size of the SQ by only requesting 1 SGE per WR. The
resulting performance is basically unaffected.
Sean Hefty [Thu, 24 May 2012 21:36:41 +0000 (14:36 -0700)]
rsocket: Reduce SQ from 2 SGE per WR to 1 SGE
We currently request 2 SGEs per WR when allocating a QP. The
second SGE is only used when sending data at the end of
the circular send buffer and the start. All other sends are
restricted to a single SGE.
Reduce the size of the SQ by only requesting 1 SGE per WR. The
resulting performance is basically unaffected.
Sean Hefty [Sat, 26 May 2012 00:24:08 +0000 (17:24 -0700)]
rsocket: Fix hang in rrecv/rsend after disconnecting
If a user calls rrecv() after a blocking rsocket has been disconnected,
it will hang. This problem and the cause was reported by Sirdhar Samudrala
<samudrala@us.ibm.com>. It can be reproduced by running netserver -f -D
using the rs-preload library. A similar issue exists with rsend().
Fix this by not blocking on a CQ unless we're connected.
Sean Hefty [Fri, 25 May 2012 19:42:12 +0000 (12:42 -0700)]
rs-preload: Handle recursive socket() calls
When ACM support is enabled in the librdmacm, it will attempt to
establish a socket connection to the ACM daemon. When the rsocket
preload library is in use, this can result in a recursive call
to socket() that results in the library hanging. The resulting
call stack is:
The second call to ucma_init() hangs because initialization is
still pending.
Fix this by checking for recursive calls to socket() in the preload
library. When detected, call the real socket() call instead of
directing the call back into rsockets(). Since rsockets is a part
of the librdmacm, it can call rsockets directly if it wants to use
rsockets instead of standard sockets.
This problem and the cause was reported by Chet Murthy <chet@watson.ibm.com>
Sean Hefty [Fri, 25 May 2012 17:48:47 +0000 (10:48 -0700)]
librdmacm: Delay ACM connection until resolving an address
Avoid creating a connection to the ACM service when
it's not needed. For example, if the user of the librdmacm
is a server application, it will not use ACM services.
Sean Hefty [Fri, 25 May 2012 01:18:43 +0000 (18:18 -0700)]
rsockets: Reduce the default inline size
Inline data consumes the same space used by the SGL. Since
we reduced the default number of SGEs per SQ entry to 1,
also reduce the default inline data size to 16 bytes.
Otherwise, the SQ size won't actually be reduced.
Although this increases the latency of small messages over
16 bytes, tests show that decreasing the inline data size
from 64 bytes to 32 or 16 bytes improves large message
bandwidth 8-10%.
projects / ~shefty / librdmacm.git / log