Sean Hefty [Fri, 25 May 2012 17:48:47 +0000 (10:48 -0700)]
librdmacm: Delay ACM connection until resolving an address
Avoid creating a connection to the ACM service when
it's not needed. For example, if the user of the librdmacm
is a server application, it will not use ACM services.
Sean Hefty [Fri, 25 May 2012 01:18:43 +0000 (18:18 -0700)]
rsockets: Reduce the default inline size
Inline data consumes the same space used by the SGL. Since
we reduced the default number of SGEs per SQ entry to 1,
also reduce the default inline data size to 16 bytes.
Otherwise, the SQ size won't actually be reduced.
Although this increases the latency of small messages over
16 bytes, tests show that decreasing the inline data size
from 64 bytes to 32 or 16 bytes improves large message
bandwidth 8-10%.
Sean Hefty [Thu, 24 May 2012 21:36:41 +0000 (14:36 -0700)]
rsocket: Reduce SQ from 2 SGE per WR to 1 SGE
We currently request 2 SGEs per WR when allocating a QP. The
second SGE is only used when sending data at the end of
the circular send buffer and the start. All other sends are
restricted to a single SGE.
Reduce the size of the SQ by only requesting 1 SGE per WR. The
resulting performance is basically unaffected.
Sean Hefty [Thu, 24 May 2012 21:31:12 +0000 (14:31 -0700)]
rsockets: Change the default QP size from 512 to 384
Simple bandwidth tests using rstream showed no difference in
performance between using a QP sized to 384 entries versus 512.
Reduce the overhead of a default rsocket by using 384 entries.
A user can request a larger size by calling rsetsockopt.
Sean Hefty [Tue, 22 May 2012 01:46:36 +0000 (18:46 -0700)]
rsocket preload: Use environment variable to set QP size
Allow the user to specify the size of the send/receive queues
and inline data size through environment variables:
RS_SQ_SIZE, RS_RQ_SIZE, and RS_INLINE.
Sean Hefty [Sat, 26 May 2012 07:02:47 +0000 (00:02 -0700)]
rsocket: Merge nonblock test with test() routine in rs_process_cq
rs_process_cq takes the following 2 parameters: nonblock and test().
These are used to control the operation of rs_process_cq. If
nonblock is true, rs_process_cq will exit without arming the CQ or
waiting on a CQ event. rs_process_cq() will also exit if test()
returns true. The only difference in the operation is the return
value that rs_process_cq() returns.
We can simplify the code by merging the nonblock test into the
caller's provided test() routine. The test() routine simply needs
to return the correct value for rs_process_cq(). This will also
simplify fixing an issue where a caller may block indefinitely
in send() or recv() after an rsocket has been disconnected. That
fix is in a subsequent patch.
Sean Hefty [Sat, 26 May 2012 07:02:47 +0000 (00:02 -0700)]
rsocket: Merge nonblock test with test() routine in rs_process_cq
rs_process_cq takes the following 2 parameters: nonblock and test().
These are used to control the operation of rs_process_cq. If
nonblock is true, rs_process_cq will exit without arming the CQ or
waiting on a CQ event. rs_process_cq() will also exit if test()
returns true. The only difference in the operation is the return
value that rs_process_cq() returns.
We can simplify the code by merging the nonblock test into the
caller's provided test() routine. The test() routine simply needs
to return the correct value for rs_process_cq(). This will also
simplify fixing an issue where a caller may block indefinitely
in send() or recv() after an rsocket has been disconnected. That
fix is in a subsequent patch.
Sean Hefty [Sat, 26 May 2012 07:02:47 +0000 (00:02 -0700)]
rsocket: Merge nonblock test with test() routine in rs_process_cq
rs_process_cq takes the following 2 parameters: nonblock and test().
These are used to control the operation of rs_process_cq. If
nonblock is true, rs_process_cq will exit without arming the CQ or
waiting on a CQ event. rs_process_cq() will also exit if test()
returns true. The only difference in the operation is the return
value that rs_process_cq() returns.
We can simplify the code by merging the nonblock test into the
caller's provided test() routine. The test() routine simply needs
to return the correct value for rs_process_cq(). This will also
simplify fixing an issue where a caller may block indefinitely
in send() or recv() after an rsocket has been disconnected. That
fix is in a subsequent patch.
Sean Hefty [Sat, 26 May 2012 00:24:08 +0000 (17:24 -0700)]
rsocket: Fix hang in rrecv/rsend after disconnecting
If a user calls rrecv() after a blocking rsocket has been disconnected,
it will hang. This problem and the cause was reported by Sirdhar Samudrala
<samudrala@us.ibm.com>. It can be reproduced by running netserver -f -D
using the rs-preload library. A similar issue exists with rsend().
Fix this by not blocking on a CQ unless we're connected.
Sean Hefty [Sat, 26 May 2012 00:24:08 +0000 (17:24 -0700)]
rsocket: Fix hang in rrecv() after disconnecting
If a user calls rrecv() after a blocking rsocket has been disconnected,
it will hang. This problem was reported by Sirdhar Samudrala
<samudrala@us.ibm.com>. It can be reproduced by running netserver -f -D
using the rs-preload library.
Signed-off-by: Pradeep Satyanarayana <pradeeps@linux.vnet.ibm.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Sean Hefty [Sat, 26 May 2012 00:24:08 +0000 (17:24 -0700)]
rsocket: Fix hang in rrecv() after disconnecting
If a user calls rrecv() after a blocking rsocket has been disconnected,
it will hang. This problem was reported by Sirdhar Samudrala
<samudrala@us.ibm.com>. It can be reproduced by running netserver -f -D
using the rs-preload library.
Signed-off-by: Pradeep Satyanarayana <pradeeps@linux.vnet.ibm.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Sean Hefty [Sat, 26 May 2012 00:24:08 +0000 (17:24 -0700)]
rsocket: Fix hang in rrecv() after disconnecting
If a user calls rrecv() after a blocking rsocket has been disconnected,
it will hang. This problem was reported by Sirdhar Samudrala
<samudrala@us.ibm.com>. It can be reproduced by running netserver -f -D
using the rs-preload library.
Signed-off-by: Pradeep Satyanarayana <pradeeps@linux.vnet.ibm.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Sean Hefty [Sat, 26 May 2012 00:24:08 +0000 (17:24 -0700)]
rsocket: Fix hang in rrecv() after disconnecting
If a user calls rrecv() after a blocking rsocket has been disconnected,
it will hang. This problem was reported by Sirdhar Samudrala
<samudrala@us.ibm.com>. It can be reproduced by running netserver -f -D
using the rs-preload library.
Signed-off-by: Pradeep Satyanarayana <pradeeps@linux.vnet.ibm.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>