Sean Hefty [Thu, 24 May 2012 21:36:41 +0000 (14:36 -0700)]
rsocket: Reduce SQ from 2 SGE per WR to 1 SGE
We currently request 2 SGEs per WR when allocating a QP. The
second SGE is only used when sending data at the end of
the circular send buffer and the start. All other sends are
restricted to a single SGE.
Reduce the size of the SQ by only requesting 1 SGE per WR. The
resulting performance is basically unaffected.
Sean Hefty [Thu, 24 May 2012 21:36:41 +0000 (14:36 -0700)]
rsocket: Reduce SQ from 2 SGE per WR to 1 SGE
We currently request 2 SGEs per WR when allocating a QP. The
second SGE is only used when sending data at the end of
the circular send buffer and the start. All other sends are
restricted to a single SGE.
Reduce the size of the SQ by only requesting 1 SGE per WR. The
resulting performance is basically unaffected.
Sean Hefty [Thu, 24 May 2012 21:36:41 +0000 (14:36 -0700)]
rsocket: Reduce SQ from 2 SGE per WR to 1 SGE
We currently request 2 SGEs per WR when allocating a QP. The
second SGE is only used when sending data at the end of
the circular send buffer and the start. All other sends are
restricted to a single SGE.
Reduce the size of the SQ by only requesting 1 SGE per WR. The
resulting performance is basically unaffected.
Sean Hefty [Thu, 24 May 2012 21:36:41 +0000 (14:36 -0700)]
rsocket: Reduce SQ from 2 SGE per WR to 1 SGE
We currently request 2 SGEs per WR when allocating a QP. The
second SGE is only used when sending data at the end of
the circular send buffer and the start. All other sends are
restricted to a single SGE.
Reduce the size of the SQ by only requesting 1 SGE per WR. The
resulting performance is basically unaffected.
Sean Hefty [Mon, 4 Jun 2012 21:51:41 +0000 (14:51 -0700)]
rsocket: Spin before blocking on an rsocket
The latency cost of blocking is significant compared to round
trip ping-pong time. Spin briefly on rsockets before calling
into the kernel and blocking.
The time to spin before blocking is read from an rsocket
configuration file %sysconfig%/rdma/rsocket/polling_time. This
is user adjustable.
As a completely unintentional side effect, this just happens to
improve application performance in benchmarks, like netpipe,
significantly. ;)
It turns out ignoring the SHUT_WR shutdown flag introduces a
separate issue with netperf. From Sridhar Samudrala <sri@us.ibm.com> :
This causes netperf to hang in recv() after shutdown(SHUT_WR)
on the data socket.
#0 0x000000805caa77e4 in __read_nocancel () from /lib64/libc.so.6
#1 0x00000fff78e6422c in .read ()
from /home/sridhar/git/librdmacm/src/preload.so
#2 0x000000805ccfe68c in .ibv_get_cq_event () from /usr/lib64/libibverbs.so.1
#3 0x00000fff78e8a9d4 in rs_get_cq_event (rs=0x1000b85f220)
at src/rsocket.c:867
#4 0x00000fff78e8c160 in rs_process_cq (rs=0x1000b85f220,
nonblock=<value optimized out>,
test=@0xfff78e9efc0: 0xfff78e899c0 <rs_conn_have_rdata>)
at src/rsocket.c:915
#5 0x00000fff78e8d2c0 in rrecv (socket=<value optimized out>,
buf=0xfffd897e428, len=1, flags=<value optimized out>)
at src/rsocket.c:1037
#6 0x00000fff78e63eec in .recv ()
from /home/sridhar/git/librdmacm/src/preload.so
#7 0x00000000100497f4 in .disconnect_data_socket ()
#8 0x000000001004c528 in .send_omni_inner ()
#9 0x00000000100502e0 in .send_tcp_stream ()
#10 0x00000000100028a4 in .main ()
Sean Hefty [Mon, 4 Jun 2012 21:51:41 +0000 (14:51 -0700)]
rsocket: Spin before blocking on an rsocket
The latency cost of blocking is significant compared to round
trip ping-pong time. Spin briefly on rsockets before calling
into the kernel and blocking.
The time to spin before blocking is read from an rsocket
configuration file %sysconfig%/rdma/rsocket/polling_time. This
is user adjustable.
As a completely unintentional side effect, this just happens to
improve application performance in benchmarks, like netpipe,
significantly. ;)
Sean Hefty [Mon, 4 Jun 2012 21:51:41 +0000 (14:51 -0700)]
rsocket: Spin before blocking on an rsocket
The latency cost of blocking is significant compared to round
trip ping-pong time. Spin briefly on rsockets before calling
into the kernel and blocking.
The time to spin before blocking is read from an rsocket
configuration file %sysconfig%/rdma/rsocket/polling_time. This
is user adjustable or may be set automatically by ibacm.
As a completely unintentional side effect, this just happens to
improve application performance in benchmarks, like netpipe,
significantly. ;)
Sean Hefty [Mon, 4 Jun 2012 20:14:42 +0000 (13:14 -0700)]
rsocket: Handle SHUT_RD/WR shutdown flags
Sridhar Samudrala <sri@us.ibm.com> reported an error (EOPNOTSUPP)
after calling select().
The issue is that rshutdown(SHUT_WR) was called before select().
As part of shutdown, rsockets switches the underlying fd from
nonblocking to blocking to ensure that previously sent data has
completed. shutdown(SHUT_WR) indicates that the socket should be
kept open for receiving data.
Delay handling the actual shutdown unless SHUT_RDWR is specified,
or the socket is closed.
projects / ~shefty / librdmacm.git / log