pmmccorm [Mon, 13 May 2013 21:03:04 +0000 (14:03 -0700)]
Enable ccl-proxy support if possible by default: yes, if nothing specified and scif.h is present no, if scif.h is not present and nothing specfied no, if --enable-mcm=no is specified yes, if --enable-mcm=yes and scif.h is present error, if --enable-mcm=yes and scif.h missing
Make the corresponding changes to the spec file so that whatever
options are specified, the RPM will contain the right files (before
we were shipping the mpxyd service and conf regardless).
Arlin Davis [Fri, 17 May 2013 19:27:36 +0000 (12:27 -0700)]
SCM: getifaddrs modfications for better out of the box experience with MIC
socket cm will now walk list of interfaces and ignore loopback
and ignore IB devices, unless the IB netdev is the only device.
Works better in a heterogenous environment with a mix of MICs.
Tested with br0, mic0, and mic0:ib netdev mixes.
Overriding with DAPL_SCM_NETDEV still works as is.
Signed-off-by: Patrick Mccormick <patrick.m.mccormick@intel.com> Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Mon, 25 Mar 2013 17:18:05 +0000 (10:18 -0700)]
mpxyd: add support for full work request or memory pool
Current implemention will fail when WR or memory is full. Change to
throttle and retry mix post_send opertions during full work queue.
New wr_pp (pst pending) added to m_qp for tracking outstanding
IB work request in flight.
Add counters for full wr and mem pool cases. Print mix-version on
startup.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Add eager completion, configurable, to signal writes or sends
after scif_readfrom is signaled and all data is local to proxy
instead of waiting for IB signal. User data on MIC is available
for reuse.
Combine sends and writes to mix_post_send command, provide
ordering guarantees between inline and dma data. Allow's
direct posting from OP thread is head of queue.
Add new counters for inline and signaled IO.
Extend m_wr to include flags for controlling eager completions
and proxy buffer and work request management.
cq event FD is now non-blocking and processed via TX thread
instead of OP thread. Allows for polling > 1 event at a time.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Fri, 15 Mar 2013 19:23:16 +0000 (12:23 -0700)]
mcm: add support for mix inline data, improve mix_poll events
mpxyd can be configured for inline data for posted
writes and sends. This will use scif_send/recv instead
of scif_readfrom based on threashold set in mpxyd.conf
change the mix_poll command to NOT issue the request
on scif and simply wait for mpxyd to write completion
back to EVD. This removes unneccesary SCIF command
traffic.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Wed, 6 Feb 2013 19:49:29 +0000 (11:49 -0800)]
install: problem with rpm update when updating from 2.0.34 or older
The postun will remove entries on older packages that incorrectly
add and remove entries instead of updating the file. When updating
from these older version we end up with an empty /etc/dat.conf.
In order to fix we have to save the dat.conf and restore during
the upgrade process with the triggerpostun
Arlin Davis [Sat, 2 Feb 2013 01:33:17 +0000 (17:33 -0800)]
mpxyd: cm scaling bug fixes and profiling
New CM thread to help with CM scale out. Testing with dtestcm
with 1000's of connections. MPI testing up to 60ppn on KNC nodes.
Add new disc timers and disconnect logging for debug.
Add cleanup for IB device during service termination.
Add profiling of device and CM operations to help debug scaling issues
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Mon, 21 Jan 2013 20:51:42 +0000 (12:51 -0800)]
mpxyd: TX thread can miss pending requests with multiple clients
Pending data variable is overwritten with multiple SCIF clients
bound to one HCA causing rdma_write to stall and not posted
on IB device. MPI running multiple ranks on a KNC can stall.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Wed, 16 Jan 2013 21:39:59 +0000 (13:39 -0800)]
mcm,mpxyd: add multi-enpoint, multi-threaded, and CPU affinity support for mpxyd and mcm clients
For performance reasons separate EP's and separate threads have been incorporated.
3 scif eps. operation, events, and transmit are created for every device open
2 threads per MIC adapter, one for operations and one for RDMA operations
CPU affinity support as been added to to assist in HCA to MIC locality
for optimum performance. This fixes some performance issues seen at scale
on HT systems.
Also added some performance profiling to help with future tunining on
various platforms.
The CPU affinity and profiling are set via new mpxyd.conf parameters.
defaults are affinity=1, affinity base cpu_id=0, profiling=0
mcm_affinity, mcm_affinity_base, mcm_profile
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
mpxyd: proxy segfaulting running as daemon, IB recv len incorrect
didn't account for IB UD packet length being larger
than actual CM message due to added 40 byte GRH
with each receive packet. Adjust each recv pkt
len according to size of GRH.
Change default log to /tmp from stdout
Cleanup debug logs
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Thu, 6 Dec 2012 01:27:17 +0000 (17:27 -0800)]
mcm: support for rdma writes with immediate plus associated fixes
Segmented writes fixed to return proper length in work completion
Mpxyd segmentation size sync'ed with configuration file
Reduced proxy WR depth for segmentation, limit to x8.
Add debug info to help profile performance stalls
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Tue, 4 Dec 2012 17:52:22 +0000 (09:52 -0800)]
mpxyd: add common scif_send_msg inline function to handle partial writes
scif_send blocking mode will not always block until
entire message is sent as documented. If will sometimes
return with partial sends. Create a common inline send
function what will handle condition and only return failure
with errors and not partial writes.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
Arlin Davis [Thu, 15 Nov 2012 02:36:56 +0000 (18:36 -0800)]
mpxyd: expose CM request and reply timers and retry count
add entries in the mpxyd.conf for timers and retry. Start with
larger default timers given small cores are processing messages
and they are proxied via SCIF.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>