shefty [Wed, 5 Aug 2009 23:43:36 +0000 (23:43 +0000)]
libibverbs/comp_mgr: fix hang during destruction
The completion manager abstraction can hang during cleanup. The hang
occurs when the user calls CompManagerCancel before calling CompManagerClose,
and the user is not waiting for events. In this case, the completion manager
thread will pull the cancel request from the IO completion port and queue it
with the manager. When CompManagerClose is called, it calls CompManagerCancel
to signal the thread to check the running state. However, the completion
manager's event structure is still marked busy from the user's CompManagerCancel
call.
The result is that the completion manager thread does not receive the
signal to check the running flag and remains asleep. Fix this by using a
different completion entry to signal the thread during destruction than that
used to cancel a CompManagerPoll event.
This fixes occasional hangs running dapltest with both the rdma_cm and socket
cm providers.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2330 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
shefty [Wed, 5 Aug 2009 23:43:10 +0000 (23:43 +0000)]
libibverbs/comp_mgr: fix hang during destruction
The completion manager abstraction can hang during cleanup. The hang
occurs when the user calls CompManagerCancel before calling CompManagerClose,
and the user is not waiting for events. In this case, the completion manager
thread will pull the cancel request from the IO completion port and queue it
with the manager. When CompManagerClose is called, it calls CompManagerCancel
to signal the thread to check the running state. However, the completion
manager's event structure is still marked busy from the user's CompManagerCancel
call.
The result is that the completion manager thread does not receive the
signal to check the running flag and remains asleep. Fix this by using a
different completion entry to signal the thread during destruction than that
used to cancel a CompManagerPoll event.
This fixes occasional hangs running dapltest with both the rdma_cm and socket
cm providers.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2329 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
stansmith [Tue, 4 Aug 2009 17:13:27 +0000 (17:13 +0000)]
[DAPL2] Fix socket CM completion handling and CNQ handling; Windows variants. Add #include <winsock2.h>
#include <ws2tcpip.h> for IPv4 & IPv6 user application support (VC env vs. WDK).
shefty [Sat, 1 Aug 2009 01:02:29 +0000 (01:02 +0000)]
netdirect/winverbs: fixes to run nd sample tests
Now that the WV ND provider can be installed, the ND sample tests can be run. This provides various fixes to get at least a couple of the sample tests running.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2324 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
shefty [Sat, 1 Aug 2009 01:01:57 +0000 (01:01 +0000)]
libibverbs: add reference counts to track CQ events
We need to track CQ events to block destroying CQ objects until all outstanding events have been acknowledged. Otherwise, the caller could reference freed memory after calling ibv_get_cq_event.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2323 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
shefty [Sat, 1 Aug 2009 00:51:05 +0000 (00:51 +0000)]
libibverbs: add reference counts to track CQ events
We need to track CQ events to block destroying CQ objects until all outstanding events have been acknowledged. Otherwise, the caller could reference freed memory after calling ibv_get_cq_event.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2322 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
shefty [Sat, 1 Aug 2009 00:45:47 +0000 (00:45 +0000)]
netdirect/winverbs: fixes to run nd sample tests
Now that the WV ND provider can be installed, the ND sample tests can be run. This provides various fixes to get at least a couple of the sample tests running.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2321 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
Running the ndmw test results in a crash in the winverbs driver. The
crash is caused by improper cleanup in the winverbs driver as a result
of a failure trying to allocate a memory window.
In studying the crash, the same situation could arise if other objects
also fail to be created (QPs, CQs, PDs, etc.) To fix the crash and
simplify the code, explicit calls to used to increment the reference
count on referenced objects, rather than simply holding onto the
reference that was acquired when the object was validated. (The latter
is slightly more efficient, but results in less maintainable code
once any fix is applied.)
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2320 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
Running the ndmw test results in a crash in the winverbs driver. The
crash is caused by improper cleanup in the winverbs driver as a result
of a failure trying to allocate a memory window.
In studying the crash, the same situation could arise if other objects
also fail to be created (QPs, CQs, PDs, etc.) To fix the crash and
simplify the code, explicit calls to used to increment the reference
count on referenced objects, rather than simply holding onto the
reference that was acquired when the object was validated. (The latter
is slightly more efficient, but results in less maintainable code
once any fix is applied.)
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2319 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
[ND] remove old, unused fake build folders and fake.c from the days before we could build ND from src. makefile.inc modified to reflect folder changes.
[WINOF] add new HPC example scripts; added new WIX include file to create %ProgramFiles(x86)%\WinOF\HPC install folder and HPC examples scripts; change wof.wxs to include ..\..\HPC\HPC.inc .
DAPL introduced the concept of directly waiting on the CQ for
events by adding a compile time flag and special handling in the
common code. Rather than using the compile time flag and modifying the
common code, let the provider implement the best way to wait for
CQ events.
This simplifies the code and allows the common openib providers to
optimize for Linux and Windows platforms independently, rather than
assuming a specific implementation for signaling events.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2310 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
[DAPL2] Implement a malloc() threshold for the completion reaping byte vector allocation in functions:
DT_handle_send_op, DT_handle_rdma_op & DT_handle_recv_op.
When allocation size is under the threshold, use a stack local allocation instead of malloc/free.
Move redundant bzero() to be called only in the case of using local stack allocation as DT_Mdep_malloc() already does a bzero().
Consolidate error handling return and free()check to a single point by using goto.
Signed-off-by: Stan Smith <stan.smith@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2305 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
[MLX4] added new Registry parameter for number of FC_EXCHs and added a sanity check, that the number of reserved QPs is less then all requested QPs. [mlnx: 4562]
It came to fix a bug: if the number of reserved QPs is greater then the number of QPs requested, the bug causes eventual allocation of reserved QPs, which is forbidden.
The bug happened due two reasons:
- absense of sanity check, comparing reserved and total number of QPs;
- a large built-in number of FC_EXCH QPs, which are not in use at all today.
The patch replaces the built-in value to Registry taken one and set it to O.
It also changes the number of MACs from 1 to 0 for there is no need in reservation for MACs.
[MLX4] 1) added support for sending large multi-parameter messages to System Event Log. [mlnx: 4561]
2) improved the wording of several most popular messages.
To print to Event Log, one has to use the new function WriteEventLogEntryData().
[WinOF] Remove disucssions for Server 2003 & XP w.r.t. selecting HCA device driver to hardware - DIFxApp/PNP simplifies and unifies HCA selection; user selects HCA vendor; Mellanox is the default.
[DAPL2] fix a problem in socket-cm with FD_SETSIZE not being uniformly defined. Add UD test options to dtestcm. Use correct -D flags argument in dt-svr.bat, do not be silent in dt-cli stop as errors were discarded. Use trunk\etc\user\gtod.c for 'gettimeofday()' function.
leonidk [Sun, 28 Jun 2009 08:37:31 +0000 (08:37 +0000)]
[IBAL] remove unnecessary query_qp request during modify_qp
HCA API of modify_qp contains an optional parameter of qp_attr.
If it is coded, modify_qp makes internally a query_qp request to low-level driver and returns QP attributes.
IBAL codes this parameter, but do not use the returned QP attributes.
It causes unnecessary query_qp request to low-level driver and possible racy scenarios.
stansmith [Wed, 24 Jun 2009 19:11:33 +0000 (19:11 +0000)]
[WinOF] Make installs with NODRV=1 work as expected: *.wxs files observe NODRV as does customactions.vbs.
Consolidate standard 'features' into a single include file; XP non-support issues handled via WIX #ifdef's.
shefty [Wed, 24 Jun 2009 17:55:07 +0000 (17:55 +0000)]
etc/dlist: eliminate dlist.c
dlist.c only contains 2 functions. Remove it and implement the
functions inline in the header file. This simplifies the use of
dlist and matches the implementation of linux list.h.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2264 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
stansmith [Mon, 22 Jun 2009 20:49:46 +0000 (20:49 +0000)]
[WinOF] remove DPInst.exe from SVN tree as svn.2261 etc\makebin.bat mods now copy DPInst.exe from WDK tree. Bottom line - having DPInst.exe in our svn tree was a poor choice on my part.
stansmith [Mon, 22 Jun 2009 20:45:17 +0000 (20:45 +0000)]
[WinOF] Use DIFxApp (Driver Install Frameworks for Applications) to install drivers instead of home-grown CustomActions using devman or dpinst.exe. DIFxApp mods resulted in rearranging WIX src files into separate as DIFxApp can not tolerate multiple .inf files in a single directory.
The big win is to use WDK supplied driver install functions which then allows all supported OS flavors to use the Driver Store which results in the user not having to choose which HCA model to install (specifically Server 2003 & XP); The 'default' driver choice is Mellanox HCA.
stansmith [Sat, 20 Jun 2009 23:18:13 +0000 (23:18 +0000)]
[WINOF] Change makebin.bat to expect a full pathname for the DDK root instead of just a DDK version; support those who install WDK in non-standard location.
leonidk [Wed, 17 Jun 2009 13:47:59 +0000 (13:47 +0000)]
[MLX4] return different MLX4 bus locations for several cards in multi-homed machine. [mlnx: 4468]
One can see on multi-homed computers, that mlx4_bus creates all its children (mlx4_hca, IPoIB et al) with built-in location string "MLX4 Bus 0" which complicates differentiation between them.
This patch makes mlx4_bus to return a location string of the structure "MLX4 bus x, device y, function z", where x,y,z is HCA card's PCI bus location.
shefty [Thu, 11 Jun 2009 04:11:24 +0000 (04:11 +0000)]
winverbs: remove QP from EP Disconnect call
The Endpoint class takes a QP parameter for Connect, Accept, and
Disconnect. To simplify the interface, store the QP internally and
remove it from the Disconnect call. Also allow it to be optional
for Accept on the active side of the connection.
Update librdmacm accordingly.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2247 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
shefty [Thu, 11 Jun 2009 04:06:23 +0000 (04:06 +0000)]
winverbs: make PD derive from IWVOverlapped
Memory registration is asynchronous and part of the PD interface.
Make the PD derive from overlapped, rather than unknown base class, and
add support for canceling asynchronous operations (i.e. register
memory) on the PD.
This is an oversight of the initial API caught by implementing the ND
provider.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2246 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
shefty [Thu, 11 Jun 2009 04:02:27 +0000 (04:02 +0000)]
winverbs: convert RDMA address to network-byte order
Change the winverbs interface to accept remote RDMA addresses in network
byte order. This keeps it consistent with all other remote/wire data fields.
Unfortunately, this means that the remote address must be swizzled between
winverbs and the HCA libraries, which take the address in host order, but
the other fields in network order. Longer term, the HCA libraries can provide
optimized interfaces for winverbs, like posting a single work request.
For now, to avoid compatibility issues later, define the winverbs interface
using what's most reasonable. Update libibverbs accordingly.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2245 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
stansmith [Wed, 10 Jun 2009 17:11:17 +0000 (17:11 +0000)]
[DOCS] trunk\docs\Manual.htm
Document new winof dapl providers: socket-cm & rdma-cm
Explain dat.conf dapl provider device specification.
Incorporate BLOCKQUOTES so selected links appear at the top of the window
Document OFED Diagnostic tools - OFED man pages reworked.
Clarify Winverbs transition from technology preview to standard WinOF/IBcore component.
shefty [Tue, 9 Jun 2009 16:40:24 +0000 (16:40 +0000)]
winverbs: process connect and accept asynchronously
Allow processing of EP:Connect and EP:Accept calls asynchronously. The
librdmacm uses events to report the completion of rdma_connect and
rdma_accept calls, which allows users of that interface to take advantage
of asynchronous operation. Modify the winverbs kernel driver to
queue connect/accept calls to a system thread for better parallelism.
This improves the measured connection rate of rdma_cmatose by 3%. The
connection rate includes address resolution, route resolution, PD/CQ/QP
creation and state transitions, memory registration, posting of receive
buffers, and CM message exchanges. This patch effectively only improves
the parallelism of modify QP.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2240 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
shefty [Tue, 9 Jun 2009 16:40:06 +0000 (16:40 +0000)]
winverbs: process connect and accept asynchronously
Allow processing of EP:Connect and EP:Accept calls asynchronously. The
librdmacm uses events to report the completion of rdma_connect and
rdma_accept calls, which allows users of that interface to take advantage
of asynchronous operation. Modify the winverbs kernel driver to
queue connect/accept calls to a system thread for better parallelism.
This improves the measured connection rate of rdma_cmatose by 3%. The
connection rate includes address resolution, route resolution, PD/CQ/QP
creation and state transitions, memory registration, posting of receive
buffers, and CM message exchanges. This patch effectively only improves
the parallelism of modify QP.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2239 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
shefty [Thu, 4 Jun 2009 18:51:56 +0000 (18:51 +0000)]
netdirect: replace 'localhost' with '..localmachine'
Use of 'localhost' does not work with getaddrinfo to return all available IP addresses, but '..localmachine' does. A bug was submitted to MS regarding the issue.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2235 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
projects / ~shefty / rdma-win.git / log