stansmith [Sat, 24 Oct 2009 00:32:28 +0000 (00:32 +0000)]
[INC] add defines and inline functions from OFED management ib_types.h in order to build OpenSM 3.3.2 using only trunk\inc\*. Tested by building & installing a WinOF release using current openSM and newer openSM; no observed differences.
stansmith [Fri, 23 Oct 2009 02:26:00 +0000 (02:26 +0000)]
[WIX] Added explicit Windows Volume (%SystemDrive%\)for DAT config & SDK directories. Required as older installers do not default TARGET dir to Windows Volume.
[WinOF] Not all installers (svr2003/XP) default TARGETDIR == WindowsVolume, explicitly set WindowsVolume so DAT\ is installed where expected '%SystemDrive%\DAT'.
[WinOF] document logic in loading the windows driver store 1st with mlx4_hca driver, then mlx4_bus driver. Story is a race between PNP and MSFT installer, as the mlx4_bus driver if installed 1st will setup for mlx4_hca and PNP will request mlx4_hca driver before the mlx4_hca driver has been installed into the driver store - net result is a failed mlx4_hca load/startup. Install into Driver Store: 1st mlx4_hca, then mlx4_bus.
dapl2
Move close socket calls to the connection thread, to prevent accessing a socket
after it has been closed.
Remove the setsockopt call to mark listen addresses as reusable. A reusable
address allows other libraries, such as MPI, to bind to the same address.
Create all sockets as IPPROTO_TCP, rather than undefined.
Listen on a specific address, rather than any address, to prevent two listens
from occurring on the same port, but using different IP addresses. This
prevents connections from going to the wrong process.
winverbs
Map select winsock errors to winverb error codes. MPI relies on specific error
codes mapped through DAPL, specifically to determine if an address is in use.
Also adds devices to the end of the device list, rather than the beginning,
which helps maintain a more natural order of the devices when matched against
system device lists.
librdmacm
Have librdmacm release all libibverbs resources when the last librdmacm
structure goes away. This allows for graceful cleanup when the libraries are
properly unloaded, which is useful for debugging application that crash or do
not cleanup all resources properly.
winverbs: fix crash accessing freed memory from async thread
If an application exits while asynchronous accept processing is queued,
it's possible for the async processing to access the IbCmId after it has
been freed. A similar problem to this was fixed that dealt with accessing
the verbs QP handle.
A simpler, more generic solution to this problem is to handle application
exit in the same manner as device removal, and lock the winverb provider
lookup lists with exclusive access. Asynchronous operations that are in
process will run to completion, and future operations will be blocked until
the provider cleanup thread has completed. Once they run, they will fail
to acquire a reference on the desired object, which should result in a
graceful failure.
This avoids more complicated locking to use handles belonging to the lower
level code. If a reference on an object can be acquired, the handle will
be available for use until the reference is released. To handle IB CM
callbacks, additional state checking is required to avoid processing
CM events when we're trying to destroy the endpoint.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2462 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
winverbs: fix crash accessing freed memory from async thread
If an application exits while asynchronous accept processing is queued,
it's possible for the async processing to access the IbCmId after it has
been freed. A similar problem to this was fixed that dealt with accessing
the verbs QP handle.
A simpler, more generic solution to this problem is to handle application
exit in the same manner as device removal, and lock the winverb provider
lookup lists with exclusive access. Asynchronous operations that are in
process will run to completion, and future operations will be blocked until
the provider cleanup thread has completed. Once they run, they will fail
to acquire a reference on the desired object, which should result in a
graceful failure.
This avoids more complicated locking to use handles belonging to the lower
level code. If a reference on an object can be acquired, the handle will
be available for use until the reference is released. To handle IB CM
callbacks, additional state checking is required to avoid processing
CM events when we're trying to destroy the endpoint.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2453 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
[DAPL2] wait for async processing thread to actually exit.
DAPL doesn't actually wait for the async processing thread to exit before
allowing the library to close. It will wait up to 10 seconds, which under
heavy load isn't enough time. Since the thread is created by an application
level thread, it will continue to run as long as the application runs. But
if the application closes the library, then all library data and code is
invalid, which can result in the thread running something that's not
library code and accessing freed memory.
With this change, I was able to run mpi ping-pong, 16 ranks on a single
system (scm provider) without crashes 1300 times.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2450 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
[DAPL2] add cleanup/release code for timer thread
dapl_set_timer() creates a thread to process timers for dat_ep_connect
but provides no mechanism to destroy/exit during dapl library unload.
Timers are initialized in library init code and should be released
in the fini code. Add a dapl_timer_release call to the dapl_fini
function to check state of timer thread and destroy before exiting.
Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2449 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
[WinVerbs] fix crash accessing freed memory from async thread
If an application exits while asynchronous accept processing is queued,
it's possible for the async processing to access the IbCmId after it has
been freed. A similar problem to this was fixed that dealt with accessing
the verbs QP handle.
A simpler, more generic solution to this problem is to handle application
exit in the same manner as device removal, and lock the winverb provider
lookup lists with exclusive access. Asynchronous operations that are in
process will run to completion, and future operations will be blocked until
the provider cleanup thread has completed. Once they run, they will fail
to acquire a reference on the desired object, which should result in a
graceful failure.
This avoids more complicated locking to use handles belonging to the lower
level code. If a reference on an object can be acquired, the handle will
be available for use until the reference is released. To handle IB CM
callbacks, additional state checking is required to avoid processing
CM events when we're trying to destroy the endpoint.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2448 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
[WinOF,ETC] When building with latest WDK, turn off OACR (Auto-Code-Review) as it slows down the build process.
Remove extra 'call' when invoking bldwo.bat from buildrelease.bat.
[WinOF] Enable Windows 7 & Svr 2008 R2 build support by making WDK 7600.16385.0 the default build env when using winof\buildrelease.bat to build a release; only affects builds using buildrelease.bat script.
[WinOF] Identify platform processor architecture [32/64 bit] for which the installer is targeted. Use the new include file WinOF\Wix\common\UserInterface.inc to define the WIX UI for all installer variants.
[WinOF] reduce Wix installer (.msi file generation) complexity & redundancy by moving bulk of Makefile to a parameterized WinOF\Wix\common\Makefile.inc
[IBAL] use non-pageable memory to prevent possible problems on power down.
IBAL uses pageable memory to create PnP context. It can create possible problems in power down flows at the time of system contention. We saw a similar case at a customer. There is no strong evidence that this is what influenced, but with this patch IBAL will be more safe and at no cost. WinOF 2.1 testing has demonstrated that with this patch, infrequent (1 out of 10) power-down BSOD have disappeared.
[WinOF] remove redundant 'root' folder specifications as TARGETDIR is the %SysVolume% 'root' (aka 'slash'). Identify 64-bit installers along with installing WinOF into ProgramFiles64 folder which == ProgramFiles; install WinOF in the same folder for all architectures.
[WinOF/DAPL2] reflect WinOF now installs into %ProgramFiles% on all systems, not %ProgramFiles(x86)%.
Move Dat/Dapl 1.1 providers to the end of the list for those MPIs which use the 1st available provider; ends up defaulting to DAT/DAPL V2.0. Thus begins the phasing out of DAT/DAPL 1.1 support.
[WinOF] Streamline driver uninstall and cleanup to play nicely with MSFT PNP; believe PNP will cleanup .inf referenced files when device references have reached zero (shutdown ND & WSD services prior to PNP device removal).
[WinOF] Place librdmacm.dll in %windir% folder along with the rest of the WinOF libs. Review folder usage to consider using system32/syswow64 at a later date.
[WinOF] remove correct WinOF shortcut folder & rename WinOF start menu Command link to 'WinOF Command' so as not to confuse the Svr 2008 start menu auto include entries.
[WinOF] Streamline WinOF uninstall such that is plays nicely with MFST PNP.
1) allow PNP to remove .inf referenced files and cleanup driver store.
2) shutdown ND & WSD prior to PNP device removal.
3) remove stale code which checks for OpenIB installs and forces a reboot
[IBBUS,COMPLIB] Eliminate re-initialization of the stop lock. Crash reported upon running “System Common Scenario” WHQL test with our stack. The crash: C4 (0xd7), which means Driver Verifier revealed a re-initializing of Remove Lock.
Signed-off by Leonid Keller leonid@mellanox.co.il
[IBBUS,COMPLIB] Eliminate re-initialization of the stop lock. Crash reported upon running “System Common Scenario” WHQL test with our stack. The crash: C4 (0xd7), which means Driver Verifier revealed a re-initializing of Remove Lock.
Signed-off by Leonid Keller leonid@mellanox.co.il
To better mimic Linux fd support, add the concept of channel sets to completion channels. This is needed to minimize changes in platform independent uDAPL code.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2412 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
[HW] allow retrieving CA attributes with pageable memory. [winof: 2408]
Modify the HCA drivers to support querying for attributes using a pageable buffer. Since the query calls block, it seems appropriate for the calls to allow pageable memory, rather than forcing the user to allocate a non-paged buffer in order to obtain a list of attributes. The problem stems from the HCA drivers accessing a user's buffer after acquiring a spinlock that raise IRQL.
This fixes kernel crashes with both the winmad and winverbs drivers.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2411 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
If an application calls Connect or Accept, their IRP is queued to a
work queue for asynchronous processing. However, if the application
crashes or exits before the work queue can process the IRP, the cleanup
code will call WvEpFree(). This destroys the IbCmId.
When the work queue finally runs, it can access a freed IbCmId.
This is bad. A similar race exists with the QP and the asynchronous
disconnect processing. The disconnect processing can access a
the hVerbsQp handle after it has been destroyed.
Additionally, in all three cases, the IRPs assume that the WV provider
is able to process IRPs. Specifically, they require that the index
tables maintained by the provider are still valid. References must
be held on the WV provider until the IRPs finish their processing to
ensure this.
Fix invalid accesses to the IbCmId and hVerbsQp handles by locking
around their use after valid state checks. In the case of the QP, we
add a guarded mutex for synchronization purposes and use that in place
where the PD mutex had been used.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2410 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
Winmad returns an incorrect error code when using send or receive in synchronous mode. The OFED MAD code ends up working, since it checks for errors by comparing the return value < 0. In this case, the return value us positive, when it should be zero. Simplify the code and return the correct error code.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2409 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
[HW] allow retrieving CA attributes with pageable memory
Modify the HCA drivers to support querying for attributes using a pageable buffer. Since the query calls block, it seems appropriate for the calls to allow pageable memory, rather than forcing the user to allocate a non-paged buffer in order to obtain a list of attributes. The problem stems from the HCA drivers accessing a user's buffer after acquiring a spinlock that raise IRQL.
This fixes kernel crashes with both the winmad and winverbs drivers.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2408 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
Winmad returns an incorrect error code when using send or receive in synchronous mode. The OFED MAD code ends up working, since it checks for errors by comparing the return value < 0. In this case, the return value us positive, when it should be zero. Simplify the code and return the correct error code.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2407 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
If an application calls Connect or Accept, their IRP is queued to a
work queue for asynchronous processing. However, if the application
crashes or exits before the work queue can process the IRP, the cleanup
code will call WvEpFree(). This destroys the IbCmId.
When the work queue finally runs, it can access a freed IbCmId.
This is bad. A similar race exists with the QP and the asynchronous
disconnect processing. The disconnect processing can access a
the hVerbsQp handle after it has been destroyed.
Additionally, in all three cases, the IRPs assume that the WV provider
is able to process IRPs. Specifically, they require that the index
tables maintained by the provider are still valid. References must
be held on the WV provider until the IRPs finish their processing to
ensure this.
Fix invalid accesses to the IbCmId and hVerbsQp handles by locking
around their use after valid state checks. In the case of the QP, we
add a guarded mutex for synchronization purposes and use that in place
where the PD mutex had been used.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2406 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86
DAPL automatically calls rdma_disconnect() when a disconnect request is
received. If the user also calls disconnect, that calls rdma_disconnect() as
well, but the connection has already been disconnected by DAPL and is no longer
valid. The result is that the user's call to rdma_disconnect() will fail. Do
not display an error message if this occurs.
Locking could be added to prevent calling rdma_disconnect() multiple times, but
since the librdmacm provides synchronization to trap this, we might as well take
advantage of it.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2405 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86