libibverbs/comp_mgr: fix hang during destruction

libibverbs/comp_mgr: fix hang during destruction

The completion manager abstraction can hang during cleanup.  The hang
occurs when the user calls CompManagerCancel before calling CompManagerClose,
and the user is not waiting for events.  In this case, the completion manager
thread will pull the cancel request from the IO completion port and queue it
with the manager.  When CompManagerClose is called, it calls CompManagerCancel
to signal the thread to check the running state.  However, the completion
manager's event structure is still marked busy from the user's CompManagerCancel
call.

The result is that the completion manager thread does not receive the
signal to check the running flag and remains asleep.  Fix this by using a
different completion entry to signal the thread during destruction than that
used to cancel a CompManagerPoll event.

This fixes occasional hangs running dapltest with both the rdma_cm and socket
cm providers.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2329 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86