rsockets: Optimize synchronization to improve performance
Performance analysis using VTune showed that pthread_mutex_unlock()
is the single biggest contributor to increasing latency for 64-byte
transfers. Unlocked was followed by get_sw_cqe(), then
__pthread_mutex_lock(). Replace the use of mutexes with an atomic
and a semaphore. When there's no contention for the lock (which
would usually be the case when using nonblocking sockets), the
code simply increments and decrements an atomic varible. Semaphores
are only used when contention occurs.