rsocket: Add APIs for direct data placement

rsocket: Add APIs for direct data placement

We introduce rsocket extensions for supporting direct
data placement (also known as zero copy).  Direct data
placement avoids data copies into network buffers when
sending or receiving data.  This patch implements zero
copies on the receive side, but adds some basic framework for
supporting it on the sending side.

Integrating zero copy support into the existing socket APIs
is difficult to achieve when the sockets are set as
nonblocking.  Any such implementation is likely to be unusable
in practice.  The problem stems from the fact that socket
operations are synchronous in nature.  Support for asynchronous
operations is limited to connection establishment.

Therefore we introduce new calls to handle direct data placement.
The use of the new calls is optional and does not affect the
use of the existing calls.  An attempt is made to have the new
routines integrate naturally with the existing APIs.  The new
functions are: riomap, riounmap, and riowrite.  The basic operation
can be described as follows:

1. App A calls riomap to register a data buffer with the local
   RDMA device.  Riomap returns an off_t offset value that
   corresponds to the registered data buffer.  The app may
   select the offset value.
2. Rsockets will transmit an internal message to the remote
   peer with information about the registration.  This exchange
   is hidden from the applications.
3. App A sends a notification message to app B indicating that
   the remote iomapped buffer is now available to receive data.
4. App B calls riowrite to transmit data directly into the
   riomapped data buffer.
5. App B sends a notification message to app A indicating that
   data is available in the mapped buffer.
6. After all transfers are complete, app A calls riounmap to
   deregister its data buffer.

Riomap and riounmap are functionally equivalent to RDMA
memory registration and deregistration routines.  They are loosely
based on the mmap and munmap APIs.

off_t riomap(int socket, void *buf, size_t len,
	     int prot, int flags, off_t offset)

Riomap registers an application buffer with the RDMA hardware
associated with an rsocket.  The buffer is registered either for
local only access (PROT_NONE) or for remote write access (PROT_WRITE).
When registered for remote access, the buffer is mapped to a given
offset.  The offset is either provided by the user, or if the user
selects -1 for the offset, rsockets selects one.  The remote peer may
access an iomapped buffer directly by specifying the correct offset.
The mapping is not guaranteed to be available until after the remote
peer receives a data transfer initiated after riomap has completed.

int riounmap(int socket, void *buf, size_t len)

Riounmap removes the mapping between a buffer and an rsocket.

size_t riowrite(int socket, const void *buf, size_t count,
		off_t offset, int flags)

Riowrite allows an application to transfer data over an rsocket
directly into a remotely iomapped buffer.  The remote buffer is specified
through an offset parameter, which corresponds to a remote iomapped buffer.
From the sender's perspective, riowrite behaves similar to rwrite.  From
a receiver's view, riowrite transfers are silently redirected into a pre-
determined data buffer.  Data is received automatically, and the receiver
is not informed of the transfer.  However, iowrite data is still considered
part of the data stream, such that iowrite data will be written before a
subsequent transfer is received.  A message sent immediately after
initiating an iowrite may be used to notify the receiver of the iowrite.

It should be noted that the current implementation primarily focused
on being functional for evaluation purposes.  Some checks have been
deferred for subsequent patches, and performance is currently limited
by linear lookups.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>