Author: Sean Hefty <sean.hefty@intel.com>
Date: 2012-10-21 14:16:03 -0700
-rsocket: Add direct data placement
+rsocket: Add APIs for direct data placement
+
+We introduce rsocket extensions for supporting direct
+data placement (also known as zero copy). Direct data
+placement avoids data copies into network buffers when
+sending or receiving data. This patch implements zero
+copies on the receive side, but adds some basic framework for
+supporting it on the sending side.
+
+Integrating zero copy support into the existing socket APIs
+is difficult to achieve when the sockets are set as
+nonblocking. Any such implementation is likely to be unusable
+in practice. The problem stems from the fact that socket
+operations are synchronous in nature. Support for asynchronous
+operations is limited to connection establishment.
+
+Therefore we introduce new calls to handle direct data placement.
+The use of the new calls is optional and does not affect the
+use of the existing calls. An attempt is made to have the new
+routines integrate naturally with the existing APIs. The new
+functions are: riomap, riounmap, and riowrite. The basic operation
+can be described as follows:
+
+1. App A calls riomap to register a data buffer with the local
+ RDMA device. Riomap returns an off_t offset value that
+ corresponds to the registered data buffer. The app may
+ select the offset value.
+2. Rsockets will transmit an internal message to the remote
+ peer with information about the registration. This exchange
+ is hidden from the applications.
+3. App A sends a notification message to app B indicating that
+ the remote iomapped buffer is now available to receive data.
+4. App B calls riowrite to transmit data directly into the
+ riomapped data buffer.
+5. App B sends a notification message to app A indicating that
+ data is available in the mapped buffer.
+6. After all transfers are complete, app A calls riounmap to
+ deregister its data buffer.
+
+Riomap and riounmap are functionally equivalent to RDMA
+memory registration and deregistration routines. They are loosely
+based on the mmap and munmap APIs.
+
+off_t riomap(int socket, void *buf, size_t len,
+ int prot, int flags, off_t offset)
+
+Riomap registers an application buffer with the RDMA hardware
+associated with an rsocket. The buffer is registered either for
+local only access (PROT_NONE) or for remote write access (PROT_WRITE).
+When registered for remote access, the buffer is mapped to a given
+offset. The offset is either provided by the user, or if the user
+selects -1 for the offset, rsockets selects one. The remote peer may
+access an iomapped buffer directly by specifying the correct offset.
+The mapping is not guaranteed to be available until after the remote
+peer receives a data transfer initiated after riomap has completed.
+
+int riounmap(int socket, void *buf, size_t len)
+
+Riounmap removes the mapping between a buffer and an rsocket.
+
+size_t riowrite(int socket, const void *buf, size_t count,
+ off_t offset, int flags)
+
+Riowrite allows an application to transfer data over an rsocket
+directly into a remotely iomapped buffer. The remote buffer is specified
+through an offset parameter, which corresponds to a remote iomapped buffer.
+From the sender's perspective, riowrite behaves similar to rwrite. From
+a receiver's view, riowrite transfers are silently redirected into a pre-
+determined data buffer. Data is received automatically, and the receiver
+is not informed of the transfer. However, iowrite data is still considered
+part of the data stream, such that iowrite data will be written before a
+subsequent transfer is received. A message sent immediately after
+initiating an iowrite may be used to notify the receiver of the iowrite.
+
+It should be noted that the current implementation primarily focused
+on being functional for evaluation purposes. Some checks have been
+deferred for subsequent patches, and performance is currently limited
+by linear lookups.
Signed-off-by: Sean Hefty <sean.hefty@intel.com>
projects / ~shefty / librdmacm.git / commitdiff