About Network Direct SPI
The Network Direct architecture allows hardware vendors to expose the advanced capabilities of their networking devices. The Network Direct SPI is closely mapped to both iWARP and InfiniBand verb interface definitions, but is applicable beyond those technologies.
Depending on the underlying hardware implementation, data transfers can be entirely handled in user-mode, without any kernel transitions. This capability is known as kernel-bypass. Explicit application buffer management allows applications to transfer data directly from one application's buffer to the remote application's buffer without an intermediate operating system copy on either side. This capability is known as zero-copy, or zCopy. These capabilities enable better scalability, efficiency, and performance than is possible through traditional socket interfaces.
Network direct uses endpoints to transfer data between peers. You can use the traditional Send and Receive requests to transfer data or you can use RDMA Read and Write requests. All data transfer requests are message-oriented. The buffers used to transfer data must be registered. RDMA Read and Write requests also require that you define a memory window into your registered memory that tells the peer where it can read or write data.
Send and Receive requests reference local, registered memory buffers. Since not all hardware devices provide buffering capabilities, you must issue a Receive request before the peer issues a corresponding Send request. Failure to have buffers pre-posted results in an I/O error and connection termination. Due to the message oriented nature of the architecture, Receive requests must reference sufficient buffer space to accommodate the largest send anticipated. Any buffer overruns result in an I/O request errors and connection termination.
RDMA Read and Write request reference both a local buffer and a memory window. For Read requests, data is transferred from the memory window on the peer to the local buffer. For Write requests, data is transferred from the local buffer to the memory window on the peer. As with Send and Receive requests, buffer overruns of either the local buffer or the memory window on the peer result in an I/O request error and connection termination. Reading or writing before the beginning or beyond the end of a memory window is considered a buffer overrun.
The Network Direct architecture uses asynchronous I/O transfers. I/O requests are issued on an endpoint and completions are reported through a completion queue. I/O requests can be issued and processed without any kernel transitions. I/O completions can also be detected and processed without any kernel transitions.
After an I/O request is submitted to an endpoint, it can be canceled only by terminating the connection.
The Network Direct interface allows a client application to receive a notification of available completions for a completion queue, and such notification leverages traditional asynchronous I/O mechanisms and incur normal overhead for such operations. However, on a busy system where many I/O requests indicate completion to a single completion queue, only a single request for notification can wake up the thread that processes all completed requests.
Network Direct providers can be entirely thread-less. All notifications from Network Direct providers are explicitly requested by the clients. This approach requires clients to issue a new request after each previous request is complete. While an I/O completion port can be used as a notification mechanism, there is no equivalent to the PostQueuedCompletionStatus function exposed to kernel clients. Since hardware related events are generally processed by the associated kernel driver, the architecture is designed to allow kernel drivers to simply use the IoCompleteRequest function to deliver a notification directly to the client. This makes notification mechanisms IRP based in kernel drivers.
Network Direct providers that rely on kernel components to implement their functionality cross trust boundaries. Network Direct providers that cross trust boundaries are responsible for ensuring the validity of handles and any other parameters passed to their respective kernel components.
Providers are required to guarantee buffer security to prevent a local or remote application from overrunning registered buffers when performing I/O transfers. The buffer overrun checks for data transfers must be performed on a byte-boundary by the hardware.
Send comments about this topic to Microsoft
Build date: 7/2/2010