350
Three basic RDMA commands are used by most RDMA-capable applications and
libraries:
RDMA Write: A one-sided operation where only the initiator supplies all of the
information required for the transfer to occur. This transfer is used to write data to the
remote target node. The write request contains all source and sink buffer information.
The remote target system is not typically interrupted and thus completely unaware of
the write operations occurring through the NIC. When the initiator’s NIC sends a write
to the target, it will generate a “software write completion interrupt.” A software write
completion interrupt means that the write message has been sent to the target NIC
and is not an indicator of the write completion. Optionally, RDMA Writes can use an
immediate option that will interrupt the target node CPU and allow software running
there to be immediately notified of the write completion.
RDMA Read: A one-sided operation where only the initiator supplies all of the
information required for the transfer to occur. This transfer is used to read data from
the remote target node. The read request contains all source buffer and target sink
buffer information, and the remote target system is not typically interrupted and thus
completely unaware of the read operations occurring through the NIC. The initiator
software read completion interrupt is an acknowledgment that the read has traversed
all the way through the initiator’s NIC, over the network, into the target system’s NIC,
through the target internal hardware mesh and memory controllers, to the DRAM or
persistent memory to retrieve the data. Then it returns all the way back to the initiator
software that registered for the completion notification.
RDMA Send (and Receive): The two-sided RDMA Send means that both the
initiator and target must supply information for the transfer to complete. This is because
the target NIC will be interrupted when the RDMA Send is received by the target NIC
and requires a hardware receive queue to be set up and pre-populated with completion
entries before the NIC will receive an RDMA Send transfer operation. Data from the
initiator application is bundled in a small, limited sized buffer and sent to the target
NIC. The target CPU will be interrupted to handle the send operation and any data it
contains. If the initiator needs to be notified of receipt of the RDMA Send, or to handle a
message back to the initiator, another RDMA Send operation must be sent in the reverse
direction after the initiator has set up its own receive queue and queued completion
entries to it. The use of the RDMA Send command and the contents of the payload
are application-specific implementation details. An RDMA Send is typically used for
bookkeeping and updates of read and write activity between the initiator and the target,
Chapter 18 remote persistent memory
Three basic RDMA commands are used by most RDMA-capable applications and
libraries:
RDMA Write: A one-sided operation where only the initiator supplies all of the
information required for the transfer to occur. This transfer is used to write data to the
remote target node. The write request contains all source and sink buffer information.
The remote target system is not typically interrupted and thus completely unaware of
the write operations occurring through the NIC. When the initiator’s NIC sends a write
to the target, it will generate a “software write completion interrupt.” A software write
completion interrupt means that the write message has been sent to the target NIC
and is not an indicator of the write completion. Optionally, RDMA Writes can use an
immediate option that will interrupt the target node CPU and allow software running
there to be immediately notified of the write completion.
RDMA Read: A one-sided operation where only the initiator supplies all of the
information required for the transfer to occur. This transfer is used to read data from
the remote target node. The read request contains all source buffer and target sink
buffer information, and the remote target system is not typically interrupted and thus
completely unaware of the read operations occurring through the NIC. The initiator
software read completion interrupt is an acknowledgment that the read has traversed
all the way through the initiator’s NIC, over the network, into the target system’s NIC,
through the target internal hardware mesh and memory controllers, to the DRAM or
persistent memory to retrieve the data. Then it returns all the way back to the initiator
software that registered for the completion notification.
RDMA Send (and Receive): The two-sided RDMA Send means that both the
initiator and target must supply information for the transfer to complete. This is because
the target NIC will be interrupted when the RDMA Send is received by the target NIC
and requires a hardware receive queue to be set up and pre-populated with completion
entries before the NIC will receive an RDMA Send transfer operation. Data from the
initiator application is bundled in a small, limited sized buffer and sent to the target
NIC. The target CPU will be interrupted to handle the send operation and any data it
contains. If the initiator needs to be notified of receipt of the RDMA Send, or to handle a
message back to the initiator, another RDMA Send operation must be sent in the reverse
direction after the initiator has set up its own receive queue and queued completion
entries to it. The use of the RDMA Send command and the contents of the payload
are application-specific implementation details. An RDMA Send is typically used for
bookkeeping and updates of read and write activity between the initiator and the target,
Chapter 18 remote persistent memory