357
How Does the Appliance Remote Replication Method Make Data
Persistent?
The combination of bypassing CPU caches on the target system for the inbound RDMA
Writes to persistent memory with the ordering semantics of the RDMA and PCIe
protocols results in an efficient mechanism to make data persistent. Since the RDMA
Read to persistent memory will force previous writes first to persistent memory and the
persistence domain, the RDMA Read completion that comes back after those writes are
complete is the initiator application’s acknowledgment that those writes are now durable.
Chapter 2 defines the persistence domain in depth, including how the platform
ensures that all writes get to the media from the persistence domain in the event of a
power loss.
Performance Implications of the Appliance Remote Replication
Method
This single extra round trip using an RDMA Read is roughly 50% lower latency than the
general-purpose server persistency method, which requires two round-trip messages
before the writes can be declared durable. As with the first method, as the size of the
writes to be made durable gets smaller, the RDMA Read round-trip overhead becomes a
significant component of the overall latency.
General Software Architecture
The software stack for the use of remote persistent memory typically uses the same
memory-mapped files discussed in Chapter 3. Persistent memory is presented to the
RDMA application as a memory-mapped file. The application registers the persistent
memory with the local NIC on both ends of the connection, and the resulting registry
key is shared with the initiator application for use in the RDMA Read and Write requests.
This is the identical process required for using traditional volatile DRAM with RDMA.
A layering of kernel and application-level software components is typically used to
allow an application to make use of both persistent memory and an RDMA connection.
The IBTA defines verbs interfaces that are typically implemented by the kernel drivers
for the NIC and the middleware software application library. Additional libraries may be
layered above the verbs layer to provide generic RDMA services via a common API- and
NIC-specific provider that implements the library.
Chapter 18 remote persistent memory
How Does the Appliance Remote Replication Method Make Data
Persistent?
The combination of bypassing CPU caches on the target system for the inbound RDMA
Writes to persistent memory with the ordering semantics of the RDMA and PCIe
protocols results in an efficient mechanism to make data persistent. Since the RDMA
Read to persistent memory will force previous writes first to persistent memory and the
persistence domain, the RDMA Read completion that comes back after those writes are
complete is the initiator application’s acknowledgment that those writes are now durable.
Chapter 2 defines the persistence domain in depth, including how the platform
ensures that all writes get to the media from the persistence domain in the event of a
power loss.
Performance Implications of the Appliance Remote Replication
Method
This single extra round trip using an RDMA Read is roughly 50% lower latency than the
general-purpose server persistency method, which requires two round-trip messages
before the writes can be declared durable. As with the first method, as the size of the
writes to be made durable gets smaller, the RDMA Read round-trip overhead becomes a
significant component of the overall latency.
General Software Architecture
The software stack for the use of remote persistent memory typically uses the same
memory-mapped files discussed in Chapter 3. Persistent memory is presented to the
RDMA application as a memory-mapped file. The application registers the persistent
memory with the local NIC on both ends of the connection, and the resulting registry
key is shared with the initiator application for use in the RDMA Read and Write requests.
This is the identical process required for using traditional volatile DRAM with RDMA.
A layering of kernel and application-level software components is typically used to
allow an application to make use of both persistent memory and an RDMA connection.
The IBTA defines verbs interfaces that are typically implemented by the kernel drivers
for the NIC and the middleware software application library. Additional libraries may be
layered above the verbs layer to provide generic RDMA services via a common API- and
NIC-specific provider that implements the library.
Chapter 18 remote persistent memory