421
© The Author(s) 2020
S. Scargall, Programming Persistent Memory, https://doi.org/10.1007/978-1-4842-4932-1
APPENDIX E
The Future of Remote
Persistent Memory
Replication
As discussed in Chapter 18, the general purpose and appliance remote persistent
memory methods are simple high-level upper-layer-protocol (ULP) changes. These
methods add a secondary RDMA Send or RDMA Read after a number of RDMA Writes
to remote persistent memory. One of the pain points with these implementations is
the Intel-specific platform feature, allocating writes, which, by default, pushes inbound
PCIe Write data from the NIC directly into the lowest-level CPU cache, speeding the
local software access to that newly written data. For persistent memory, it is desirable
to turn off allocating writes to persistent memory, elevating the need to flush the CPU
cache to guarantee persistence. However, the platform limitations on the control over
allocating writes only imprecise control over the behavior of writes for an entire PCIe
Root complex. All devices connected to a given root complex will behave the same way.
The implications to other software running on the system can be difficult to determine
if access to the write data is delayed by bypassing caches. These are contradictory
requirements since allocating writes should be disabled for writes to persistent memory,
but for writes to volatile memory, allocating writes should be enabled.
To make this per IO steering possible, the networking hardware and software
need to have native support for persistent memory. If the networking stack is aware
of the persistent memory regions, it can select whether the write is steered toward the
persistent memory subsystem or the volatile memory subsystem on a per IO basis,
completely removing the need to change global PCIe Root complex allocating-write
settings.
© The Author(s) 2020
S. Scargall, Programming Persistent Memory, https://doi.org/10.1007/978-1-4842-4932-1
APPENDIX E
The Future of Remote
Persistent Memory
Replication
As discussed in Chapter 18, the general purpose and appliance remote persistent
memory methods are simple high-level upper-layer-protocol (ULP) changes. These
methods add a secondary RDMA Send or RDMA Read after a number of RDMA Writes
to remote persistent memory. One of the pain points with these implementations is
the Intel-specific platform feature, allocating writes, which, by default, pushes inbound
PCIe Write data from the NIC directly into the lowest-level CPU cache, speeding the
local software access to that newly written data. For persistent memory, it is desirable
to turn off allocating writes to persistent memory, elevating the need to flush the CPU
cache to guarantee persistence. However, the platform limitations on the control over
allocating writes only imprecise control over the behavior of writes for an entire PCIe
Root complex. All devices connected to a given root complex will behave the same way.
The implications to other software running on the system can be difficult to determine
if access to the write data is delayed by bypassing caches. These are contradictory
requirements since allocating writes should be disabled for writes to persistent memory,
but for writes to volatile memory, allocating writes should be enabled.
To make this per IO steering possible, the networking hardware and software
need to have native support for persistent memory. If the networking stack is aware
of the persistent memory regions, it can select whether the write is steered toward the
persistent memory subsystem or the volatile memory subsystem on a per IO basis,
completely removing the need to change global PCIe Root complex allocating-write
settings.