370
00001020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 |................|
*
00002000
Summary
It is important to know that neither the general-purpose remote replication method
nor the appliance remote replication method is ideal because vendor-specific platform
features are required to use non-allocating writes, adding the complication of effecting
performance on an entire PCI root complex. Conversely, flushing remote writes using
allocating writes requires a painful interrupt of the target system to intercept an RDMA
Send request and flush the list of regions contained within the send buffer. Waking the
remote node is extremely painful in a cloud environment because there are hundreds
or thousands of inbound RDMA requests from many different connections; avoid this if
possible.
There are cloud service providers using these two methods today and getting
phenomenal performance results. If the persistent memory is used as a replacement for
a remotely accessed SSD, huge reductions in latency can be achieved.
As the first iteration of remote persistence support, we focused on application/
library changes to implement these high-level persistence methods, without hardware,
firmware, driver, or protocol changes. At the time of publication, IBTA and IETF drafts
for a new wire protocol extension for persistent memory is nearing completion. This will
provide native hardware support for RDMA to persistent memory and allow hardware
entities to route each I/ O to its destination memory device without the need to change
allocating write mode and without the potential to adversely affect performance on
collateral devices connected to the same root port. See Appendix E for more details on
the new extensions to RDMA, specifically for remote persistence.
RDMA protocol extensions are only one step into further remote persistent memory
development. Several other areas of improvement are already identified and shall be
addressed to the remote persistent memory users community, including atomicity of
remote operations, advanced error handling (including RAS), dynamic configuration of
remote persistent memory and custom setup, and real 0% CPU utilization on remote/
target replication side.
Chapter 18 remote persistent memory
00001020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 |................|
*
00002000
Summary
It is important to know that neither the general-purpose remote replication method
nor the appliance remote replication method is ideal because vendor-specific platform
features are required to use non-allocating writes, adding the complication of effecting
performance on an entire PCI root complex. Conversely, flushing remote writes using
allocating writes requires a painful interrupt of the target system to intercept an RDMA
Send request and flush the list of regions contained within the send buffer. Waking the
remote node is extremely painful in a cloud environment because there are hundreds
or thousands of inbound RDMA requests from many different connections; avoid this if
possible.
There are cloud service providers using these two methods today and getting
phenomenal performance results. If the persistent memory is used as a replacement for
a remotely accessed SSD, huge reductions in latency can be achieved.
As the first iteration of remote persistence support, we focused on application/
library changes to implement these high-level persistence methods, without hardware,
firmware, driver, or protocol changes. At the time of publication, IBTA and IETF drafts
for a new wire protocol extension for persistent memory is nearing completion. This will
provide native hardware support for RDMA to persistent memory and allow hardware
entities to route each I/ O to its destination memory device without the need to change
allocating write mode and without the potential to adversely affect performance on
collateral devices connected to the same root port. See Appendix E for more details on
the new extensions to RDMA, specifically for remote persistence.
RDMA protocol extensions are only one step into further remote persistent memory
development. Several other areas of improvement are already identified and shall be
addressed to the remote persistent memory users community, including atomicity of
remote operations, advanced error handling (including RAS), dynamic configuration of
remote persistent memory and custom setup, and real 0% CPU utilization on remote/
target replication side.
Chapter 18 remote persistent memory