423
InfiniBand Trade Association. Both protocols track each other architecturally and have
essentially added an RDMA Flush and RDMA Atomic Write commands to the existing
volatile memory support.
RDMA Flush – Is a protocol command that flushes a portion of a memory region.
The completion of the flush command indicates that all of the RDMA Writes within the
domain of the flush have made it to the final placement. Flush placement hints allow
the initiator software to request flushing to globally visible memory (could be volatile or
persistent memory regions) and separately whether the memory is volatile or persistent
memory. The scope of the RDMA Write data that is included in the RDMA Flush domain
is driven by the offset and length for the memory region being flushed. All RDMA Writes
covering memory regions contained in the RDMA Flush command shall be included in
the RDMA Flush. That means that the RDMA Flush command will not complete on the
initiator system until all previous remote writes for those regions have reached the final
requested placement location.
RDMA Atomic Write – Is a protocol command that instructs the NIC to write a
pointer update directly into persistent memory in a pipeline efficient manner. This
allows the preceding RDMA Write, RDMA Flush, RDMA Atomic Write, and RDMA
Flush sequence to occur with only one single complete round-trip latency incurred by
software. It simply needs to wait for the final RDMA Flush completion.
Platform hardware changes are required to efficiently make use of the new network
protocol additions for persistent memory support. The placement hints provided in the
RDMA Flush command allows four possible routing combinations:
• Cache Attribute
• No-cache Attribute
• Volatile Destination
• Persistent memory destination
The chipset, CPU, and PCIe root complexes need to understand these placement
attributes and steer or route the request to the proper hardware blocks as requested.
On upcoming Intel platforms, the CPU will look at the PCIe TLP Processor Hint
fields to allow the NIC to add the steering information to each PCIe packet generated
for the inbound RDMA Writes and RDMA Flush. The optional use of this PCIe steering
mechanism is defined by the PCIe Firmware Interface in the ACPI specification and
allows NIC kernel drivers and PCI bus drivers to enable the IO steering and essentially
Appendix e The FuTure oF remoTe persisTenT memory replicATion
InfiniBand Trade Association. Both protocols track each other architecturally and have
essentially added an RDMA Flush and RDMA Atomic Write commands to the existing
volatile memory support.
RDMA Flush – Is a protocol command that flushes a portion of a memory region.
The completion of the flush command indicates that all of the RDMA Writes within the
domain of the flush have made it to the final placement. Flush placement hints allow
the initiator software to request flushing to globally visible memory (could be volatile or
persistent memory regions) and separately whether the memory is volatile or persistent
memory. The scope of the RDMA Write data that is included in the RDMA Flush domain
is driven by the offset and length for the memory region being flushed. All RDMA Writes
covering memory regions contained in the RDMA Flush command shall be included in
the RDMA Flush. That means that the RDMA Flush command will not complete on the
initiator system until all previous remote writes for those regions have reached the final
requested placement location.
RDMA Atomic Write – Is a protocol command that instructs the NIC to write a
pointer update directly into persistent memory in a pipeline efficient manner. This
allows the preceding RDMA Write, RDMA Flush, RDMA Atomic Write, and RDMA
Flush sequence to occur with only one single complete round-trip latency incurred by
software. It simply needs to wait for the final RDMA Flush completion.
Platform hardware changes are required to efficiently make use of the new network
protocol additions for persistent memory support. The placement hints provided in the
RDMA Flush command allows four possible routing combinations:
• Cache Attribute
• No-cache Attribute
• Volatile Destination
• Persistent memory destination
The chipset, CPU, and PCIe root complexes need to understand these placement
attributes and steer or route the request to the proper hardware blocks as requested.
On upcoming Intel platforms, the CPU will look at the PCIe TLP Processor Hint
fields to allow the NIC to add the steering information to each PCIe packet generated
for the inbound RDMA Writes and RDMA Flush. The optional use of this PCIe steering
mechanism is defined by the PCIe Firmware Interface in the ACPI specification and
allows NIC kernel drivers and PCI bus drivers to enable the IO steering and essentially
Appendix e The FuTure oF remoTe persisTenT memory replicATion