359
libpmemobj uses a synchronous write model, meaning that the local initiator write
and all of the remotely replicated writes must complete before the local write will be
completed back to the application. The libpmemobj library also implements a simple
active-passive replication architecture, where all persistent memory transactions are
driven through the active initiator node and the remote targets passively standby,
replicating the write data. While the passive target systems have the latest write data
replicated, the implementation makes no attempt to fail over, fail back, or load balance
using the remote systems. The following sections describe the significant performance
drawbacks to this implementation.
libpmemobj uses the local memory pool configuration information provided in a
configuration file to describe the remote network–connected memory-mapped files.
A remote rpmemd program installed on each remote target system is started and
connected to the librpmem library on the initiator using a secure encrypted socket
connection. Through this connection, librpmem, on behalf of libpmemobj, will set up the
RDMA point-to-point connection with each target system, determine the persistence
method the target supports (general purpose or appliance method), allocate remote
memory-mapped persistent memory files, register the persistent memory on the remote
NIC, and retrieve the resulting memory keys for the registered memory.
Once all the RDMA connections to all the targets are established, all required
queues are instantiated, and memory buffers have all been allocated and registered,
the libpmemobj library is ready to begin remotely replicating all application write
data to its local memory-mapped file. When the application calls pmemobj_persist()
in libpmemobj, the library will generate a corresponding rpmem_persist() call
into librpmem which, in turn, calls the libfabric fi_write() to do the RDMA Write.
librpmem then initiates the RDMA Read or Send persistence method (as governed
by an understanding of the currently enabled target node’s current configuration) by
calling libfabric fi_read() or fi_send(). RDMA Read is used in the appliance remote
replication method, and RDMA Send is used in the general-purpose remote replication
method.
Figure 18-5 outlines the high-level components and interfaces described earlier and
used by both the initiator and remote target system using librpmem and libpmemobj.
Chapter 18 remote persistent memory
libpmemobj uses a synchronous write model, meaning that the local initiator write
and all of the remotely replicated writes must complete before the local write will be
completed back to the application. The libpmemobj library also implements a simple
active-passive replication architecture, where all persistent memory transactions are
driven through the active initiator node and the remote targets passively standby,
replicating the write data. While the passive target systems have the latest write data
replicated, the implementation makes no attempt to fail over, fail back, or load balance
using the remote systems. The following sections describe the significant performance
drawbacks to this implementation.
libpmemobj uses the local memory pool configuration information provided in a
configuration file to describe the remote network–connected memory-mapped files.
A remote rpmemd program installed on each remote target system is started and
connected to the librpmem library on the initiator using a secure encrypted socket
connection. Through this connection, librpmem, on behalf of libpmemobj, will set up the
RDMA point-to-point connection with each target system, determine the persistence
method the target supports (general purpose or appliance method), allocate remote
memory-mapped persistent memory files, register the persistent memory on the remote
NIC, and retrieve the resulting memory keys for the registered memory.
Once all the RDMA connections to all the targets are established, all required
queues are instantiated, and memory buffers have all been allocated and registered,
the libpmemobj library is ready to begin remotely replicating all application write
data to its local memory-mapped file. When the application calls pmemobj_persist()
in libpmemobj, the library will generate a corresponding rpmem_persist() call
into librpmem which, in turn, calls the libfabric fi_write() to do the RDMA Write.
librpmem then initiates the RDMA Read or Send persistence method (as governed
by an understanding of the currently enabled target node’s current configuration) by
calling libfabric fi_read() or fi_send(). RDMA Read is used in the appliance remote
replication method, and RDMA Send is used in the general-purpose remote replication
method.
Figure 18-5 outlines the high-level components and interfaces described earlier and
used by both the initiator and remote target system using librpmem and libpmemobj.
Chapter 18 remote persistent memory