326
memory. Additionally, the deallocation path often returns the memory block to the
thread cache from which it originated, again preserving locality.
We mentioned earlier that volatile allocators manage operating system–provided
pages but did not explain how they acquire those pages. This will become very important
later as we discuss how things change for persistent memory. Memory is usually
requested on demand from the operating system either through sbrk(), which moves
the break segment of the application, or anonymous mmap(), which creates new virtual
memory mapping backed by the page cache. The actual physical memory is usually not
assigned until the page is written to for the first time. When the allocator decides that it
no longer needs a page, it can either completely remove the mapping using unmap() or
it can tell the operating system to release the backing physical pages but keep the virtual
mapping. This enables the allocator to reuse the same addresses later without having to
memory map them again.
How does all of this translate into persistent memory allocators and libpmemobj
specifically?
The persistent heap must be resumable after application restart. This means that
all state information must be either located on persistent memory or reconstructed on
startup. If there are any active bookkeeping processes, those need to be restarted from
the point at which they were interrupted. There cannot be any volatile state held in
persistent memory, such as thread cache pointers. In fact, the allocator must not operate
on any pointers at all because the virtual address of the heap can change between
restarts.
In libpmemobj, the heap is rebuilt lazily and in stages. The entire available memory
is divided into equally sized zones (except for the last one, which can be smaller than
the others) with metadata at the beginning of each one. Each zone is subsequentially
divided into variably sized memory blocks called chunks. Whenever there is an
allocation request, and the runtime state indicates that there is no memory to satisfy it,
the zone’s metadata is processed, and the corresponding runtime state is initialized.
This minimizes the startup time of the pool and amortizes the cost of rebuilding the
heap state across many individual allocation requests.
There are three main reasons for having any runtime state at all. First, access
latency of persistent memory can be higher than that of DRAM, potentially impacting
performance of data structures placed on it. Second, separating the runtime state from
the persistent state enables a workflow where the memory is first reserved in runtime
state and initialized, and only then the allocation is reflected on the persistent state.
Chapter 16 pMDK Internals: IMportant algorIthMs anD Data struCtures
memory. Additionally, the deallocation path often returns the memory block to the
thread cache from which it originated, again preserving locality.
We mentioned earlier that volatile allocators manage operating system–provided
pages but did not explain how they acquire those pages. This will become very important
later as we discuss how things change for persistent memory. Memory is usually
requested on demand from the operating system either through sbrk(), which moves
the break segment of the application, or anonymous mmap(), which creates new virtual
memory mapping backed by the page cache. The actual physical memory is usually not
assigned until the page is written to for the first time. When the allocator decides that it
no longer needs a page, it can either completely remove the mapping using unmap() or
it can tell the operating system to release the backing physical pages but keep the virtual
mapping. This enables the allocator to reuse the same addresses later without having to
memory map them again.
How does all of this translate into persistent memory allocators and libpmemobj
specifically?
The persistent heap must be resumable after application restart. This means that
all state information must be either located on persistent memory or reconstructed on
startup. If there are any active bookkeeping processes, those need to be restarted from
the point at which they were interrupted. There cannot be any volatile state held in
persistent memory, such as thread cache pointers. In fact, the allocator must not operate
on any pointers at all because the virtual address of the heap can change between
restarts.
In libpmemobj, the heap is rebuilt lazily and in stages. The entire available memory
is divided into equally sized zones (except for the last one, which can be smaller than
the others) with metadata at the beginning of each one. Each zone is subsequentially
divided into variably sized memory blocks called chunks. Whenever there is an
allocation request, and the runtime state indicates that there is no memory to satisfy it,
the zone’s metadata is processed, and the corresponding runtime state is initialized.
This minimizes the startup time of the pool and amortizes the cost of rebuilding the
heap state across many individual allocation requests.
There are three main reasons for having any runtime state at all. First, access
latency of persistent memory can be higher than that of DRAM, potentially impacting
performance of data structures placed on it. Second, separating the runtime state from
the persistent state enables a workflow where the memory is first reserved in runtime
state and initialized, and only then the allocation is reflected on the persistent state.
Chapter 16 pMDK Internals: IMportant algorIthMs anD Data struCtures