247
Out-of-Order Writes
When developing software for persistent memory, remember that even if a cache line is
not explicitly flushed, that does not mean the data is still in the CPU caches. For example,
the CPU could have evicted it due to cache pressure or other reasons. Furthermore, the
same way that writes that are not flushed properly may produce bugs in the event of an
unexpected application crash, so do automatically evicted dirty cache lines if they violate
some expected order of writes that the applications rely on.
To better understand this problem, explore how flushing works in the x86_64
and AMD64 architectures. From the user space, we can issue any of the following
instructions to ensure our writes reach the persistent media:
• CLFLUSH
• CLFLUSHOPT (needs SFENCE)
• CLWB (needs SFENCE)
• Non-temporal stores (needs SFENCE)
The only instruction that ensures each flush is issued in order is CLFUSH because
each CLFLUSH instruction always does an implicit fence instruction (SFENCE). The other
instructions are asynchronous and can be issued in parallel and in any order. The CPU
can only guarantee that all flushes issued since the previous SFENCE have completed
when a new SFENCE instruction is explicitly executed. Think of SFENCE instructions as
synchronization points (see Figure 12-6). For more information about these instructions,
refer to the Intel software developer manuals and the AMD software developer manuals.
Chapter 12 Debugging persistent MeMory appliCations
Out-of-Order Writes
When developing software for persistent memory, remember that even if a cache line is
not explicitly flushed, that does not mean the data is still in the CPU caches. For example,
the CPU could have evicted it due to cache pressure or other reasons. Furthermore, the
same way that writes that are not flushed properly may produce bugs in the event of an
unexpected application crash, so do automatically evicted dirty cache lines if they violate
some expected order of writes that the applications rely on.
To better understand this problem, explore how flushing works in the x86_64
and AMD64 architectures. From the user space, we can issue any of the following
instructions to ensure our writes reach the persistent media:
• CLFLUSH
• CLFLUSHOPT (needs SFENCE)
• CLWB (needs SFENCE)
• Non-temporal stores (needs SFENCE)
The only instruction that ensures each flush is issued in order is CLFUSH because
each CLFLUSH instruction always does an implicit fence instruction (SFENCE). The other
instructions are asynchronous and can be issued in parallel and in any order. The CPU
can only guarantee that all flushes issued since the previous SFENCE have completed
when a new SFENCE instruction is explicitly executed. Think of SFENCE instructions as
synchronization points (see Figure 12-6). For more information about these instructions,
refer to the Intel software developer manuals and the AMD software developer manuals.
Chapter 12 Debugging persistent MeMory appliCations