Efficient Remote Memory Ordering for Non-Coherent Systems

Wei Siew Liew, Md Ashfaqur Rahaman, Adarsh Patil, Ryan Stutsman and Vijay Nagarajan

When a CPU talks to a network card over PCIe, some operations must happen in a specific order (send packet A before packet B; check a lock before reading the object it guards), but PCIe has no way to say "keep these in order," so today software forces order by stalling at the source — a fence after every write, or a full round-trip wait before each read — which cripples throughput; this paper (University of Utah and Arm — Wei Siew Liew, Md Ashfaqur Rahaman, Ryan Stutsman, Vijay Nagarajan, and colleagues) moves the ordering job to hardware at the destination, letting operations fly out concurrently while a queue in the PCIe Root Complex enforces the order, reaching line-rate transmit and up to 50.9 times faster ordered reads.

The argument in one glance

Problem

PCIe cannot say "keep these in order," so software stalls to force it

Modern servers connect their CPUs to devices like network cards (NICs) and GPUs over non-coherent interconnects, most commonly PCI Express (PCIe). "Non-coherent" means the interconnect does not automatically keep everyone's view of shared memory consistent the way a CPU's caches do; software has to manage that itself. Many communication patterns need certain remote operations to happen in a specific order. When a CPU hands packets to a NIC, packet A must arrive before packet B (a write-then-write ordering). When a network card looks up a value in a remote key-value store over RDMA, it must first check a lock and only then read the object the lock guards (a read-then-read ordering) — reading in the wrong order could return stale or corrupt data. The problem is that PCIe has no way to express "these two operations must stay ordered." Its rules give ordering to some cases (writes stay ordered relative to writes) but not to reads relative to reads. So to get the fine-grained ordering software needs, systems fall back on serializing at the source: the sender does one operation, waits until it is fully done, and only then starts the next. For writes that means a store fence after every write; for reads it means a stop-and-wait for a full round-trip across the interconnect before issuing the next read. This source-side serialization is brutally slow, and it is baked into how CPUs and NICs talk today.
What people did

Serialize at the source with fences, doorbells, and stop-and-wait

The standard fixes all serialize at the sending side. For CPU-to-NIC transmit, software puts a store fence after each memory-mapped write so the writes cannot be reordered, or it avoids direct writes altogether with elaborate workarounds — writing a "doorbell" register that tells the NIC to come fetch the data itself with a DMA read. For ordered remote reads, the NIC issues one read, waits for the whole round-trip completion, and only then issues the dependent read.
Why it fell short

Source-side stalls throttle both transmit and remote reads

Serializing at the source is exactly what kills performance, because each stall pays the full latency of the interconnect before anything else can proceed. On the transmit path, a store fence after every packet adds about 100 nanoseconds per packet, which caps 64-byte packets at roughly 5 gigabits per second — a small fraction of a modern 100-gigabit link — so a direct write-based transmit path is impractical and everyone resorts to the complex doorbell-and-DMA workarounds. On the remote-read path, the stop-and-wait forces the NIC to eat the full round-trip latency (about 500 nanoseconds) for every ordered read, which makes ordered remote reads more than ten times slower than unordered ones and caps throughput at only about 2 million operations per second. Worse, even if the interconnect preserved order, the host's own memory system can complete parallel requests out of order — a read that hits in the cache can pass a read that misses — so ordering has to be enforced against host memory too, not just across the wire. The root cause is a mismatch: the consistency model software needs is finer-grained than anything PCIe can express or enforce, so software is forced into the one blunt tool it has, source-side serialization. What was missing was a way for software to state its ordering intent precisely and have hardware enforce it without stalling the source.
This paper's idea

Express ordering explicitly, and enforce it at the destination

The paper proposes a destination-based ordering model with a hardware-software co-design. The key move is to make memory ordering an explicit, first-class thing all the way from the instruction set down to the interconnect, and then shift the job of enforcing it from the source to hardware near the destination. Concretely, it adds acquire and release semantics to the PCIe specification — a bit in the transaction header that marks a read or write as ordered, distinguishing ordered from unordered reads the way PCIe already distinguishes ordered from unordered writes — and it adds acquire/release memory-mapped load and store instructions to the host ISA, so the CPU can signal precisely which operations must stay ordered, and tag write streams with sequence numbers. Because ordering is now expressed rather than implied, the sender no longer has to stall: it fires operations off concurrently. The enforcement moves into the PCIe Root Complex, the hub between the CPU and devices, via a new Remote Load-Store Queue (RLSQ) and a reorder buffer. For writes, the reorder buffer uses the sequence numbers to reconstruct the correct order before forwarding to the NIC, so no fence is needed; for reads, the RLSQ pipelines the requests and enforces the expressed order against the host's coherent memory, and an advanced version borrows speculative ordering from out-of-order processors to remove almost all remaining serialization. The result: a simple, fence-free transmit path that hits line rate (100 gigabits per second on a single core), near-zero-penalty ordered DMA reads, and an RDMA key-value store that is simpler than the state of the art yet up to 50.9 times faster for small objects and 1.6 times faster than FaRM.

Why this problem is worth solving

The interconnect between CPUs and devices is now a primary server bottleneck, and remote ordering sits on the critical path of the most important I/O patterns.

Modern servers are increasingly limited not by their CPUs but by the interconnects that link those CPUs to network cards and GPUs, and PCIe is the interconnect that carries almost all of today’s I/O. Two of the most performance-critical things a server does run straight through this bottleneck. Sending network packets from the CPU to the NIC is the heart of every networked service, and it needs writes to stay ordered. Looking up values in a remote key-value store over RDMA — a building block of distributed storage, caching, and databases — needs reads to stay ordered so that a lock is checked before the data it protects is read. When the only way to get that ordering is to stall the sender, these operations run far below the hardware’s line rate, wasting the very interconnect bandwidth that servers are starved for. So fixing remote ordering is not a niche concern; it directly determines how fast a server can move packets and serve remote data, which are among the most common and valuable things servers do.

The reason it is hard is that the problem is a layering mismatch: software reasons in a fine-grained consistency model the interconnect simply cannot speak.

Programmers reason about memory using consistency models with precise notions of what must happen before what — the same acquire/release vocabulary used within a multicore CPU. But PCIe was not built to express that vocabulary for remote operations. Its ordering rules are coarse and fixed: writes are kept in order relative to other writes, but reads are not ordered relative to other reads, and there is no bit a device can set to say “this read must not be reordered past that one.” Because the interconnect cannot represent the ordering software needs, software has exactly one fallback: do the operations one at a time, waiting for each to fully complete before the next, which forces the full interconnect latency onto the critical path of every ordered operation. And the mismatch runs deeper than the wire, because even a perfectly ordered interconnect would not be enough — the host’s memory system can itself reorder parallel accesses (a cache hit overtaking a cache miss), so ordering must be enforced where the requests actually touch host memory. Closing this gap means giving software a way to say precisely what it needs and giving hardware a place to enforce it cheaply, rather than leaving software to hammer everything flat with serialization. That is the design problem the paper takes on.

What people did before, and where it broke

Fence after every memory-mapped write. Put a store fence between writes so they cannot be reordered on the transmit path. Where it broke. The fence stalls the CPU about 100 nanoseconds per packet, capping small-packet transmit at roughly 5 gigabits per second, far below a 100-gigabit link.
Use doorbells and DMA reads instead of direct writes. Write a doorbell register so the NIC fetches the data itself with a DMA read, sidestepping ordered MMIO writes. Where it broke. This is a complex workaround that adds latency and machinery precisely because a direct, ordered write path is impractical today.
Stop-and-wait for ordered remote reads. Have the NIC issue one read, wait for the full round-trip, then issue the dependent read. Where it broke. Paying the full interconnect round-trip (about 500 nanoseconds) per read makes ordered reads more than ten times slower than unordered ones, capping throughput near 2 million operations per second.
Rely on PCIe’s built-in ordering. Hope the interconnect’s existing rules suffice. Where it broke. PCIe orders writes relative to writes but not reads relative to reads, and cannot express fine-grained intent, so the common read-then-read pattern gets no help.
The gap. No mechanism let software express fine-grained remote ordering and have hardware enforce it near the destination, so systems were stuck serializing at the source.

How the design works

Turning source-side stalls into destination-side enforcement

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Make memory ordering an explicit, first-class thing across the whole stack. The design’s foundation is to lift ordering out of hidden, implied rules and give it precise names — acquire and release — from the instruction set down to the PCIe transaction, so software can say exactly what must stay ordered instead of over-serializing. Treating the memory-consistency model as something to state and reason about explicitly, rather than leaving order implicit and coping with whatever the hardware happens to provide, is the same undertaking the work on automatically migrating code between memory consistency models (Atomig) covered elsewhere on this site pursues, taking the ordering semantics seriously as a first-class concern. This paper carries that discipline into the I/O interconnect, where ordering had no vocabulary at all.

Co-design the ISA, the interconnect, and the microarchitecture to express intent without stalling. The paper does not patch one layer; it adds new ISA instructions, new PCIe header semantics, and new Root Complex hardware together, so software’s ordering intent flows through and gets enforced without the source ever having to stop and wait. Adding a small amount of instruction-set and microarchitectural support so software can convey what it wants and avoid an expensive fallback like polling or serialization is the same principle the work on extended user interrupts for fast notification without polling (XUI) covered elsewhere on this site is built on, co-designing the ISA and hardware to replace a costly stall with an efficient signal. Here the costly stall is source-side serialization, and the co-design replaces it with expressed intent plus destination-side enforcement.

Let operations run out of order, then reconstruct the correct order where it matters. The efficiency comes from allowing requests to fly concurrently and be completed out of order, with the Root Complex using sequence numbers and, in the advanced design, speculation to restore the program order — exactly the trick a modern CPU uses internally. Extracting parallelism by executing operations out of their strict order while guaranteeing the final result respects that order is the same idea the work on formally verified out-of-order execution in dataflow circuits (Graphiti) covered elsewhere on this site develops, reaping concurrency without breaking ordering. The paper applies it to remote memory operations, so the interconnect gets the throughput of unordered traffic with the correctness of ordered traffic.

Following an ordered remote read through the design

An RDMA key-value lookup that must check a lock before reading

A trace, drawn from the paper, of how a read-then-read lookup runs under destination-based ordering.

Step 1 of 5

A lookup must check a lock, then read the object

An RDMA key-value get requires a check-before-read pattern: the NIC must read a lock or status flag and, only after seeing it, read the object the lock guards. Reading in the wrong order could return stale or corrupt data.

This is a read-then-read ordering the software must guarantee.

Why it works

Moving enforcement to the destination lets the source pipeline. Serializing at the source pays the full round-trip per operation → but enforcing order at the Root Complex lets the source issue operations concurrently → so the long interconnect latency is amortized across many in-flight requests.
Explicit acquire and release let software ask for exactly what it needs. PCIe cannot express fine-grained order → but adding acquire/release bits to transactions and instructions to the ISA → so software marks only the operations that must stay ordered and nothing is over-serialized.
Sequence numbers plus a reorder buffer replace the store fence. A fence flushes and stalls after every write → but tagging writes with sequence numbers and reordering them at the Root Complex → so the CPU streams packets fence-free and reaches line rate.
Ordering against host memory keeps parallel reads correct. Parallel reads to host memory can complete out of order, a cache hit passing a cache miss → but the Remote Load-Store Queue enforces the expressed order where the requests touch memory → so correctness holds, not just order on the wire.
Speculation removes the last serialization. Even destination enforcement can stall on sequential host-memory access → but speculatively executing reads and checking order afterward, as out-of-order CPUs do → so throughput approaches the ideal unordered rate.

What it costs (trade-offs and scope)

It requires changing the PCIe standard, the host ISA, and the Root Complex. This is a broad hardware-software co-design that touches an industry standard (PCIe transaction semantics), the processor’s instruction set, and the interconnect microarchitecture; it is not a software-only fix, so realizing it depends on adoption across those layers.
It adds new hardware and speculation logic. The Remote Load-Store Queue, the reorder buffer, and the speculative-ordering machinery are new structures in the Root Complex, adding area and complexity, and speculation brings its own correctness-checking and recovery machinery.
Some headline results are from simulation. The largest speedups (for example, the up to 50.9 times for 64-byte RDMA key-value objects) are measured in simulation with a single RDMA queue pair; real hardware and multi-queue behavior could shift the numbers, though the transmit-path line-rate result is demonstrated on real NICs.
It targets non-coherent PCIe specifically. Coherent I/O interconnects such as CXL are an alternative direction; the paper argues that thoroughly optimizing existing non-coherent systems is a prerequisite to justifying the cost of full coherence, but this is a bet that PCIe-style non-coherent interconnects remain central.
The benefit is for ordering-bound I/O patterns. The gains come from workloads whose performance is limited by remote ordering — packet transmit and RDMA key-value lookups; workloads without fine-grained remote ordering requirements would see little from this machinery.

The evaluation

Enforcing remote memory ordering over PCIe

Property	Source-side serialization (today)	PCIe built-in ordering	Destination-based (this work)
Transmit path (write ordering)
Small-packet transmit rate
Ordered remote reads
Ordered-read throughput
RDMA key-value store

A comparison of ways to enforce remote memory ordering over PCIe. By expressing ordering explicitly (acquire/release in PCIe and the ISA) and enforcing it at the destination with a Remote Load-Store Queue and reorder buffer in the Root Complex, the design eliminates source-side stalls: a fence-free transmit path at line rate (100 gigabits per second on one core), ordered remote reads up to 50.9 times faster for small objects in simulation, and an RDMA key-value store simpler than and 1.6 times faster than FaRM.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ copy hosted at the University of Utah, of the ASPLOS 2026 paper; the hosted title reads “Efficient Remote Memory Ordering for Non-Coherent Interconnects”). The design described is the paper’s: software over non-coherent interconnects like PCIe needs fine-grained remote memory ordering, but the interconnect forces costly source-side serialization. The paper proposes a destination-based ordering model with a hardware-software co-design: acquire and release semantics added to the PCIe specification (an ordered-read distinction, via a header bit), acquire/release memory-mapped load and store instructions in the host ISA plus sequence numbers on writes, and new microarchitecture at the PCIe Root Complex — a Remote Load-Store Queue and a reorder buffer that enforce the expressed ordering while maximizing parallelism, with an advanced variant using speculative ordering from out-of-order processors. The paper reports that source-side ordering caps 64-byte transmit at roughly 5 gigabits per second (a store fence adds about 100 nanoseconds per packet) and ordered remote reads near 2 million operations per second (a full round-trip of about 500 nanoseconds each), and that the design enables a fence-free MMIO transmit path at line rate (100 gigabits per second on a single core), near-zero-penalty ordered DMA reads, RDMA key-value performance up to 50.9 times higher for 64-byte objects using a single RDMA queue pair in simulation, and a key-value store 1.6 times faster than FaRM. Exact PCIe and ISA extensions, the microarchitecture, the correctness argument, and full results should be read from the paper.

Background

Why the interconnect keeps becoming the place where consistency is decided

It helps to see this work within a long arc in which the boundary between processors and everything else becomes the hard part of system design. For decades, memory consistency was a story told inside a single machine: caches, coherence protocols, and fences gave software a well-understood, if subtle, contract about what happens before what. As computing spread out — across sockets, across devices on an interconnect, across machines on a network — that contract had to be re-established at each new boundary, and interconnects that were designed mainly to move bytes fast, like PCIe, often lacked the vocabulary to express the ordering software had come to rely on. The result is a recurring pattern: performance-critical I/O runs into a consistency gap at the interconnect, and systems paper over it with expensive serialization until someone redesigns the interconnect and the hardware around it to make ordering first-class. The stakes keep rising because the interconnect, not the CPU, is increasingly the bottleneck, and because new coherent interconnects like CXL are being proposed partly to close exactly these gaps. The broad effort to build fast, purpose-designed interconnect fabrics for tightly-coupled systems is the same terrain the work on an ultra-low-latency Ethernet fabric for memory disaggregation (EDM) covered elsewhere on this site occupies, rethinking the fabric so the semantics systems need are efficient rather than an afterthought. This paper makes the case that even without moving to full coherence, giving a non-coherent interconnect the ability to express and enforce ordering at the destination reclaims most of the performance that source-side serialization throws away.

If you remember only three things

1. Over non-coherent interconnects like PCIe, some remote operations must stay ordered — packets sent CPU-to-NIC (write-then-write), and an RDMA key-value lookup that checks a lock before reading the object (read-then-read). But PCIe cannot express fine-grained ordering (it orders writes relative to writes, not reads relative to reads), so software serializes at the source: a store fence after every write, or a stop-and-wait full round-trip before each dependent read. This caps 64-byte transmit near 5 gigabits per second (versus a 100-gigabit link) and makes ordered remote reads more than ten times slower than unordered ones.
2. The paper proposes destination-based ordering with a hardware-software co-design. It makes ordering explicit by adding acquire and release semantics to the PCIe specification (a header bit distinguishing ordered from unordered reads) and acquire/release memory-mapped instructions plus sequence numbers to the host ISA, so software states exactly what must stay ordered and the source stops stalling. Enforcement moves into the PCIe Root Complex via a Remote Load-Store Queue and a reorder buffer: for writes the reorder buffer reconstructs order from sequence numbers, for reads the queue pipelines requests and orders them against host coherent memory, and an advanced version uses speculative ordering from out-of-order processors to remove nearly all remaining serialization.
3. The result is a fence-free transmit path at line rate (100 gigabits per second on a single core), near-zero-penalty ordered DMA reads, and an RDMA key-value store that is simpler than the state of the art yet up to 50.9 times faster for 64-byte objects (in simulation, single queue pair) and 1.6 times faster than FaRM. The trade-offs: it requires changing the PCIe standard, the host ISA, and the Root Complex microarchitecture (needing broad adoption), adds new hardware and speculation logic, has some headline numbers from simulation, targets non-coherent PCIe specifically, and helps only ordering-bound I/O patterns.

Check yourself

Check your understanding

1. Why does enforcing remote memory ordering over PCIe cripple performance today?
2. What does "destination-based ordering" mean in this work, and why does it help?
3. Why is it not enough to preserve order only across the PCIe wire?
4. Predict: the advanced Remote Load-Store Queue borrows "speculative memory ordering" from out-of-order CPUs for ordered reads. What is it doing, and what is the payoff?

Sources

The paper is “Efficient Remote Memory Ordering for Non-Coherent Systems” by Wei Siew Liew, Md Ashfaqur Rahaman, Adarsh Patil, Ryan Stutsman, and Vijay Nagarajan (University of Utah and Arm), published at ASPLOS 2026, DOI 10.1145/3779212.3790156. It is licensed CC BY, with an author copy hosted at the University of Utah (under the title “Efficient Remote Memory Ordering for Non-Coherent Interconnects”) and code at github.com/icsa-caps/efficient-remote-memory-ordering.

This explainer is written from the paper’s full text (the authors’ hosted copy). Software over non-coherent interconnects like PCIe requires fine-grained memory ordering, but current hardware mandates costly source-side serialization, which the paper shows severely limits two applications: CPU-to-NIC packet transmission (needing write-to-write ordering) and RDMA-based key-value lookups (needing read-to-read ordering). The paper proposes a destination-based ordering model and a hardware-software co-design of PCIe extensions and ISA extensions — adding acquire/release semantics to PCIe and acquire/release memory-mapped instructions to the host ISA (with sequence numbers on writes) so software expresses ordering intent — enforced by novel microarchitecture at the PCIe Root Complex, a Remote Load-Store Queue and reorder buffer that eliminate source-side stalls, with an advanced variant using speculative ordering from out-of-order processors. The paper reports a fence-free MMIO transmit path at line rate (100 gigabits per second on a single core), near-zero-penalty ordered DMA reads, RDMA key-value performance up to 50.9 times higher for 64-byte objects using a single RDMA queue pair in simulation, and a key-value store 1.6 times faster than FaRM. Numbers are the paper’s own; exact extensions, microarchitecture, and full results should be confirmed in the paper.