Wave: Offloading Resource Management to SmartNIC Cores

Jack Tigar Humphries, Neel Natu, Kostis Kaffes, Stanko Novakovic, Paul Turner, Henry M. Levy and 2 others

Cloud servers now ship with SmartNICs that carry their own small ARM processors, and every host core those chips can take over is a core a provider can sell — but the operating system work worth moving there, like deciding which thread to run next, needs answers in well under a microsecond, and the SmartNIC sits across a slow PCIe link; this paper (Google, Stanford, Washington, Berkeley) builds Wave, a framework that runs the OS's decision-making as agents on the SmartNIC and hides the link latency with the right memory tricks and by computing decisions before they are asked for, recovering as many as 16 host cores at a cost of only a few percent.

The argument in one glance

Problem

A SmartNIC has spare processors, but the useful OS work to move there is too latency-sensitive

Cloud servers increasingly ship with SmartNICs — network cards that are really small computers, carrying their own ARM processors, memory, and accelerators. Providers already offload networking, storage, and security onto these cards, because every host CPU core reclaimed is a core they can rent to a paying customer. The cards also include a handful of general-purpose ARM cores, sixteen on a recent one, that largely go underused. A tempting workload to move onto them is the operating system's own decision-making: choosing which thread a core should run next, which physical page frame to hand out for a virtual page, how to route an incoming remote procedure call. This is a good match — it runs on every machine, so the work of offloading it pays off fleet-wide, and it is trusted code the provider controls, so it can safely run in a privileged spot on the card. But there is a hard obstacle. These decisions are needed extremely fast — a scheduler must answer in well under a microsecond, or a core sits idle waiting — and the SmartNIC sits across a slow PCIe link from the host. Moving the decision-maker to the far side of that link threatens to make everything slower, not faster.
What people did

Offload to the SmartNIC using existing frameworks

There are established ways to push work onto a SmartNIC. Frameworks like Floem and iPipe move application pipelines to the card, and a body of work offloads the host network stack's data plane to SmartNIC accelerators. These systems build host-to-SmartNIC communication out of direct-memory-access queues, tuned to push a lot of data across the PCIe link efficiently.
Why it fell short

Throughput-optimized offload does not meet system software's latency and reliability needs

Those frameworks optimize primarily for throughput, which is the wrong target for operating-system decision-making. System software has strict, diverse requirements the existing machinery does not satisfy: a thread scheduler or an RPC stack needs sub-microsecond, reliable communication because host cores stall until the decision comes back, while a memory manager instead needs high throughput to ship an entire address space of page-table entries over and get a decision per page in return. The PCIe interconnect is fundamentally slow for this — a single memory-mapped read across it costs hundreds of nanoseconds because of packet protocols, flow control, bridge logic, and low clock rates — and it is non-coherent, so the host cannot simply cache the SmartNIC's memory the way it caches its own. The userspace operating-system frameworks that already separate decision-making from the kernel, like ghOSt, were built for tightly-coupled coherent CPUs, not for a decision-maker sitting across PCIe. Nothing existing put the interconnect in the critical path of microsecond-scale system software and still went fast.
This paper's idea

Run the OS decision-maker as an agent on the NIC, and hide the link with the right mechanisms

Wave offloads userspace system software to agents running on the SmartNIC, keeping the kernel's enforcement on the host. It borrows the ghOSt model, in which decisions are sent back and committed atomically, and implements it with a pair of one-way queues across PCIe. Its central move is to match the communication mechanism to each workload: direct memory access for the high-throughput memory manager, memory-mapped IO for the low-latency scheduler and RPC stack — and within that, different page-table-entry types (write-combining to batch messages out, write-through to cache decisions coming back), plus software cache management to stay correct over the non-coherent link. Above all, Wave lets agents precompute and prestage decisions before the host asks, and prefetch them, so the PCIe round trip is hidden. The result recovers up to sixteen host cores at a cost of only a few percent.

Why this problem is worth solving

The economics are simple, and they point straight at the operating system.

Cloud providers relentlessly minimize the host resources their own infrastructure consumes, because every core and every gigabyte not spent on management can be sold to a customer. That is why virtualization, networking, and security have already been pushed off the host CPU onto SmartNICs. The operating system’s decision-making is an especially attractive next target for three reasons. First, it is universal: every server in the fleet runs a scheduler, a memory manager, and an RPC stack, so a one-time effort to offload them yields benefits on every machine. Second, it is a natural fit for the SmartNIC’s mid-tier ARM cores, since these decisions are largely simple heuristics that balance well against the cores’ modest compute, and offloading them removes their interference with the customer workloads on the host. Third, it is safe: because the provider writes and controls this system software, it is trusted, so it can run in a privileged environment on the card, unlike arbitrary customer code, which providers forbid on these cores. And a great deal of prior work — microkernels, multikernels, and userspace subsystems like ghOSt and Snap — has already split OS decision-making out of the kernel into userspace, so the structure needed to relocate it is largely in place.

The obstacle is that the interconnect is now inside the fast path, and it is slow.

The frameworks that run system software in userspace, and the microkernels and multikernels that split the OS across cores, were designed for CPUs joined by a fast, cache-coherent on-chip interconnect. A SmartNIC is joined to the host by PCIe instead, which is a different animal: communication is packet-based with flow control and bridge logic, clocked lower than an internal interconnect, so a single memory-mapped operation across it takes on the order of a microsecond and a read costs hundreds of nanoseconds. Worse, PCIe is non-coherent — the host cannot transparently cache the SmartNIC’s memory, because there is no hardware protocol keeping the two caches consistent. This collides with what system software needs. A thread scheduler must return a decision in well under a microsecond, because until it does the host core has nothing to run; an RPC stack is similar; a memory manager instead needs to move whole address spaces of page-table entries, demanding throughput. Prior offload frameworks built throughput-oriented direct-memory-access queues, which do not meet the strict latency and reliability that decision-making in the critical path requires. Bridging that latency and coherence gap, for workloads with opposite requirements, is exactly what Wave sets out to do.

What people did before, and where it broke

Throughput-oriented offload frameworks (Floem, iPipe). Move application pipelines to the SmartNIC over direct-memory-access queues tuned for bandwidth. Where it broke. They optimize for throughput, not the sub-microsecond latency and reliability that system-software decisions in the critical path demand.
Offload the network data plane to accelerators. Push the host network stack’s fast path onto SmartNIC accelerators and datapaths. Where it broke. This targets fixed-function acceleration, not the general-purpose OS decision-making that runs fleet-wide and needs flexible compute on the ARM cores.
Userspace OS frameworks on coherent CPUs (ghOSt, Snap, microkernels). Split OS decision-making into userspace agents communicating with the kernel over a fast on-chip interconnect. Where it broke. They assume a low-latency coherent interconnect; placing the agent across a slow, non-coherent PCIe link would put that latency directly in the decision path.
Just add more on-host cores. Run the system software on host efficiency cores rather than offloading it. Where it broke. It still consumes host resources the provider wants to sell, and forgoes the uniform, cross-vendor management view and the network insight a SmartNIC gives.
The gap. No framework let latency-critical, microsecond-scale system software run on SmartNIC cores across PCIe without paying the interconnect’s latency, while also serving throughput-bound components like memory management. Building that framework is what Wave contributes.

How Wave works

Running OS decisions on the NIC while hiding the PCIe round trip

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Put the OS’s decision-making on the NIC’s own cores, and keep enforcement on the host. Wave’s premise is that a SmartNIC is not just an accelerator but a small general-purpose computer whose spare ARM cores can host real host-OS work, and the right work to give them is the decision-making of operating-system subsystems — which thread, which page, which route — precisely because it is universal, trusted, and heuristic. The kernel’s enforcement machinery stays on the host; only the policy brain moves across the link, as a userspace agent, leaving applications and the Linux kernel unmodified. Treating the network card’s onboard, general-purpose resources as a place to run real system work rather than as a fixed-function offload is the same repurposing the work on the benefits of general-purpose on-NIC memory covered elsewhere on this site argues for, discovering that the NIC’s ordinary compute and memory are useful for far more than moving packets.

Reuse a proven split-OS contract, and match every mechanism to its workload. Rather than inventing an interface, Wave adopts the ghOSt model in which a userspace agent proposes decisions and the kernel commits them atomically, a contract that becomes more valuable, not less, once the agent sits across a lossy, high-latency link. On top of that contract it refuses a one-size transport: memory management, which must ship whole address spaces, gets throughput-oriented direct memory access, while the scheduler and RPC stack, which stall cores until they answer, get low-latency memory-mapped IO — and even there, Wave splits the outgoing and return paths into write-combining and write-through memory to speed each, and manages the resulting cache staleness in software because PCIe offers no coherence. Relocating the operating system’s scheduling decisions into a cooperative agent that proposes and the system commits, with a carefully engineered channel between them, is the same restructuring the work on cooperative para-virtualized scheduling covered elsewhere on this site performs, moving the scheduling brain out of its usual seat while preserving a disciplined handshake with the mechanism that enforces it.

Hide the interconnect by answering before you are asked. The deepest idea is that the PCIe round trip need not be on the critical path at all if the decision is already made. Wave lets an agent precompute and prestage decisions in SmartNIC memory ahead of the host’s request — a memory manager that sees strided access preallocates the frames it will soon need; a scheduler with a deep run queue stages the next runnable thread per core — so when the host needs a decision it finds one waiting, and, because the return queue is write-through, it prefetches that decision early enough to hide the read entirely, driving the effective cost of a prestaged decision toward zero. Wringing latency out of a slow host-to-NIC channel through careful use of caching, prefetching, and memory-type discipline is the same battle the work on a cache-coherent interface to the NIC covered elsewhere on this site fights, attacking the host-NIC interconnect as the thing standing between offload and performance.

Following a scheduling decision through Wave

Scheduling a thread from across the PCIe link

A trace, following the paper's example, of how a Wave agent on the SmartNIC schedules a host thread and how the latency is hidden.

Step 1 of 5

A host thread triggers a scheduling event

Thread A, running on host core 0, does something that needs a scheduling decision — it blocks on a futex, say. The host kernel packages the relevant state into a message and sends it toward the SmartNIC agent over the outgoing queue.

The outgoing queue is mapped write-combining, so the host batches the message into a fast-draining buffer.

Why it works

The offloaded work is universal and trusted, so the effort pays off and is safe. Offloading arbitrary code to SmartNIC cores is forbidden and one-off → but OS decision-making runs on every machine and is provider-controlled → so a single offloading effort benefits the whole fleet and can run in a privileged spot on the card.
Matching transport to workload serves opposite needs at once. One communication mechanism cannot be both low-latency and high-throughput → but Wave routes the scheduler and RPC stack through memory-mapped IO and the memory manager through direct memory access → so latency-critical and bandwidth-critical components are each served well.
Page-table-entry types turn a non-coherent link into a fast one. The host cannot cache SmartNIC memory coherently, making reads cost hundreds of nanoseconds → but write-combining batches outgoing messages and write-through caches incoming decisions a cache line at a time → so both directions of the memory-mapped path get much cheaper, with software flushes keeping caches correct.
Prestaging removes the round trip from the critical path. Requesting a decision on demand forces the host to wait several microseconds while the core idles → but an agent that predicts and precomputes decisions has one waiting when the host asks → so host utilization is decoupled from decision latency, improving workload performance markedly.
Prefetching a write-through decision hides the read entirely. Even a staged decision normally costs a slow PCIe read to fetch → but issuing a prefetch about a microsecond before the read lets it complete in the background → so the subsequent load hits the cache and the effective read cost of a prestaged decision falls to zero.

What it costs (trade-offs and scope)

It only works for trusted, provider-controlled system software. The safety argument — running in a privileged environment on the card — depends on the offloaded code being the provider’s own trusted system software; it does not extend to arbitrary or customer code, which providers deliberately keep off these cores.
PCIe stays in the critical path, so a residual cost remains. Even with every optimization the interconnect is still there, leaving a 1.1 to 7.4 percent slowdown versus on-host; the latency-hiding only fully pays off when decisions can be prestaged, which requires the agent to predict them (strided memory access, a deep run queue) — genuinely unpredictable decisions pay more of the round trip.
Correctness leans on manual, software-managed coherence. Because PCIe is non-coherent, Wave keeps the host cache correct with explicit flushes on interrupts and careful use of write-combining and write-through memory; this is hand-managed discipline rather than hardware coherence, and it is specific to the mechanisms used.
It presupposes a split, userspace-separable OS. Wave builds on the fact that ghOSt, Snap, and similar systems already separate decision-making from enforcement; system software that is monolithically embedded in the kernel would need refactoring into this agent-and-mechanism structure first.
It is prototyped on one SmartNIC, and the benefit is provider-side. The evaluation is on a specific card with mid-tier ARM cores, and the payoff is infrastructure efficiency — recovering host cores and reducing interference — rather than making the offloaded component itself faster, which it makes slightly slower.

The evaluation

Placing OS decision-making: host, throughput-offload, or Wave

Property	On-host system software	Throughput-oriented offload	Wave
Where decisions run
Meets sub-microsecond latency
Handles the PCIe round trip	n/a
Host cores recovered
Cost of offloading	Baseline

From the paper's evaluation on an Intel Mount Evans SmartNIC. Wave offloads three preexisting components (thread scheduling, memory management, RPC) with a 1.1 to 7.4 percent slowdown versus on-host, its optimizations improving RocksDB throughput by about 350 percent over unoptimized offload; it recovers 16 host cores from ML-based memory management (which also cuts RocksDB's memory footprint 79 percent), 8 from the RPC stack, and improves virtual-machine turbo performance by 11.2 percent by reducing interference.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The contribution is a practical demonstration that even the most latency-sensitive operating-system work — thread scheduling — can be moved onto a SmartNIC’s cores across PCIe without wrecking performance, by engineering the communication rather than accepting the interconnect’s raw latency. Wave keeps enforcement on the host and runs decision-making as ghOSt-style agents on the card, routes throughput-bound and latency-bound components through direct memory access and memory-mapped IO respectively, exploits write-combining and write-through page-table types with software cache management to speed the non-coherent link, and above all precomputes and prefetches decisions so the round trip is hidden. The payoff is real infrastructure efficiency: up to sixteen host cores recovered and better VM performance, at a 1.1 to 7.4 percent cost. Read it within scope. It relies on the offloaded code being trusted provider software, not customer code; PCIe stays in the path so latency-hiding only fully works when decisions can be predicted and prestaged; correctness depends on hand-managed software coherence; it presupposes an OS already split into userspace decision-making plus in-kernel enforcement; and it is prototyped on one card, with the benefit accruing to the provider rather than speeding up the offloaded component itself. Within those bounds it is a strong argument for rethinking where system software runs.

Background

Why reclaiming host resources drives datacenter design

It helps to see the force behind this work. In a cloud datacenter the host’s own resources are in constant tension with the customer’s: every CPU core, and every gigabyte of memory, that the infrastructure spends on management, networking, or the operating system is a resource that cannot be rented out. At fleet scale this tax is enormous, so a recurring theme in datacenter systems is to claw back host resources transparently — without the customer’s applications noticing or needing to change — and hand them back to revenue-generating work. Sometimes that means moving cold memory to cheaper tiers behind the application’s back; here it means moving the operating system’s own housekeeping off the host CPU entirely. Transparently reclaiming host resources that infrastructure would otherwise consume, so more of the machine is available to paying customers, is the same objective the work on transparent memory offloading in datacenters covered elsewhere on this site pursues, quietly relocating what does not need to sit on the critical, expensive resource. Wave applies that instinct to compute, arguing that the natural home for much of the operating system’s decision-making is not the host at all, but the small computer already sitting on the network card.

If you remember only three things

1. SmartNICs carry mid-tier ARM cores that largely go unused, and offloading the operating system's decision-making (thread scheduling, memory management, RPC handling) onto them would free host cores for paying customers — a good fit because that software runs fleet-wide and is trusted provider code — but the decisions are needed in well under a microsecond while the host-to-SmartNIC PCIe interconnect is slow and non-coherent, and prior offload frameworks optimized for throughput rather than the strict latency and reliability system software demands.
2. Wave runs userspace system software as agents on the SmartNIC (keeping in-kernel enforcement on the host), reuses the ghOSt contract of atomically-committed decisions over two one-way PCIe queues, and matches each transport to its workload: direct memory access for the throughput-bound memory manager, memory-mapped IO for the latency-bound scheduler and RPC stack — mapping the outgoing queue write-combining to batch messages and the incoming queue write-through to cache decisions, with software cache flushes for correctness over the non-coherent link, and above all letting agents precompute and prestage decisions before the host asks and prefetch them, so the PCIe round trip is hidden.
3. Prototyped on an Intel Mount Evans SmartNIC, Wave offloads three preexisting components with only a 1.1 to 7.4 percent slowdown versus on-host (its optimizations improving RocksDB throughput about 350 percent over unoptimized offload, with a 426-nanosecond notify), recovering 16 host cores from ML-based memory management (which also cut RocksDB's memory footprint 79 percent), 8 from the RPC stack, and improving VM turbo performance 11.2 percent by reducing interference; the approach requires trusted provider software, an OS already split into userspace decisions plus in-kernel enforcement, and predictable decisions to fully hide latency, with a residual cost because PCIe stays in the critical path.

Check yourself

Check your understanding

1. Why is the operating system's decision-making an especially good workload to offload to SmartNIC ARM cores?
2. What makes offloading latency-critical system software across PCIe hard, and how does Wave match mechanisms to workloads?
3. How does Wave hide the remaining PCIe latency from the critical path?
4. What did Wave achieve, and what are its limits?

Sources

The paper is “Wave: Offloading Resource Management to SmartNIC Cores” by Jack Tigar Humphries, Neel Natu, Kostis Kaffes, Stanko Novakovic, Paul Turner, Henry M. Levy, David Culler, and Christos Kozyrakis (Stanford University, Google, Columbia University, University of Washington, and University of California, Berkeley), published at ASPLOS 2025 (Volume 3), DOI 10.1145/3676642.3736113. It is released under a Creative Commons Attribution license.

This explainer is written from the full text of the authors’ arXiv version (arXiv:2408.17351). Everything above comes from the paper’s own sections: the introduction on SmartNICs (Infrastructure Processing Units) in datacenters and why cloud providers offload to them, the underused mid-tier ARM cores (for example sixteen on Intel Mount Evans), and the obstacles of weak ARM compute, high host-SmartNIC PCIe latency, and security restrictions on running arbitrary code; the argument for offloading userspace system software — its decision-making tasks (thread scheduling, memory frame allocation, RPC routing), its uniform fleet-wide deployment, its fit to ARM cores and heuristics, its trusted status, and the prior microkernel, multikernel, exokernel, and Linux userspace work that already separates OS decision-making; the challenge of PCIe latency and non-coherence and the differing needs of latency-critical scheduling and RPC (sub-microsecond) versus throughput-critical memory management, and why prior throughput-oriented frameworks (Floem, iPipe) fall short; the Wave design reusing the ghOSt API with atomic commits over two unidirectional queues, the lifecycle of a scheduling decision (message, poll, decide, decision queue, MSI-X interrupt, atomic commit and enforce), and the Wave API; the mechanisms of DMA for throughput and MMIO for latency, the write-combining and write-through page-table-entry types with batching, caching, and software cache-flush coherence, and the prestaging and prefetching that hide the remaining latency (including a reported 32 percent workload improvement from prestaging); and the evaluation on Intel Mount Evans reporting a 426-nanosecond notify, a roughly 350 percent RocksDB throughput improvement from the optimizations, a 1.1 to 7.4 percent slowdown versus on-host, recovery of 16 host cores from machine-learning-based memory management (with a 79 percent RocksDB memory-footprint reduction) and 8 from the RPC stack, and an 11.2 percent virtual-machine turbo-performance improvement from reduced interference.