The benefits of general-purpose on-NIC memory

A whole class of network applications — firewalls, load balancers, address translators, key-value stores — only ever reads the small HEADER of each packet and passes the large payload through untouched, yet today's servers drag every payload across the PCIe bus into main memory and back out again just to forward it, saturating the interconnect and DRAM long before the CPU is busy; this paper notices that modern network cards already carry a few megabytes of fast internal memory that sits mostly idle, and proposes exposing it to software so these "data movers" can park payloads ON the card — the header goes to the host, the payload never leaves — cutting network-function latency up to 23% and, for the hot items of a skewed key-value store, memcached latency up to 43% and throughput up to 80%.

The argument in one glance

Problem

The network outran the plumbing, not the processor

Network cards now push 100, 200, and 400 gigabits per second, and the pressure has shifted off the CPU onto the interconnect and memory. A huge class of network applications — network address translators, load balancers, firewalls, and key-value stores answering "get" requests — are DATA MOVERS: they read a packet's small header (or a request's key), decide where it goes, and forward the large payload (or value) completely untouched. Yet the standard server drags that payload across the PCIe bus into main memory, lets the application ignore it, and drags it back out to transmit. For a load balancer, that is megabytes per second of pure waste — and up to ~60% of datacenter traffic passes through exactly these functions.
What people did

Cache the packets on-chip with DDIO

Intel's Data Direct I/O (DDIO) already tries to spare DRAM: incoming DMA writes land in the last-level cache (LLC) instead of memory, and outgoing reads are served from cache if the data is still there. When it works, payloads never touch DRAM and the CPU sees them hot. It is the deployed answer to I/O-bandwidth pressure, and for modest rates it holds.
Why it fell short

DDIO is capped by cache, and high-rate networking overflows it

DDIO can only cache what fits in the slice of LLC it is allowed to use, and high-throughput multi-core networking needs enormous receive buffers: one receive ring per core, each entry sized for a maximum 1500-byte packet, and the rings must be large (≥1024 entries) or bursts drop packets. A single 1024-entry ring already holds ~2 MiB — a whole cache way. Push the rate up and DDIO writes evict still-unprocessed packets to DRAM before the CPU reads them (the "leaky DMA problem"), so the payload traffic spills into memory anyway, contends with the application's own memory accesses, and throttles everything. Shrinking rings to fit the cache just trades one bottleneck (memory) for another (dropped packets). And even off-bottleneck, the round trip inflates latency.
This paper's idea

Park the payload on the card, which already has spare memory

The paper (Technion and NVIDIA) makes three observations: data movers touch only metadata; modern ASIC NICs already ship a few megabytes of fast internal SRAM (NVIDIA's ConnectX-6 Dx has 4 MiB); and that memory sits mostly idle (typically under 15% used). So expose the unused part to software as ordinary memory-mapped memory — "nicmem" — and let data movers keep payloads there. Using the NIC's existing packet-splitting, headers go to host memory (where the CPU works on them) and payloads stay on the card (where the NIC transmits them directly, never crossing PCIe or touching DRAM). A "header inlining" trick tucks the tiny header into the ring descriptor itself, saving a PCIe round trip. A "split rings" fallback spills to host memory when nicmem fills. For key-value stores, hot items (workloads are Zipf-skewed) live on nicmem with a zero-copy racesafe protocol. Results: network-function latency down 23%, throughput up 19%; memcached latency down 43%, throughput up 80% — and a system with DDIO OFF and nicmem ON beats one with maximum DDIO and no nicmem.

Why this problem is worth solving

The economics of fast networking have inverted. For years the CPU was the scarce resource and everyone optimized cycles; now, at 100-gigabit-and-beyond line rates, the processor often sits partly idle while the

the bus connecting the network card to the CPU and memory. Every byte a NIC delivers to the host, or reads back to transmit, crosses it. At modern line rates PCIe becomes a first-class bottleneck: the paper shows a single 100-gigabit NIC saturating its ~125 Gbps of available PCIe outbound bandwidth to 99.8%, at which point transmitted packets stall and drop. Every payload byte that need not cross this bus is bandwidth handed back to useful work

and host DRAM saturate first. The paper’s motivating measurements are stark: run a plain layer-3 forwarder and the system cannot reach line rate — sometimes because PCIe is 99.8% full, sometimes because the transmit ring backs up, sometimes because fewer than ten random memory reads per packet exhaust DRAM bandwidth. None of these is a CPU-cycles problem. They are all consequences of moving data that the application never even reads.

The deep observation is that a

a network application that routes unchanged data to its destination based solely on the associated metadata. Network functions like address translation and load balancing read and maybe rewrite packet HEADERS but never touch payloads; a key-value store’s “get” returns the VALUE (data) selected by its KEY (metadata). In every case the large part rides along untouched — so hauling it into the CPU’s memory hierarchy buys nothing and costs interconnect and DRAM bandwidth. As end-to-end encryption spreads, even fewer functions inspect payloads, making data movers ever more dominant

is a real and growing category — up to ~60% of datacenter traffic flows through address translators or load balancers alone. For these applications, the entire journey of the payload through the host is ceremony. The paper’s insight is that the hardware to skip the ceremony already exists, unused, on the card itself: NIC vendors ship small fast SRAM for internal offload features, applications rarely enable those features, and the memory idles. Turning that idle silicon into a place to park payloads is close to free, and it attacks the exact resource — data movement — that line-rate networking is choking on.

What people did before, and where it broke

Full offload to the NIC / SmartNIC. Move the whole application onto the card — key-value stores, network functions and all. It eliminates host data movement, but surrenders the ease of general-purpose programming, ossifies the network stack, and adds security and maintenance burden. A heavy hammer for a specific nail.
Faster host stacks (kernel bypass, zero-copy, DPDK/RDMA). Cut software abstraction cost on the host — genuinely helpful and fully compatible with this work, but they still move the payload across PCIe into memory; they attack CPU overhead, not the data-movement bottleneck.
DDIO (cache the I/O). The deployed mitigation, and the paper’s main foil. Capped by the LLC ways it may use, it loses to the large receive buffers high-rate multi-core networking demands (leaky DMA), and tuning it — more ways for I/O, smaller rings — just moves contention between cache, memory, and packet loss. It cannot make payloads the NIC never needs to hand over simply not travel.
The gap. No prior approach lets a general-purpose, host-resident application keep the payload OFF the host entirely, on memory the NIC can reach without PCIe — while the application keeps working on the header in ordinary host memory with ordinary code.

How nicmem works

Header to the host, payload on the card

Expose the idle SRAM as memory new

NIC firmware carves the unused portion of on-card SRAM into a software-visible region — "nicmem" — after internal NIC features have taken what they need. The OS sees it via the NIC's capabilities and hands it to applications through mmap, isolating processes to disjoint ranges (an on-NIC IOMMU and memory keys enforce isolation). Applications map it and reference it in NIC queue descriptors like any buffer.
↓ →
Split each packet at arrival new

The NIC already can write a packet's header and payload to two different buffers. nmNFV arms the receive ring so headers land in HOST memory (a few tens of bytes, where the CPU will process them) and payloads land in NICMEM (they stay on the card). A flag on the descriptor tells the NIC "this address is nicmem," so it reads/writes that SRAM directly instead of going over PCIe.
↓ →
Forward without moving the payload new

The network function reads the header from host memory, makes its decision (rewrite an address, pick a server), and transmits — handing back the SAME nicmem pointer it received. The payload never crossed PCIe on the way in and never does on the way out. PCIe traffic, DRAM bandwidth, and per-packet CPU work all drop.
↓ →
Header inlining new

A second optimization: because headers are small, write the header's CONTENT directly into the ring descriptor instead of a separate host buffer. Now the NIC fetches just the descriptor to send a packet — no second PCIe round trip to chase the header's address — and the receive ring needs one scatter-gather entry per packet again (for the payload), not two. Costs the CPU a tiny header copy from Rx to Tx descriptor, cheap because the header is hot in cache.
↓ →
Split rings: spill when nicmem is full new

Nicmem is only megabytes, and bursty high-rate traffic can overrun it. So each receive ring is split into a PRIMARY (nicmem buffers) and a SECONDARY (host-memory buffers). An arriving packet takes a nicmem buffer if the primary ring has one; if empty, it falls back to host memory. As long as the working set fits in nicmem, everything is served from the card — and nothing drops when it does not. Occupancy checks read indices already on nicmem, so the extra queue is nearly free.
↓ →
nmKVS: hot values on the card, race-free new

A key-value store cannot fit its whole item set (as large as host DRAM) in megabytes of nicmem, but its access is Zipf-skewed, so put the HOT items there and serve them zero-copy on transmit. The hazard: the NIC may still be transmitting an item's value when a "set" overwrites it, sending a torn mix. nmKVS keeps a STABLE buffer (on nicmem, never overwritten mid-transmit, reference-counted) and a PENDING buffer (new writes); updates go to pending and invalidate stable, and a later "get" lazily promotes pending into stable once no transmit references it.

Built on NVIDIA ConnectX-5 NICs within DPDK; nmNFV implemented in l3fwd and the FastClick NF framework, nmKVS on the MICA key-value store. Modest code: ~400 lines in the DPDK driver, ~200 in FastClick, a few hundred for nmKVS's zero-copy protocol. The evaluation NIC exposes only 256 KiB of SRAM and lacks split-ring hardware, so the authors emulate larger nicmem by reusing one payload buffer (safe, since data movers never inspect payloads).

The idea

The cheapest byte to move is the one you never move. Every prior technique — faster stacks, bigger caches, more DDIO ways — accepts that the payload travels into the host and tries to make the travel cheaper. nicmem asks why it travels at all. For a data mover the answer is: it does not have to. The payload’s only destination is back out the wire, and the NIC can reach on-card memory without the interconnect, so the payload can sit still while only the metadata makes the trip the CPU actually needs. This reframes the bottleneck from “how fast can we move data” to “how much data must move,” and for the ~60% of datacenter traffic that is header-only, the answer drops by the payload-to-header ratio — roughly an order of magnitude, since payloads cluster around 1400 bytes and headers around a few tens.

Exploit hardware that already exists and already idles. The proposal’s leverage is that it invents almost nothing. The SRAM is on the card; the packet-splitting is a shipping feature; the descriptor-inlining is a shipping feature; MMIO exposure is a firmware change plus an mmap. The paper even prices the scale-up: SRAM costs about $0.21 per megabyte at 7nm, so ~$ 16 (2% of a cheap 100-gigabit NIC’s price) buys 80 MB — more than a top server’s entire LLC — and vendors say enlarging it is easy given a use case. This is the opposite of a clean-slate SmartNIC: a minimal, cost-trivial change that keeps the application general-purpose and host-resident.

Respect the asymmetry: nicmem is fast for the NIC, slow for the CPU. The one hard constraint is that CPU access to nicmem is routed over PCIe and is much slower than host DRAM — and worse, it is marked write-combining, so the CPU can cache writes to it but not reads (a read from nicmem costs 50–528×). The whole design is organized around never letting the CPU touch payloads on nicmem: the header (which the CPU does read) lives in host memory; the payload (which only the NIC touches) lives on nicmem. nmKVS’s zero-copy path is careful to have the CPU write hot values to nicmem (tolerable) but never read them back on the fast path. Placement follows the access pattern — put each piece where its actual consumer is fast — which is the same match-the-medium-to-the-consumer discipline the storage pages on this site keep applying, here across the PCIe boundary.

A load balancer at 200 gigabits

One packet through nmNFV, and why the baseline chokes

A load balancer forwarding 1500-byte packets at line rate across many cores. Numbers from the paper's macrobenchmarks and bottleneck study.

Step 1 of 5

The baseline drowns in its own payloads

Every arriving packet is DMAed whole into host memory. With one receive ring per core, each holding ~2 MiB of buffers, the receive rings blow past the LLC slice DDIO is allowed — DDIO writes evict unprocessed packets to DRAM (leaky DMA), the DDIO hit rate collapses from 83% to 27%, and memory bandwidth jumps from 5 to 55 GB/s. The load balancer never reads a single payload byte, yet payloads are what saturate PCIe and DRAM. It cannot reach line rate; latency climbs with every core added.

up to ~60% of datacenter traffic runs through load balancers or address translators — all header-only
fewer than 10 random memory reads per packet already exhaust DRAM bandwidth on top of the I/O

Why it works

Data movers never read the payload → keeping it on nicmem removes it from the PCIe bus and DRAM entirely → the exact resource line-rate networking saturates is freed, without touching the CPU’s view of the header.
The payload’s only consumer is the NIC’s transmit engine, which reaches nicmem without PCIe → parking it there costs nothing on the fast path and saves both the inbound and outbound DMA → latency and interconnect load drop together.
Headers are tiny and cache-hot → inlining them into descriptors removes a PCIe round trip and collapses two scatter-gather entries to one → PCIe hit rate reaches 100%, helping even payload-less small packets.
Nicmem is bounded but split rings fall back to host memory → the design degrades gracefully instead of dropping packets when the working set exceeds SRAM → benefit scales with how much fits, with no cliff.
KVS access is Zipf-skewed → hot items on nicmem capture most requests from a small footprint, and the stable/pending buffer protocol serves them zero-copy without torn reads → large wins (43%/80%) from megabytes of memory.

What it costs

CPU access to nicmem is punishingly slow. Routed over PCIe and write-combining (writes cacheable, reads not), a CPU read from nicmem costs 50–528× a host read. The entire design must keep the CPU off payloads-on-nicmem; a workload that needs to INSPECT payloads (deep packet inspection, a VPN) gains nothing here.
Capacity is small and shared. Only megabytes today, and the NIC’s own features claim some; benefit is bounded by how much of the working set fits, and split-ring spillover reintroduces host-memory cost for the overflow.
Prototype scaffolding. The evaluation NIC exposes just 256 KiB, lacks split-ring hardware and receive-side inlining, and cannot split by parsed headers — so larger nicmem is emulated (safe for data movers) and some costs (receive-side splitting) are still paid. Real gains assume future silicon removes these limits.
Zero-copy complexity for KVS. The stable/pending protocol and reference counting add real concurrency machinery, and “set”-heavy workloads (which must write both buffers) see up to ~5% loss; the wins are on read-heavy, skewed loads.
Not for everyone. SmartNIC-style full offload or payload-touching applications are out of scope; nicmem is a targeted accelerator for the data-mover pattern, valuable exactly in proportion to how header-only the traffic is.

The evaluation

DPDK-based prototype on NVIDIA ConnectX-5 NICs, driving 100- and 200-gigabit links: synthetic NF sweeps, real address translation and load balancing (nmNFV in FastClick), a CAIDA packet trace, and the MICA key-value store (nmKVS), against a host-memory baseline and a header-payload “split” variant without nicmem.

Where the payload lives, and what it costs

Property	Host-memory baseline (DDIO)	Full NIC offload (SmartNIC)
Payload data path
Programming model	Ordinary host application, general-purpose
Behavior at high rate		Line rate, but inflexible
Network-function results	Reference; exceeds per-core cycle budget on 46% of synthetic NFs	—
Key-value store results	Reference (MICA on DDIO)	—

Cost of scale-up: SRAM ≈ $0.21/MB at 7nm, so ~$16 (2% of a $795 100-gigabit NIC) buys 80 MB — larger than a top Intel server's LLC. Even one nicmem queue of seven clears the PCIe bottleneck; benefit grows with how much of the working set fits on the card.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

nicmem is a targeted accelerator, not a general speedup: its entire premise is that the application never reads the payload, so payload-inspecting functions (deep packet inspection, VPN termination) gain nothing, and the CPU must be kept off nicmem because reads from it cost 50–528× a host read. Capacity is small and shared with the NIC’s own features; split-ring spillover reintroduces host-memory cost for whatever overflows, so the win scales with how much of the working set fits. The prototype leans on emulation — the available NIC exposes only 256 KiB, lacks split-ring and receive-inline hardware, and cannot split by parsed headers — so the cleanest results presume future silicon. And for key-value stores the gains are on read-heavy, skewed loads; write-heavy workloads pay for the racesafe dual-buffer protocol.

Background

Move less, and put each byte where its consumer is fast

nicmem belongs to two threads this site keeps pulling. The first is the storage/data-placement lineage: like the heterogeneous key-value store covered elsewhere on this site, it decides WHERE each piece of data should live by matching the medium to its actual consumer — there, index on low-latency memory and bulk values on high-bandwidth flash; here, headers in CPU-fast host memory and payloads in NIC-fast on-card SRAM, with the write-combining asymmetry making the CPU-vs-NIC access gap the governing constraint. The second is the recurring move of reclaiming a resource that already exists and already idles — the GPU-memory defragmentation and qubit-reuse pages turn stranded capacity back into usable capacity, and nicmem does the same with the under-15%-utilized SRAM sitting on every modern NIC. Its deepest lesson is the simplest: the fastest data movement is the one you avoid — the same “don’t pay for what the workload never uses” logic that the accuracy-scaling and worst-case- provisioning pages apply to compute, applied here to the bytes crossing PCIe.

If you remember only three things

1. A dominant class of network applications — address translators, load balancers, firewalls, key-value "get" — are DATA MOVERS that read only the header (or key) and forward the payload untouched, yet servers haul every payload across PCIe into DRAM and back just to forward it, saturating the interconnect and memory (PCIe 99.8% full, DDIO hit rate crashing 83%→27%, DRAM 5→55 GB/s) long before the CPU is the limit — and DDIO caching cannot fix it because high-rate multi-core receive buffers overflow the cache (leaky DMA).
2. nicmem exposes the few megabytes of fast SRAM that modern NICs already carry and leave under 15% used, so data movers park payloads ON the card: packet-splitting sends headers to host memory and payloads to nicmem (never crossing PCIe), header inlining tucks the tiny header into the descriptor to save a round trip, split rings spill to host memory when SRAM fills, and a stable/pending dual-buffer protocol serves a key-value store's hot (Zipf-skewed) items zero-copy — all while the CPU stays off nicmem, whose reads cost 50–528× a host read.
3. On real prototypes: network-function latency −23% and throughput +19% (reaching 200-gigabit line rate where the baseline cannot), memcached latency −43% and throughput +80%, and a system with DDIO disabled plus nicmem beating one with maximum DDIO and no nicmem — from a change costing ~$16 to scale to 80 MB, with the honest limits that payload-inspecting apps gain nothing, capacity is small, and the cleanest numbers assume future NIC silicon.

Check yourself

Check your understanding

1. A plain layer-3 forwarder at 100 gigabits cannot reach line rate even though its CPU is not saturated. Predict the actual bottleneck and why DDIO — designed to solve exactly this — fails to.
2. CPU reads from nicmem cost 50–528× a normal host read, yet the paper reports large speedups. Predict how the design turns this apparently disqualifying constraint into a non-issue.
3. Header inlining writes the packet header into the ring descriptor. Curiously, tiny 64-byte packets — which fit ENTIRELY inline and gain nothing from payload-on-nicmem — improve as much as large ones. Predict why.
4. A key-value store's item set is as large as host DRAM, dwarfing nicmem's few megabytes — yet nmKVS improves memcached throughput up to 80%. Predict what makes such small memory so effective, and the concurrency hazard it must handle.

Sources

ACM DL: The Benefits of General-Purpose On-NIC Memory — ASPLOS 2022