A network card today plugs into the CPU over PCIe, which is not cache-coherent, so the two talk through a clumsy mailbox of ring buffers and "doorbell" registers — and because every doorbell is a slow PCIe write that stalls the CPU, NIC drivers are tuned to save CPU cycles at the cost of latency, leaving packet round-trips near 2 microseconds; new interconnects (CXL, Intel's UPI) instead let the NIC join the CPU's cache-coherence protocol, sharing memory the way two CPU sockets do — but this paper (Google, UW, Utah, Virginia) shows you cannot just run the old PCIe interface over coherence (a naive port is 79% slower); CC-NIC redesigns every part of the host-NIC interface for coherence — inlining the signal into the descriptor so it arrives with the data in one cache-line fetch, homing each queue in the socket that WRITES it so the reader gets a fast cache-to-cache transfer, and sharing the buffer pool between host and NIC — reaching 1.5 billion packets per second and 980 Gbps at 490ns latency (77% lower than PCIe, 88% lower under load) while freeing up to half the application cores.
The argument in one glance
-
Problem
The NIC and the CPU talk through a slow, non-coherent mailbox
A network interface card (NIC) is what moves packets between the network and the CPU, and at high line rates how the CPU and NIC coordinate is a first-order performance question. Today the NIC attaches over PCIe, which is not cache-coherent. So the two sides cannot simply share memory; they coordinate through a producer-consumer interface: the host fills descriptor rings (lists of work), signals the NIC by writing a "doorbell" register with a memory-mapped I/O (MMIO) write, and the NIC reaches into host memory with direct memory access (DMA). Every one of these crosses the PCIe bus, which is slow, and a doorbell write stalls the CPU while it completes. The result is a stilted, high-latency conversation between two devices that cannot see each other's memory.
-
What people did
Tune the PCIe interface to spend CPU cycles wisely
NIC drivers have been carefully optimized around PCIe's costs. Since a doorbell (an MMIO write) stalls the CPU and each PCIe transaction is expensive, the interface is designed to conserve CPU cycles even if that raises latency. It BATCHES: the host writes many descriptors, then rings one doorbell for the whole batch. It keeps data structures HOST-LOCAL and signals updates explicitly, so the CPU is not reading across the bus. And it makes the HOST do all buffer management — pre-allocating receive buffers, freeing transmit buffers — because without coherence the two sides cannot safely share the buffer-pool data structure. These are sensible choices given PCIe.
-
Why it fell short
Those choices trade away latency, and a naive port to coherence keeps the tax
The cost of "conserve CPU cycles" is latency. A host-to-NIC-to-host round-trip on a modern PCIe NIC takes about 2 microseconds at low load and 6 at 80% load — nearly an order of magnitude more than the raw interconnect traversal — because each doorbell is a costly MMIO write, each descriptor fetch is a separate DMA round-trip after the signal, and nothing is shared. Now emerging coherent interconnects (CXL, and Intel's UPI, which already links CPU sockets) let a device join the CPU's cache-coherence protocol and share memory directly, with the coherence hardware doing the signaling. That should help — but the paper shows you cannot just run the existing PCIe interface over a coherent link. A naive port of today's NIC interface onto UPI performs 79% WORSE than a redesign, because the old signaling and data layouts trigger extraneous coherence traffic and leave data cached in the wrong place. The interface itself was shaped by PCIe's constraints and does not fit the new abstraction.
-
This paper's idea
Redesign every part of the host-NIC interface for coherence
This paper (Google with UW, Utah, and Virginia — Schuh et al.) presents CC-NIC, a host-NIC interface designed from scratch for coherent interconnects. It rethinks signaling, data layout, homing, caching, and buffer management to exploit what coherence offers and avoid what it penalizes. It INLINES the signal into the descriptor — a ready flag the NIC polls — so the signal and the descriptor content arrive together in a single cache-line access, eliminating the separate doorbell and the extra fetch round-trip. It HOMES each queue in the socket that writes it (transmit ring on the host, receive ring on the NIC), so the reader gets a fast cache-to-cache transfer from the writer's L2 rather than a slow DRAM access. It writes packet data with ordinary caching stores to ride the cache-to-cache path, RECYCLES buffers so they stay in the writer's cache and prefetchers do not contend, and SHARES the buffer pool between host and NIC so neither side does redundant bookkeeping. Modeled on UPI, CC-NIC reaches 1.5 billion packets per second and 980 Gbps, with minimum latency of 490ns (77% lower than PCIe, 88% lower under load), and frees up to half the application cores — and the benefits hold across a range of interconnect speeds, including CXL-like ones.
Why this problem is worth solving
At datacenter line rates, the network is often the bottleneck not
because the wire is slow but because the CONVERSATION between the CPU and
the NIC is expensive, and that conversation is dictated by the interface
between them. Understanding today’s interface is the starting point. A
the machinery by which a CPU and a
PCIe network card exchange work, built entirely around the fact that
PCIe is not cache-coherent. The host and NIC share ring buffers of
DESCRIPTORS — small records (typically 16 bytes) that point to packet
buffers and carry metadata like length. To submit work, the host writes
descriptors into a ring in its own memory and then rings a DOORBELL: an
MMIO (memory-mapped I/O) write to a register on the device, which is an
uncacheable write that crosses PCIe and stalls the CPU. To move the
actual bytes, the NIC uses DMA (direct memory access), reading and
writing host DRAM directly, bypassing the CPU cache. Head and tail
registers track the producer and consumer positions. Every coordination
step — signaling, fetching a descriptor, moving a packet — is a separate
PCIe transaction, and PCIe transactions are slow (an MMIO read is
roughly 982 nanoseconds), which is why the whole interface is designed to
minimize their number even at the cost of latency
What changes the picture is a fundamentally different kind of link. A
a link, such as Intel’s UPI (which
connects CPU sockets) or the emerging CXL, over which a device
participates in the CPU’s cache-coherence protocol instead of using
PCIe’s separate read/write transactions. This gives host and device a
symmetric, shared-memory relationship: both access shared memory through
the CPU’s ordinary, highly optimized memory datapath and caches, and the
coherence protocol handles signaling in HARDWARE — when one side writes a
line, the protocol invalidates the other side’s cached copy so its next
read fetches the new value, with no explicit doorbell. Crucially, a read
of data that the other side holds Modified in its L2 cache is served by a
fast CACHE-TO-CACHE transfer, far quicker than going to DRAM, and both
sides can share a data structure with normal memory operations. The
catch is that coherence has its own cost model — moving cache-line
ownership across the link is not free, and data cached in the wrong
place adds work to a later access — so the interface must be laid out to
exploit the cheap operations and avoid the expensive ones
What people did before, and where it broke
- Optimized PCIe NIC interfaces (today’s drivers). Batch doorbells, keep data host-local, let the host manage all buffers. Highly tuned, but they trade latency for CPU efficiency, so a round-trip is about 2 microseconds and 6 under load — nearly an order of magnitude over the raw interconnect.
- Software NIC-interface optimizations (TinyNF, NIQ, PacketMill). Strip driver and stack overheads to speed up the PCIe interface. Real gains, but they keep the same PCIe packet-queue model, so they inherit its signaling and DMA round-trips.
- New CPU-NIC data paths (NanoPU, Direct Cache Access, semi-coherent DMA). Integrate the NIC more tightly with the CPU caches. A step toward coherence, but they add specific mechanisms rather than redesigning the whole interface around a coherent link.
- Naively running the PCIe interface over a coherent link. Take the existing NIC data structures and register signaling and put them on UPI. This does beat native PCIe on packet rate, but it performs 79% worse than a coherence-aware redesign, because the old layouts trigger extraneous coherence traffic and leave data cached in the wrong place.
- The gap. Nobody had redesigned the host-NIC interface itself — signaling, layout, homing, caching, buffer management — for a coherent interconnect, so its potential (cache-to-cache transfers, hardware signaling, shared structures) went unrealized. That is the gap CC-NIC fills.
How CC-NIC redesigns the interface for coherence
Inline the signal, home data at its writer, manage caching, and share the pool
-
The PCIe interface and its latency tax
PCIe NICs signal with MMIO doorbells, fetch descriptors and packets with DMA, and keep structures host-local — every step a slow PCIe transaction. Drivers batch and conserve CPU cycles at the cost of latency, so a round-trip is about 2 microseconds and 6 under load.
-
Do not port the old interface — redesign for coherence new
A coherent link lets the NIC share the CPU cache hierarchy, but simply running the PCIe interface over UPI is 79% slower than a redesign, because the old signaling and layouts cause extraneous coherence traffic. CC-NIC rethinks every part of the interface for the coherence cost model.
-
Inline the signal into the descriptor new
Instead of a separate MMIO doorbell register, CC-NIC puts a ready flag inside the descriptor and has the NIC poll the next descriptor in the ring. Once the flag is set (after the fields are written), the NIC gets the signal AND the descriptor in ONE cache-line access — no separate doorbell and no extra round-trip to fetch the metadata.
-
Home each queue in the socket that writes it new
A read of a line another socket holds Modified in its L2 is a fast cache-to-cache transfer; a read from DRAM is slow. Since each descriptor has one writer and one reader, CC-NIC homes the transmit ring on the host (the host writes it) and the receive ring on the NIC, so the reader always gets the cache-to-cache path — 1.7 to 2.4x faster than reader-homed memory. Packet data is written with caching stores to ride the same path.
-
Manage caching: recycle buffers, avoid prefetch contention new
A buffer left in the consumer's cache after use hurts, because a later producer write must pay a cross-socket invalidation, and x86 has no cheap way to purge a remote cache line. So CC-NIC recycles freed transmit buffers as receive buffers and vice versa, keeping buffer memory in the writer's cache, and lays out the pool so sequential allocations are not sequential addresses, preventing a prefetcher from contending with the producer.
-
Share the buffer pool and batch per cache line new
Coherence lets host and NIC access the buffer pool concurrently, so the NIC frees transmit buffers and allocates receive buffers itself — a symmetric design that removes host bookkeeping passes. Descriptors are packed several to a 64-byte cache line with one shared signal, filling the line in high-throughput bursts while avoiding cache-line thrashing at low load.
The idea
When the abstraction changes, redesign for it — porting squanders the win. The load-bearing insight is negative: coherence is not a faster PCIe that you can drop the old interface onto. Running today’s NIC data structures and register signaling over UPI leaves most of the benefit unrealized — 79% slower than a redesign — because the PCIe interface was shaped by constraints that no longer apply and by no awareness of the new cost model. CC-NIC’s whole value is redesigning signaling, layout, homing, caching, and buffer management for coherence. This redesign-for-the-new-substrate-do-not-port discipline is exactly the one the site’s FHE-accelerator page makes when it argues that fully using an accelerator requires redesigning the encryption scheme rather than accelerating the CPU version, and the accelerator-companion page makes when it specializes for what the hardware actually offers. The recurring lesson: the biggest gains live in the interface between two systems, and a port carries the old interface’s assumptions across a boundary where they have become liabilities.
Fuse the signal with the data, and lay data out for the cost model. Two of CC-NIC’s moves come straight from reading the coherence cost model honestly. PCIe separates the doorbell from the data, forcing an extra round-trip to fetch the descriptor after the signal; CC-NIC inlines the ready flag into the descriptor so one cache-line access carries both — the same eliminate-the-extra-round-trip, fuse-two-operations move the site’s accelerator pages make when they combine a notification with its payload. And because a read served from the writer’s L2 is a fast cache-to-cache transfer while a read from DRAM is slow, CC-NIC homes each queue in its writer’s socket, turning every consumer read into the cheap case. Knowing the substrate’s real cost model and placing data to hit its fast path is the same discipline the site’s pages on interconnect contention and I/O scheduling use when they measure the true cost of an operation rather than assuming a uniform one, and the same locality-aware placement the tiered-memory pages rely on.
The optimization that usually helps can hurt — so manage it. The subtlest part is that caching and prefetching, normally pure wins, become hazards in a coherent host-NIC interface. A buffer retained in the consumer’s cache after use forces a cross-socket invalidation on the next producer write; a hardware prefetcher reading ahead into the next buffer contends with the producer writing it. CC-NIC actively counteracts both — recycling buffers to keep them writer-cached, and scattering pool addresses so prefetchers do not run ahead — and indeed on the unoptimized interface, enabling prefetching makes things up to 7% WORSE. This know-when-the-optimization-backfires awareness is the same observer-effect-flavored caution the site’s pages on cheap monitoring and runtime instrumentation raise: a mechanism that speeds the common case can quietly tax a different one, and a coherence-aware design has to decide, per access pattern, whether to invite it or suppress it.
Why coherence needs a new interface, not the old one
From a 2-microsecond round-trip to 490 nanoseconds
Redesigning the host-NIC interface for a coherent interconnect. Numbers from the paper's UPI-based evaluation.
Step 1 of 5
The PCIe latency tax
On PCIe, a doorbell is a slow MMIO write, fetching a descriptor after the signal is a separate DMA round-trip, and the host and NIC share no data structures. Drivers respond by batching and keeping data host-local, conserving CPU cycles but paying latency: a round-trip is about 2 microseconds at low load and 6 at 80% load.
an MMIO read is roughly 982 nanoseconds; the host does all buffer management because PCIe cannot safely share the pool nearly an order of magnitude of the round-trip is the interface, not the wire
Why it works
- PCIe separates the doorbell signal from the descriptor data → the NIC needs an extra round-trip to fetch metadata after each signal → inlining the flag into the descriptor delivers both in one cache-line access, cutting minimum latency 37%.
- A read of a line held Modified in another socket’s L2 is a fast cache-to-cache transfer, but a DRAM read is slow → homing each queue in its writer’s socket makes every consumer read the cheap case → writer-homed placement is 1.7 to 2.4x lower latency than reader-homed.
- A buffer left in the consumer’s cache costs a cross-socket invalidation on the next producer write, and x86 cannot cheaply purge it → recycling transmit and receive buffers keeps them in the writer’s cache → later writes avoid the extra coherence traffic.
- Coherence lets host and NIC share the buffer pool with ordinary memory operations → the NIC frees and allocates buffers itself instead of the host doing it all → redundant host bookkeeping passes disappear, freeing CPU cores.
- The naive port keeps PCIe’s signaling and layouts, which fight the coherence protocol → redesigning signaling, homing, caching, and buffer management for coherence removes the extraneous traffic → the redesign is 79% faster than the port, hitting terabit rates at sub-microsecond latency.
What it costs (trade-offs and scope)
- It is modeled, not built as a coherent NIC ASIC. CC-NIC uses one CPU socket as the NIC over UPI; the authors argue hardware-initiated transfers map to equivalent coherence operations, and note that an event-driven ASIC could optimize signaling beyond the cache-polling model — but a real device may differ from the software model.
- Part of the throughput rides a fatter link. UPI has more bandwidth than current PCIe, so absolute numbers benefit from that; the paper isolates the design with a same-link naive-port baseline, which shows 3.3x from the design alone, but the headline rates depend on the interconnect.
- Coherence hazards must be actively managed. Caching and prefetching help or hurt depending on layout (prefetching costs up to 7% on the unoptimized interface), and because x86 lacks a cheap consumer-cache invalidate, CC-NIC leans on buffer recycling — a workload that retains transmit buffers for retransmission falls back to slower standard allocation.
- Writer-homing assumes a single writer. The clean homing rule works for descriptors (one writer each), but packet buffers can be accessed arbitrarily by the application, so they are host-homed and rely on recycling and locality heuristics rather than a tidy placement rule.
- It is a poll-mode data-plane interface for packets. Benefits are largest for small packets and latency-sensitive, high-fan-out workloads; bulk large-packet transfers might still want a DMA-like path, which the paper leaves to future work with CXL.io and dedicated copy engines.
The evaluation
CC-NIC is modeled on the coherent UPI interconnect by using one socket of a dual-socket Intel server as the NIC, on Ice Lake (3× 11.2 GT/s UPI) and Sapphire Rapids (terabit UPI), compared against Intel E810 and Nvidia CX6 PCIe NICs and a naive port of the PCIe interface onto UPI, using loopback microbenchmarks and key-value-store and TCP-RPC applications.
How the host and NIC talk
| Property | PCIe NIC (today) | Naive port to coherence | CC-NIC (this work) |
|---|---|---|---|
| Signaling | |||
| Descriptor fetch | |||
| Data placement | |||
| Buffer management | |||
| Result |
Check yourself
Check your understanding
1. The paper shows that simply running today's PCIe NIC interface over a coherent link (UPI) performs 79% worse than CC-NIC. Predict why coherence alone does not deliver the benefit.
2. CC-NIC homes the transmit ring in the host and the receive ring in the NIC — always in the socket that WRITES each ring. Predict why this placement is fastest.
3. CC-NIC goes out of its way to RECYCLE buffers and to scatter pool addresses so prefetchers do not run ahead — deliberately limiting caching behaviors that normally speed things up. Predict why.
4. PCIe NICs batch many descriptors behind a single doorbell, trading latency for CPU efficiency. Predict why CC-NIC needs far less batching, and what that enables.
Sources
- CC-NIC: a Cache-Coherent Interface to the NIC (author PDF, UW) — ASPLOS 2024 (full text read).
- ACM DL: CC-NIC (DOI 10.1145/3617232.3624868) — published version.
- Author page (Brent Stephens, Utah) — project and paper links.