An accelerator that wants to talk to the network has three bad options — go through the host processor and waste its cycles, rebuild an entire network stack in its own silicon, or sit between the card and the wire and lose access to everything the card can do; this paper (NVIDIA and Technion) takes a fourth route, putting a small hardware module on the accelerator that drives a commodity network card directly, and makes it fit by noticing that control structures which exist only to be read over a bus need not be stored at all — they can be manufactured at the moment the card asks for them, shrinking 85 megabytes to under a megabyte.
The argument in one glance
-
Problem
Accelerators need the network, and the network is complicated
Two pressures push accelerators towards the network at once. Datacenters want to pool them — a rack of them shared by whoever needs one, reached remotely rather than sitting in a particular machine. And network speeds have outrun processor speeds, so the packet processing that used to run on the host has to be done by something else. Both require an accelerator that can send and receive on its own, and the network is not a wire but a stack of protocols with a great deal of machinery behind it.
-
What people did
Three ways to attach an accelerator to a network
The established designs are: let the host processor mediate, so the accelerator receives data the processor has already fetched; put a complete network stack into the accelerator itself, which is what network-attached reconfigurable cards do; or place the accelerator between the card and the wire, so packets pass through it on their way in and out. Each works, and each is deployed commercially.
-
Why it fell short
Each gives up one of three things you want
Mediating through the processor burns the very cycles that were meant to be saved. Building the stack into the accelerator consumes silicon that was supposed to hold the accelerator, and reimplements what the network card already does well. Sitting between card and wire keeps the silicon cost but forfeits most of the card's offloads, since packets have been intercepted before the card could apply them. The three designs occupy three corners of a trade-off and nobody had the middle.
-
This paper's idea
Let the accelerator drive the card, the way a driver does
This paper (Eran, Fudim, Malka, Shalom, Cohen, Hermony, Levi, Liss and Silberstein) puts a hardware module on the accelerator that operates an unmodified commodity card over its ordinary control interface — the same interface the operating system driver uses, so the same offloads become available. Two things make it practical: only the performance-critical half of a driver goes into hardware, and the control structures the card expects to read are generated on demand rather than stored.
Why this problem is worth solving
The value at stake is not connectivity but everything a modern card does beyond it.
A
modern network card is a substantial processor in its own right. It implements a reliable
transport in hardware, so two machines can move data into each other’s memory without either
processor being involved. It wraps and unwraps tunnel headers for virtual networks, computes
and checks packet checksums, spreads incoming traffic across processor cores by hashing packet
headers, steers particular flows to particular destinations, and shapes bandwidth so one
tenant cannot starve another. Using any of this means driving the card the way an operating
system does — writing descriptors into ring buffers that say where data lives and what should
happen to it, ringing a doorbell register to tell the card there is work, and reading
completion entries the card writes back. That protocol is the interface, and an accelerator
that cannot speak it gets a wire and nothing else.
The trade-off the paper draws is worth stating precisely, because everything follows from it.
THERE
are three costs and the existing designs each pay one in full. SILICON — networking logic on
the accelerator die is area not spent on the accelerator, and reimplementing a network stack
in reconfigurable logic is expensive. PROCESSOR TIME — if the host is in the path, the cycles
saved by offloading are spent again on moving data. OFFLOAD ACCESS — the card’s hardware
features only apply if traffic passes through the card in the expected way, and interrupting
that path forfeits them. Mediating through the host pays in processor time; building the
stack into the accelerator pays in silicon; sitting between the card and the wire pays in
offloads. There is also a subtler version of the last one, since offloads chain in a fixed
order and a task that cannot be offloaded breaks everything after it in the chain.
What people did before, and where it broke
-
Host-mediated networking. The processor runs the network stack and hands data to the accelerator, sometimes with the data path optimised so transfers land directly in accelerator memory.
-
The gap. The host stays in the control path, so its cycles are consumed on exactly the work that motivated the accelerator, and the accelerator cannot be pooled independently of a machine willing to babysit it.
-
A complete network stack in the accelerator. Build the physical layer, the protocol processing, and everything above it into the accelerator itself, which is what network-attached reconfigurable cards do.
-
The gap. Silicon. The stack occupies area that was supposed to hold the accelerator, and it reimplements — usually less well — what a commodity card already does in dedicated hardware.
-
Sitting between the card and the wire. Put the accelerator in the path so packets flow through it, reusing parts of the card for talking to the host.
-
The gap. The card’s offloads are largely unavailable, because the accelerator sees traffic at a point where they have not been or cannot be applied. Anything the accelerator needs done before its own processing has to be built into the accelerator too.
-
Porting a driver into hardware directly. The obvious way to let an accelerator drive a card is to reimplement the driver in logic.
-
The gap. Drivers are large, intricate, and specific to a device. A verbatim port would be hard to build, hard to maintain, and would occupy far more area than the accelerator can spare — which is why the partitioning question matters more than the idea.
How an accelerator ends up driving the card
From an accelerator with no network to one using the card's full feature set
-
-
new
-
new
-
new
-
new
-
new
The idea
Speak the card’s existing language instead of replacing it. The design decision that opens everything up is refusing to build a network stack. The card already implements one, in dedicated silicon, better than reconfigurable logic could. What the accelerator lacks is not capability but the ability to ask for it, and asking is a matter of writing the right control structures over the bus. So the accelerator gets a module that does exactly what a driver does, and immediately inherits the card’s reliable transport, its tunnel handling, its steering and its shaping. The subtler benefit is about ordering: card offloads chain in a fixed sequence, and traditionally a step that cannot be offloaded breaks every step after it, which is why an accelerator placed between the card and the wire loses so much. Here the card can hand a packet to the accelerator partway through the chain and take it back, so accelerator processing interleaves with offloads rather than terminating them. Reconsidering the shape of the host-to-card interface rather than accepting it is the same instinct as the cache-coherent card interface covered elsewhere on this site, which redesigns the descriptor protocol once the bus underneath it changes.
A driver is mostly control plane, and control plane does not need to be hardware. Porting a driver verbatim into logic would be enormous, because the bulk of a driver is setup — create queues, register memory, configure features, handle errors and reconfiguration. All of that is infrequent and complicated, which is the worst combination for hardware. The data plane is the opposite: simple, repetitive, and on the critical path. So the split is drawn there. Ordinary host software does the setup through the usual interfaces, and the hardware module handles only posting and completing packets. This also keeps the result compatible with existing software, since the control side is still a normal driver talking to a normal card. Moving the bookkeeping to wherever it is cheapest to hold, rather than putting it where the data is, is the same restructuring as the disaggregated memory work covered elsewhere on this site, which strips per-client state out of the memory device and pushes it to the clients.
A structure that exists only to be read across a bus does not have to exist. This is the observation that makes the whole thing fit. A card learns what to transmit by reading descriptors from rings in memory, and with many queues at high speed those rings total tens of megabytes — the analysis here reaches 85 megabytes, against 10 available on the prototype’s chip. Storing them in the accelerator’s external memory would work but would contend with the accelerator’s own traffic, which is exactly the interference the design is trying to avoid. The way out is that these reads arrive as bus transactions, and a bus transaction can be answered by logic as easily as by memory. The module therefore synthesises each descriptor when the card asks for it, in the format the card expects, from a compact internal representation — a transmit descriptor shrinks from 64 bytes to 8, a completion entry from 64 to 15. A hash-based address translation then lets many queues share one small pool of real descriptors, so what is stored scales with traffic in flight rather than with the number of queues. Together these take 85 megabytes to under one, a factor of about 105. Using memory the card already has rather than moving data to where software expects it is the same reframing as the on-card memory work covered elsewhere on this site.
Answering a read instead of storing an answer
How a 64-megabyte descriptor ring becomes 32 kilobytes
Following one transmit descriptor from the card's request to the packet leaving.
Step 1 of 5
The card asks for the next descriptor
Having been told there is work waiting, the card issues a read across the bus for the descriptor at a particular position in a particular queue's ring — exactly as it would if a processor had written it there.
From the card's side nothing is unusual. It is following the same protocol it follows with an operating system driver, which is the entire reason the accelerator gets access to the card's features.
Why it works
- The card’s features come free because the protocol is unchanged. Driving the card through its ordinary interface → traffic reaches it the way it expects → reliable remote transfers, tunnel handling, steering and shaping all apply to accelerator traffic with no accelerator logic at all.
- Interleaving beats interception. Letting the card hand a packet to the accelerator partway through its offload chain and take it back → accelerator processing no longer terminates the chain → offloads can run both before and after the accelerator, which a bump-in-the-wire design cannot do.
- Splitting at the control-plane boundary keeps hardware small. Leaving setup and configuration in host software and putting only packet posting and completion in logic → the complicated, infrequent majority of a driver never becomes hardware → the module is comparable in area to alternatives that offer far less.
- Bus reads can be answered by logic. Generating each descriptor on demand in the card’s format from a compact internal form → 64-byte structures need only 8 bytes of real storage → the ring stops being a memory cost and becomes a computation.
- Sharing storage across queues decouples space from scale. Translating per-queue addresses into one pool sized by packets in flight → adding queues no longer adds megabytes → 85 megabytes of structures fit in under one, on a chip with ten in total.
What it costs (trade-offs and scope)
- Latency is slightly worse than intercepting the wire. Packets now traverse the card and the bus rather than being caught in transit. The paper argues the added microsecond is worth the card’s full feature set, which is a judgement rather than a measurement.
- The bus is now in the path, and it is not free. Control traffic shares the link with data, so line rate is not achievable for every packet size. The model shows about 95 percent of Ethernet line rate at moderate packet sizes and full rate at the prototype’s speed.
- The hardware module is vendor-specific. It speaks one manufacturer’s control structures, and a layer converts between those and its internal format. Another card means rewriting that layer, though the paper argues the architecture carries over.
- The control plane still needs a host. Setup, queue creation and configuration remain in ordinary software, so an accelerator is autonomous during operation but not at initialisation.
- The prototype is limited by its platform. It is built on a reconfigurable card whose link runs at tens of gigabits, and scaling to the fastest links is argued rather than shown.
The evaluation
| Property | What was measured | The result | What it tells you |
|---|---|---|---|
| Memory for control structures | |||
| Where the compression comes from | |||
| Inline acceleration of a real problem | |||
| Isolating tenants without accelerator logic | |||
| Throughput and latency |
Check yourself
Check your understanding
1. Why does placing an accelerator between the network card and the wire cost access to the card's offloads?
2. Why is a driver split at the control-plane boundary rather than ported into hardware wholesale?
3. How can descriptor rings that would occupy 85 megabytes fit on a chip with 10?
4. Predict what limits throughput once the accelerator drives the card over the bus rather than intercepting the wire.
Sources
FlexDriver: A Network Driver for Your Accelerator. Haggai Eran, Maxim Fudim, Gabi Malka, Gal Shalom, Noam Cohen, Amit Hermony, Dotan Levi, Liran Liss, Mark Silberstein. ASPLOS 2022, pages 1115 to 1129. DOI 10.1145/3503222.3507776. Full text read from the authors’ copy hosted at the Technion. Note that some indexed copies of the abstract render the compression factor as 5 times; the paper states two orders of magnitude and computes 105 times for its worked example.