Toasty: Speeding Up Network I/O with Cache-Warm Buffers

Fast NICs deliver incoming packets straight into the CPU's cache so the software finds them without a slow trip to main memory, but under load the flood of new packets keeps evicting older ones before they are processed — and shrinking the buffer pool to fit the cache risks the NIC running out of buffers during a burst; this paper (IIT Bombay — Preeti, Nitish Bhat, Ashwin Kumar, Mythili Vutukuru) presents Toasty, a software-only fix that reuses recently-freed cache-warm buffers first (a stack, not a queue) and grows the pool only during bursts, lifting network throughput by up to 78 percent on commodity hardware.

The argument in one glance

Problem

Delivering packets into the cache backfires when they overflow it

To make networking fast, modern NICs use a feature like Intel's Data Direct I/O (DDIO) to DMA incoming packets straight into the CPU's last-level cache (LLC) rather than main memory. The idea is that the software processing a packet then finds it already in cache, avoiding a slow fetch from DRAM. This works beautifully when the set of packet buffers in active use fits within the cache. But under load it can backfire, through what prior work calls the leaky-DMA problem: when the working set of packet buffers is larger than the LLC, each incoming packet the NIC writes into cache evicts an older packet that the application has not finished processing yet. That evicted packet is pushed out to main memory, so when the application finally gets to it, it must be fetched back from DRAM — exactly the slow access DDIO was supposed to avoid. The natural fix is to use a smaller pool of packet buffers, small enough to fit in the cache, so nothing spills. But that creates a new danger: with only a few buffers, the NIC can run out of free buffers to DMA into when a burst of packets arrives faster than the application drains them, and packets are then dropped. So sizing the packet buffer pool becomes a dilemma with no good static answer.
What people did

Pick a buffer pool size, or add special hardware to manage the cache

Two kinds of response exist. One is to simply choose a pool size and live with the tradeoff: a small pool keeps buffers cache-warm for high throughput but risks dropping packets during bursts, while a large pool absorbs bursts but overflows the cache and triggers leaky DMA. The other is to change the hardware: prior work has proposed hardware-based mechanisms to manage which packet data occupies the cache, so that the cache is used well without forcing the software to make the buffer-pool tradeoff by hand.
Why it fell short

A fixed size cannot satisfy both goals, and hardware fixes are not deployable today

Neither response is satisfactory. Picking a fixed pool size forces a permanent compromise: whatever you choose is wrong for part of the traffic, because steady-state traffic wants a small cache-warm pool while bursts want a large one, and no single size is right for both — the tension between high throughput and resilience to packet loss is baked into the choice. Hardware-based solutions can break this tradeoff by managing the cache more cleverly, but they require changes to the NIC or CPU that are not present in commodity hardware, so they cannot be deployed on the machines datacenters actually run today. What was missing was a way to get both the throughput of a small cache-resident buffer set and the burst-resilience of a large pool, without new hardware — purely in software, on the NICs already in the field.
This paper's idea

Reuse warm buffers first with a stack, and grow the pool only during bursts

Toasty resolves the tradeoff in software, deployable on commodity NIC hardware, with two cooperating changes to how the packet buffer pool is managed. First, it manages the pool of free buffers as a LIFO stack rather than a FIFO queue. When the application finishes with a buffer and frees it, that buffer is the most recently touched and therefore the one most likely still warm in the cache. A FIFO queue would put it at the back and cycle through every other buffer before reusing it, so the whole pool gets touched and the working set is large; a LIFO stack hands that just-freed warm buffer straight back to the NIC for the next incoming packet, so in steady state the same small set of buffers is recirculated, staying cache-warm, while the rest of the pool sits idle. Second, Toasty adapts the number of buffers it populates into the NIC's hardware receive (RX) ring based on the incoming packet load and how fast the application is processing them. Under steady traffic it keeps only a small, cache-fitting set of buffers in play; when a burst arrives, it expands the set, drawing on the larger reserve pool so the NIC does not run out of buffers and drop packets. Together, these let Toasty recirculate a small working set of cache-warm buffers normally and fall back to a large pool during bursts. Implemented over the AF_XDP kernel-bypass framework, Toasty improves network throughput for a variety of network functions by up to 78 percent over AF_XDP's default buffer pool, matching a small cache-fitting pool's performance while being far more resilient to bursts.

Why this problem is worth solving

Cache behavior is the hidden determinant of high-speed network performance, and a feature meant to help can silently hurt when its assumptions break.

At the packet rates modern datacenters run, the software touching each packet is on a very tight time budget, and the single biggest swing in how long that takes is whether the packet’s data is in cache or in main memory. DDIO exists precisely to keep it in cache: by DMA-ing packets straight into the LLC, it lets the processing code hit cache instead of paying a DRAM access on every packet, which at line rate is the difference between keeping up and falling behind. But this benefit is conditional — it holds only while the data the NIC and the application are juggling fits in the cache. When it does not, DDIO does not just stop helping; it actively hurts, because the NIC’s writes evict useful, not-yet-processed packets, converting would-be cache hits into DRAM fetches and adding cache-eviction traffic on top. So the same mechanism is a large win or a real loss depending on a working-set condition that shifts with load. That makes managing cache warmth not a micro-optimization but a first-order performance lever for any high-speed networking system, and it makes the buffer pool — the thing that determines the working set — a critical knob. Getting it right is what lets a server actually realize the throughput its NIC and cores are capable of, which at datacenter scale translates into how much traffic each machine can carry.

The deeper insight is that the throughput-versus-resilience tradeoff is not fundamental but an artifact of treating the pool as one fixed thing, and separating steady state from bursts dissolves it.

The dilemma looks unavoidable if you think of the buffer pool as a single quantity that must be sized once: small for warmth, large for bursts, pick one. Toasty’s reframing is that these are two different needs at two different times, and you can serve each without compromising the other by controlling which buffers are actually in play moment to moment. In steady state, only a handful of buffers need to be circulating between the NIC and the application, so the trick is to make sure it is the same handful each time — which is exactly what reusing the most-recently-freed buffer accomplishes, since that buffer is the one still warm in cache. This turns a large pool into a small hot working set plus a cold reserve, at no cost, just by changing the reuse order from a queue to a stack. Then the reserve exists precisely for the moments the hot set is not enough: when a burst outpaces the application, more buffers are brought into play so the NIC never starves, and when the burst subsides the working set shrinks back. The elegance is that both properties come from software policy over the existing buffers and rings, with no hardware support, so the benefit is available on commodity NICs immediately. Recognizing that the warm working set and the burst reserve are separable, and that ordering plus adaptive ring-filling separates them, is what lets one system have both throughput and resilience.

What people did before, and where it broke

A large packet buffer pool. Provision plenty of buffers so the NIC never runs out during bursts. Where it broke. The working set exceeds the LLC, so incoming packets evict unprocessed ones (leaky DMA) and the application refetches them from DRAM, cutting throughput.
A small cache-fitting pool. Shrink the pool so all buffers stay in cache. Where it broke. With few buffers, a traffic burst can outrun the application and leave the NIC with no free buffers to DMA into, so packets are dropped.
Picking a fixed size for the tradeoff. Choose one pool size balancing throughput and burst resilience. Where it broke. Steady traffic and bursts want opposite sizes, so any fixed choice is wrong part of the time.
Hardware-based cache management. Change the NIC or CPU to control which packet data occupies the cache. Where it broke. It requires hardware not present in commodity machines, so it cannot be deployed on the servers datacenters run today.
The gap. No software-only approach gave both the throughput of a small cache-resident working set and the burst-resilience of a large pool on commodity NICs.

How Toasty works

From a fixed FIFO buffer pool to a stack of warm buffers with a burst reserve

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Keep fast networking fast by respecting the NIC-to-cache path. Toasty is fundamentally about the interaction between the NIC delivering packets and the CPU cache that processes them, ensuring packets stay in cache where the software can reach them cheaply. Designing the data path so that what the NIC delivers lands and stays usefully in the CPU cache, rather than thrashing it, is the same NIC-cache co-design concern the work on a cache-coherent interface to the NIC (CC-NIC) covered elsewhere on this site addresses, tightening how the NIC and cache cooperate. Toasty pursues it purely in software, by managing which buffers the NIC writes into.

Take control of what stays resident in the cache. Toasty’s core effect is to keep a small, hot working set of buffers in the cache and keep the rest out, rather than letting I/O traffic pollute the cache indiscriminately. Deliberately controlling which data occupies the limited cache, so the useful working set stays resident instead of being evicted, is the same cache-control instinct the work on taking control of your cache (Skip It) covered elsewhere on this site is built on, managing residency rather than leaving it to chance. Toasty achieves that control through buffer reuse order, not a cache directive, making it deployable without hardware support.

Provision for the steady state but absorb the bursts. Toasty runs lean when traffic is steady and expands only when a burst demands it, so it does not permanently pay for burst capacity it rarely needs. Sizing a resource for the common case while keeping headroom to absorb occasional bursts, rather than statically over-provisioning, is the same burst-adaptive allocation idea the work on a resource allocator for oversubscribing bursty workloads (Audible) covered elsewhere on this site pursues, matching capacity to demand over time. Toasty applies it to the NIC receive ring, growing the in-play buffer set during bursts and shrinking it back afterward.

Following a packet through Toasty

From steady traffic through a burst and back

A trace, drawn from the paper's abstract and the authors' description, of how Toasty manages buffers across steady traffic and a burst. The full paper was not available for this explainer.

Step 1 of 5

Steady traffic: a small warm set circulates

Under steady load, only a few buffers are in play. The NIC DMAs each incoming packet into a buffer that the application soon processes and frees, and that just-freed, cache-warm buffer is reused for the next packet, so the same small set stays resident in the LLC.

The application reads each packet from cache, not DRAM.

Why it works

Reusing the just-freed buffer keeps it cache-warm. A queue cycles through the whole pool before reuse → but a stack hands back the most-recently-freed buffer first → so the next packet lands in a buffer still resident in cache.
A small recirculated set avoids leaky DMA. A large touched working set overflows the LLC and evicts unprocessed packets → but circulating only a few buffers in steady state → so incoming packets no longer evict older ones to DRAM.
Adaptive ring-filling prevents burst drops. A small fixed pool starves the NIC during a burst → but growing the in-play buffers when load outpaces processing → so the NIC always has a free buffer and packets are not dropped.
Separating warm set from reserve breaks the tradeoff. One fixed size cannot be both small and large → but a hot working set plus a cold reserve, chosen by policy → so throughput and burst-resilience are obtained together.
Doing it in software makes it deployable now. Hardware cache-management fixes need new NICs or CPUs → but reordering reuse and adapting the ring in software → so the benefit runs on commodity hardware already in the field.

What it costs (trade-offs and scope)

It adds buffer-management logic to the fast path. Managing a stack and adapting the RX ring introduces software policy where a plain pool had none; the design keeps this cheap enough to net a large gain, but it is extra work on the per-packet path.
The adaptation must track load accurately. Growing and shrinking the in-play set depends on sensing the incoming load and the application’s processing rate; a poor estimate could expand too slowly (dropping burst packets) or too eagerly (enlarging the working set and re-triggering leaky DMA).
The warm-set benefit assumes prompt processing. Reuse stays warm because the application frees buffers soon after they are filled; a workload that holds buffers a long time before freeing them keeps a larger set in flight and gains less warmth.
It targets the receive path on DDIO-style systems. Toasty addresses packet reception into an LLC-DMA cache path via AF_XDP; systems without DDIO-like direct-cache DMA, or very different processing models, may see different effects.
This explainer is built from the abstract and public materials. The paper’s full text was not reachable; the mechanism here comes from the complete abstract and the authors’ open-source description, so the detailed adaptation algorithm, the evaluation setup, and the per-function results should be read in the full paper.

The comparison

Sizing the network packet buffer pool

Property	Large FIFO pool	Small cache-fitting pool	Toasty
Working set versus cache
Leaky DMA
Burst resilience
Reuse order
Deployment and result

Ways to size the network packet buffer pool. A large FIFO pool resists bursts but overflows the cache and triggers leaky DMA; a small cache-fitting pool stays warm but drops packets during bursts; and prior hardware fixes need non-commodity NICs. Toasty is software-only: it reuses the most-recently-freed (cache-warm) buffer first via a LIFO stack, recirculating a small warm working set in steady state, and adaptively grows the NIC receive ring during bursts to draw on a larger reserve — improving throughput by up to 78 percent over default AF_XDP while matching a small pool's performance with far better burst resilience. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s complete published abstract (obtained via Semantic Scholar) together with the authors’ open-source repository description (github.com/networkedsystemsIITB/toasty); the paper’s full text was not reachable at the time of writing (it is behind the ACM Digital Library with no open preprint), so this page is marked a draft. What is reported is what those authoritative sources state: modern NICs DMA packets into the last-level cache via DDIO to cut latency, but when the packet-buffer working set exceeds the LLC, the leaky-DMA problem arises — incoming packets evict older, unprocessed packets, forcing refetches from main memory; shrinking the pool to fit the cache risks the NIC running out of buffers during bursts, so pool sizing trades throughput against packet-loss resilience. Toasty is a software-only solution deployable on commodity NICs that manages the free-buffer pool as a LIFO stack (so the most-recently-freed, cache-warm buffer is reused first, recirculating a small warm working set) and adapts how many buffers it populates into the NIC RX ring based on incoming load and application processing rate (falling back to a larger pool during bursts). Implemented over AF_XDP, the abstract reports up to 78 percent higher throughput across a variety of network functions than AF_XDP’s default buffer pool, matching a small cache-fitting pool while being more burst-resilient. Beyond these claims — the detailed adaptation algorithm, the evaluation methodology, and the per-function results — the paper should be consulted. Background on DDIO, LLC, DMA, AF_XDP kernel-bypass, and NIC RX rings is standard context, and the walkthrough is an illustration at the abstract’s level, not a verbatim trace of the paper’s mechanism.

Background

Why fast networking keeps fighting over the last-level cache

It helps to see Toasty within a growing line of work on the boundary between the network card and the CPU cache. As network speeds outran the latency of main memory, keeping packet data in cache became essential to processing it at line rate, and features like DDIO were built to deliver packets directly into the cache. But the cache is small and shared, so the moment the traffic’s footprint exceeds it, the same feature that helped starts to hurt, evicting useful data and forcing memory accesses — the leaky-DMA problem. A cluster of recent research therefore studies how to keep the network’s working set cache-resident without starving other users of the cache or the NIC of buffers, through smarter buffer management, cache partitioning, and adaptive policies, some in hardware and some in software. The recurring theme is that a static configuration cannot serve both steady traffic and bursts, so the winning designs separate the common case from the exceptional one and adapt between them. This effort to make high-speed network I/O fast by carefully governing how packet data flows through the cache hierarchy is the same terrain the work on a tiered network buffer architecture for chiplet CPUs (TiNA) covered elsewhere on this site occupies, which attacks the very same leaky-DMA problem from the hardware side by tiering packets across chiplet caches. Toasty’s distinctive contribution is to solve it purely in software on commodity NICs, by reusing warm buffers first and sizing the receive ring to the load.

If you remember only three things

1. Modern NICs use DDIO to DMA incoming packets straight into the CPU's last-level cache, so the software processing each packet finds it in cache rather than paying a slow main-memory access — crucial at datacenter packet rates. But this only helps while the active packet buffers fit in the cache. When the working set is larger, the leaky-DMA problem strikes: each incoming packet the NIC writes into cache evicts an older packet the application has not finished with, pushing it to DRAM so it must be refetched. The obvious fix — a small pool that fits in cache — creates the opposite danger: with few buffers, a traffic burst can outrun the application and leave the NIC with no free buffer to DMA into, dropping packets. So sizing the buffer pool trades throughput against burst resilience, and prior fixes that break this tradeoff required new NIC or CPU hardware not present on commodity machines.
2. Toasty resolves the tradeoff purely in software on commodity NICs, with two cooperating changes. It manages the free-buffer pool as a LIFO stack rather than a FIFO queue: when the application frees a buffer, that buffer is the most-recently-touched and most likely still cache-warm, so handing it back first means the NIC and application recirculate the same small set of buffers in steady state, keeping them cache-resident while the rest of the pool sits idle. And it adapts how many buffers it populates into the NIC's hardware receive ring based on the incoming load and the application's processing rate: lean in steady state, expanding into the larger reserve pool during a burst so the NIC never runs out of buffers, then shrinking back. Together these give a small cache-warm working set normally and a large pool exactly when bursts demand it.
3. The key insight is that the throughput-versus-resilience tradeoff is not fundamental but an artifact of treating the pool as one fixed-size thing; the warm working set and the burst reserve are separable needs at different times, and reuse order plus adaptive ring-filling separates them — all in software, so the benefit runs on hardware already in the field. Implemented over the AF_XDP kernel-bypass framework, Toasty improves network throughput for a variety of network functions by up to 78 percent over AF_XDP's default buffer pool, matching a small cache-fitting pool's performance while being far more burst-resilient. The trade-offs: it adds buffer-management logic to the per-packet path, its adaptation must track load accurately, its warmth benefit assumes the application frees buffers promptly, and it targets the receive path on DDIO-style systems. This explainer is drawn from the complete abstract and the authors' open-source description, since the paper's full text was not reachable.

Check yourself

Check your understanding

1. What is the leaky-DMA problem?
2. Why can you not simply fix the leaky-DMA problem by using a small buffer pool?
3. What are Toasty's two changes to buffer management?
4. What did Toasty achieve, and why is being software-only significant?

Sources

The paper is “Toasty: Speeding Up Network I/O with Cache-Warm Buffers” by Preeti, Nitish Bhat, Ashwin Kumar, and Mythili Vutukuru (Indian Institute of Technology Bombay), published at ASPLOS 2026, DOI 10.1145/3779212.3790235. Code is available at github.com/networkedsystemsIITB/toasty.

This explainer is written from the paper’s complete published abstract (obtained via the Semantic Scholar API) together with the authors’ open-source repository description; the paper’s full text was not reachable, as it is behind the ACM Digital Library with no open preprint, so this page is marked a draft. The abstract states that modern NICs DMA packets into the LLC via DDIO, that the leaky-DMA problem arises when the packet-buffer working set exceeds the LLC (incoming packets evict unprocessed ones, forcing DRAM refetches), and that a small cache-fitting pool risks the NIC running out of buffers during bursts — a throughput-versus-resilience tradeoff. Toasty is a software-only solution for commodity NICs that manages the buffer pool as a LIFO stack (reusing the most-recently-freed cache-warm buffer first, recirculating a small warm working set) and adapts the number of buffers in the NIC RX ring to the incoming load and application processing rate (expanding during bursts). Implemented over AF_XDP, the abstract reports up to 78 percent higher throughput than AF_XDP’s default buffer pool across a variety of network functions, matching a small cache pool while being more burst-resilient. Background on DDIO, LLC, DMA, AF_XDP, and NIC RX rings is standard context. All quantitative claims are the paper’s own; the detailed algorithm and full evaluation should be confirmed in the paper.