TiNA: Tiered Network Buffer Architecture for Fast Networking in Chiplet-based CPUs

Siddharth Agarwal, Tianchen Wang, Jinghan Huang, Saksham Agarwal and Nam Sung Kim

Big server CPUs are now built from several chiplets in one package, which makes memory further away and more variable — bad news for microsecond networking that reads memory many times per packet — and a little-known mode called Sub-NUMA Clustering fixes the latency by pinning work to one chiplet, but its shrunken cache then chokes under packet bursts; this paper (UIUC, Cornell — Siddharth Agarwal, Saksham Agarwal, Nam Sung Kim, and colleagues) presents TiNA, an enhanced NIC and network stack that keeps packets in the fast local chiplet cache normally and spills only long bursts into other chiplets' caches, cutting mean latency by about a quarter.

The argument in one glance

Problem

Building a big CPU from chiplets pushes memory further away, hurting fast networking

Because a single monolithic die can no longer grow — reticle-size and yield limits, and the end of Moore's law — large server CPUs are now assembled from several chiplets in one package. Intel's Sapphire Rapids, for example, is up to four chiplets, each a small but complete CPU with its own cores, last-level cache (LLC) slices, DRAM controllers, and I/O lanes, stitched together by an on-package interconnect. By default the chiplets act as one big CPU: a core can reach any LLC slice or DRAM controller, and addresses are interleaved across all of them. That gives each core the full cache capacity and DRAM bandwidth, but it means most of a core's memory accesses land on another chiplet and must cross the on-package interconnect — each crossing adding tens of nanoseconds, several times the on-chip latency. The paper measures that only about one in four accesses stays local; the rest pay this cross-chiplet penalty, making memory latency both higher and far more variable even within a single socket. This is a serious problem for microsecond-scale networking, the kind used in datacenters with kernel-bypass stacks like DPDK, because processing a single packet requires many memory accesses, and the end-to-end latency is dominated by how fast those accesses complete. On a chiplet CPU, that fast path is no longer reliably fast.
What people did

Turn on Sub-NUMA Clustering to keep each core's work local to its chiplet

These CPUs offer a little-used mode, Sub-NUMA Clustering (SNC), that logically splits the CPU at chiplet boundaries so the operating system sees each chiplet as its own NUMA node. With SNC on, a core's memory allocations and accesses are confined to its local chiplet unless explicitly placed elsewhere, so they stay on-chip and avoid the cross-chiplet interconnect. For latency-sensitive packet processing, the appeal is obvious: keep the packets and the code touching them on one chiplet, and every access is a fast local one. The paper shows SNC delivers on this, cutting median and tail packet-processing latency substantially compared to the default non-SNC mode.
Why it fell short

SNC shrinks the cache to a quarter, so packet bursts overflow it and DCA fails

The catch is that confining a core to its own chiplet also confines it to that chiplet's resources: with SNC on, a packet-processing core gets only a quarter of the CPU's total LLC capacity and DRAM bandwidth. That is fine for light traffic but disastrous under bursts — periods, common in datacenters and lasting hundreds of microseconds, when the NIC receives packets at line rate. Modern NICs use Direct Cache Access (DCA), which DMA-writes incoming packets straight into specific LLC ways instead of DRAM, so the core finds each packet already in cache and processes it with low latency. DCA only works if the cache can hold the packets until the core reads them. With only a quarter of the LLC, a long burst overruns the DCA region: packets are written into cache but evicted before they are processed (a DMA leak, forcing a slow DRAM re-fetch), and the DCA data crowds out other useful data (a DMA bloat). The paper measures the reversal precisely: SNC beats non-SNC for bursts shorter than about 400 microseconds, cutting latency by up to 45 to 50 percent, but for longer bursts SNC becomes worse, raising tail latency by up to 50 percent, because its small cache makes DCA ineffective. So neither mode wins outright — SNC is best for short bursts, non-SNC for long ones — and you cannot know in advance which a given moment will bring.
This paper's idea

A tiered network buffer that spills only long bursts into other chiplets' caches

TiNA gets the benefit of SNC without its drawback by building a tiered network buffer across chiplets, using an enhanced NIC (TiNA-NIC) and network stack (TiNA-stack). The stack sets up two tiers of receive buffers where the NIC writes incoming packets: a Local-tier, backed by memory in the processing core's own chiplet and cached in that chiplet's DCA ways, and a Remote-tier, backed by and cached in a neighboring chiplet. Normally, packets go to the Local-tier, giving SNC's fast local access. But when a burst arrives that would overflow the local DCA region, TiNA-NIC diverts the excess into the Remote-tier: those packets are DMA-written into another chiplet's cache, which is farther away but keeps them in cache rather than spilling to DRAM. The NIC decides per-packet which tier to use, based on the measured size of the incoming traffic and an estimate — supplied by TiNA-stack — of how much room remains in each tier's DCA ways, and it steers packets to the right tier's cache ways using the NIC's existing Receiver-Side Scaling (RSS) mechanism. In effect, TiNA treats the local chiplet's cache as a fast first tier and other chiplets' caches as a larger, slightly-slower second tier, using the remote capacity only when a burst demands it. Across diverse network applications and traces, TiNA reduces mean latency by 25 percent and tail (99th-percentile) latency by 18 percent versus SNC, and by 28 percent and 22 percent versus non-SNC.

Why this problem is worth solving

Microsecond networking is the backbone of modern datacenter services, and it is exquisitely sensitive to exactly the memory latency that chiplet CPUs make worse.

In a datacenter, a huge fraction of work is short remote requests — a key-value lookup, a microservice call — whose whole round trip is measured in microseconds, and the software processing each packet is on the critical path. Kernel-bypass stacks like DPDK strip away operating-system overhead precisely so that this path is as short as possible, at which point what remains is dominated by memory accesses: the core must read packet descriptors, headers, and payloads, walk data structures, and update state, and each of these is a memory access whose latency directly adds to how long the packet takes. When those accesses hit cache, the packet flies through; when they miss to DRAM, or worse to a remote chiplet’s DRAM, the latency balloons. This is why Direct Cache Access exists at all — getting the packet into cache before the core needs it is one of the highest-leverage things a system can do for networking latency. A chiplet CPU threatens this by making memory both slower and more variable, and variability is especially damaging because tail latency, not average latency, is what service-level objectives are written against. So the problem is not a niche one: it sits under the performance of the interactive services that fill datacenters, on the hardware those datacenters are increasingly built from. Keeping fast networking fast as CPUs go multi-chiplet is worth real effort because so much depends on it.

The deeper insight is that the two available modes each solve half the problem, and the right design is not to choose between them but to combine their strengths at fine granularity.

The characterization reveals a genuine dilemma: SNC gives low latency by keeping everything local but shrinks the cache, so it wins for short bursts and loses for long ones; non-SNC keeps the full cache but scatters accesses across chiplets, so it does the reverse. A static choice is doomed, because real traffic mixes short and long bursts unpredictably, so whichever mode you pick is wrong part of the time. TiNA’s reframing is that locality and capacity are not a single either-or knob but two resources that can be blended: use the fast local cache as the default, and reach into the larger pool of other chiplets’ caches only for the overflow that would otherwise miss to DRAM. This turns the chiplet structure from a liability into a hierarchy — a small fast tier plus a larger slower tier — and the design problem becomes deciding, moment to moment, how much of each to use. The decision is well-posed because it has a clear trigger: a packet should go remote only when the local DCA region is about to overflow, since a remote cache hit is still far cheaper than a local cache miss to DRAM. Realizing this needs the NIC and the stack to cooperate, because the NIC places the packets but only the stack knows the cache occupancy, and it needs to be cheap enough to do at line rate. Recognizing that the two modes are endpoints of a tunable spectrum, and building the mechanism to move along it per burst, is what lets TiNA beat both.

What people did before, and where it broke

Default (non-SNC) mode. Treat all chiplets as one CPU with interleaved, full-capacity cache and DRAM. Where it broke. Most accesses cross the on-package interconnect, so memory latency is higher and more variable, hurting microsecond networking even though the full cache helps under bursts.
Sub-NUMA Clustering (SNC). Confine each core’s memory to its local chiplet for fast local access. Where it broke. It shrinks the packet-processing core’s cache to a quarter, so under long bursts DCA overflows and suffers DMA leaks and bloats, making it slower than non-SNC.
Direct Cache Access alone. DMA incoming packets into LLC to avoid DRAM latency. Where it broke. It only helps while the cache can hold the packets; a small SNC cache under a long burst evicts packets before they are processed, defeating the purpose.
Picking one mode statically. Choose SNC or non-SNC for the deployment. Where it broke. Traffic mixes short and long bursts unpredictably, so a fixed choice is wrong for a large fraction of the traffic.
The gap. No design combined SNC’s local-cache speed with non-SNC’s larger capacity by tiering across chiplet caches and steering only the overflow to remote caches when a burst demands it.

How TiNA works

From a single fixed mode to a per-burst tiered network buffer

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Keep microsecond packet processing fast on the NIC. TiNA is fundamentally about low-latency networking: getting incoming packets to the processing core as quickly as possible on a NIC-and-CPU system. Engineering the NIC and memory path so that packet processing stays in the microsecond regime, by controlling where incoming data lands relative to the core, is the same fast-networking goal the work on the benefits of general-purpose on-NIC memory (general-purpose on-NIC memory) covered elsewhere on this site pursues, using the NIC’s placement of data to cut networking latency. TiNA does this by choosing which chiplet’s cache each packet is delivered into.

Deliver packets straight into the cache, and manage that carefully. TiNA’s mechanism rests on Direct Cache Access — the NIC writing packets into the CPU’s cache rather than DRAM — and on managing the interaction between the NIC and the cache so DCA stays effective. Treating the boundary between the NIC and the CPU cache as a first-class thing to design, so that data the NIC delivers lands usefully in cache, is the same NIC-cache co-design idea the work on a cache-coherent interface to the NIC (CC-NIC) covered elsewhere on this site is built on, tightening how the NIC and the cache cooperate. TiNA extends that cooperation across chiplets, steering DMA into whichever chiplet’s cache has room.

Tier across near and far memory, spilling to the far tier under pressure. TiNA’s core structure is a fast local cache tier plus a larger, slightly-slower remote cache tier, using the remote one only when the local overflows. Organizing memory as a fast near tier and a larger far tier and adaptively moving data to the far tier only when the near one is under pressure is the same memory-tiering idea the work on an adaptive and lightweight memory tiering system (HybridTier) covered elsewhere on this site is built on, balancing capacity against latency across tiers. TiNA applies tiering not to DRAM and a far memory pool but to the caches of different chiplets, for network receive buffers.

Following a burst through TiNA

From a packet burst to low-latency processing across two cache tiers

A trace, drawn from the paper, of how TiNA handles incoming packets during light traffic and during a long burst.

Step 1 of 5

Light traffic arrives

The NIC receives packets at a modest rate. TiNA-NIC places them in the Local-tier, DMA-writing them into the processing core's own chiplet's DCA ways, so the core reads each from local cache at the lowest latency — the SNC benefit.

Normal traffic never leaves the fast local tier.

Why it works

Local placement keeps normal traffic at SNC speed. Cross-chiplet accesses are slow → but delivering light traffic into the local chiplet’s cache → so ordinary packets get the fast local-access latency SNC provides.
Remote cache beats a DRAM miss. A small local cache overflows under bursts → but spilling excess packets into another chiplet’s cache → so they stay a cache hit rather than falling all the way to DRAM.
Spilling only the overflow limits the remote penalty. Sending everything remote would pay cross-chiplet latency always → but diverting only the burst that would not fit locally → so the remote-tier cost is incurred only when it prevents a worse cost.
NIC-and-stack cooperation makes the decision correct. The NIC alone cannot see cache occupancy → but the stack estimating each tier’s free DCA ways and sharing it → so packets go remote exactly when the local tier is about to overflow.
Reusing RSS makes steering cheap at line rate. Directing packets to specific cache ways sounds expensive → but repurposing the NIC’s existing Receiver-Side Scaling → so the tier choice is enforced with a mechanism the hardware already has.

What it costs (trade-offs and scope)

Remote-tier hits still cost more than local ones. Packets diverted to another chiplet’s cache pay a cross-chiplet access penalty; TiNA wins because that is cheaper than a DRAM miss, but the remote tier is a fallback, not free, so its use is deliberately limited to overflow.
It needs an enhanced NIC and stack. TiNA requires changes to both the NIC (TiNA-NIC) and the networking stack (TiNA-stack), including the burst measurement and occupancy estimation, so it is not a pure software tweak on commodity hardware.
The occupancy estimate must be good and cheap. The decision to spill depends on estimating how full each tier’s DCA ways are; a poor or costly estimate would either spill too late (missing to DRAM anyway) or too eagerly (paying remote latency needlessly).
It targets chiplet CPUs with SNC and DCA. The design assumes a chiplet-based CPU exposing sub-NUMA clustering and a DCA-capable NIC; systems without these features, or with different interconnect characteristics, would need the approach adapted.
Gains depend on burst behavior and configuration. The reported 25 and 18 percent (versus SNC) and 28 and 22 percent (versus non-SNC) improvements are averages across the paper’s applications and traces; workloads with different burst distributions or cache pressure may see different results.

The comparison

Handling packet bursts on a chiplet CPU

Property	Non-SNC (default)	SNC	TiNA
Memory access latency
Cache capacity for packets
DCA under long bursts
Best for which traffic
Reported latency

Handling packet bursts on a chiplet-based CPU. Non-SNC keeps the full cache but scatters accesses across chiplets (good for long bursts, worse latency otherwise); SNC keeps accesses local but shrinks the cache to a quarter, so long bursts overflow Direct Cache Access. TiNA keeps packets in the fast local chiplet cache by default and spills only burst overflow into a neighboring chiplet's cache, keeping DCA effective without the cross-chiplet penalty on normal traffic — reducing mean and tail latency by 25 and 18 percent versus SNC, and 28 and 22 percent versus non-SNC. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ hosted copy of the ASPLOS 2026 paper, at saksham.web.illinois.edu). The described system is the paper’s: on chiplet-based CPUs (the paper uses Intel Sapphire Rapids), the default mode interleaves memory across chiplets so most accesses cross the on-package interconnect, raising latency and variance for microsecond-scale networking; Sub-NUMA Clustering (SNC) confines a core to its local chiplet for lower latency but reduces its LLC and DRAM to a quarter, which the paper shows helps for bursts shorter than about 400 microseconds (cutting latency up to 45 to 50 percent) but hurts for longer bursts (raising tail latency up to 50 percent) because Direct Cache Access overflows via DMA leaks and bloats. TiNA (TiNA-NIC plus TiNA-stack) builds a tiered network buffer with a Local-tier and a Remote-tier backed and cached in the local and a neighboring chiplet; the NIC places packets by tier based on measured burst size and estimated DCA-way availability (from the stack), steering via Receiver-Side Scaling. The reported results — reducing mean and 99th-percentile latency by 25 and 18 percent versus SNC and 28 and 22 percent versus non-SNC across diverse applications and traces — are from the paper’s evaluation. Background on chiplet CPUs, NUMA, DCA/DDIO, DPDK kernel-bypass networking, and RSS is standard context. The step-by-step walkthrough illustrates the described mechanism. Exact hardware details, the placement policy, and full results should be read from the paper.

Background

Why fast systems keep fighting the growing distance to memory

It helps to see TiNA within the long struggle to keep data close to where it is computed as machines get bigger and more disaggregated. For decades, performance has hinged on the memory hierarchy — registers, caches, DRAM — and on keeping the working set in the fastest tier that can hold it. As systems scale out, new levels keep appearing and the distances grow: multi-socket NUMA, then sub-socket effects within chiplet packages, and now pooled and disaggregated memory reached over fabrics. Each new level brings the same tension between capacity and latency, and the same recurring answer: treat the levels as a tier, put the hot or latency-critical data in the near tier, and reach into the far tier only when necessary. Networking is an acute case because its latency budget is tiny and its data — arriving packets — must be delivered into the hierarchy at just the right level, which is why features like Direct Cache Access exist. Chiplet CPUs add a twist by making the cache itself non-uniform, so even a cache hit can be near or far. This effort to preserve low latency as the memory hierarchy grows more levels and more distance, by tiering data intelligently across them, is the same terrain the work on workload-aware page-table management for NUMA servers (WASP) covered elsewhere on this site occupies, coping with non-uniform memory within a machine. TiNA’s specific contribution is to turn the chiplet cache non-uniformity into a tiered network buffer, so fast networking survives the move to chiplet CPUs.

If you remember only three things

1. Large server CPUs are increasingly built from several chiplets in one package, because a single die can no longer grow. By default the chiplets act as one CPU with cache and DRAM interleaved across them, so most of a core's memory accesses cross the on-package interconnect, adding tens of nanoseconds and much variance — a problem for microsecond-scale datacenter networking, where processing each packet takes many memory accesses on the critical path. A little-known mode, Sub-NUMA Clustering (SNC), confines each core to its local chiplet for fast local access, which the paper shows cuts packet-processing latency substantially for short bursts. But SNC also shrinks the core's cache to a quarter of the CPU's total, so during long bursts (hundreds of microseconds, common in datacenters) Direct Cache Access — the NIC writing packets straight into cache to avoid DRAM latency — overflows, suffering DMA leaks and bloats, and SNC becomes slower than the default. Neither mode wins for all traffic.
2. TiNA gets SNC's benefit without its drawback by building a tiered network buffer across chiplets, using an enhanced NIC (TiNA-NIC) and stack (TiNA-stack). It sets up a Local-tier of receive buffers backed and cached in the processing core's own chiplet and a Remote-tier backed and cached in a neighboring chiplet. Normal traffic goes to the Local-tier for fast local access; when a burst would overflow the local Direct Cache Access region, the NIC diverts the excess to the Remote-tier, writing those packets into another chiplet's cache — farther for the core to reach but still a cache hit, avoiding a costly DRAM miss. The NIC decides per packet based on the measured burst size and the stack's estimate of each tier's remaining cache room, and it steers packets to the right tier's cache ways using the NIC's existing Receiver-Side Scaling.
3. The key insight is that SNC and non-SNC are two endpoints — locality versus capacity — and the right design blends them per burst rather than choosing one: use the fast local cache as a first tier and other chiplets' caches as a larger, slightly-slower second tier, spilling only the overflow that would otherwise miss to DRAM. Across diverse applications and traces, TiNA reduces mean and 99th-percentile latency by 25 and 18 percent versus SNC and 28 and 22 percent versus non-SNC. The trade-offs: remote-tier hits still cost more than local ones (so they are limited to overflow), it requires an enhanced NIC and stack with a good, cheap cache-occupancy estimate, it assumes a chiplet CPU with SNC and a DCA-capable NIC, and the gains depend on the traffic's burst behavior.

Check yourself

Check your understanding

1. Why do chiplet-based CPUs create a problem for microsecond-scale networking?
2. What does Sub-NUMA Clustering (SNC) do, and why does it backfire under long packet bursts?
3. What is TiNA's core idea?
4. What did TiNA achieve, and what is the trade-off of its remote tier?

Sources

The paper is “TiNA: Tiered Network Buffer Architecture for Fast Networking in Chiplet-based CPUs” by Siddharth Agarwal, Tianchen Wang, Jinghan Huang, Saksham Agarwal, and Nam Sung Kim (University of Illinois Urbana-Champaign, with Cornell University), published at ASPLOS 2026, DOI 10.1145/3760250.3762224.

This explainer is written from the paper’s full text (an author-hosted copy at saksham.web.illinois.edu). On chiplet-based CPUs, the default mode interleaves memory across chiplets, so most accesses cross the on-package interconnect and raise latency and variance for microsecond-scale networking; Sub-NUMA Clustering confines a core to its local chiplet for lower latency but reduces its cache and DRAM to a quarter, which helps short bursts but hurts long ones because Direct Cache Access overflows (DMA leaks and bloats). TiNA (TiNA-NIC plus TiNA-stack) builds a tiered network buffer with a Local-tier and a Remote-tier cached in the local and a neighboring chiplet; the NIC places packets per tier based on measured burst size and the stack’s estimate of DCA-way availability, steering via Receiver-Side Scaling, and spilling to the remote tier only for burst overflow. The paper reports reducing mean and 99th-percentile latency by 25 and 18 percent versus SNC and 28 and 22 percent versus non-SNC across diverse applications and traces. Background on chiplet CPUs, NUMA, DCA, DPDK, and RSS is standard context. The numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.