PagedAttention fixed KV-cache fragmentation by building demand paging in user space — but that made the cache non-contiguous in virtual memory, forcing every attention kernel to be rewritten (paged kernels run 12–42% slower, and FlashAttention-3 shipped with no paging support at all) and every serving framework to duplicate the OS's address translation with Block-Tables — so vAttention asks why the application is doing the memory system's job, reserves the KV cache as contiguous VIRTUAL tensors, and maps physical pages on demand through CUDA's virtual-memory APIs, hiding allocation latency behind decode's one-token-per-iteration predictability and shrinking pages to 64 KB via an open-source driver extension — unmodified kernels work out of the box (FlashAttention-3 on day one, 1.26–1.5× over paged FlashAttention-2), decode beats vLLM by up to 1.99×, and end-to-end serving gains up to 1.23×.
The argument in one glance
-
Problem
A cache that grows one token at a time, to an unknown size
LLM serving throughput lives and dies by batch size, and batch size is capped by GPU memory — most of which goes to the KV cache, the stored key/value vectors of every token processed so far. Two properties make allocating it miserable: it grows by exactly one token per request per iteration, and nobody knows a request's final length in advance. Early systems (Orca, FasterTransformer) reserved each request's worst case — the model's maximum context length, 200K tokens for Yi-34B — while real chat requests average ~415 decode tokens. The internal fragmentation strangled batch sizes.
-
What people did
PagedAttention: demand paging, rebuilt in user space
vLLM's PagedAttention borrowed the OS playbook: split the KV cache into fixed-size blocks, allocate blocks on demand as tokens accumulate, track which blocks belong to which request. Fragmentation nearly vanished, batch sizes soared, and the approach became the industry default — TensorRT-LLM, HuggingFace TGI, LightLLM all adopted it.
-
Why it fell short
The cure changed the cache's shape, and everything downstream paid
Dynamically allocated blocks land wherever the allocator puts them — so the KV cache became NON-CONTIGUOUS in virtual memory, breaking the assumption every attention kernel is written against. Consequences cascade: every kernel must be rewritten to chase block pointers (paged prefill kernels run 37% slower in FlashAttention-2, 42% in FlashInfer; vLLM's own paged kernel is up to 2.8× slower than FlashAttention-2, and 1.9× sensitive to block size); cutting-edge kernels ship without paging support at all (FlashAttention-3, cuDNN-9 at release — early FlashAttention paging attempts failed unit tests); and the serving framework must maintain Block-Tables mapping requests to scattered blocks — user-space page tables that once cost vLLM 30% of decode latency in CPU preparation. The framework is doing, badly and repeatedly, what the OS and GPU hardware already do.
-
This paper's idea
Keep virtual contiguity; defer only the physical commitment
vAttention (Microsoft Research India) observes that fragmentation is a PHYSICAL memory problem and contiguity a VIRTUAL memory property — and the two can be separated exactly as real demand paging separates them. Using CUDA's virtual memory management APIs, it reserves each layer's K and V caches as huge contiguous virtual tensors up front (virtual address space is effectively free — 128 TB per process) and maps physical pages on demand as tokens arrive. Unmodified attention kernels see the contiguous tensors they were written for; no Block-Tables exist. Two LLM-specific fixes make it fast: allocation latency hides behind decode's perfect predictability (a background thread maps next iteration's pages during the current one), and a modified open-source NVIDIA driver adds 64–256 KB page-groups where stock CUDA offers only 2 MB. Results: decode up to 1.99× over vLLM's kernel, end-to-end serving up to 1.18×/1.23× over paged FlashAttention-2/FlashInfer — and FlashAttention-3 works on day one, yielding 1.26–1.5×.
Why this problem is worth solving
Serving is where LLM economics concentrate, and the
the per-request store of key and value
vectors for every token processed so far, reused each
iteration so attention need not recompute them. It is the
dominant consumer of GPU memory in inference — and it grows
by one token per iteration to a length nobody knows in
advance, which is what makes allocating it hard
vLLM’s user-space answer:
split the KV cache into fixed-size blocks allocated on
demand, with the serving framework tracking each request’s
scattered blocks in Block-Tables that are handed to specially
rewritten attention kernels. It solved fragmentation so well
it became the de facto standard across the industry’s serving
stacks
The diagnosis is architectural, not incidental: PagedAttention
implements
the OS technique of allocating
physical memory only when it is actually needed, while
programs see stable, contiguous virtual addresses — the
separation is maintained by page tables and translated
invisibly by hardware on every access
What people did before, and where it broke
- Static worst-case reservation (Orca, FasterTransformer). One contiguous tensor per request sized at maximum context length. Kernels are simple and fast; memory is catastrophic — a 200K-token reservation against a 415-token average conversation strands almost everything, capping batch size and throughput.
- PagedAttention (vLLM, then everyone). Near-perfect fragmentation control, at the price of the non-contiguity cascade: kernel rewrites (7–13% more instructions executed, register spilling from cached page indices, L1-cache sensitivity to block size), framework-side memory management in the critical path, and a permanent porting treadmill as attention-kernel research advances — production systems falling behind the state of the art is a measured outcome, not a hypothetical (vLLM’s paged kernel vs FlashAttention-2: up to 2.8×).
- The GPU runtime’s default (cudaMalloc). Allocates virtual and physical memory together — reservation-based, no demand paging. This conflation is the real root cause: with cudaMalloc as the only interface, dynamic physical allocation NECESSARILY meant scattered virtual blocks.
- CUDA unified memory. Page-fault-driven migration could oversubscribe, but fault handling costs far too much for a serving hot path (the paper’s discussion, and a companion page on this site, treat UM’s fault tax in depth).
How vAttention works
Virtual tensors, physically filled on demand
-
Reserve virtual, defer physical new
At startup, vAttention reserves 2×N contiguous VIRTUAL tensors per worker (K and V per layer), each sized for the maximum batch at maximum context length. For Yi-34B that is 12 TB of virtual address space — irrelevant, since a 64-bit process has 128 TB. No physical page backs any of it yet. Request reqId's sub-tensor sits at offset reqId × S: pure arithmetic, no tables.
-
CUDA VMM APIs (the mechanism that was already there) new
Recent CUDA exposes what OSes always had: cuMemAddressReserve (virtual only), cuMemCreate (physical only), cuMemMap + cuMemSetAccess (bind them). vAttention extends PyTorch's caching allocator with "virtual tensors" allocated through these APIs instead of cudaMalloc, which always couples the two.
-
The serving-framework contract: one call per iteration new
The framework calls step(seq_lens) before each iteration; vAttention maps whatever new page-groups each active request needs and returns. alloc_reqid/free_reqid manage request slots. Attention kernels receive ordinary contiguous tensors — FlashAttention-2, FlashInfer, and FlashAttention-3 run UNMODIFIED; continuous batching's holes are handled by FlashAttention's existing cache_batch_idx argument, no Block-Tables anywhere.
-
Hiding allocation latency (decode) new
Mapping pages via CUDA VMM costs ~40 microseconds per call — 5 ms to grow one Yi-34B request, intolerable synchronously. But decode demand is PERFECTLY predictable: one token per request per iteration. A background thread maps iteration i's pages while iteration i−1 computes (iterations take tens to hundreds of milliseconds — ample cover). Measured: even worst-case 2 MB-page allocation vanishes from the latency timeline; synchronous allocation shows 5–15 ms spikes.
-
Deferred reclamation + eager allocation (prefill) new
When a request completes, its mapped pages are NOT unmapped — the reqId and its already-backed tensor go to the next arriving request, which usually needs zero critical-path allocation for its prefill. One spare reqId is kept eagerly pre-mapped as a buffer. Reclamation triggers only when cached pages dip below a threshold. Ablation: kills the 1.03–1.15× synchronous-prefill overhead entirely.
-
64 KB page-groups (a driver patch where CUDA said no) new
Stock CUDA VMM allocates only 2 MB pages — real internal fragmentation at large batch. The VMM implementation is closed-source, but NVIDIA's unified-memory driver code is open: the authors implemented parallel vMem* APIs there supporting 64/128/256 KB page-groups. Measured: no attention-kernel slowdown at 64 KB (no TLB-thrashing evidence), and allocation bandwidth of 7.6 GB/s per GPU — an order of magnitude above the ≤750 MB/s any model/batch combination actually demands.
The idea
Fragmentation is physical; contiguity is virtual; only a conflated allocator forces you to choose. PagedAttention’s whole cascade of costs flows from one inherited assumption: that allocating memory dynamically means the addresses the kernel sees are scattered. That is true of cudaMalloc, which welds virtual and physical allocation together — and false of memory systems generally, whose entire design separates the two. vAttention’s insight is to relocate the problem to the layer built for it: reserve the shape (virtual, contiguous, worst-case-sized, free) once, and let the substance (physical pages) arrive on demand. Every downstream cost then simply never exists — kernels need no rewrite because the tensor looks exactly as their authors assumed, Block-Tables need no preparation because offset arithmetic replaces them, and address translation happens where it always did, in hardware.
The workload’s predictability funds the mechanism’s latency. The honest objection to CUDA VMM is cost: a kernel-crossing API on the critical path of a millisecond-scale serving loop. The paper’s two workload measurements dissolve it. Decode grows every request by exactly one token per iteration, so demand for iteration i is knowable during iteration i−1 — allocation overlaps compute behind a background thread, and iterations are three orders of magnitude longer than a mapping call. And total allocation bandwidth saturates with batch size at ≤750 MB/s — a trickle against the 7.6 GB/s the patched driver sustains even at 64 KB granularity. Prefill, the one bursty moment, is served by recycling: a completed request’s still-mapped pages are handed wholesale to the next arrival. The same one-token-per-iteration regularity this site has seen fund prefetching elsewhere funds allocation here.
Portability is the compounding dividend. The performance numbers (1.1–1.2× end-to-end) are solid but not the headline. The headline is structural: attention-kernel research moves fast, paged variants lag by months or forever, and every paged port is new code with new bugs in the hottest kernel in the industry. vAttention consumes new kernels the day they ship — demonstrated with FlashAttention-3, which launched Hopper-optimized with no paging support and ran under vAttention out of the box for 1.26–1.5× over paged FlashAttention-2. A memory-management design choice turns into a permanent free ride on the kernel community’s roadmap.
A request’s memory, iteration by iteration
Serving Yi-34B with vAttention on two A100s
Virtual tensor = contiguous per-layer K (or V) cache for the whole batch, reserved once. Page-group = unit of physical mapping (64 KB–2 MB). step() = the per-iteration API call. Numbers from the paper's evaluation.
Step 1 of 6
Startup: 12 terabytes that cost nothing
vAttention reserves 120 virtual tensors (K and V for 60 layers), each sized for 500 requests × 200K tokens — 12 TB of virtual address space against the process's 128 TB. Zero physical pages are committed. A pool of physical page-groups is pre-created, unmapped, waiting.
virtual memory is the one resource worst-case sizing cannot waste request reqId's slice = offset reqId × S — no tables, ever
Why it works
- Virtual address space is five orders of magnitude larger than GPU memory → worst-case-sized contiguous reservations are free in the only currency that is abundant → kernels keep the layout assumption they were all written against, and fragmentation moves to physical pages where demand mapping dissolves it.
- Address translation already exists in hardware, maintained by the OS/driver → replacing Block-Tables with page tables deletes user-space translation from both the GPU kernels (7–13% instruction overhead, register pressure) and the CPU runtime (Block-Table preparation) → the framework shrinks instead of growing.
- Decode adds exactly one token per request per iteration → memory demand is knowable one iteration early, and iterations (tens to hundreds of ms) dwarf mapping calls (tens of microseconds) → a background thread hides all allocation latency; measured timelines are flat.
- Token generation rate saturates with batch size → required allocation bandwidth caps at ≤750 MB/s → even 64 KB page-groups (7.6 GB/s) leave an order-of-magnitude margin, so fine granularity costs nothing it needs.
- Completed requests’ pages are exactly the right size and place for successors → deferred reclamation turns most prefill allocation into reuse → the one bursty allocation moment usually touches no CUDA API at all.
What it costs
- A patched driver for small pages. The 64–256 KB page-groups live in modified open-source NVIDIA driver code; stock CUDA gives only 2 MB. Deployments unwilling to run a patched driver keep 2 MB granularity and its fragmentation at large batch sizes.
- Worst-case virtual reservations are still bookkeeping. Max batch size must be configured up front, and each worker’s virtual footprint (terabytes) is harmless but unusual — tooling and debuggers may be surprised.
- Physical-page pools and caching need headroom. Deferred reclamation holds memory that an admission-controlled system might want back instantly; the threshold policy is a knob, and KV-cache swapping to CPU memory is explicitly future work.
- Prediction covers decode, not everything. Prefill bursts lean on recycling and eager pre-mapping — heuristics, not guarantees; a cold start with many long prompts and nothing to recycle pays synchronous mapping (measured up to 1.15× prefill overhead without the optimizations).
- Scope of the win. Where paged kernels are already well-tuned and contexts short, vAttention ties rather than wins — its decode matches FA2_Paged rather than beating it; the throughput gains concentrate in long contexts, and the strongest argument is portability rather than raw speed.
The evaluation
Three models across one and two A100s (plus H100s for portability), prefill and decode phases isolated then combined in dynamic traces, four kernel back-ends compared inside one common serving framework (vLLM v0.2.7) with each baseline at its best-tuned block size, and ablations for every optimization.
Three answers to KV-cache growth
| Property | Static worst-case reservation (Orca / FasterTransformer) | PagedAttention (vLLM and the industry default) | vAttention |
|---|---|---|---|
| Memory layout the kernel sees | |||
| Fragmentation | |||
| Attention kernels | Stock | ||
| Framework-side memory management | Trivial | ||
| End-to-end serving | Throughput-bound by tiny batches | Reference |
Check yourself
Check your understanding
1. FlashAttention-3 launches tomorrow, Hopper-optimized, with no PagedAttention support — as actually happened. Predict the adoption timeline for a PagedAttention-based serving stack versus a vAttention-based one.
2. The paper claims PagedAttention "duplicates what the OS already does." Trace the duplication precisely — what plays the role of page tables, of the TLB, and of the page-fault handler in a PagedAttention stack?
3. CUDA VMM calls cost tens of microseconds each and a KV-cache extension for Yi-34B needs 120 of them (~5 ms) — seemingly fatal for a serving loop. Why does vAttention's latency vanish anyway?
4. Stock CUDA VMM allocates only 2 MB pages, and NVIDIA's VMM implementation is closed-source. Why does granularity matter here, and how did the authors get 64 KB anyway?