REPA: Reconfigurable PIM for the Joint Acceleration of KV Cache Offloading and Processing

As a language model generates text it keeps a growing "KV cache" of every past token, which both eats huge GPU memory and slows each step because the model must re-read the whole cache — and existing fixes solve only one problem (offload the cache to bigger memory, or split the work) while paying to shuttle data back; this paper (Shanghai Jiao Tong University — Yang Hong, Jianguo Yao, and colleagues) presents REPA, which parks the KV cache in reconfigurable processing-in-memory hardware that both stores it and computes attention on it in place, running 1.5 to 6.5 times faster than an A100.

The argument in one glance

Problem

The KV cache is both too big and too slow to read

When a large language model generates text, it does so one token at a time, and at each step it must attend to every token it has already produced. To avoid recomputing everything, it stores the keys and values of past tokens in a KV cache. This cache is central to fast inference, but it creates two distinct problems at once. First, footprint: the cache grows with the sequence length and the batch size, so for long contexts and many concurrent requests it becomes enormous and can overflow the fast but limited memory on a GPU. Second, decoding speed: at every single decoding step the model has to read the entire KV cache to compute attention, so decoding is bottlenecked by memory bandwidth — the model spends its time streaming the cache rather than computing. These two limitations pull in different directions. If you shrink or move the cache to fit more of it, you tend to make it slower to access; if you keep it fast, you cannot fit long contexts. So the KV cache simultaneously strains capacity and caps performance, and doing well on one usually means doing worse on the other.
What people did

Offload the cache, or split inference into stages

Prior work tackles one of the two problems at a time. To relieve the footprint, systems offload the KV cache to larger, cheaper memory — CPU DRAM or storage — so long contexts and big batches fit, moving the needed parts back to the GPU when attention must be computed. To improve decoding performance, other systems use stage-split inference, separating the compute-heavy prefill phase from the memory-heavy decode phase and mapping each to suitable resources. Each approach picks its battle: offloading buys capacity, stage-splitting buys speed.
Why it fell short

Solving one problem re-creates the other through data movement

The trouble is that these are point solutions that leave the other limitation, and often the underlying tension, unaddressed. Offloading fixes the footprint but not the speed: the KV cache now lives in slow, distant memory, and to actually compute attention the relevant cache must be transferred back to the GPU, so decoding pays a heavy data-movement cost and the memory-bandwidth bottleneck reappears as an interconnect bottleneck. Stage-split inference improves how the phases are scheduled but does not shrink the cache or remove the need to stream it. Underneath both is the same fundamental issue: the KV cache and the computation over it are in different places, so whenever you make room for the cache by moving it away, you pay to bring it back to be computed on. The two goals — large capacity for the cache and fast processing over it — keep fighting because storage and compute are separated. What was missing was a way to store the huge KV cache and compute attention over it in the same place, so that adding capacity does not cost speed and processing does not require moving the cache.
This paper's idea

Put the cache in memory that also computes attention on it

REPA is a GPU-plus-processing-in-memory (PIM) hybrid system that pursues a joint solution to both problems at once. Its core move is to offload the KV cache into reconfigurable ReRAM PIM — a memory technology that is both dense enough to hold a large cache and able to perform computation in place — so the cache is persisted there and attention is computed directly on it, without shuttling it back to the GPU. Because the PIM is reconfigurable, REPA can balance the competing demands of processing speed and memory capacity, dedicating the device to storage or to compute as needed. To fully exploit the parallelism this makes possible, REPA adds optimizations at three levels. In architecture, it introduces bulk-wise memory instructions and multi-level controllers to enable finer-grained parallelism inside the PIM device. In data mapping, it uses locality-aware placement to make the most of that architecture and to reduce long-range data transfer on the chip. In pipelining, it uses sub-batch pipelining to reduce idle time within batches, and transfer overlapping to hide the KV cache transfer behind computation. The result, the paper reports, is high inference speed, energy efficiency, and integratability: REPA is 1.5 to 6.5 times faster and 8 to 10 times more energy-efficient than an NVIDIA A100, outperforms state-of-the-art DRAM-PIM systems by up to 1.4 times for long-context inference, and, when integrated into existing offloading systems, delivers 1.4 to 2.0 times faster offloading and 1.2 to 1.4 times end-to-end speedup.

Why this problem is worth solving

The KV cache is the pinch point of modern LLM serving, and its twin costs — memory and bandwidth — set how long a context you can run and how fast you can decode.

Serving large language models is dominated by the decode phase, where tokens are generated one at a time, and that phase is bottlenecked by the KV cache in two compounding ways. On capacity: the cache grows linearly with context length and batch size, so the demand for long contexts and high throughput runs straight into the limited high-bandwidth memory on accelerators, capping how much you can serve at once. On bandwidth: every decoding step reads the whole cache to compute attention, so decode is memory-bound, and the accelerator’s compute sits underused while it waits on memory. These are not niche concerns — they are the reason long-context inference is expensive and slow, and why so much systems work targets the KV cache specifically. Crucially, the two costs interact: the usual way to relieve capacity is to move the cache somewhere bigger, which worsens bandwidth, and the usual way to keep bandwidth is to keep the cache close, which limits capacity. A solution that addresses both at once, rather than trading one for the other, therefore attacks the central economic constraint of LLM serving. That is why a joint approach matters: it can raise the ceiling on context length and batch size while also speeding up decoding, instead of buying one with the other.

The key realization is that the capacity-versus-speed tension comes from separating storage and compute, and it dissolves if you compute where the cache lives.

The reason offloading trades speed for capacity is a structural one: the KV cache is data, the attention over it is computation, and in a conventional system these happen in different physical places — the cache in some memory, the compute in the GPU. So any time you move the cache to a larger memory to gain capacity, you incur a transfer to bring it back for computation, and that transfer is exactly the cost you were trying to avoid. Processing-in-memory breaks this by doing the computation inside the memory that holds the data, so the cache never has to travel to be computed on. REPA’s insight is to use this specifically for the KV cache: park the cache in a PIM device that is dense enough to hold a lot of it and capable of computing attention in place, and the capacity gain no longer costs a bandwidth penalty, because there is no round-trip. Reconfigurability is what makes it practical, because the balance between how much of the device acts as capacity and how much acts as compute can be tuned to the workload. And once compute and storage are unified, the remaining performance is a parallelism problem — extracting fine-grained parallelism from the device, placing data to keep transfers local, and pipelining so the unavoidable transfers hide behind computation. So the deep move is architectural: collapse the storage-compute separation for the KV cache, and the long-standing capacity-speed tradeoff turns into an engineering problem of parallelizing a single, co-located system.

What people did before, and where it broke

Keep the KV cache in GPU memory. Store the cache in fast high-bandwidth memory and compute attention normally. Where it broke. The cache grows with context and batch, overflowing limited GPU memory and capping context length and throughput.
Offload the KV cache to bigger, cheaper memory. Move the cache to CPU DRAM or storage to fit long contexts, transferring parts back to compute. Where it broke. Attention still runs on the GPU, so the cache must be moved back, turning the memory-bandwidth bottleneck into an interconnect bottleneck.
Split inference into stages. Separate the compute-heavy prefill from the memory-heavy decode and schedule each on suitable resources. Where it broke. It improves scheduling but does not shrink the cache or remove the need to stream it, leaving the footprint problem.
Address only one limitation at a time. Pick either capacity or speed and optimize it. Where it broke. The two are linked through data movement, so a point fix on one often worsens the other.
The gap. No approach stored the large KV cache and computed attention over it in the same place, so capacity and speed kept trading off against each other through the cost of moving the cache.

How REPA works

From offloading-and-moving-back to computing on the cache in place

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Compute attention where the KV cache lives instead of moving it to the GPU. REPA’s foundation is processing-in-memory for the KV cache: keep the cache in a PIM device and run attention on it in place, so the data-movement cost that dominates offloaded decoding disappears. Bringing the computation to the KV cache inside memory, rather than streaming the cache out to a separate accelerator to compute attention, is the same principle the work on unleashing processing-in-memory for batched transformer generation (AttAcc) covered elsewhere on this site is built on, putting the attention computation next to the data it consumes. REPA applies this to the offloaded, persisted KV cache so that capacity and computation coexist.

Use a memory substrate that both holds a huge working set and computes on it. REPA’s reconfigurable ReRAM PIM is dense enough to store a large KV cache and able to compute, letting it balance the scarce-capacity and processing demands in one device. Using a compute-capable memory to hold a workload’s large data footprint and process it in place, carefully managing the trade-off between how much of the substrate is capacity and how much is compute, is the same undertaking the work on wafer-scale SRAM computing-in-memory for large language model inference (Ouroboros) covered elsewhere on this site pursues, turning memory into a place that both stores and computes under tight capacity. REPA does this reconfigurably and specifically for KV-cache persistence and attention.

Overlap the unavoidable transfers so they hide behind compute. Even with in-memory computation, some KV cache transfer remains, so REPA pipelines sub-batches and overlaps transfer with computation to keep the device busy and hide movement latency. Scheduling data transfer to run concurrently with computation so the communication is shadowed rather than stalling the pipeline is the same lever the work on efficient scheduling for communication-computation overlap (Centauri) covered elsewhere on this site pulls, hiding movement behind useful work. REPA’s transfer overlapping and sub-batch pipelining are its way of ensuring the remaining data movement does not sit on the critical path.

Following a decoding step through REPA

Serving one decoding step over an offloaded KV cache

An illustration, drawn from the paper's abstract, of how REPA processes the KV cache in place. Internal mechanism details are the paper's.

Step 1 of 5

The KV cache is persisted in ReRAM PIM

REPA has offloaded the growing KV cache into the reconfigurable ReRAM PIM device, which holds it densely and is configured to balance storage capacity against compute for the current workload.

The cache lives where it can also be computed on.

Why it works

In-memory computation removes the KV round-trip. Offloading forces the cache back to the GPU to compute attention → but computing attention inside the memory that holds the cache → so the dominant data-movement cost of offloaded decoding disappears.
A dense compute-capable memory unifies capacity and speed. Storage and compute in different places make capacity and speed trade off → but ReRAM PIM both holds a large cache and computes on it → so adding capacity no longer costs a bandwidth penalty.
Reconfigurability tunes the storage-compute balance. A fixed split would waste the device on one workload or another → but reconfiguring how much acts as capacity versus compute → so REPA matches the balance to processing and footprint demands.
Architecture and mapping expose the device’s parallelism. In-memory compute is wasted if it runs serially → but bulk-wise instructions, multi-level controllers, and locality-aware mapping → so the PIM runs attention with fine-grained parallelism and little long-range transfer.
Pipelining hides the transfers that remain. Any residual KV movement could stall the pipeline → but sub-batch pipelining and overlapping transfer with computation → so the movement is shadowed behind compute and the device stays busy.

What it costs (trade-offs and scope)

It requires ReRAM PIM hardware in the system. REPA’s gains depend on a GPU-plus-reconfigurable-ReRAM-PIM hybrid; it is a hardware-software co-design, not a change that runs on commodity GPUs alone.
Reconfiguration and mapping add design complexity. Balancing capacity against compute, the bulk-wise instructions and multi-level controllers, and the locality-aware mapping are non-trivial machinery that must be gotten right to realize the parallelism.
It targets the KV cache and attention specifically. REPA jointly accelerates KV cache offloading and the attention processing over it; the rest of the model still runs on the GPU, so its benefit is bounded by how much of inference the KV cache dominates.
Some transfer remains, only hidden. Transfer overlapping shadows the KV cache movement behind computation rather than eliminating it, so the benefit depends on there being enough compute to hide it under.
Reported figures are against specific baselines. The 1.5-to-6.5-times speedup and 8-to-10-times efficiency versus an A100, the up-to-1.4-times over DRAM PIM, and the offloading and end-to-end speedups are measured on the paper’s workloads and baselines; results depend on context length, batch, and hardware.

The comparison

Handling the KV cache in LLM decoding

Property	KV cache in GPU memory	Offload to CPU/storage	REPA (reconfigurable PIM)
Fits long contexts / big batches
Where attention is computed
Data-movement cost
Capacity vs speed
Performance

A comparison of ways to handle the KV cache in LLM decoding. Keeping it in GPU memory is fast but caps context; offloading it to CPU or storage fits long contexts but must move the cache back to the GPU to compute attention. REPA persists the cache in reconfigurable ReRAM processing-in-memory that both stores and computes on it, balancing capacity and speed and hiding residual transfers behind compute — running 1.5 to 6.5 times faster and 8 to 10 times more efficiently than an A100. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s abstract and public metadata, not its full text, which was not available at the time of writing. The high-level claims are the authors’: REPA is a GPU-PIM hybrid system that jointly accelerates KV cache offloading and processing, motivated by the observation that the KV cache in LLM inference causes both a large memory footprint and sub-optimal decoding performance, which prior work addresses only one at a time (via offloading or stage-split inference). REPA leverages reconfigurable ReRAM PIM to achieve fast KV cache persistence and to balance processing speed against memory capacity, and adds optimizations at three levels: (1) architecture — bulk-wise memory instructions and multi-level controllers for finer-grained parallelism; (2) data mapping — locality-aware mapping to exploit that architecture and reduce long-range on-chip transfer; and (3) pipelining — sub-batch pipelining to reduce batch idleness and transfer overlapping to shadow KV cache transfer behind computation. The reported results are: 1.5 to 6.5 times faster and 8 to 10 times more energy-efficient than an NVIDIA A100; up to 1.4 times better than state-of-the-art DRAM PIM systems for long-context inference; and, integrated into existing offloading systems, 1.4 to 2.0 times faster offloading and 1.2 to 1.4 times end-to-end speedup. The background above on the KV cache, decoding, processing-in-memory, ReRAM, and offloading is standard context provided to make the approach intuitive. The Flow, the step-by-step walkthrough, and the “Why it works” reasoning are illustrations built from the abstract’s description, not detailed accounts from the paper — the exact PIM microarchitecture, the instructions and controllers, the mapping and pipelining, and the full evaluation should be read from the paper. Do not treat specific procedural details here as established beyond what the abstract states.

Background

Why the memory wall keeps pushing computation toward the data

It helps to see REPA within a broad and accelerating shift in computer architecture: for data-intensive workloads, moving data costs far more than computing on it, and the response, again and again, is to move the computation toward the data rather than the data toward the computation. Large language model inference is a textbook case, because the KV cache is huge and must be read constantly, so decoding is dominated not by arithmetic but by streaming memory. The field has attacked this from many sides — smarter caching, offloading to bigger memory, compressing the cache, disaggregating memory over fast networks — but each conventional approach ultimately still ferries the cache to wherever the compute is. Processing-in-memory is the more radical response: collapse the separation entirely, so the memory that stores the data also computes on it. REPA is a clean instance aimed at the KV cache, using reconfigurable memory to both persist the cache and run attention on it, which turns the usual capacity-versus-speed tradeoff into a parallelization problem. This drive to relieve the memory bottleneck of LLM serving by rethinking where the KV cache lives and how it is accessed is the same terrain the work on network-accelerated memory offloading for large language models (AQUA) covered elsewhere on this site occupies, attacking the cost of getting the cache to the compute. REPA’s particular contribution is to show that persisting the cache in compute-capable memory, and jointly optimizing the architecture, mapping, and pipelining around it, can solve the footprint and the speed problem together rather than trading one for the other.

If you remember only three things

1. A large language model's KV cache — the stored keys and values of all past tokens — causes two problems at once during decoding: a large memory footprint (it grows with context length and batch size, overflowing limited GPU memory) and sub-optimal decoding speed (every step reads the whole cache to compute attention, so decode is memory-bandwidth-bound). These pull against each other: moving the cache to bigger memory relieves footprint but slows access, and keeping it close keeps speed but limits context. Prior work fixes only one — offloading the cache to CPU or storage for capacity, or stage-splitting inference for scheduling — and offloading in particular still has to transfer the cache back to the GPU to compute attention, so it converts the memory bottleneck into a data-movement one rather than resolving the underlying tension.
2. REPA is a GPU-plus-processing-in-memory hybrid that solves both at once by parking the KV cache in reconfigurable ReRAM PIM — memory dense enough to hold a large cache and able to compute on it in place — so attention is computed directly on the persisted cache without moving it back to the GPU, and the device can be reconfigured to balance storage capacity against processing speed. To exploit the parallelism this enables, REPA optimizes at three levels: architecture (bulk-wise memory instructions and multi-level controllers for finer-grained parallelism), data mapping (locality-aware placement that reduces long-range on-chip transfer), and pipelining (sub-batch pipelining to cut idle time, and transfer overlapping to hide the remaining KV cache movement behind computation).
3. The paper reports REPA is 1.5 to 6.5 times faster and 8 to 10 times more energy-efficient than an NVIDIA A100, outperforms state-of-the-art DRAM PIM systems by up to 1.4 times for long-context inference, and, integrated into existing offloading systems, achieves 1.4 to 2.0 times faster offloading and 1.2 to 1.4 times end-to-end speedup. The trade-offs and scope: it requires reconfigurable ReRAM PIM hardware, its reconfiguration and mapping add design complexity, it targets the KV cache and attention specifically (the rest of the model still runs on the GPU), and it hides rather than eliminates residual transfers. This explainer is based on the paper's abstract; the PIM microarchitecture and full evaluation should be confirmed in the paper.

Check yourself

Check your understanding

1. Why is the KV cache a double problem in LLM decoding?
2. Why do offloading and stage-split inference not fully solve the problem?
3. What is REPA's core idea, and how does the reconfigurable ReRAM PIM help?
4. What does REPA achieve, and what are the limits of the approach?

Sources

The paper is “REPA: Reconfigurable PIM for the Joint Acceleration of KV Cache Offloading and Processing” by Yang Hong, Junlong Yang, Bo Peng, and Jianguo Yao (Shanghai Jiao Tong University), published at ASPLOS 2026, DOI 10.1145/3779212.3790212.

This explainer is written from the paper’s abstract and public metadata; the full text was not available at the time of writing, so the page is marked a draft. As reported there, REPA is a GPU-PIM hybrid system that jointly accelerates KV cache offloading and processing, using reconfigurable ReRAM PIM to persist the KV cache and compute attention on it in place while balancing processing speed against memory capacity. It adds optimizations in architecture (bulk-wise memory instructions and multi-level controllers), data mapping (locality-aware placement to reduce long-range on-chip transfer), and pipelining (sub-batch pipelining and transfer overlapping to hide KV cache transfer behind computation). The paper reports REPA is 1.5 to 6.5 times faster and 8 to 10 times more energy-efficient than an NVIDIA A100, up to 1.4 times better than state-of-the-art DRAM PIM for long context, and 1.4 to 2.0 times faster offloading (1.2 to 1.4 times end-to-end) when integrated into offloading systems. Background on the KV cache, processing-in-memory, ReRAM, and offloading is standard context. The specific numbers are the paper’s own; the mechanism details and full evaluation should be confirmed in the paper.