Accelerating LLM Serving for Multi-turn Dialogues with Efficient Resource Management

When you chat with an AI over many turns, the system has to remember the whole conversation, and today's LLM servers do it wastefully — either recomputing the entire history from scratch on every new message, or stalling when a long prompt does not fit in GPU memory while shorter requests wait behind it; this work from Korea University builds FlashGen, which stashes each turn's attention cache across GPU, host memory, and SSD so history is fetched rather than recomputed, and reorders requests to fill the idle GPU memory instead of blocking on the front of the line, for up to 2.85 times the throughput of vLLM at the same latency.

The argument in one glance

Problem

Multi-turn chat forces the server to remember everything, expensively

When you hold a conversation with a chatbot, it has to keep the context of everything said so far, or its answers stop making sense. For a transformer model that context is the attention keys and values — the history — of every token in the conversation. But GPU memory is small and precious, so retaining all of it across a long chat is prohibitively expensive. A session in a real dataset has a median of three turns and an average of seven, and once the history is included the prompt grows 99 times longer than the current question alone.
What people did

Recompute the history, and serve requests first-come-first-served

The standard workaround is simple. Rather than store the old attention values, the server just prepends the entire chat history to the new question and runs the model over the whole thing, recomputing everything from scratch. And it schedules requests in arrival order, first come first served, as mature serving frameworks like vLLM do. Both choices are reasonable defaults that keep the system simple.
Why it fell short

Both defaults waste the two scarcest resources, compute and memory

Recomputing the history means re-running the model over tokens it already processed, and since history is often more than half of every prompt, that is a huge, growing tax on compute. Meanwhile, first-come-first-served scheduling causes head-of-line blocking: when the long prompt at the front of the queue does not fit in the free GPU memory, the scheduler stalls and leaves the rest of the memory idle, even though shorter requests waiting behind it could run right now. The system burns compute it should not and wastes memory it has.
This paper's idea

Cache the history across a memory hierarchy, and reorder to fill the gaps

FlashGen attacks both. First, it caches each turn's attention values across a hierarchy of GPU, host memory, and SSD, so when a follow-up arrives its history is fetched from the nearest tier instead of recomputed — and it hides the fetch behind computation, or falls back to recomputing when fetching would be slower. Second, its scheduler stops stalling on a too-big head-of-queue request and instead promotes shorter, runnable requests to fill the idle GPU memory, preempting them later so the long request never starves.

Why this problem is worth solving

The object that has to be remembered, and that overflows the GPU, is the model’s own scratchpad of the conversation.

A transformer generates text one token at a time, and to produce each new token it attends to every previous token in the sequence — weighing how relevant each one is. To avoid recomputing that relationship from scratch every step, the model saves, for each past token, a pair of vectors called its key and value; together these are the KV cache. In a chatbot conversation the KV cache holds the whole history of the dialogue, because the model must attend back over everything said before to stay coherent. The problem is size: these key-value pairs are large and grow with the length of the conversation — a single multi-turn session can need gigabytes of them — so they cannot all fit in the limited memory of a GPU. What to do with this history cache — keep it, drop it and recompute, or move it somewhere cheaper — is the central question of serving long conversations efficiently.

Understanding why memory, not raw speed, is the binding constraint explains the whole design.

Serving a large language model means running it as an online service that many users hit at once, and it has a peculiar performance profile. Each request has two phases: a prefill phase that processes the input prompt in one shot, and a decode phase that generates the answer one token at a time. A server batches many requests together to use the GPU well, and the scarce resource that limits how many it can batch is not compute but memory — specifically the memory holding each request’s KV cache. Modern frameworks like vLLM manage this memory in fixed-size blocks so it can be packed tightly, and schedule work at the granularity of single generation steps. The whole art of LLM serving is fitting as many requests as possible into that scarce memory and keeping the GPU busy — which is exactly what multi-turn conversations, with their enormous growing histories, make hard.

What people did before, and where it broke

Recompute the history each turn (vLLM, TensorRT-LLM). Prepend the whole conversation to the new question and re-run the model over all of it. Where it broke. It re-processes tokens it already computed, and since history is often more than half of every prompt, this is a large and growing compute tax that spikes the time to first response.
Keep all history in GPU memory. Cache every turn’s attention values on the GPU so nothing is recomputed. Where it broke. GPU memory is far too small — a session needs gigabytes — and under load the GPU cache hit rate collapses, falling below a quarter with only twenty concurrent users, worsened by the long human pauses between turns that destroy temporal locality.
Schedule first-come-first-served. Serve requests strictly in arrival order, the default in mature frameworks. Where it broke. A long prompt at the head of the queue that does not fit in free memory stalls the whole queue and idles the rest of the GPU memory, dropping utilization and making shorter runnable requests wait for no reason.
Offload the history to SSD (CachedAttention). Archive the KV cache to cheap storage and load it back on demand. Where it broke. SSD read bandwidth is low, so loading history from it can be slower than just recomputing, and relying solely on SSD caps throughput.
The gap. Each approach fixed one axis and paid on another: recomputation wastes compute, GPU-only caching runs out of memory, first-come scheduling wastes the memory it has, and SSD-only offload is bandwidth-bound. What was missing was a design that treats the history cache and the scheduler together — moving history through a full memory hierarchy while reshaping the request order to keep the scarce GPU memory full — which is exactly what multi-turn serving needs.

How FlashGen works

Caching history across tiers and reordering to fill GPU memory

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Keep the conversation’s history in a memory hierarchy, not on the GPU alone. The whole trouble is that the KV cache of a long chat is gigabytes but GPU memory is not, so the field either threw the history away and recomputed it or tried to cram it all onto the GPU and thrashed. FlashGen instead treats GPU memory, host DRAM, and SSD as one deep, inclusive cache: it proactively copies each turn’s values down into host memory as they are generated, spills host memory to cheap SSD in the background, and pre-stages a request’s history up from storage while it waits in the queue. So a follow-up almost always finds its history in a nearby tier, and the GPU is freed the instant a request completes because a copy already sits below it. Managing a multi-level cache so that only the immediately-needed data occupies the scarce fast tier while the bulk streams up from cheap storage, its load hidden behind other work, is the same discipline the out-of-core random-walk work covered elsewhere on this site relies on, reserving a succinct working set in fast memory while the rest lives on SSD and prefetching it before it is needed.

Reorder requests to fill the idle memory that strict ordering wastes, but preempt to stay fair. The second waste is scheduling: first-come-first-served leaves the GPU memory half empty whenever the front-of-queue prompt is too big, even as shorter requests behind it could run immediately. FlashGen opportunistically promotes those runnable requests into the idle space to keep the GPU full, then guards against starving the long request by tracking the promoted requests’ memory as reclaimable and preempting them the moment enough space frees to schedule the deferred one. Opportunistically repurposing an idle resource to do useful work while preserving fairness by taking it back when the deferred work can finally proceed is the same balance the NUMA page-table work covered elsewhere on this site strikes, reclaiming space it had lent out and preempting the borrower so the request that yielded its turn is never left waiting forever.

Decide, per situation, whether to fetch or to recompute. A subtler move is that FlashGen does not blindly prefer its cache. Because SSD is slow, loading history from it can be slower than recomputing the history outright, and pulling from host memory only helps if the transfer can hide behind computation. So FlashGen weighs the two and chooses: it fetches when the transfer overlaps useful work and recomputes when it would not, and it stages from SSD only when a waiting request exists to hide the read behind. Refusing to apply an optimization when it would actually cost more than the thing it replaces, and cheaply choosing the better of the two options for the current conditions, is the same guard the post-deduplication compression work covered elsewhere on this site builds in, dropping a would-be saving whenever it would not in fact pay off.

Following a follow-up message through FlashGen

Answering the third turn of a conversation

Tracing one follow-up message shows how the cache and the scheduler cooperate.

Step 1 of 5

The user sends turn three

To answer coherently, the model needs the attention history of turns one and two. In a baseline server that history would be recomputed from scratch — a large cost, since across a conversation the history is often the majority of the prompt and grows every turn.

On a real dataset the prompt with history is 99 times longer than the current question alone.

Why it works

Fetching cached history beats recomputing what is mostly repeated tokens. Prepending the whole conversation and re-running the model re-processes tokens it already computed, and history is often more than half the prompt → but saving each turn’s attention values and reloading them replaces that recomputation with a memory transfer → so as conversations lengthen the saving grows, cutting tail time-to-first-token from seconds to a fraction of a second.
A GPU-host-SSD hierarchy makes the cache both fast and big enough. GPU memory alone cannot hold gigabytes of history per session, and its cache hit rate collapses under load → but proactively copying each turn’s values down through host memory and onto cheap SSD in the background keeps a deep, inclusive cache → so a follow-up’s history is usually found in a nearby tier, and freeing GPU space is instant because a copy already lives below.
Pipelining the transfer behind computation hides the cost of a cache miss. Moving history from host memory to the GPU takes time the request would otherwise wait on → but transferring the next layer’s values while the current layer computes overlaps the two → so the restoration disappears into work already happening, and when it cannot be hidden the server simply recomputes instead.
Reordering fills the idle memory that first-come-first-served leaves stranded. When a long prompt at the front does not fit, the scheduler stalls and the rest of GPU memory sits unused → but promoting a shorter runnable request into that idle space lets it run now instead of waiting → so GPU utilization climbs past 98 percent and batches grow, lifting throughput without waiting for the big request.
Preempting the promoted request keeps the long one from starving. Letting younger requests jump ahead could leave a memory-hungry prompt waiting forever → but once enough space frees, the scheduler counts the promoted request’s memory as reclaimable and preempts it to schedule the deferred one → so fairness is preserved, and the preempted request resumes cheaply by restoring its values from the host cache.

What it costs (trade-offs and scope)

It trades GPU memory for a lot of host memory and SSD. The multi-level cache needs real capacity beneath the GPU — the testbed devotes about 224 gigabytes of host DRAM plus two NVMe SSDs to it. The idea moves the KV cache off the scarce GPU, but only if you have a deep, well-provisioned memory hierarchy to move it into.
The headline gain is model- and load-specific. The 2.85-times throughput is Llama-2 70B, whose grouped-query attention and heavy recomputation cost make caching especially valuable; the other models land around 1.55 to 1.63 times. And the scheduler barely helps at low load, where head-of-line blocking rarely occurs.
The SSD tier only pays off under queueing. Loading from SSD is slower than recomputing, so FlashGen stages storage data only when a waiting request exists to hide the read behind, and otherwise recomputes. For small-cache models where host memory holds everything, the SSD tier is rarely used at all.
The margin over the closest prior work is modest. FlashGen’s caching resembles the concurrent CachedAttention; its distinct advantages — the recompute-versus-fetch choice and the GPU caching — yield about 1.13 to 1.19 times over it, with the full system’s larger wins coming mostly from adding the scheduler.
It requires modified attention kernels. Because cached history blocks are not contiguous in GPU memory, the Flash-Attention and Flash-Decoding kernels had to be revised, so it is not a drop-in on an unmodified stack.
Key parameters are configured, not learned. The host-memory split for the cache, the SSD staging budget, and the reading-speed model that sets the pauses between turns are all set by hand, and the results depend on them.
It is evaluated at one scale on one instance type. Two A100 GPUs up to a 70B model, with the longest contexts extrapolated from synthetic traces; larger models, more GPUs, and other hardware are not shown.

The evaluation

Serving multi-turn conversations

Property	vLLM	CachedAttention	FlashGen-Sched only	FlashGen
How history is handled
Scheduling		First-come-first-served
GPU memory utilization		Similar to vLLM		Above 98 percent
Throughput versus vLLM	Baseline	Better, SSD-limited
Tail time to first token (OPT 30B)		7.8 seconds	14.2 seconds

Multi-turn LLM serving approaches, with FlashGen built on vLLM and evaluated on two A100 GPUs across ShareGPT, Alpaca, and HumanEval for OPT and Llama-2 models from 13B to 70B. FlashGen reaches 1.55 to 2.85 times vLLM's throughput at equal latency, holds GPU memory above 98 percent (versus about 88 percent for vLLM), and cuts P95 time-to-first-token from 16.9 seconds to 1.3 on OPT 30B; its caching gives roughly 1.13 to 1.19 times over the concurrent CachedAttention, with the scheduler adding the rest.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

FlashGen is a well-engineered pair of ideas for a real and growing problem: it stops recomputing conversation history by caching it across GPU, host memory, and SSD with the transfers hidden behind computation, and it stops wasting GPU memory by reordering requests to fill the gaps that first-come-first-served leaves, preempting to keep the long prompts fair — for up to 2.85 times vLLM’s throughput and a tail time-to-first-token cut from seconds to a fraction. Read the numbers with their conditions. The headline gain is the 70B model with its heavy recomputation cost; the smaller models see 1.55 to 1.63 times, and the scheduler helps mainly under load. The design assumes a deep, well-provisioned memory hierarchy — hundreds of gigabytes of host DRAM and SSDs beneath the GPU. Its edge over the closest concurrent work is a modest 1.13 to 1.19 times from the cache alone, it needs modified attention kernels, and several knobs are hand-tuned. It was measured on two A100 GPUs up to 70B, with the longest contexts extrapolated. The durable idea is the combination: treat the history cache and the scheduler as one problem, and choose per situation whether to fetch or recompute.

Background

Why memory movement, not math, now bounds LLM serving

It is worth stepping back to see why a paper about caching and scheduling, rather than faster arithmetic, is where the gains in LLM serving now live. For a decade the story of AI hardware was about making the dense matrix multiply faster, and it worked. But serving a conversational model at scale is not bottlenecked by how fast the model can multiply; it is bottlenecked by memory — how much KV cache fits on the GPU, how fast it can be moved between tiers, and how well the scarce memory is packed with useful work. A multi-turn chatbot makes this vivid, because its true cost is not the computation of each answer but the storing, moving, and re-deriving of an ever-growing conversation history. So the interesting engineering has shifted from the compute engine to the data around it: the caches, the memory hierarchy, and the scheduler that decides what occupies the precious space. The same observation drives the deep-learning-on-GPUs retrospective covered elsewhere on this site, which measures whole real applications and concludes that the era of simply accelerating the matrix multiply is over and that data movement is the new frontier. The broader pattern is that as models mature and deploy, performance stops being about faster math and becomes about orchestrating the memory that feeds it.

If you remember only three things

1. Serving multi-turn chatbots is hard because the model must remember the whole conversation as a KV cache that grows gigabytes-large and cannot fit in GPU memory; today's servers therefore recompute the entire history on every new turn (a huge, growing compute tax, since history is often more than half of each prompt) and schedule first-come-first-served, which causes head-of-line blocking — a long prompt that does not fit stalls the queue and idles GPU memory while shorter runnable requests wait behind it.
2. FlashGen introduces two cooperating techniques: a multi-level KV cache (FlashGen-Cache) that proactively copies each turn's attention values down through GPU, host memory, and SSD, pre-stages history from storage while requests queue, restores it with transfers pipelined behind layer computation, and opportunistically recomputes when fetching would be slower; and a reordering scheduler (FlashGen-Sched) that promotes shorter runnable requests into the idle GPU memory a stalled long prompt leaves, then preempts them once space frees so the deferred long request never starves, resuming the promoted request cheaply from the host cache.
3. Built on vLLM and evaluated on two A100 GPUs across ShareGPT and other datasets for OPT and Llama-2 from 13B to 70B, FlashGen reaches 1.55 to 2.85 times vLLM's throughput at equal latency, holds GPU memory above 98 percent (versus about 88 percent), and cuts P95 time-to-first-token from 16.9 seconds to 1.3 — but the biggest gain is the 70B model specifically, the scheduler helps mainly under load, the design needs a deep host-memory and SSD hierarchy, its edge over the concurrent CachedAttention is a modest 1.13 to 1.19 times from caching alone, and it requires modified attention kernels.

Check yourself

Check your understanding

1. Why are multi-turn conversations expensive for current LLM servers?
2. What are FlashGen's two cooperating techniques?
3. Why does FlashGen sometimes recompute the history instead of loading it from cache?
4. Which statement best reflects the limits of FlashGen?

Sources

The paper is “Accelerating LLM Serving for Multi-turn Dialogues with Efficient Resource Management” by Jinwoo Jeong and Jeongseob Ahn, published at ASPLOS 2025, DOI 10.1145/3676641.3716245. The authors are at Korea University.

This explainer is written from the full 15-page text. Everything above comes from the paper’s own sections: the introduction identifying the prompt-amplification and recomputation problems in multi-turn serving; the background on autoregressive prefill and decode phases, the attention KV cache, transformer chatbots and history KVs, and the ShareGPT characterization (median three and average seven turns, prompts 99 times longer with history, history exceeding half of tokens even at the median, and per-session history-KV sizes from 2.8 to 16.2 gigabytes across OPT models) plus the microbenchmark showing host caching far cheaper than recomputation and SSD; the FlashGen-Cache design of a GPU-host-SSD multi-level cache with proactive inclusive copying, layer-by-layer background eviction, pipelined and batch-aware restoration, SSD archiving with pre-staging, and the opportunistic recompute-versus-retrieve fallback; the FlashGen-Sched design attacking head-of-line blocking with request reordering and starvation-free preemption; and the evaluation on an Azure two-A100 instance with 440 gigabytes of host DRAM and two NVMe SSDs, across ShareGPT, Alpaca, and HumanEval and OPT and Llama-2 models, reporting 1.55-to-2.85-times throughput over vLLM, above-98-percent memory utilization versus about 88 percent, P95 time-to-first-token from 16.87 to 1.29 seconds on OPT 30B, roughly 1.13-to-1.19-times gains over the concurrent CachedAttention, and the multi-level cache hit-rate, batch-size, and long-context sensitivity analyses.