FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning

Hao Mark Chen, Zhiwen Mo, Guanxi Lu, Shuang Liang, Lingxiao Ma, Wayne Luk and 1 other

A small language model that fits on an edge device reasons poorly, and the fix — test-time scaling, where the model explores many reasoning paths and a verifier prunes them — is far too slow on a constrained GPU because the parallel paths finish their thinking steps at wildly different lengths, two models fight over memory, and shared reasoning prefixes get recomputed; this paper (Imperial College London and Microsoft Research — Hao Mark Chen, Wayne Luk, and colleagues) builds FastTTS, a vLLM library that speculatively fills the idle time behind straggler paths, splits memory between the generator and verifier by their real compute patterns, and orders execution to reuse the KV cache across shared prefixes, giving 2.2 times higher goodput and 38 to 68 percent lower latency so a 7-billion-parameter model on a single 24 GB GPU can match cloud accuracy and latency.

The argument in one glance

Problem

A small model that fits on your device cannot reason well enough

Running a language-model agent right on an edge device — a laptop or a single consumer GPU — is increasingly desirable: it keeps your data private, works offline, and can react to local context. But an edge device has strict memory limits, so it can only hold a small model, and small models reason far worse than the big models that live in the cloud. That gap is the barrier to practical edge agents. There is a known way to close it without a bigger model, called test-time scaling: instead of trusting one answer, have the model spend extra compute at inference time exploring many candidate reasoning paths and use a separate scorer to keep the promising ones and prune the rest. It genuinely lifts a small model's reasoning toward cloud quality. The catch is that all that extra exploration is slow, and on a memory-constrained device it is slow enough to be impractical.
What people did

Run test-time scaling on a standard LLM serving system

The natural approach is to take the structured search of test-time scaling — generate a batch of parallel reasoning paths, have a process reward model score each new thinking step, replicate the top paths and drop the losers, repeat — and run it on a mature LLM serving framework like vLLM, which already provides KV-cache management, paged attention, and preemptive scheduling for ordinary token generation.
Why it fell short

General serving systems are built for streams of tokens, not trees of reasoning

A standard serving system is tuned for independent token-generation requests, not the structured, multi-path search that test-time scaling needs, and three mismatches make it slow on constrained hardware. The thinking steps of parallel paths differ enormously in length, so a batch stalls waiting for the longest straggler while every finished path sits idle, wasting the GPU. Two models — the generator and the verifier — must share the scarce memory, and splitting it evenly ignores that they stress the hardware differently. And the many search paths share long reasoning prefixes, whose cached key-value state gets evicted and recomputed instead of reused. No serving system natively handles this.
This paper's idea

Fill the idle time, split memory by workload, and reuse shared prefixes

FastTTS is a serving library, plug-and-play on top of vLLM, built specifically for test-time-scaling reasoning on memory-constrained devices, with three techniques matched to the three bottlenecks. Speculative Beam Extension uses the idle GPU time behind a straggler to speculatively generate future tokens for the paths that already finished their step, hiding the straggler's latency. Asymmetric Multi-Model Memory Allocation profiles the generator and the verifier — which have different compute patterns — and splits the KV-cache memory between them to maximize overall throughput rather than dividing it evenly. Dynamic Prefix-Aware Scheduling orders the execution of reasoning paths so that consecutively processed paths share as much prefix as possible, minimizing KV-cache evictions and recomputation. Together they give an average 2.2 times higher goodput and 38 to 68 percent lower latency, letting a 7-billion-parameter model on one 24 GB GPU match cloud accuracy and latency.

Why this problem is worth solving

The appeal of test-time scaling is that it buys reasoning quality with compute rather than model size, which is exactly what an edge device can offer.

Test-time scaling is a family of methods that improve a model’s answers by spending more computation at inference, instead of by training or loading a bigger model. The dominant form for reasoning is a structured, verifier-guided search: the model generates several parallel reasoning paths (beams), and at each step it produces a new thinking step — an arbitrary-length chunk of tokens — for each path; a process reward model, or verifier, then scores each new step, and the top-scoring paths are replicated to spawn the next round of beams while the rest are pruned. This generate-verify loop repeats until the paths terminate. Simple variants explore independent chains (like sampling many answers and picking the best), while more powerful ones build a reasoning tree with intermediate pruning (beam search and its relatives). Because the extra work is inference compute rather than parameters, a small model that fits on an edge device can, in principle, reason far better than its size suggests by scaling this search up — which is precisely why it is the lever for closing the edge-versus-cloud reasoning gap, and why making it fast enough to actually use is worth a dedicated serving system.

The trouble is that the workload this search creates is nothing like what serving systems were built for.

Systems like vLLM are engineered for high-throughput streaming of independent generation requests, with optimizations — KV-cache management, paged attention, preemptive scheduling — that assume roughly uniform, token-by-token work. Test-time-scaling reasoning violates those assumptions in three structural ways. First, its unit of work is a thinking step whose length is unpredictable: on real reasoning benchmarks the number of tokens in a step varies dramatically across the parallel paths, so a batch cannot advance until its single longest path finishes, and meanwhile every shorter path that already finished sits idle — severe hardware underutilization from stragglers. Second, it runs two models at once, a generator and a verifier, which must divide the device’s scarce memory between their KV caches; but the two have different compute profiles (the verifier processes prompts in large batches, compute-bound, while the generator decodes token by token, memory-bound), so an even or arbitrary split is suboptimal. Third, the parallel paths are not independent — they branch from shared ancestors, so they share long reasoning prefixes whose cached key-value state could be reused, but a scheduler unaware of this evicts and recomputes it. The right metric here is also different: since many generated tokens are pruned and never used, goodput (useful work) matters more than raw throughput. A general serving system optimizes none of this, which is why test-time scaling on it is impractically slow on an edge device.

What people did before, and where it broke

Deploy a small model directly on the edge device. Fit the largest model memory allows and run it normally. Where it broke. Small models reason far worse than cloud models, so on hard multi-step tasks their accuracy is simply inadequate, which is the gap test-time scaling exists to close.
Run test-time scaling on a stock serving system (vLLM). Layer the structured search on top of a mature framework’s KV-cache and scheduling machinery. Where it broke. The framework is built for uniform token streams, so straggler thinking steps idle the GPU, the two models split memory blindly, and shared prefixes are recomputed — making the search too slow to be practical on constrained hardware.
Even or static memory split between generator and verifier. Give each model a fixed share of the KV-cache memory. Where it broke. The generator and verifier have different compute characteristics and throughput sensitivities to memory, so a fixed split leaves overall throughput on the table.
Prefix caching without prefix-aware scheduling. Rely on the serving system’s KV-cache reuse without controlling the order paths are processed. Where it broke. Under a tight memory budget, if paths sharing a prefix are not processed close together, their shared key-value state is evicted between them and must be recomputed, so the reuse opportunity is lost.
The gap. No serving system was designed for the specific structure of verifier-guided test-time scaling on memory-constrained devices — hiding straggler latency, allocating memory between two dissimilar models by their real workloads, and ordering execution to actually capture the shared-prefix reuse.

How FastTTS works

Reshaping a general serving system into a fast test-time-scaling engine

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Turn the wait behind a straggler into useful work you can do now. The first bottleneck is that parallel reasoning paths finish their thinking steps at wildly different lengths, so a batch is held hostage by its longest path while the finished ones idle — the classic straggler problem, made worse here because step lengths are genuinely unpredictable. FastTTS’s move is to refuse to waste that idle time: while the batch waits for the straggler, it speculatively generates future tokens for the paths that already finished, work those paths would have needed in the next iteration anyway. If speculation guesses right, that future work is already done when the next round begins; the danger — corrupting the algorithm or adding tail latency — is defused by verifying only the non-speculative tokens and strictly killing speculation the instant the standard steps complete, so speculation is pure upside. Using idle hardware to run ahead on work that is likely to be needed, gated so a wrong guess costs nothing, is the same bet the tree-based speculative-inference work covered elsewhere on this site is built on, generating candidate continuations opportunistically and keeping only the ones that turn out to be correct.

Give each of the two models the memory its workload actually wants. The second bottleneck is that test-time scaling runs two models on one device — a generator that produces reasoning tokens and a verifier that scores steps — and they must share the scarce KV-cache memory. The naive split is even, but that ignores a real asymmetry: the verifier processes prompts in large batches and is compute-bound, while the generator decodes token by token and is memory-bound, so a marginal megabyte of KV cache helps them by very different amounts. FastTTS profiles each model’s throughput as a function of its memory using a roofline view and allocates the asymmetric split that maximizes total system throughput, rather than treating the two as interchangeable. Recognizing that co-located models with different compute-versus-memory characteristics should be provisioned according to their individual bottlenecks, not uniformly, is the same resource-allocation discipline the constraint-aware LLM-inference scheduling work covered elsewhere on this site relies on, tuning each component’s share of a fixed budget to where it yields the most throughput.

Schedule so that what you cached stays worth caching. The third bottleneck is that the parallel paths branch from common ancestors, so they share long reasoning prefixes whose key-value state could be reused — but under a tight memory budget, if two paths sharing a prefix are processed far apart in time, that prefix is evicted between them and recomputed, wasting the opportunity. FastTTS treats execution order as the lever: it models the paths as a tree, defines the cost of switching between batches as the number of cached nodes that must be evicted, and orders the paths to minimize those evictions, greedily grouping paths that descend from the same parent so their shared prefix stays resident. The reuse the system was already capable of is thus actually realized, because the schedule keeps co-dependent work adjacent in time. Ordering computation so that data already in a cache is consumed before it is displaced, turning a latent reuse opportunity into a realized one, is the same temporal-locality principle the multi-turn-dialogue KV-cache-reuse work covered elsewhere on this site is built on, arranging requests so shared context is served from cache rather than rebuilt.

Following a reasoning batch through FastTTS

Running one generate-verify round without stalling, wasting memory, or recomputing

A trace, drawn from the paper's design, of how FastTTS processes a batch of parallel reasoning paths on a constrained GPU.

Step 1 of 5

Parallel paths generate thinking steps of uneven length

A batch of reasoning paths each begins generating its next thinking step. Some paths finish in a few tokens; one straggler runs much longer. On a standard system the whole batch would wait for the straggler while the finished paths leave the GPU idle.

The paper measures that step lengths vary dramatically across paths on reasoning benchmarks, so this straggler gap is the norm, not an edge case.

Why it works

Speculation converts straggler idle time into completed future work. A batch stalls on its longest path while finished paths idle → but speculatively generating those paths’ future tokens fills the GPU during the wait → so next-iteration work is already done, hiding straggler latency with no correctness or tail-latency cost.
Verifying only real tokens keeps speculation free. Speculative tokens could change the search outcome or add latency → but verification ignores them and speculation is killed when standard steps finish → so a wrong guess wastes only spare cycles and the algorithm’s result is provably unchanged.
Workload-matched memory beats an even split. The compute-bound verifier and memory-bound generator gain differently from KV cache → but profiling each model’s throughput-versus-memory curve and splitting asymmetrically → so the same memory yields higher total system throughput than dividing it blindly.
Prefix-aware ordering realizes reuse the cache already allows. Paths sharing a prefix recompute it if processed far apart under a tight budget → but scheduling children of the same parent consecutively keeps the shared prefix resident → so the key-value state is reused instead of evicted and recomputed, cutting redundant work.
LookAhead verification exploits the locality speculation creates. A speculated future step would normally need a second, separate verifier call → but concatenating it with the current step into one call reuses the verifier’s KV cache → so verification latency drops by avoiding an eviction and recomputation.

What it costs (trade-offs and scope)

Speculation can be wasted work. When a speculative guess is not ultimately chosen, the tokens generated for it are discarded, so the technique spends GPU cycles that yield nothing; FastTTS mitigates this by speculating only into otherwise-idle capacity and selecting promising candidates, but under workloads with little idle time or poor predictability the payoff shrinks.
The gains depend on the search having structure to exploit. The benefits come from straggler gaps, two-model memory contention, and shared prefixes; a test-time-scaling method with uniform step lengths, a single model, or independent paths would offer less for FastTTS to optimize, so the improvements are tied to verifier-guided tree search specifically.
Memory allocation rests on profiling. The asymmetric split is chosen from profiled throughput-versus-memory behavior, so it assumes that profile is representative; workloads or model pairs that behave differently from the profiled regime could get a suboptimal split until re-profiled.
It is a serving-system optimization, not a smarter algorithm. FastTTS accelerates existing test-time-scaling methods without changing what they compute, so it inherits their accuracy and their fundamental compute cost; it makes cloud-level reasoning affordable at the edge in latency, but the underlying search is still far more expensive than a single forward pass.
Evidence is specific models, datasets, and one GPU class. Results come from small reasoning models with a process-reward-model verifier on reasoning benchmarks (such as math and code datasets) on a single 24 GB consumer GPU; the 2.2-times-goodput and 38-to-68-percent-latency figures are for these settings against a vLLM baseline, and behavior on other hardware, larger models, or other search methods is inferred rather than exhaustively shown.

The evaluation

Serving verifier-guided test-time scaling on a constrained edge GPU

Property	vLLM baseline	Even memory split	FastTTS
Straggler idle time
Generator/verifier memory
Shared-prefix reuse
Verifier calls on speculated steps
Reported outcome

A comparison drawn from the paper. FastTTS is a plug-and-play library on top of vLLM, evaluated with small reasoning models and a process-reward-model verifier on reasoning benchmarks on a single 24 GB consumer GPU; the 2.2-times-goodput and 38-to-68-percent-latency figures are the paper's reported results versus the vLLM baseline.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The full paper text was available (the authors’ arXiv version, 2509.00195), so this account of the mechanism — the verifier-guided generate-verify test-time-scaling pattern, the three bottlenecks (straggler underutilization, two-model memory contention, shared-prefix recomputation), Speculative Beam Extension with its two-phase preemptible scheduler and LookAhead Verification, Asymmetric Multi-Model Memory Allocation via roofline-guided profiling, and Dynamic Prefix-Aware Scheduling framed as minimizing KV-cache evictions and solved greedily — follows the paper’s own description. The headline numbers are the paper’s reported results: an average 2.2 times higher goodput and 38 to 68 percent lower latency than the vLLM baseline, letting a model of up to 7 billion parameters on a single 24 GB consumer GPU match cloud-model accuracy and cloud-measured latency. Those figures are for specific small reasoning models, a process-reward-model verifier, and reasoning benchmarks on one GPU class; results vary with the model pair, search method, and hardware. Exact per-configuration numbers, the scheduling and allocation formulas, and the selection heuristics should be read from the paper itself.

Background

Why goodput, not throughput, is the right yardstick

It is worth dwelling on a subtle shift this work makes in how success is measured. A conventional LLM serving system is judged on throughput: how many tokens per second it can generate. That is the right metric when every generated token is part of an answer someone wants. But test-time scaling deliberately generates far more tokens than it keeps — it explores many reasoning paths precisely so it can prune most of them, and a token on a pruned path is computed and then thrown away. So raw throughput can look high while much of that work is discarded, which means it is the wrong target to optimize. FastTTS instead centers goodput: the rate of useful work, counting only the tokens that contribute to a surviving path. This reframing matters because it changes what a good optimization looks like — reusing a shared prefix, or hiding a straggler, raises goodput by eliminating waste, whereas naively cranking throughput might just generate more soon-to-be-pruned tokens. Measuring a serving system by the useful work it delivers rather than its gross output, especially when accuracy can be traded against cost, is the same evaluation stance the accuracy-scaling inference-serving work covered elsewhere on this site is built on, judging the system by delivered quality per unit resource rather than undifferentiated volume. It is what makes the case that FastTTS truly brings cloud-level reasoning within an edge device’s budget.

If you remember only three things

1. Edge deployment of language-model agents needs small models, which reason far worse than cloud models; test-time scaling closes the gap by spending inference compute on a verifier-guided search — parallel reasoning paths each generate a thinking step, a process reward model scores and prunes them, and the survivors branch — but on a memory-constrained GPU this is impractically slow because thinking-step lengths vary enormously (a batch stalls on the longest straggler while finished paths idle), the generator and verifier models contend for scarce KV-cache memory with different compute patterns, and the search paths share reasoning prefixes whose cached state is evicted and recomputed.
2. FastTTS is a plug-and-play vLLM library with three techniques matched to those bottlenecks: Speculative Beam Extension fills the idle time behind a straggler by speculatively generating future tokens for finished paths (verifying only real tokens and killing speculation when standard steps finish, so it adds no error or tail latency, with LookAhead Verification merging a speculated step into one verifier call); Asymmetric Multi-Model Memory Allocation profiles the compute-bound verifier and memory-bound generator and splits KV-cache memory to maximize total throughput instead of evenly; and Dynamic Prefix-Aware Scheduling orders paths so those sharing a prefix run consecutively, keeping the shared KV state resident and reused rather than recomputed.
3. On a single 24 GB consumer GPU, FastTTS lets a model of up to 7 billion parameters match cloud-model accuracy and cloud-measured latency, achieving an average 2.2 times higher goodput and 38 to 68 percent lower latency than the vLLM baseline. It optimizes the serving of existing test-time-scaling methods without changing their results, so it inherits their accuracy and cost; its gains depend on the search having stragglers, two-model contention, and shared prefixes to exploit, and it centers goodput (useful work) rather than raw throughput since most generated tokens are pruned.

Check yourself

Check your understanding

1. What is test-time scaling, and why is it attractive for edge devices?
2. Why does running test-time scaling on a standard serving system like vLLM perform poorly on a constrained GPU?
3. How does Speculative Beam Extension speed things up without changing the result or adding tail latency?
4. What are FastTTS's main results, and what is the nature of its contribution?

Sources

The paper is “FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning” by Hao Mark Chen, Zhiwen Mo, Guanxi Lu, Shuang Liang, Wayne Luk, and Hongxiang Fan (Imperial College London) and Lingxiao Ma (Microsoft Research), published at ASPLOS 2026, DOI 10.1145/3779212.3790161. It is licensed CC BY-NC-ND.

This explainer is based on the paper’s full text, obtained from the authors’ arXiv preprint (arXiv:2509.00195). The account of the mechanism — the edge-versus-cloud reasoning gap and test-time scaling as its remedy; the verifier-guided generate-verify pattern (parallel reasoning paths, a process reward model scoring thinking steps, replication of top paths and pruning of the rest); the three bottlenecks on constrained hardware (straggler underutilization from irregular step lengths, two-model KV-cache memory contention, and shared-prefix eviction and recomputation); Speculative Beam Extension with its two-phase preemptible scheduler, verification of only non-speculative tokens, and LookAhead Verification; Asymmetric Multi-Model Memory Allocation via roofline-guided profiling of the generator and verifier; and Dynamic Prefix-Aware Scheduling framed as minimizing KV-cache evictions over a tree of paths and solved by a greedy grouping of same-parent paths — follows the paper’s own description. The reported results cited here are the paper’s own: an average 2.2 times higher goodput and a 38 to 68 percent latency reduction versus the vLLM baseline, enabling models of up to 7 billion parameters on a single 24 GB consumer GPU to match cloud-model accuracy and cloud-measured latency, evaluated with small reasoning models and a process-reward-model verifier on reasoning benchmarks. Exact per-configuration numbers, the scheduling and allocation formulas, and the candidate-selection heuristics should be read from the paper itself.