DFVG: A Heterogeneous Architecture for Speculative Decoding with Draft-on-FPGA and Verify-on-GPU

Shaoqiang Lu, Yangbo Wei, Junhong Qian, Dongge Qin, Shiji Gao, Yizhi Ding and 4 others

Speculative decoding speeds up large language models by having a cheap draft model guess several tokens ahead and a big target model verify them all at once — but drafting and verifying want very different hardware, so running both on the same GPU wastes it, the fixed guess-trees generate mostly-rejected candidates, and the constant back-and-forth between the two stages leaves each idle waiting on the other; this paper (Shaoqiang Lu, Lei He, and colleagues) presents DFVG, which runs the draft stage on an FPGA and the verify stage on a GPU, adapts how much it guesses to the model's confidence, and overlaps the two so neither stalls — reporting up to 3.26 times faster decoding and 5.8 times better energy efficiency.

The argument in one glance

Problem

Speculative decoding is fast, but its two stages fight over one GPU

Large language models generate text one token at a time, and each step is a full pass through a huge model, which is slow and leaves the GPU underused because there is not much work per step. Speculative decoding is a clever fix: a small, cheap draft model guesses several of the next tokens ahead, and then the big, expensive target model verifies all of those guesses at once in a single pass, keeping the ones it agrees with and discarding the rest. Because the target model checks many tokens in parallel instead of producing them one by one, the whole system runs faster while still producing exactly what the target model would have. But three problems hold it back. First, drafting and verifying are very different jobs — the draft model is small and latency-sensitive, the verification is a big parallel matrix computation — so running both on the same GPU underutilizes it and wastes energy, because while one stage runs the hardware suited to the other sits idle. Second, to raise the chance that guesses are accepted, systems propose a whole tree of candidate continuations, but these trees use a fixed shape that generates many candidates of which only a few paths are ever valid, so much of the drafting work is thrown away. Third, the two stages alternate constantly and asynchronously, so each spends time idle waiting for the other, and when a guess is wrong the system must roll back, adding overhead.
What people did

Run speculative decoding on the GPU with fixed token trees

The standard approach keeps everything on the GPU: the draft model and the target model both run there, and the drafts are structured as fixed-pattern token trees — a preset branching shape that proposes many candidate next-token sequences at once, so that the verification pass has a good chance of accepting a long run of tokens.
Why it fell short

One device, fixed trees, and constant alternation waste work and energy

This leaves large gains on the table. Putting both models on one GPU means their mismatched resource needs collide: the lightweight, sequential drafting cannot fill the GPU, and while it runs the GPU's verification muscle is idle, so utilization and energy efficiency suffer. The fixed-pattern token tree is wasteful because it commits to the same branching regardless of how confident the draft model actually is at each step: when the model is confident, most of the tree's branches are unnecessary; when it is unsure, the fixed shape may still miss the right continuation — either way, many candidates are generated that verification will reject, so drafting effort is wasted. And because drafting and verification alternate in lockstep, each waits on the other, and mispredicted drafts force rollbacks, so the pipeline stalls repeatedly instead of keeping both kinds of hardware busy. What was missing was a design that gives each stage the hardware it actually wants, adapts the amount of drafting to the model's confidence, and overlaps the stages so neither idles.
This paper's idea

Draft on an FPGA, verify on a GPU, and overlap the two

DFVG is a heterogeneous speculative-decoding architecture that offloads draft generation to an FPGA and verification to a GPU, exploiting their complementary strengths, and it rests on three ideas. First, a heterogeneous architecture partitions speculative decoding into FPGA-based drafting (well suited to the small, latency-sensitive, custom-dataflow draft model) and GPU-based verification (well suited to the large parallel matrix computation), coordinated by an overlap processor for high-throughput execution — so each stage runs on the hardware it fits, and the GPU is freed to focus on verifying. Second, hardware-aware dynamic draft generation predicts, at each step, how many speculative branches and how many tokens to draft based on the draft model's confidence, while respecting the FPGA's parallelism limits — so it drafts more where the model is confident and less where it is not, instead of a wasteful fixed tree. Third, a tightly-coupled heterogeneous pipeline with stage-decoupled scheduling allocates execution windows between the drafting and verifying stages so they overlap rather than idle-wait, and adds lightweight cross-device alignment and rollback prediction to keep the FPGA-to-GPU handoff cheap and to reduce the cost of wrong guesses. Evaluated on mainstream models (OPT, LLaMA, Qwen), the authors report DFVG achieves up to 3.26 times speedup and 5.8 times better energy efficiency over existing approaches.

Why this problem is worth solving

Serving large language models is dominated by the slow, one-token-at-a-time decoding step, so anything that speeds it up without changing the output is enormously valuable.

Generating text from a large language model is inherently sequential: each token depends on the ones before it, so the model runs a full forward pass per token, and because each pass does little arithmetic relative to the memory it must move, the GPU is poorly utilized and latency is high. Speculative decoding is one of the most effective known remedies, because it converts many cheap sequential draft steps plus one parallel verification into fewer effective steps, and it does so losslessly — the output matches what the target model would have produced on its own. That makes it broadly applicable across models and deployments. But its benefit is capped by a hardware reality: the two stages have opposite appetites. Drafting is small, latency-bound, and does not need a GPU’s full width, while verification is a wide, compute-heavy matrix operation that does. Forcing both onto the same device means one of them always leaves the hardware underused, which shows up as low utilization and wasted energy — exactly the costs that matter at data-center scale. So closing the gap between speculative decoding’s promise and its realized efficiency is worth real architectural effort.

The deeper inefficiencies are that the guessing is one-size-fits-all and the two stages keep waiting on each other.

Two structural choices in typical speculative decoding leave performance behind. The first is the fixed-pattern token tree: to raise the odds that verification accepts a long run of tokens, the draft stage proposes a tree of candidate continuations, but with a fixed branching shape applied at every step regardless of context. This ignores a signal the draft model already has — its own confidence. When the model is highly confident about the next token, most of the tree’s alternative branches are wasted; when it is uncertain, a fixed shape may still fail to include the token that would have been accepted. Either way, the stage produces many candidates that verification discards, spending draft compute on paths that never pay off. The second is the lockstep alternation: drafting and verification run as separate phases that hand off to each other, so while one runs the other’s hardware is idle, and every time a draft is rejected the system must roll back and redo work. Both problems are about coordination and adaptivity — matching the amount of speculation to the confidence, and keeping both stages busy at once — and both are hard to fix when everything shares one device and one rigid schedule.

What people did before, and where it broke

Run both draft and target models on the GPU. Keep the whole speculative-decoding loop on one device. Where it broke. The two stages have mismatched resource needs, so the lightweight drafting cannot fill the GPU and leaves it idle while it runs, hurting utilization and energy efficiency.
Use fixed-pattern token trees to raise acceptance. Propose a preset tree of candidate continuations each step. Where it broke. The fixed shape ignores the draft model’s confidence, so it generates many candidate paths that verification rejects, wasting drafting effort.
Alternate drafting and verification asynchronously. Switch back and forth between the two stages. Where it broke. The stages wait on each other and mispredicted drafts trigger rollbacks, so the pipeline stalls repeatedly instead of keeping hardware busy.
Scale up the GPU or the draft model to compensate. Throw more of the same hardware at the problem. Where it broke. It does not address the underlying mismatch, the wasted candidates, or the idle waiting, so the extra resources are themselves underused.
The gap. No speculative-decoding design gave each stage hardware matched to its needs, adapted the amount of drafting to model confidence, and overlapped the stages to eliminate idle waiting.

How DFVG works

Splitting speculative decoding across an FPGA and a GPU

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Give each stage the hardware it actually fits. DFVG’s foundation is heterogeneity: rather than forcing drafting and verifying onto one device, it maps each to the accelerator whose strengths match it, using an FPGA for the small custom draft model and a GPU for the wide verification. Composing complementary accelerators and routing each part of a workload to the hardware that runs it best is the same undertaking the work on a unified framework for heterogeneous FPGA acceleration (Harmonia) covered elsewhere on this site pursues, treating an FPGA as a first-class partner in a heterogeneous system. DFVG applies this split to speculative decoding, so the GPU is freed to do only what it is best at.

Guess cheaply, then confirm — and only keep the good guesses. The whole speculative-decoding paradigm DFVG accelerates is a draft-then-verify loop: produce candidate tokens quickly with a small model, then have the large model check them in parallel and accept only the ones it agrees with. Speeding a task up by generating speculative candidates cheaply and then verifying them against the authoritative computation, keeping only what passes, is the same principle the work on a draft-then-verify exploration mechanism (Pruner) covered elsewhere on this site is built on, trading cheap speculation for expensive confirmation. DFVG makes this loop efficient by putting the drafting and verifying on different, well-matched devices and overlapping them.

Adapt the amount of work to the input at hand. DFVG’s dynamic draft generation refuses to spend a fixed amount of speculation everywhere; it reads the draft model’s confidence and drafts more or fewer branches and tokens accordingly. Tailoring how much computation to do to the specific input, rather than provisioning uniformly for the worst case, is the same idea the work on input-adaptive algebraic transformations for high performance (Voyager) covered elsewhere on this site develops, letting the data decide how much effort is worthwhile. DFVG uses the model’s own confidence as that signal, avoiding the wasted candidates of a fixed token tree while respecting the FPGA’s parallelism limits.

Following a decoding step through DFVG

Producing the next tokens across two devices

An illustrative trace, based on the paper's abstract and the authors' repository, of DFVG's flow. Exact mechanisms and results should be read in the paper.

Step 1 of 5

The FPGA drafts candidate tokens

The small draft model runs on the FPGA, generating candidate next tokens quickly and at low energy, using the FPGA's custom dataflow rather than tying up the GPU with a poorly-matched small-model workload.

Drafting is latency-sensitive and light, a good fit for an FPGA.

Why it works

Matching each stage to its hardware raises utilization. One GPU cannot suit both a small draft model and a wide verification → but running drafting on an FPGA and verification on a GPU gives each stage well-fitted hardware → so neither device sits idle waiting for a mismatched workload.
Freeing the GPU lets it focus on verification. Drafting on the GPU steals cycles from the work it is best at → but moving drafting to the FPGA leaves the GPU for the parallel verification matmul → so the GPU’s strength is spent where it counts.
Confidence-driven drafting cuts wasted candidates. A fixed token tree generates many paths verification rejects → but predicting branches and lengths from the draft model’s confidence → drafts more only where acceptance is likely, so less effort is thrown away.
Overlapping the stages removes idle waiting. Lockstep alternation makes each stage wait on the other → but stage-decoupled scheduling gives them overlapping execution windows → so the FPGA drafts the next step while the GPU verifies the current one.
Cheap alignment and rollback prediction keep the split efficient. A heterogeneous design risks costly handoffs and rollbacks → but lightweight cross-device alignment and rollback prediction limit those costs → so the overhead of splitting across devices stays small.

What it costs (trade-offs and scope)

It requires an FPGA alongside the GPU. DFVG’s gains come from heterogeneous hardware, so it needs an FPGA (with its synthesis and bitstream toolchain) in addition to the GPU; deployments without an FPGA, or where adding one is impractical, cannot use this design.
The draft model must fit and map well onto the FPGA. Running the draft model on an FPGA means it has to fit the device’s resources and parallelism limits, and the dynamic draft generation explicitly respects those limits, so very large draft models or unusual architectures may be harder to accommodate.
Cross-device coordination adds complexity. Splitting the loop across two devices introduces a handoff, alignment, and scheduling machinery (the overlap processor, stage-decoupled scheduling, rollback prediction) that a single-device design avoids; these are engineered to be lightweight, but they are real added complexity.
The benefit depends on speculative decoding’s own dynamics. Speedups from speculative decoding vary with how often drafts are accepted, which depends on the model pair and the workload; DFVG’s reported gains are on particular mainstream models, and other models or tasks could shift them.
This explainer is drawn from limited source material. The full paper text was not accessible for this write-up; the description here is based on the paper’s abstract and the authors’ public repository, so the precise microarchitecture, scheduling algorithm, and detailed results should be confirmed in the paper.

How it compares

Running speculative decoding: one device versus DFVG

Property	GPU-only, fixed token trees	Larger GPU / draft model	DFVG
Where drafting runs
Where verification runs
Draft amount
Stage coordination
Reported gains

A qualitative comparison based on the paper's abstract and the authors' repository. DFVG splits speculative decoding into FPGA-based drafting and GPU-based verification, adapts the amount of drafting to the draft model's confidence, and overlaps the stages with decoupled scheduling, reporting up to 3.26 times speedup and 5.8 times better energy efficiency over existing approaches on OPT, LLaMA, and Qwen. Specific per-model figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s abstract and the authors’ public code repository, not its full text, which was not accessible for this write-up. The system and claims described are the authors’: DFVG is a heterogeneous speculative-decoding architecture that offloads draft generation to FPGAs and verification to GPUs. Its three stated contributions are a heterogeneous architecture partitioning speculative decoding into FPGA-based drafting and GPU-based verification with an overlap processor for high-throughput execution; hardware-aware dynamic draft generation that predicts speculative branches and token lengths from the draft model’s confidence while respecting hardware parallelism limits; and a tightly-coupled heterogeneous pipeline with stage-decoupled scheduling that allocates execution windows between stages, plus lightweight cross-device alignment and rollback prediction. The authors report that, evaluated on mainstream models (OPT, LLaMA, Qwen), DFVG achieves up to 3.26 times speedup and 5.8 times energy-efficiency improvement over existing approaches. The 3.26 times and 5.8 times figures are the authors’ reported headline results; no other quantitative results, figure contents, or precise microarchitectural or scheduling details are asserted here, because the full paper was unavailable. Exact mechanisms, the experimental setup, per-model results, and ablations should be read from the paper and the repository. This page is marked as a draft for that reason.

Background

Why LLM inference keeps pushing past a single accelerator

It helps to see DFVG within a broad shift in how large language models are served: the realization that a single kind of accelerator, run in a single rigid way, leaves large efficiency gains unclaimed. Autoregressive decoding is awkward for GPUs because it is sequential and light per step, so a whole family of techniques has grown up to reshape the work — batching many requests, overlapping the compute-heavy and memory-heavy phases, and speculating ahead so the expensive model can verify many tokens at once. Each of these is really about the same thing: keeping expensive hardware busy with useful work instead of stalling on dependencies. As models and deployments scale, the pressure grows to match different parts of the pipeline to different hardware and to schedule them so they overlap, rather than treating one GPU as a monolith that does everything in turn. The broad effort to squeeze more useful throughput out of the hardware serving large language models is the same terrain the work on boosting GPU utilization for LLM serving (Bullet) covered elsewhere on this site occupies, attacking the same idle-hardware problem from the scheduling side. DFVG contributes the heterogeneous-hardware angle, arguing that drafting and verifying are different enough jobs that they deserve different devices, coordinated to run at the same time.

If you remember only three things

1. Speculative decoding speeds up large language model inference by having a small draft model guess several tokens ahead and a large target model verify them all in one parallel pass, keeping only the accepted ones — losslessly faster than one-token-at-a-time decoding. But three problems limit it: drafting and verifying need very different resources, so running both on one GPU underuses it and wastes energy; fixed-pattern token trees generate many candidate paths that verification rejects, ignoring the draft model's confidence; and the two stages alternate in lockstep, causing idle waiting and rollback overhead.
2. DFVG is a heterogeneous architecture that offloads drafting to an FPGA and verification to a GPU. Its three contributions are: a heterogeneous partition of the two stages onto complementary hardware with an overlap processor; hardware-aware dynamic draft generation that predicts how many branches and tokens to draft from the model's confidence while respecting FPGA parallelism limits (instead of a fixed tree); and a tightly-coupled pipeline with stage-decoupled scheduling that overlaps the stages, plus lightweight cross-device alignment and rollback prediction to keep handoffs and wrong guesses cheap.
3. On mainstream models (OPT, LLaMA, Qwen), the authors report DFVG achieves up to 3.26 times speedup and 5.8 times better energy efficiency over existing approaches. The trade-offs: it requires an FPGA alongside the GPU (with its toolchain), the draft model must fit the FPGA's resources and parallelism, cross-device coordination adds complexity, and the benefit depends on speculative decoding's acceptance dynamics for the given models. This explainer is a draft based on the abstract and the authors' repository; the precise microarchitecture, scheduling, and full results should be confirmed in the paper.

Check yourself

Check your understanding

1. What is speculative decoding, and why does it speed up large language model inference?
2. Why does running both stages of speculative decoding on the same GPU waste resources?
3. How does DFVG's dynamic draft generation improve on fixed-pattern token trees?
4. Predict: DFVG runs the FPGA drafting and GPU verification with stage-decoupled scheduling instead of strict alternation. What is the intended effect, and what remaining cost must it manage?

Sources

The paper is “DFVG: A Heterogeneous Architecture for Speculative Decoding with Draft-on-FPGA and Verify-on-GPU” by Shaoqiang Lu, Yangbo Wei, Junhong Qian, Dongge Qin, Shiji Gao, Yizhi Ding, Qifan Wang, Chen Wu, Xiao Shi, and Lei He, published at ASPLOS 2026, DOI 10.1145/3779212.3790153. The code is available at github.com/ShaoqiangLu/DFVG.

This explainer is written from the paper’s abstract and the authors’ public repository, because the full paper text was not accessible for this write-up (the ACM version was not retrievable and no author preprint was found). DFVG is a heterogeneous speculative-decoding architecture that offloads draft generation to FPGAs and verification to GPUs, exploiting their complementary strengths. Its three contributions, as stated by the authors, are: a heterogeneous architecture partitioning speculative decoding into FPGA-based drafting and GPU-based verification with an overlap processor for high-throughput execution; hardware-aware dynamic draft generation that dynamically predicts speculative branches and token lengths based on model confidence while considering hardware parallelism limits; and a tightly-coupled heterogeneous pipeline with stage-decoupled scheduling that allocates execution windows between stages, combined with lightweight cross-device alignment and rollback prediction strategies. The authors report that, evaluated on mainstream models (OPT, LLaMA, Qwen), DFVG achieves up to 3.26 times speedup and 5.8 times energy-efficiency improvement over existing approaches. The 3.26 times and 5.8 times figures are the authors’ reported headline results; no other quantitative results or precise mechanisms are asserted here, and they should be confirmed in the paper and repository. This page is marked draft because the full text was unavailable.