PF-LLM: Large Language Model Hinted Hardware Prefetching

Ceyu Xu, Xiangfeng Sun, Weihang Li, Chen Bai, Bangyan Wang, Mengming Li and 2 others

A CPU hides slow memory by prefetching — guessing what data it will need next and fetching it early — but the little on-chip predictor has to decide instantly, from runtime signals alone, which of several prefetch strategies to use, and it often guesses wrong; this paper (HKUST and Duke — Ceyu Xu, Zhiyao Xie, Yuan Xie, and colleagues, an ASPLOS 2026 Best Paper) trains a large language model to read the assembly around each load instruction offline — the way an expert programmer would — and emit hints telling the hardware exactly how to prefetch, lifting instructions-per-cycle by 9.8 percent over the best hardware prefetchers.

The argument in one glance

Problem

The prefetcher must guess what memory the CPU needs, instantly

A modern CPU spends much of its time waiting on memory: a single DRAM access can take hundreds of cycles, and this "memory wall" is the biggest obstacle to faster single-thread performance. The main defense is data prefetching — the hardware speculatively pulls data into fast on-chip caches before the core actually asks for it, so that when the demand comes, the data is already there. A prefetcher works by recognizing memory access patterns: researchers have built specialist prefetchers for streams (sequential reads), strides (fixed-step loops), spatial locality (nearby fields of a struct), and irregular pointer-chasing sequences. But real programs are not monolithic — they move through many phases, each with its own dominant memory behavior — so no single specialist covers a whole workload well. Recent designs therefore combine several specialists into an ensemble, which shifts the hard problem from designing individual prefetchers to orchestrating them: at every moment, a selection policy must pick the right sub-prefetcher for the current phase. Pick wrong and you actively hurt performance, evicting useful cache data and wasting memory bandwidth. And all of this must be decided on-chip, in real time, under severe area and latency budgets. The prefetcher is being asked to make a subtle, context-dependent choice with almost no room to think.
What people did

Learn which prefetcher to use online, at runtime

The standard approach to orchestrating an ensemble is online learning: the hardware watches how each candidate sub-prefetcher is doing at runtime and gradually learns a selection policy that matches program phases to the best specialist. It observes accuracy and coverage as the program runs and adjusts which prefetcher it trusts, trying to converge on a good policy for whatever phase the workload is currently in. Everything is derived on the fly, from runtime signals, inside the chip.
Why it fell short

Runtime-only learning is slow, shallow, and blind to the code

Deciding the policy purely at runtime runs into two hard walls. First, online learning needs a trial-and-error convergence period: it has to try selections and observe their effects before it settles on a good policy, and that lag means it adapts poorly to rapid phase changes — by the time it has learned the right specialist for a phase, the phase may be over. Second, the on-chip area and latency budgets are brutal, so whatever learning happens must be a simple heuristic; there is no room in the pipeline to run a rich analysis or to consider the broader program context. The result is a selection policy that is both slow to adapt and shallow in what it can consider. Yet here is the striking thing the runtime never sees: the right prefetching strategy is often plainly written in the static code. An experienced developer, looking at the source or assembly around a load, can usually tell at a glance whether it is a sequential string scan (stream), a fixed-step array traversal (stride), a struct field access (spatial), or an atomic lock acquisition that should not be prefetched at all. That context — the structure of the code surrounding each load instruction — carries most of the answer, but it is exactly what the tiny, real-time, runtime-only hardware is unable to read and reason about. What was missing was a way to bring deep, code-aware analysis to bear on the prefetch decision without paying for it at runtime.
This paper's idea

Let an LLM read the code offline and hand the hardware the answer

PF-LLM's idea is to move the hard "when, how, and how aggressively to prefetch" decisions out of the runtime hardware and into an offline analysis powered by a large language model — reading the code the way an expert programmer would. The authors fine-tune an LLM, PF-LLM, to analyze the assembly context surrounding each load instruction and predict the best prefetching strategy for it. It is trained on a large dataset of assembly snippets paired with ground-truth best strategies obtained from architectural simulation, so it learns to map code structure to prefetch behavior. Because this analysis happens offline, it can use a large compute budget to inspect the code far more deeply than any runtime mechanism could. For each load instruction, PF-LLM emits three hints: which sub-prefetcher to use (for example stride or stream), how aggressively to prefetch (the degree), and a demand-request filter that improves the accuracy of certain sub-prefetchers. At runtime, a lightweight hardware prefetcher ensemble called the LMHint Prefetcher simply consumes these precomputed hints to guide its decisions — so it needs no slow online training period and never has to figure out the policy itself. In effect the on-chip prefetcher becomes a near-zero-latency, oracle-level system that follows the best policy for every single load. On memory-intensive SPEC 2017 benchmarks, this delivers a 9.8 percent instructions-per-cycle improvement over state-of-the-art hardware prefetchers and 18.9 percent over state-of-the-art ensemble methods. The paper received an ASPLOS 2026 Best Paper Award.

Why this problem is worth solving

Prefetching is one of the main levers left for single-thread speed, and single-thread speed still governs how fast most software actually runs.

Even in an era of many cores, GPUs, and accelerators, Amdahl’s Law means single-thread performance keeps setting the ceiling on how fast real applications run, because the serial parts of a program dominate as everything else is parallelized. And the dominant limit on single-thread speed is the memory wall: the processor can execute instructions far faster than memory can supply data, and a cache miss to DRAM stalls the core for hundreds of cycles. Prefetching is the primary way to hide that latency without changing the program — if the hardware fetches the right data early, the miss never stalls the core. So improvements in prefetching translate fairly directly into faster software across the board, on the exact workloads (irregular, memory-bound, phase-shifting) that are hardest to speed up any other way. But prefetching is a double-edged tool: a wrong guess does not just fail to help, it evicts useful data and burns bandwidth, so getting the decision right matters as much as making a decision at all. A method that reliably picks the right strategy per load instruction therefore attacks a first-order performance problem, which is why an approach that meaningfully raises instructions-per-cycle on memory-intensive benchmarks is significant.

The key realization is that the information the runtime lacks is sitting in the source code, and it can be extracted once, offline, at leisure.

The deep constraint on runtime prefetch decisions is not a lack of cleverness but a lack of budget: the hardware must decide in a cycle or two, with a few gates of logic, using only what it can observe as the program runs. That budget rules out any rich reasoning, which is why online policies stay simple and slow to adapt. PF-LLM’s insight is that the very information those policies are missing — what kind of access each load performs — is largely determined by the static structure of the code, which never changes and can be read ahead of time. So instead of forcing a tiny real-time circuit to rediscover, over and over, something a programmer could read off the code, you analyze the code once, offline, with as much compute as you want, and simply record the answer as a hint attached to each load. This flips the economics of the problem: the expensive part (understanding the code) happens where compute is essentially free, and the runtime part (following the hint) becomes trivial and instant. It also removes the convergence lag entirely, because the policy is already known before the program runs — there is nothing to learn on the fly. Turning a hard real-time decision into a cheap lookup of a precomputed answer is what lets a heavyweight model like an LLM improve something as latency-critical as a hardware prefetcher.

What people did before, and where it broke

Use a single specialist prefetcher. Build one prefetcher tuned for streams, or strides, or spatial locality. Where it broke. Real programs shift through phases with different access patterns, so any single specialist covers only part of a workload well.
Combine specialists into an ensemble. Run several sub-prefetchers and select among them. Where it broke. The hard problem just moves to orchestration — a selection policy must match each phase to the right specialist, and a wrong pick evicts useful cache data and wastes bandwidth.
Learn the selection policy online. Have the hardware observe each sub-prefetcher’s runtime behavior and learn which to trust. Where it broke. Online learning needs a trial-and-error convergence period, so it adapts too slowly to rapid phase changes.
Keep the runtime mechanism tiny. Fit the policy within on-chip area and latency limits. Where it broke. The budget forces simple heuristics that cannot consider the broader program context, leaving good information on the table.
The gap. No approach used the static code context — which usually reveals the right strategy and which an expert reads easily — to guide prefetching, because that deep, code-aware analysis was infeasible inside real-time hardware.

How PF-LLM works

Moving the prefetch decision from runtime hardware to an offline LLM

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Read the code around each load to decide how to prefetch it. PF-LLM grounds the prefetch decision in the static code context surrounding a load instruction — the structure that tells you whether it is a stream, a stride, a spatial access, or something not to prefetch — rather than trying to infer everything from runtime address streams. Analyzing the specific code that leads up to a memory access, and using that code-level understanding to drive prefetching, is the same principle the work on critical slice prefetching (CRISP) covered elsewhere on this site is built on, extracting the program structure responsible for misses to guide what to fetch. PF-LLM broadens this from a hand-crafted slice analysis to a learned model that reads the assembly context directly.

Do the expensive analysis once, offline, and consume it cheaply at runtime. PF-LLM spends a large offline compute budget running an LLM over the code, then reduces the result to compact per-load hints that a tiny runtime mechanism can follow instantly — so the heavy lifting never touches the critical path. Paying for a powerful language-model analysis a single time, offline, and then serving its distilled output at runtime with negligible cost is the same structure the work on skeleton-guided SMT solver fuzzing with an LLM (Once4All) covered elsewhere on this site uses, investing in the model once and reusing the cheap artifact thereafter. PF-LLM applies that offline-to-online split to a hardware prefetcher, turning an LLM’s code analysis into hints an on-chip ensemble consumes.

Predict the policy in advance instead of learning it reactively. By determining the best strategy for every load before the program runs, PF-LLM removes the slow trial-and-error convergence of online ensemble learning, so the hardware never has to react its way to a good policy. Replacing a reactive runtime control loop that must observe and correct with a decision made ahead of time from foresight is the same shift the work on predicting rather than reacting for fine-grain GPU frequency control (predict, not react) covered elsewhere on this site makes, acting on a precomputed answer rather than converging to one. PF-LLM’s hints are exactly that precomputed answer, so the LMHint Prefetcher adapts to phase changes immediately.

Following a load instruction through PF-LLM

From assembly context to a prefetched cache line

A trace, drawn from the paper, of how PF-LLM turns static code into runtime prefetch hints.

Step 1 of 5

Offline: gather assembly around each load

For a program, PF-LLM takes each load instruction together with the assembly context around it — the surrounding code that determines what kind of memory access this load performs.

This is the same context an expert developer would read to judge the pattern.

Why it works

The answer is in the code, which never changes. A runtime policy must rediscover each load’s pattern from address streams → but the static code around a load already reveals whether it is a stream, stride, spatial, or non-prefetchable access → so reading the code gives the strategy directly.
Offline analysis can be as deep as needed. On-chip logic has a cycle or two and a few gates → but an offline LLM has an essentially unlimited compute budget → so it can inspect code far more thoroughly than any real-time mechanism.
Precomputed hints remove the convergence lag. Online learning needs a trial-and-error period and adapts slowly to phase shifts → but the best strategy per load is decided before the program runs → so the runtime prefetcher is correct immediately, even across rapid phase changes.
The runtime mechanism stays tiny and fast. Rich reasoning cannot fit in the prefetch path → but the hardware only has to look up and follow a hint → so the LMHint Prefetcher remains lightweight and near-zero-latency.
Per-load guidance approaches an oracle. A single heuristic is wrong for many loads → but a hint tailored to each individual load instruction → so the prefetcher follows the best policy everywhere, raising IPC and avoiding wasted bandwidth.

What it costs (trade-offs and scope)

It requires an offline analysis pass over the code. PF-LLM must run the LLM on a program’s loads ahead of time to generate hints; the benefit assumes you can afford this offline step and attach the resulting hints to the program.
Hints are tied to the analyzed code. The guidance is generated for specific assembly, so it applies to code that has been analyzed; the approach targets scenarios where offline analysis of the binary is available, and the hardware must be extended to carry and consume the hints.
The runtime needs the LMHint Prefetcher. Realizing the gains requires the lightweight hint-consuming ensemble in hardware, plus a channel to deliver the hints; it is not a drop-in for an unmodified existing prefetcher.
The LLM is only as good as its training. PF-LLM learns from assembly paired with simulation-derived best strategies, so its predictions depend on that dataset and on how well simulation ground truth transfers; a load whose behavior is not discernible from static context is inherently harder.
Reported gains are on specific benchmarks and baselines. The 9.8 percent and 18.9 percent IPC improvements are averages on memory-intensive SPEC 2017 against particular hardware and ensemble baselines; results on other workloads and hardware may differ.

The comparison

Deciding how to prefetch for each load instruction

Property	Single specialist prefetcher	Online-learned ensemble	PF-LLM + LMHint
Where the decision is made
Uses static code context
Adapts to phase changes
Runtime cost
Performance

A comparison of ways to decide prefetching per load instruction. A single specialist covers only some phases; an online-learned ensemble adapts slowly and cannot read the code. PF-LLM fine-tunes an LLM to analyze the assembly context around each load offline and emit hints (which prefetcher, how aggressively, and a demand filter), which the lightweight LMHint Prefetcher follows at runtime — acting like a near-oracle and raising instructions-per-cycle by 9.8 percent over state-of-the-art hardware prefetchers and 18.9 percent over ensemble methods on SPEC 2017. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ hosted copy at zhiyaoxie.com). The described system is the paper’s: PF-LLM is a large language model fine-tuned to analyze the assembly code context surrounding a load instruction and generate prefetching hints, trained on a dataset of assembly paired with ground-truth best prefetching strategies obtained through architectural simulation. For each load, it emits three hints — the prefetcher selection (e.g. stride or stream), the prefetch degree, and a demand-request filter. These offline-generated hints are consumed at runtime by the LMHint Prefetcher, a lightweight hardware prefetcher ensemble, which thereby avoids the online trial-and-error convergence of conventional ensemble methods. The paper frames this as moving the “when, how, and how aggressively to prefetch” decisions out of runtime hardware into offline LLM-powered analysis, turning the on-chip prefetcher into a near-zero-latency, oracle-level system. Reported results are a 9.8 percent average instructions-per-cycle (IPC) improvement on memory-intensive SPEC 2017 over state-of-the-art hardware prefetching baselines and 18.9 percent over state-of-the-art ensemble methods; the paper received an ASPLOS 2026 Best Paper Award. Background on the memory wall, data prefetching, specialist prefetchers (stream, stride, spatial, irregular), and ensembles is standard context provided to make the approach intuitive. The step-by-step walkthrough is an illustration of the described mechanism. Exact model, training, hint encoding, hardware, and evaluation details should be read from the paper.

Background

Why architecture keeps offloading hard runtime decisions to offline intelligence

It helps to see PF-LLM within a broad and growing pattern: many of the hardest decisions in computer architecture are made by tiny circuits in real time, under punishing budgets, using only what they can observe on the fly — and a recurring way to do better is to move the intelligence off the critical path. Branch predictors, cache replacement, prefetchers, and frequency governors all face the same bind: the decision matters enormously for performance, but the mechanism that makes it has almost no time or logic to think. The escape is to precompute: analyze the program ahead of time, where compute is cheap and abundant, distill the result into compact guidance, and let the fast path simply follow it. What is new here is the analyst — a large language model that can read code with something approaching a developer’s understanding — and the target, a hardware prefetcher, which is about as latency-critical as microarchitecture gets. The broader current PF-LLM rides, using machine learning to make the low-level decisions that architects once hand-tuned, is the same question the keynote on whether machine learning for systems has reached an inflection point (ML-for-systems keynote) covered elsewhere on this site takes up, weighing when learned models genuinely beat hand-crafted heuristics in systems. PF-LLM’s contribution is a concrete, strong data point: by letting an LLM read the code offline and hand the hardware per-load hints, it turns a hard real-time orchestration problem into a near-oracle lookup and measurably raises single-thread performance.

If you remember only three things

1. A CPU hides the hundreds-of-cycles cost of a memory access by prefetching — speculatively pulling data into cache before the core needs it — and a prefetcher works by recognizing access patterns (streams, strides, spatial locality, irregular sequences). Because real programs shift through phases with different patterns, no single specialist suffices, so modern designs use an ensemble of specialists and must decide, per load and per phase, which one to use. But that decision is made on-chip in real time under tight area and latency budgets, so ensembles rely on online learning that needs a slow trial-and-error convergence period and cannot consider the broader program context — even though the right strategy is often plainly visible in the static code, which the runtime hardware never gets to read.
2. PF-LLM moves the hard "when, how, and how aggressively to prefetch" decision out of runtime hardware and into an offline large language model. The authors fine-tune an LLM to read the assembly context around each load instruction and predict the best prefetching strategy, training it on assembly snippets paired with ground-truth strategies from architectural simulation. Because the analysis is offline, it can use a large compute budget to inspect the code far more deeply than any runtime circuit. For each load, PF-LLM emits three hints — which sub-prefetcher to use, the prefetch degree, and a demand-request filter — and at runtime a lightweight hardware ensemble, the LMHint Prefetcher, simply follows them. This removes the online convergence lag entirely and turns the on-chip prefetcher into a near-zero-latency, oracle-level system that follows the best policy for every load.
3. On memory-intensive SPEC 2017 benchmarks, PF-LLM improves instructions-per-cycle by 9.8 percent on average over state-of-the-art hardware prefetchers and 18.9 percent over state-of-the-art ensemble methods, and the paper won an ASPLOS 2026 Best Paper Award — a strong demonstration that LLMs can guide microarchitectural decisions. The trade-offs: it requires an offline LLM analysis pass over the code and a way to attach hints to the program, the hardware must include the hint-consuming LMHint Prefetcher, and the model is only as good as its training data and how well static context reveals a load's behavior. This explainer is based on the paper's full text; the model, hardware, and evaluation details should be confirmed in the paper.

Check yourself

Check your understanding

1. Why is deciding how to prefetch hard for conventional hardware prefetchers?
2. What is PF-LLM's key insight, and what does it do with it?
3. How does the offline-to-runtime split make this practical, and what is the LMHint Prefetcher?
4. What does PF-LLM achieve, and what are the limits of the approach?

Sources

The paper is “PF-LLM: Large Language Model Hinted Hardware Prefetching” by Ceyu Xu, Xiangfeng Sun, Weihang Li, Chen Bai, Bangyan Wang, Mengming Li, Zhiyao Xie, and Yuan Xie (The Hong Kong University of Science and Technology, and Duke University), published at ASPLOS 2026, DOI 10.1145/3779212.3790202. It received an ASPLOS 2026 Best Paper Award, and the authors host a copy at zhiyaoxie.com.

This explainer is written from the paper’s full text (the authors’ hosted copy). PF-LLM is a large language model fine-tuned to analyze the assembly context around each load instruction and generate prefetching hints — the prefetcher selection, the prefetch degree, and a demand-request filter — trained on assembly paired with ground-truth strategies from architectural simulation. These offline hints are consumed at runtime by the LMHint Prefetcher, a lightweight hardware ensemble, so it avoids the online trial-and-error convergence of conventional ensembles and follows a near-oracle policy per load. The paper reports a 9.8 percent average IPC improvement on memory-intensive SPEC 2017 over state-of-the-art hardware prefetchers and 18.9 percent over ensemble methods. Background on the memory wall, data prefetching, and specialist prefetchers is standard context. The numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.