Limoncello: Prefetchers for Scale

Hardware prefetchers guess what a program will read next and fetch it early — they are designed and tuned against benchmark suites, where memory bandwidth is effectively free, and then deployed in datacentres where bandwidth is the scarcest thing on the machine; this paper from Google turns the prefetchers off across hundreds of thousands of production servers to find out what they were really costing, discovers they were adding 15 percent to memory latency at peak, and builds a system that switches them off exactly when bandwidth is tight while hand-inserted software prefetches cover the handful of library functions that genuinely needed them.

Why this problem is worth solving

The mechanism in question is a bet, and the stake is not what people usually count.

The resource it spends is the one a fleet actually runs out of.

What people did before, and where it broke

• Ship prefetchers always on and tune them on benchmark suites. The industry default, and the reason prefetchers work as well as they do on the workloads they were tuned against. Where it broke. Benchmark suites do not reproduce a datacentre’s bandwidth pressure. Measured on one such suite across three processor generations, enabling prefetchers raises memory bandwidth usage by 30 to 40 percent — and the newest generation is the worst of the three, so the mismatch is widening rather than closing.
• Throttle prefetchers in hardware based on contention or accuracy. A well-studied family of designs that dials aggressiveness up and down at runtime. Where it broke. These are coarse — they apply to all code regardless of which parts matter — so they cannot help when prefetch-friendly and prefetch-hostile code are interleaved finely. They are also reactive rather than anticipatory.
• Predict per-load whether prefetching will pay. Finer-grained hardware proposals use learned filters or criticality prediction to suppress useless prefetches. Where it broke. They remain limited by how good the underlying prefetcher is. If the hardware only predicts a region moderately well, suppressing it loses coverage that nothing else replaces.
• Let the compiler insert software prefetches. Static analysis finds simple access patterns and emits prefetch instructions, with extensions for pointer-based structures. Where it broke. Static analysis has no dynamic execution information, so it cannot know how far ahead to fetch or how often a path is taken.
• Use profile-guided software prefetching. Run representative benchmarks, collect profiles, insert prefetches where the profile says they help. Where it broke. It depends on having representative benchmarks and inputs. In a fleet of tens of thousands of distinct workloads with enormous input diversity, that representativeness does not exist.
• The gap. Every one of these decides in the wrong place. Hardware throttling can see contention but not which code matters; compilers can see code but not contention; profile-guided tools need a benchmark that stands in for a fleet. Nobody had used the fleet itself as the measuring instrument, and nobody had treated the choice between hardware and software prefetching as a runtime decision rather than a design-time one.

How a fleet decides when prefetching is worth it

The idea

The prefetcher is optimising the wrong thing, and only production can show it. A prefetcher spends memory bandwidth to buy cache hits. Whether that is a good deal depends entirely on how scarce bandwidth is, and the environment where prefetchers are designed is not the environment where they run. The measured consequence is stark: the load-to-use latency curve roughly doubles as bandwidth utilisation rises, and having prefetchers on lifts the high end of that curve further still, through queueing delays and inaccurate fetches that a lightly loaded machine would never notice. Bandwidth spent on a speculative fetch is also bandwidth denied to a co-located workload’s actual demand. The insight that the right prefetching decision depends on conditions no design-time analysis can observe, and therefore has to be made while the program is running, is the same one the runtime prefetch generator covered elsewhere on this site reaches from the single-machine direction, where the invisible variables are input data and processor model rather than fleet load.

A fleet-wide ablation is a measuring instrument, and a better one than a benchmark. The usual way to find software prefetch targets is to run a profile on a benchmark believed to represent production. This paper substitutes something more direct: disable the hardware feature on one large group of production machines, leave it enabled on another, profile both at once, and read off which functions bleed. Because the fleet is large, a sampling profiler that runs only in short bursts on random machines still accumulates a meaningful picture. What comes back is a per-function ledger of exactly who depended on the prefetcher — and it is not uniform, since many functions run faster with prefetching off, from lower latency and less cache pollution. Treating the production system itself as the experimental apparatus, rather than trying to build a benchmark that stands in for it, is the same methodological move the hyperscale architecture search covered elsewhere on this site makes when it searches on live traffic instead of a curated dataset.

Software prefetching is precise for a reason that only applies to a few functions. The functions that turned out to need help are the datacentre tax — compression, serialising and deserialising remote calls, hashing, and memory copying — leaf functions that every distributed application uses and that account for 30 to 40 percent of large-fleet cycles. They are ideal software prefetch targets because their signatures tell you what you need to know: the function knows the source address, the destination address, and exactly how many bytes it will touch. So a software prefetch never over-fetches and always knows when to stop, whereas a hardware prefetcher has to learn the pattern from scratch each time and has no idea where the stream ends. That difference matters most for short streams, which the profiling shows dominate; the workloads that regressed had memory copies 26 percent larger on average than those that did not, because a hardware prefetcher only recovers its warm-up cost on longer copies. Splitting the work so that software does the analysis it is uniquely positioned to do and hardware handles only what needs to be fast is the same division of labour that critical-slice prefetching covered elsewhere on this site arranges through a one-byte instruction prefix.

Walking one socket through a traffic peak

The signal is genuinely noisy. A minute-by-minute trace from a representative machine is described as volatile and hard to predict, which is why the controller cannot simply act on the latest reading.

Why it works

• The two regimes want opposite things, and only one of them was ever being served. When bandwidth is plentiful, extra memory traffic is free and minimising cache misses is the right objective → when bandwidth is scarce, every speculative fetch queues ahead of a real one and minimising latency is the right objective → a controller that switches between them gets the correct answer in both, which no fixed setting can.
• Hysteresis converts a noisy signal into a stable decision. Bandwidth measured every second swings enough that a single threshold would toggle the prefetchers constantly → separate on and off thresholds mean a socket that dips briefly stays in its current state → and a dwell requirement means only sustained conditions cause a switch, so the controller tracks load rather than noise.
• The ablation finds targets that a benchmark could not. Profile-guided prefetching needs a benchmark representative of production, which does not exist across tens of thousands of diverse workloads → running the experiment on the production fleet itself removes the need for a stand-in → and because the profiler compares an experimental group against a control group function by function, the output is a direct list of who depended on the prefetcher.
• Library functions know things a prefetcher has to guess. A copy routine has the source pointer, the destination pointer and the byte count in its arguments → so it can prefetch exactly the right addresses, never over-fetch, and stop precisely at the end of the stream → whereas hardware must detect the pattern from observed accesses and has no signal for where it ends, which is why short streams are where it loses.
• A small number of functions carry a disproportionate share of the fleet. The datacentre tax accounts for 30 to 40 percent of cycles across a handful of well-contained library routines → so hand-tuning a few of them is enough to cover most of the damage from disabling the hardware → and the same argument explains why the approach scales at all, since inserting prefetches everywhere across a fleet would be infeasible.

What it costs (trade-offs and scope)

• Half the system is a regression on its own. Disabling hardware prefetchers costs an average of 5 percent across the fleet. The paper reports this plainly. The two halves only work as a pair, and the software half exists to repair damage the hardware half causes.
• The software prefetching was slow, manual work. The authors state that inserting the prefetches took many days of extensive testing and careful tuning, and attribute the toil to two things they lacked: visibility into application memory access patterns, and benchmarks that reflect a datacentre deployment. Both appear in their own list of open problems.
• It is all or nothing per socket. For a given platform, all prefetchers are disabled together. There is no selectivity among prefetcher types, and no per-core or per-workload granularity, even though the machines run hundreds of services each.
• Only four function categories are covered. Some non-tax functions also regress when prefetching is disabled, and those were deliberately left alone as not hot enough to justify individual optimisation. Workloads that lean on them presumably still lose.
• The thresholds are empirical and per-platform. 80 and 60 percent of saturation came out of a two-week controlled experiment on 20,000 machines across geographic zones. Nothing suggests those numbers transfer to a different processor, and the saturation point itself is established per socket during machine qualification.
• A one-second sample against a volatile signal, damped by a dwell time. Bursts shorter than the dwell requirement are invisible to the controller by construction. That is the correct engineering choice for stability, and it is also a real limit on how quickly the system can respond.
• The software half contributes a small share of the headline. Adding software prefetches reduces fleet cycles spent in the targeted functions by about 2 percent. Most of the 10 percent throughput gain comes from turning the hardware off; the software half is what makes doing so safe.
• The tuning is fitted to today’s traffic. Prefetch distances and degrees were chosen against current call-size distributions. The argument that they will keep working is that these functions have well-understood access patterns and that profiling continues — an argument rather than a measurement over time.
• None of it is reproducible outside Google. The fleet, the telemetry, the profiler, the platforms and the thresholds are all internal, and no artefact is offered.

The evaluation

Check yourself

Sources

The paper is “Limoncello: Prefetchers for Scale” by Akanksha Jain, Hannah Lin, Carlos Villavieja, Baris Kasikci, Chris Kennelly, Milad Hashemi and Parthasarathy Ranganathan, published at ASPLOS 2024, pages 577-590, DOI 10.1145/3620666.3651373. Akanksha Jain and Hannah Lin contributed equally.

This explainer is written from the full 14-page text. Everything above comes from the paper’s own sections: the motivation, covering the plateau in per-core memory bandwidth against pin limits, the roughly 1.4 times growth in fleet bandwidth usage over four years at about 10 percent a year, the roughly 3 gigabytes per second per core achievable on the two studied platforms, the finding that bandwidth saturates at 40 to 60 percent processor utilisation against a 70 to 80 percent target, the table of bandwidth reductions from disabling prefetchers on both platforms at average, tail and peak, and the benchmark-suite measurement showing 30 to 40 percent more bandwidth with prefetching enabled across three processor generations; the hardware component, covering the per-second socket telemetry, the model-specific register actuation, the two forms of hysteresis and the worked timeline example, and the 20,000 machine two-week controlled experiment used to set the 80 and 60 percent thresholds; the software component, covering the function-level ablation profiling methodology, the mixed workload results including the average 5 percent regression alongside gains of more than 10 and more than 30 percent on specific services, the identification of the four datacentre tax categories and the reasons they are amenable to software prefetching, the three design parameters of address, distance and degree, the copy-size distribution with its short calls and long tail, the 26 percent larger copies in regressing workloads, and the microbenchmark sweeps and load tests used for tuning; and the evaluation, covering throughput by utilisation band, median and tail memory latency reductions, socket bandwidth and saturated socket counts, the shift in where saturation occurs, and the roughly 2 percent cycle reduction attributable to the software prefetches, together with the discussion section on what would make hardware and software prefetching better.