A Prediction System Service

Countless performance tricks in software boil down to a guess — take the lock or gamble on a transaction, compile this now or later, evict this page or keep it — and today each program hides that guess in its own brittle hand-tuned rule; this work from UC Santa Barbara proposes making prediction a shared operating-system service with just two calls, predict and update, so programmers write an optimization and let a fast, feedback-learning predictor decide when to use it, matching or beating hand-crafted heuristics in lock elision, compiler tuning, and memory management with far less code.

The argument in one glance

Problem

Countless performance decisions are really hidden predictions

Squeezing out performance means answering trade-off questions — lock or optimistic transaction, optimize now or later, keep a page or reclaim it. Each is really a prediction, but today that prediction is hidden inside a hand-tuned parameter or ad-hoc heuristic that is brittle, needs deep expertise, and cannot adapt as conditions change.
What people did

Let each application embed its own machine learning

Since machine learning is good at navigating these trade-offs, have each application include its own machine-learning framework to dynamically tune its few parameters.
Why it fell short

Per-app ML duplicates effort and cannot share

Every application would carry its own machine-learning code base just to control a handful of knobs, duplicating work and eliminating any chance to share training data or hardware resources. And deploying prediction on the critical path is hard, because a slow prediction can cost more than it saves.
This paper's idea

Make prediction a shared, low-latency system service (PSS)

The Prediction System Service exposes prediction to the whole software stack with as few as two functions, predict and update. Programmers write optimizations that help in specific cases and leave the decision of when to apply them to the service, which learns from feedback with a fast online perceptron accessed via the low-latency vDSO path. It matches or beats hand-crafted heuristics for lock elision (34 percent faster), compiler tuning (15 percent faster), and page reclaim (33 percent lower memory latency) with a fraction of the complexity.

Why this problem is worth solving

A surprising amount of performance work comes down to guessing, and those guesses hide everywhere.

If you catalog the performance optimizations in a modern system, a huge fraction of them are driven by prediction. The operating system predicts which disk pages a program will need next and prefetches them into memory. It learns a thread’s memory access pattern and migrates pages closer to the processor using them. A lock spins for a while, predicting the holder will release soon, before falling back to heavyweight blocking. Transactional memory speculatively adjusts to how much contention it observes. All of these rest on a prediction — a guess about the near future that, if right, buys performance. But most of them are not explicit about being predictions. Instead, they hide their predictive nature inside the choice of a parameter or a set of selection criteria. That is a problem, because such parameter choices are usually ad-hoc, relying on limited use cases and hard-won domain expertise, which makes them fragile and hard to scale; and even when informed by data, they are typically static, so they cannot adapt when the workload, the machine state, or the objective changes. The result is a landscape of brittle, hand-tuned heuristics, each reinventing prediction in its own corner.

The natural fix, letting each program learn for itself, does not scale, and speed is the hidden obstacle.

Machine learning is demonstrably good at navigating exactly these complex trade-offs, so the tempting response is for each application to embed its own machine-learning framework to control its parameters dynamically. But that makes little sense at scale: every application would have to carry and maintain its own machine-learning code base just to tune a few knobs, and doing so duplicates effort and forecloses any sharing of training information or hardware resources between programs. A better abstraction is to make prediction a shared system service — provided by the operating system, callable with as few as two functions. As a service, it can be reused across the whole stack and across programming languages, the operating system can share training information across applications when appropriate or restrict it by policy, and it can even be used inside the kernel by tasks that otherwise rely on domain-specific heuristics. There is one catch that makes this genuinely hard: because these predictions target performance-critical code, they sit directly on the critical path, so their usefulness depends on both their accuracy and their latency. A prediction service must be cheap enough that asking it does not cost more than the optimization saves, and good enough that it rivals a hand-tuned approach.

What people did before, and where it broke

Bury prediction inside a hand-tuned parameter or heuristic. Encode the guess as a fixed threshold or selection rule chosen by an expert. Where it broke. These are brittle, require deep domain knowledge, and are static, so they cannot adapt when the workload or machine changes.
Profile offline to pick better parameters. Gather profiles to inform the parameter choices. Where it broke. Profiling needs well-understood use cases or low-overhead production measurement, and the chosen parameters are still static once deployed.
Embed a bespoke machine-learning framework in each application. Let every program learn to control its own knobs. Where it broke. It duplicates machine-learning code across applications, prevents any sharing of training or resources, and is often too slow to sit on the critical path.
The gap. No shared, low-latency, adaptive abstraction let any part of the stack turn a decision into a learned prediction cheaply. Providing that as a system service is what PSS does.

How the Prediction System Service works

From per-application hand-tuned heuristics to a shared learning service

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Turn scattered, hand-rolled prediction into one shared service. The core proposal is to stop having every application reinvent prediction and instead provide it once, as an operating-system service with as few as two functions, predict and update, reusable across languages and even inside the kernel. This lets the system share training across programs, apply policy, and give every optimization the same simple lever. Taking a capability that each system would otherwise build for itself and providing it once as a minimal shared substrate that many different systems reuse is the same move the trusted-network-card work covered elsewhere on this site makes, distilling a needed guarantee into one small, common building block rather than rebuilding it, differently and redundantly, in every system that needs it.

Make the prediction cheap enough to sit on the critical path. Because these decisions guard performance-critical code, a prediction is only useful if it is both accurate and fast, so PSS uses a simple online perceptron — a weighted sum of features compared to a threshold — and reaches it through the vDSO, a low-latency path that avoids the full cost of a system call. Getting out of the slow, general operating-system path so that a frequent, latency-sensitive operation runs at close to hardware speed is the same technique the direct-storage-access work covered elsewhere on this site uses, bypassing the heavy kernel route for the common case so the operation is cheap enough to do constantly.

Learn the decision from feedback instead of hand-tuning it. Rather than fixing a parameter by expert intuition, PSS lets the perceptron learn from observed outcomes: after each decision, update tells the service whether it paid off, and the weights adjust, so the predictor tracks the actual workload and machine state. Replacing hand-picked static configuration with a mechanism that searches for and adapts to the good setting from real feedback is the same shift the tinyML design-exploration work covered elsewhere on this site embodies, letting an automatic optimizer discover the effective configuration rather than trusting a person to tune it by hand.

Following a lock-elision decision through PSS

Deciding whether to gamble on a hardware transaction or take the lock

A trace of PSS guiding transactional lock elision, following the paper's example.

Step 1 of 5

A choice: lock, or optimistic transaction

A program must protect a shared data structure. It can take a normal lock (safe, but slower when contention is low) or optimistically use a hardware transaction (fast if it commits, wasted work if it aborts).

This is a classic fast-path versus slow-path decision.

Why it works

Many optimizations are really hidden predictions. Because a huge share of performance techniques come down to guessing whether a fast path will pay off → those guesses are usually buried in brittle static parameters → so exposing them as explicit, learnable predictions is a broadly useful change.
A shared service beats per-application machine learning. Because each application would otherwise embed its own machine-learning code to tune a few knobs → that duplicates effort and prevents sharing across programs → so one reusable prediction service lowers the barrier and lets the system share training and resources.
Learning from feedback beats static tuning. Because a hand-tuned parameter cannot adapt to changing workloads or machine state → a predictor that updates from observed outcomes tracks the real conditions → so it matches or beats hand-crafted heuristics without their fragility.
Latency matters as much as accuracy here. Because these predictions sit directly on the critical path of performance-critical code → a slow prediction would cost more than it saves → so making the service cheap, with a simple perceptron and the vDSO path, is what makes it usable at all.
Simplicity generalizes. Because even a straightforward perceptron works across three very different scenarios → the same tiny service helps user code, a language runtime, and the kernel alike → so the approach is broadly applicable rather than tuned to a single problem.

What it costs (trade-offs and scope)

The proof-of-concept predictor is deliberately simple. PSS uses a single online perceptron making one-dimensional (true or false) predictions with up to 16 features. The interface allows swapping in richer models (decision trees, or heavier ones like gradient boosting or neural networks), but the demonstrated results come from the simple version, chosen for low latency.
Latency, not just accuracy, bounds what is usable. Because predictions are on the critical path, a more accurate but slower model can be a net loss. The approach depends on keeping the prediction cheap (the vDSO path, a light model), which constrains how sophisticated the predictor can be.
The programmer still supplies the features and the optimization. PSS decides when to apply an optimization, but a person must still write the optimization that helps in specific cases and choose the features to feed the predictor. Poor features or a poorly-conceived optimization limit what the service can do.
Online learning has a warm-up and can mispredict. The perceptron learns from feedback over time, so early decisions may be poor, and mispredictions still happen; the design leans on optimizations where a wrong prediction is a modest performance cost rather than a correctness problem.
The evidence is three case studies. Generality is argued from three scenarios (hardware lock elision, PyPy compiler tuning, Linux page reclaim) spanning user and kernel mode and multiple languages. These are strong, varied demonstrations rather than a proof that every optimization fits the two-function pattern.
It is a system-service change. Realizing the full vision means providing the service in the operating system (and ideally sharing training and adding hardware support), which is a change to the platform, not just a library an application links.

The evaluation

Three ways to make a performance decision

Property	Hand-tuned static heuristic	Per-application embedded ML	Prediction System Service
Adapts to changing workload		Yes	Yes, learns from feedback
Reusable across apps and languages
Cheap enough for the critical path
Programmer complexity	Low but brittle
Shares training across programs	Not applicable
Demonstrated result	Baseline	Varies

Three ways to make a runtime performance decision, as framed by the paper. A hand-tuned static heuristic is cheap but brittle and cannot adapt. Embedding a machine-learning framework in each application adapts but duplicates code across programs, shares nothing, and is often too slow for the critical path. The Prediction System Service exposes prediction to the whole stack with two functions, predict and update, learning the decision with a fast online perceptron reached through the low-latency vDSO path. Across three very different scenarios it meets or beats the best-known hand-crafted methods with a fraction of the complexity: guiding hardware lock elision for a 34 percent average speedup, tuning PyPy's just-in-time compiler parameters for a 15 percent average speedup, and guiding Linux kernel page reclaim to cut average memory latency by 33 percent — the last outperforming human-optimized heuristics developed within the prior year. The results hold as long as the service latency is kept low.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This paper makes a clean conceptual argument and backs it with varied evidence. The insight is that an enormous number of runtime performance decisions — lock or transaction, optimize now or later, keep a page or reclaim it — are really predictions in disguise, hidden inside brittle, static, hand-tuned parameters that each application reinvents. Rather than have every program embed its own machine-learning framework, the authors propose prediction as a shared operating-system service with just two functions, predict and update, so a programmer writes an optimization that helps in specific cases and lets the service learn when to apply it. The design respects the hard constraint that these predictions sit on the critical path: it uses a simple online perceptron reached through the low-latency vDSO path, keeping each prediction cheap. The demonstrations are convincing precisely because they are so different from one another — guiding hardware lock elision (34 percent average speedup), tuning a just-in-time compiler (15 percent), and steering Linux page reclaim (33 percent lower memory latency, beating heuristics tuned within the previous year) — spanning user and kernel mode and multiple languages, each matching or beating hand-crafted methods with far less complexity. Read the scope precisely. The predictor is deliberately simple, and latency, not just accuracy, bounds how sophisticated it can be; the programmer still supplies the optimization and the features; online learning has a warm-up and can mispredict, so it suits decisions where a wrong guess costs performance, not correctness; and realizing the full vision means changing the operating system, not just linking a library. The durable idea is that prediction deserves to be a first-class, shared, low-latency system abstraction — that separating the writing of an optimization from the learned decision of when to use it can make performance programming both simpler and better.

Background

Why separating policy from mechanism, and learning the policy, keeps paying off

It helps to place PSS in a long-running theme: systems work better when the mechanism (what an optimization does) is separated from the policy (when to apply it), and when that policy is learned from real behavior rather than fixed by hand. Here the mechanism is any optimization a programmer writes, and the policy is the perceptron’s learned decision about when it helps. The secure-undervolting work covered elsewhere on this site reflects the same instinct on the hardware side, replacing a single conservative, worst-case setting with a dynamic decision that adapts to what the running code actually needs, made cheaply enough to happen constantly. The common thread is that static, one-size-fits-all choices leave performance on the table, because the right choice depends on conditions that change; a cheap, adaptive decision maker that watches outcomes and adjusts can capture that lost value, provided it is fast enough not to become the bottleneck itself. There is a second lesson in PSS about where such a decision maker belongs — not duplicated inside every application, but factored out into a shared service that the whole stack, including the kernel, can reuse and that the system can govern. As software and hardware grow more complex and the number of tunable decisions multiplies, both instincts — learn the policy from feedback, and share the machinery for doing so — are likely to become more important, not less.

If you remember only three things

1. A large fraction of runtime performance optimizations are really predictions — guesses about whether a fast path will pay off, like taking a lock versus an optimistic hardware transaction, compiling now versus later, or keeping a memory page versus reclaiming it. Today that predictive nature is hidden inside hand-tuned parameters and ad-hoc heuristics that are brittle, need deep expertise, are static (so they cannot adapt), and are reinvented separately in every application. Having each program embed its own machine-learning framework to tune a few knobs does not scale: it duplicates code, shares nothing, and is often too slow for the critical path.
2. The Prediction System Service (PSS) makes prediction a shared operating-system service with as few as two functions: predict, which takes a feature vector and returns a decision, and update, which reports whether the decision paid off so the model learns. The proof-of-concept uses a simple online perceptron (a weighted sum of up to 16 features against a threshold, with weights nudged up on correct predictions and down on wrong ones), chosen because predictions sit on the critical path and must be fast as well as accurate; it is reached through the low-latency vDSO path to avoid full system-call overhead, and the interface allows swapping in richer models. As a service it is reusable across languages and inside the kernel, and the system can share training across programs or restrict it by policy.
3. Across three deliberately different scenarios, PSS meets or beats the best-known hand-crafted methods with a fraction of the complexity: it guides hardware transactional lock elision for a 34 percent average speedup, tunes PyPy's just-in-time compiler parameters for a 15 percent average speedup, and steers Linux kernel page reclaim to cut average memory latency by 33 percent (beating heuristics tuned within the previous year), spanning user and kernel mode and multiple languages. Scope: the predictor is deliberately simple and latency-bounded, the programmer still supplies the optimization and features, online learning has a warm-up and can mispredict (so it suits performance, not correctness, decisions), and the full vision is an operating-system change. The durable idea is that prediction deserves to be a first-class, shared, low-latency abstraction that separates writing an optimization from the learned decision of when to use it.

Check yourself

Check your understanding

1. What is the central observation behind the Prediction System Service?
2. Why make prediction a shared system service rather than have each application embed its own machine learning?
3. Why does PSS use a simple perceptron reached through the vDSO, rather than a heavier model behind a normal system call?
4. What do the three case studies (lock elision, JIT tuning, page reclaim) demonstrate about PSS?

Sources

The paper is “A Prediction System Service” by Zhizhou Zhang, Alvin Oliver Glova, Timothy Sherwood, and Jonathan Balkind, published at ASPLOS 2023, DOI 10.1145/3575693.3575714. The authors are at UC Santa Barbara.

This explainer is based on the full text of the paper, obtained as the open-access author copy hosted on the authors’ university website. The problem framing (the prevalence of prediction-driven optimizations such as prefetching, page migration, adaptive locking, and transactional memory; the observation that their predictive nature is hidden in brittle, static, hand-tuned parameters; and the argument against each application embedding its own machine-learning framework), the design (prediction as a system service with the predict, update, and reset functions; the online perceptron predictor computing a weighted sum against a threshold, with up to 16 hashed features and 1024 entries each, updated by reinforcing or penalizing weights; the extensibility of the interface to other models; and the use of the vDSO for low-latency access), and the evaluation results (the 34 percent average speedup for hardware transactional lock elision, the 15 percent average speedup for PyPy just-in-time compiler parameter tuning, and the 33 percent average reduction in memory latency for Linux kernel page reclaim, each meeting or beating hand-crafted methods and the last outperforming heuristics developed within the prior year) are all drawn directly from the paper. Background explanations of transactional memory, just-in-time compilation, page reclaim, perceptrons, and the vDSO are standard and provided to make the paper’s contribution understandable.