LAIKA: Machine Learning-Assisted In-Kernel APU Acceleration

Operating-system kernels increasingly want to run small machine-learning models to make smart decisions, but offloading them to a discrete GPU means shipping data across the PCIe bus, which can eat more than 93 percent of the time; this paper (South China Normal University — Haoming Zhuo, Dingding Li, Yong Tang, and colleagues) argues the fix is not a faster GPU but the integrated GPU in an APU, whose memory is shared with the CPU, and presents LAIKA, a kernel framework that exploits that unified memory for true zero-copy inference — cutting latency up to 9.7 times and power to under a third.

The argument in one glance

Problem

Running ML in the kernel is killed by shipping data to a discrete GPU

There is growing interest in putting small machine-learning models inside the operating-system kernel, so that the kernel can make smarter decisions — which page to evict, how to schedule, whether to admit an I/O request — by learning from data rather than following a fixed heuristic. These kernel decisions are latency-sensitive: they sit on the hot path of the system, so the inference that guides them has to be fast. The obvious way to run a model fast is to offload it to a GPU. But in a typical machine that GPU is a discrete GPU (dGPU), a separate card connected over the PCIe bus with its own memory, and that is exactly where the plan falls apart. To run an inference on a dGPU, the kernel must copy the input data from host memory across PCIe to the GPU's memory, run the model, and copy the result back. For the tiny models and small inputs typical of kernel tasks, the actual computation is trivially fast — but the data transfer across PCIe dominates, consuming over 93 percent of the total execution time. In other words, the bottleneck is not compute but I/O: the system spends almost all its time moving data to and from the accelerator, not accelerating. So a more powerful dGPU does not help at all, because it makes the fast part faster and leaves the slow part — the transfer — untouched.
What people did

Reach for a more powerful discrete GPU

The instinctive response to slow acceleration is to get a better accelerator: a faster, higher-end discrete GPU with more compute and more memory bandwidth. This is the standard path for GPU workloads, and for compute-bound tasks it works. The assumption is that if inference is slow, the GPU needs to be stronger, so upgrading the card should shorten the time each kernel decision takes.
Why it fell short

A faster GPU cannot fix an I/O bottleneck

This response misdiagnoses the problem. When over 93 percent of the time is spent transferring data across PCIe, the computation is not the bottleneck — the movement of data to and from the accelerator is. A more powerful discrete GPU speeds up only the small fraction of time actually spent computing, while the dominant cost, the PCIe transfer, is fixed by the bus and the architecture, not by how strong the GPU is. So upgrading the card yields almost nothing for these latency-sensitive kernel tasks; you can double the GPU's power and barely move the end-to-end latency. The deeper issue is architectural: a discrete GPU is separated from the CPU and kernel by a bus, so every use requires crossing it. What was missing was the recognition that for I/O-dominated kernel inference the answer is not a stronger accelerator but a different architecture — one where the accelerator and the CPU share memory, so there is no transfer to pay for in the first place.
This paper's idea

Use the integrated GPU's unified memory for zero-copy in-kernel inference

LAIKA argues that the right accelerator for latency-sensitive kernel ML is not a discrete GPU but the integrated GPU (iGPU) built into modern APUs — chips that put a CPU and a GPU on the same die. The decisive feature of an APU is unified memory: the iGPU and the CPU share the same physical memory, so data the kernel already holds can be used by the GPU with no copy across any bus. That erases the 93-percent-plus transfer cost that crippled the dGPU. LAIKA is a kernel-space acceleration framework that fully exploits this unified memory, built from three mechanisms. First, a lightweight API proxy, AProxy, safely bridges the kernel-user boundary: GPU runtimes live in user space, but the ML task is triggered in the kernel, so AProxy lets kernel code invoke the GPU through the user-space API cheaply and safely. Second, a tri-domain shared memory manager, AShm, provides true zero-copy data exchange across all three domains that touch the data — the kernel, user space, and the GPU — so an input is never copied as it passes between them. Third, an APU Persistent Kernel, APK, eliminates control-path overhead: instead of launching a fresh GPU kernel for every inference (which carries dispatch cost), a persistent kernel stays resident on the iGPU and is simply signaled to process each new request. Across a suite of representative kernel workloads, by eliminating the I/O bottleneck LAIKA slashes end-to-end inference latency by up to 9.7 times and reduces total system power to as little as 28.9 percent of an optimized discrete-GPU baseline — identifying iGPUs as a practical, highly efficient way to accelerate machine learning inside the OS kernel.

Why this problem is worth solving

Learned kernel policies could make the operating system smarter across the board, but only if the inference behind them is fast enough to sit on the hot path.

The operating system makes countless small decisions — caching, scheduling, memory management, I/O admission — that today rely on hand-tuned heuristics, and there is strong evidence that learning-based policies can do better. But a learned policy is only usable in the kernel if evaluating the model is cheap, because these decisions are on the critical path of the system: a slow inference does not just cost the model’s own time, it stalls whatever the kernel was deciding about. If offloading a tiny model to an accelerator takes far longer than the heuristic it replaces, the learned policy is a net loss no matter how good its predictions are. So the practicality of the whole idea of a machine-learning-assisted kernel hinges on driving inference latency down to where it fits on the hot path. Because the bottleneck turns out to be data movement rather than compute, solving it does not require exotic hardware — it requires the right architecture. Making in-kernel ML fast enough therefore unlocks a broad space of smarter OS policies on commodity machines, which is a large payoff for a targeted architectural insight.

The key realization is diagnostic: the bottleneck is I/O, not compute, so the answer is architecture, not horsepower.

The entire argument turns on correctly identifying what is slow. It is natural to assume that a slow accelerated task needs a faster accelerator, and for genuinely compute-bound work that is right. But kernel ML tasks use tiny models on small inputs, so the computation is nearly free; what dominates is shuttling the data to the accelerator and back. On a discrete GPU that shuttle crosses the PCIe bus, and the paper measures it at over 93 percent of the total time. Once you see this, the conclusion is forced: making the compute faster with a stronger GPU attacks the 7 percent and ignores the 93 percent. The real lever is to remove the transfer entirely, which is an architectural property, not a performance tier. An APU’s integrated GPU shares memory with the CPU, so there is no separate device memory to copy into and no bus to cross — the transfer that dominated simply does not exist. This is why LAIKA frames its contribution as a shift to an I/O-efficient architecture rather than a faster one: the problem was never a lack of compute, so the solution is not more of it, but an arrangement where the data the kernel holds is already where the GPU can use it. Everything else in the design exists to realize that zero-copy potential without reintroducing overhead elsewhere.

What people did before, and where it broke

Offload kernel inference to a discrete GPU. Send the model and data to a separate GPU card to run fast. Where it broke. Copying data across the PCIe bus to and from the dGPU consumes over 93 percent of execution time, so the transfer, not the compute, dominates.
Upgrade to a more powerful discrete GPU. Buy a stronger card to shorten inference. Where it broke. A faster GPU only speeds the small compute fraction; the PCIe transfer that dominates is fixed by the architecture, so latency barely improves.
Copy data between kernel, user space, and the GPU as needed. Move the input through each domain that must touch it. Where it broke. Each copy adds latency, and for small kernel inferences these copies are a large part of the cost.
Launch a fresh GPU kernel per inference. Dispatch the model to the GPU each time a decision is needed. Where it broke. The per-launch control-path overhead is significant relative to the tiny computation of a kernel-sized model.
The gap. No approach removed the data-transfer bottleneck itself by using an architecture where the accelerator and CPU share memory, with the kernel-user bridging and control overhead also eliminated.

How LAIKA works

Accelerating kernel ML on an integrated GPU with shared memory

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Put learning inside the kernel, where the decisions are made. LAIKA’s premise is that the operating system’s own hot-path decisions should be guided by learned models, run in kernel space rather than shipped elsewhere. Bringing machine learning into the operating-system kernel so its policies can be learned rather than hand-tuned is the same direction the work on whether machine learning for systems has reached an inflection point covered elsewhere on this site surveys, arguing that learned policies are becoming practical throughout the stack. LAIKA tackles the concrete obstacle that has held this back — making the inference fast enough to belong on the kernel’s hot path.

Eliminate the transfer by sharing memory instead of moving data. LAIKA’s core architectural move is to use an integrated GPU whose memory is unified with the CPU, so the data the kernel holds is already where the GPU can read it, with no copy. Removing the cost of moving data between a processor and an accelerator by having them share the same memory, rather than transferring across a bus, is the same principle the work on tensor migration and prefetching in unified memory (DeepUM) covered elsewhere on this site is built on, exploiting unified memory so data need not be shipped. LAIKA applies it to kernel inference, where the shared-memory zero-copy is what erases the 93-percent transfer cost.

Keep the worker resident so you never pay to start it up. LAIKA’s persistent kernel stays alive on the iGPU and is merely signaled per request, avoiding the dispatch overhead of launching a fresh kernel each time. Holding an execution context resident and ready so repeated work avoids the cost of cold startup is the same idea the work on warming serverless functions with heterogeneity (IceBreaker) covered elsewhere on this site develops, keeping the executor warm rather than paying launch cost on every invocation. LAIKA does this on the GPU, so each tiny kernel inference triggers a ready worker instead of a fresh launch.

Following a kernel inference through LAIKA

Running an in-kernel model with zero copies and no launch

An illustrative walkthrough, based on the paper's abstract (this explainer is written without the full paper text; exact details are in the paper).

Step 1 of 5

The kernel needs a prediction on the hot path

A kernel decision — say, whether to admit an I/O request — calls for a model inference. The input data already lives in memory the kernel owns; on a discrete GPU it would now have to be copied across PCIe.

Latency here is on the critical path of the system.

Why it works

Shared memory removes the transfer that dominated. A discrete GPU forces copying data across PCIe, over 93 percent of the time → but an APU’s integrated GPU shares memory with the CPU → so the data is already where the GPU can use it and the transfer disappears.
Fixing I/O beats making compute faster. A stronger GPU speeds only the small compute fraction → but removing the data movement attacks the dominant cost → so end-to-end latency falls where a faster card could not move it.
Zero-copy across domains avoids per-copy latency. Passing data between kernel, user space, and GPU normally copies it each time → but a tri-domain shared buffer exposes one memory to all three → so a small input is never copied as it moves between them.
A resident kernel removes launch overhead. Launching a fresh GPU kernel per inference carries dispatch cost that dwarfs a tiny model → but a persistent kernel is just signaled per request → so the control-path overhead per inference is eliminated.
The architecture also cuts power. A discrete GPU draws large power for its compute and its transfers → but an integrated GPU doing zero-copy work → so total system power drops to as little as 28.9 percent of the discrete-GPU baseline.

What it costs (trade-offs and scope)

The integrated GPU is weaker for compute-bound work. An iGPU has far less raw compute and memory bandwidth than a high-end discrete GPU; LAIKA wins because kernel ML is I/O-bound, so for genuinely compute-heavy models a discrete GPU would still be faster and the architecture argument would not hold.
It requires an APU with unified memory. The whole approach depends on the CPU and GPU sharing physical memory on one die; systems with only a discrete GPU, or without an APU, cannot use the zero-copy path, so this is hardware-specific.
Bridging the kernel-user boundary must be done carefully. AProxy lets kernel code invoke a user-space GPU runtime, which crosses a protection boundary; the design has to keep this safe and lightweight, and that bridging is machinery a purely user-space or purely kernel solution would not need.
A persistent kernel occupies the GPU. Keeping a kernel resident on the iGPU to avoid launch cost means it holds GPU resources between requests, a trade of occupancy for latency that suits latency-sensitive kernel tasks but is not free.
Reported gains are workload- and baseline-specific. The up-to-9.7-times latency reduction and 28.9-percent power figure are measured on a suite of representative kernel workloads against an optimized discrete-GPU baseline; the exact improvement depends on the model, the input sizes, and the specific hardware.

The evaluation

Accelerating latency-sensitive machine learning inside the OS kernel

Property	Discrete GPU	More powerful discrete GPU	LAIKA (integrated GPU)
Data path to the accelerator
Effect of faster compute
Copies between kernel, user, GPU
Per-inference launch overhead
Latency and power

A comparison of ways to run latency-sensitive ML inside the OS kernel. By shifting from a discrete GPU (dominated by PCIe transfers) to an APU's integrated GPU with unified memory, and adding a lightweight kernel-user API proxy (AProxy), a tri-domain zero-copy shared memory manager (AShm), and a persistent GPU kernel (APK), LAIKA removes the I/O bottleneck, cutting end-to-end inference latency by up to 9.7 times and total system power to as little as 28.9 percent of an optimized discrete-GPU baseline. This explainer is written from the paper's abstract; exact figures and mechanisms should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is written from the paper’s abstract (the full text was not accessible at the time of writing — the ASPLOS 2026 paper is open access under CC BY, but the copy was reachable only through a publisher page that could not be retrieved, and no author-hosted copy or preprint was found). The described system is the paper’s: LAIKA is a kernel-space acceleration framework arguing that for latency-sensitive in-kernel machine-learning tasks, whose cost on a discrete GPU is dominated by PCIe data transfer (over 93 percent of total execution time), the solution is the I/O-efficient architecture of the integrated GPU (iGPU) in modern APUs, which shares unified memory with the CPU. LAIKA combines three mechanisms: a lightweight API proxy (AProxy) to safely bridge the kernel-user boundary; a tri-domain shared memory manager (AShm) for true zero-copy data exchange among the kernel, user space, and GPU; and an APU Persistent Kernel (APK) to eliminate control-path (per-launch) overhead. The paper reports that, by eliminating the I/O bottleneck, LAIKA reduces end-to-end inference latency by up to 9.7 times and total system power to as little as 28.9 percent of an optimized discrete-GPU baseline, across a suite of representative kernel workloads, and identifies iGPUs as a practical, efficient alternative for machine-learning acceleration within OS kernels. The background above on APUs, unified memory, PCIe transfers, and persistent GPU kernels is standard architecture context provided to make the approach intuitive; the paper’s own account of AProxy, AShm, and APK, and the full evaluation, are the authority. The step-by-step walkthrough is illustrative, not a claim about the paper’s exact implementation.

Background

Why the accelerator's bottleneck keeps turning out to be data movement, not compute

It helps to see LAIKA within a broad pattern: over and over, when a workload is offloaded to an accelerator and turns out to be slow, the culprit is not the accelerator’s compute but the cost of getting data to it and back. Discrete accelerators are separated from the processor by a bus, and for small or latency-sensitive work that separation dominates, because the transfer time does not shrink as the compute gets faster. The instinctive fix — a bigger, stronger accelerator — attacks the part that was already cheap and leaves the expensive part untouched, which is why so much progress instead comes from rearchitecting around where the data actually lives: sharing memory, co-locating compute with storage, or removing copies from the path. LAIKA is a clean instance of this reframing for the operating-system kernel: it recognizes that in-kernel ML is I/O-bound, so it chooses an architecture where the accelerator and the CPU already share memory, rather than a faster one that would still have to cross a bus. This willingness to rearchitect a system around its true bottleneck — data movement rather than raw speed — is the same terrain the work on re-architecting I/O caches for emerging fast storage devices covered elsewhere on this site occupies, redesigning the path once the hardware balance shifts. LAIKA’s contribution is to show that, for machine learning inside the kernel, the integrated GPU’s unified memory turns a transfer-dominated task into a zero-copy one, making learned kernel policies fast and low-power enough to be practical on commodity hardware.

If you remember only three things

1. There is growing interest in running small machine-learning models inside the OS kernel to make its hot-path decisions (caching, scheduling, I/O admission) smarter, but these are latency-sensitive, so the inference must be fast. Offloading to a discrete GPU fails at this: for the tiny models and small inputs of kernel tasks, the computation is trivial, but copying the data across the PCIe bus to the GPU and back consumes over 93 percent of total execution time. The bottleneck is I/O, not compute, so a more powerful discrete GPU barely helps — it speeds only the small compute fraction while the dominant transfer, fixed by the bus and architecture, is unchanged.
2. LAIKA argues the right accelerator is not a faster discrete GPU but the integrated GPU (iGPU) in a modern APU, where the CPU and GPU share the same physical memory (unified memory), so the data the kernel holds is already where the GPU can use it — eliminating the transfer. It is a kernel-space framework built on three mechanisms: a lightweight API proxy (AProxy) that safely bridges the kernel-user boundary so kernel code can drive the user-space GPU runtime; a tri-domain shared memory manager (AShm) that exposes one buffer to the kernel, user space, and GPU for true zero-copy data exchange; and an APU Persistent Kernel (APK) that stays resident on the iGPU and is signaled per request, eliminating the per-launch control-path overhead that would otherwise dominate a tiny inference.
3. By removing the transfer, the copies, and the launch overhead, the paper reports LAIKA cuts end-to-end inference latency by up to 9.7 times and total system power to as little as 28.9 percent of an optimized discrete-GPU baseline, across a suite of representative kernel workloads, identifying iGPUs as a practical way to accelerate in-kernel ML. The trade-offs: an integrated GPU is weaker for genuinely compute-bound work (where a discrete GPU would still win), the approach requires an APU with unified memory, safely bridging the kernel-user boundary adds machinery, and a persistent kernel occupies the GPU between requests. This explainer is based on the abstract; full details are in the paper.

Check yourself

Check your understanding

1. Why is running machine learning inside the OS kernel so slow when offloaded to a discrete GPU?
2. What is LAIKA's central architectural argument?
3. What do LAIKA's three mechanisms — AProxy, AShm, and APK — each do?
4. What does LAIKA achieve, and what are the limits of the approach?

Sources

The paper is “LAIKA: Machine Learning-Assisted In-Kernel APU Acceleration” by Haoming Zhuo, Dingding Li, Ronghua Lin, and Yong Tang (South China Normal University), published at ASPLOS 2026, DOI 10.1145/3779212.3790181. The paper is open access under CC BY.

This explainer is written from the paper’s abstract; the full text was not accessible at the time of writing (the open-access copy was reachable only through a publisher page that could not be retrieved, and no author-hosted copy or preprint was found). LAIKA is a kernel-space acceleration framework arguing that latency-sensitive in-kernel machine-learning tasks, whose discrete-GPU cost is dominated by PCIe data transfer (over 93 percent of total execution time), are best served by the integrated GPU in modern APUs, which shares unified memory with the CPU. It combines a lightweight API proxy (AProxy) to safely bridge the kernel-user boundary, a tri-domain shared memory manager (AShm) for true zero-copy data exchange among the kernel, user space, and GPU, and an APU Persistent Kernel (APK) to eliminate control-path overhead. The paper reports that, by eliminating the I/O bottleneck, LAIKA cuts end-to-end inference latency by up to 9.7 times and total system power to as little as 28.9 percent of an optimized discrete-GPU baseline across a suite of representative kernel workloads, identifying iGPUs as a practical, efficient alternative for machine-learning acceleration within OS kernels. The background on APUs, unified memory, PCIe, and persistent kernels is standard architecture context; exact mechanisms and full results should be confirmed in the paper.