Stramash: A Fused-Kernel Operating System For Cache-Coherent, Heterogeneous-ISA Platforms

Tong Xing, Cong Xiong, Tianrui Wei, April Sanchez, Binoy Ravindran, Jonathan Balkind and 1 other

A machine can now mix CPUs that speak different instruction sets — an x86 chip and an Arm chip sharing one coherent memory — but the operating systems built for such machines still treat each CPU as a separate computer, passing messages and copying memory pages back and forth; this work from Edinburgh, UC Santa Barbara and Virginia Tech builds the fused-kernel OS, where the separate kernels reach directly into each other's data through the shared coherent memory instead of messaging, prototypes it as Stramash-Linux on top of Linux, tests it on a new fused QEMU simulator, and runs up to 2.1 times faster than the state-of-the-art multiple-kernel OS.

The argument in one glance

Problem

New hardware shares memory across different-ISA CPUs, but the OS ignores it

A heterogeneous-ISA machine mixes CPUs that speak different instruction sets, such as x86 and Arm. New coherent interconnects like CXL now give those CPUs cache-coherent shared memory, so the machine behaves like a shared-memory multiprocessor with mismatched cores. But the operating systems built for such machines were designed before this and do not use the shared memory at all.
What people did

Run a multiple-kernel OS that passes messages

Existing multiple-kernel operating systems, like Popcorn-Linux, run one kernel per CPU and coordinate by passing messages, deliberately sharing nothing. To give a migrating application one consistent view of memory, they replicate and copy memory pages between the kernels in software.
Why it fell short

Message passing and page replication are slow and complex

Treating a coherent shared-memory machine as a distributed system is wasteful: every shared operation becomes a message with serialization, copying, and coordination, and keeping memory consistent means replicating pages and running a software consistency protocol. This overhead is a large part of why heterogeneous-ISA systems have been considered a liability.
This paper's idea

Fuse the kernels through coherent shared memory (Stramash)

The fused-kernel OS goes beyond the multiple-kernel design: instead of sharing nothing, the kernels are shared-mostly, reaching directly into each other's data structures through cache-coherent shared memory and nearly eliminating messages. The prototype, Stramash-Linux, shares physical pages instead of replicating them and lets one kernel walk the other's page tables directly. Tested on a new fused QEMU simulator, it runs up to 2.1 times faster than the state-of-the-art multiple-kernel OS.

Why this problem is worth solving

There is a growing class of machines that mix instruction sets, and they have been held back by their software.

Most computers use processors that all speak the same instruction set — the low-level vocabulary of operations a CPU understands, such as x86 or Arm. A heterogeneous-ISA platform instead puts CPUs of different instruction sets into one machine, for instance an x86-64 processor next to an Arm one. This is attractive as a lighter, more programmable alternative to bolting on specialized accelerators: general-purpose code can run on whichever instruction set executes it most efficiently, and a program can even migrate from one CPU to the other mid-execution. The catch has always been the software. Historically these platforms were loosely coupled, connected over a bus like PCIe with no hardware cache coherence between the different CPUs, so each CPU had to run its own separate operating system and its own software stack. From the application’s point of view the machine looked like a distributed system of separate computers rather than one machine, which meant it could not take advantage of per-platform optimizations and paid constant coordination overhead. As a result, ISA heterogeneity came to be seen as a liability that adds complexity, rather than an opportunity — and because the software was painful, there was little motivation to build more advanced hardware.

The software that does exist was built for a world without shared memory, and that world is ending.

Because a classic operating system for a multiprocessor is compiled for a single instruction set, it cannot run across CPUs of different ISAs. To bridge them, researchers built multiple-kernel operating systems, such as Popcorn-Linux, which run one kernel instance per CPU and have those instances coordinate by passing messages to one another. Crucially, these designs are shared-nothing: they were deliberately built to avoid using shared memory between kernel instances, both for scalability and because, on the loosely-coupled hardware of the time, coherent shared memory simply did not exist. To give a single application a consistent view of memory as it migrates between CPUs, a multiple-kernel OS uses software distributed shared memory, replicating memory pages between the kernels and passing them as messages, often with a copy-on-write policy to reduce the traffic. Now the hardware is changing underneath these assumptions. New coherent interconnects — CXL, OpenCAPI, CCIX — and integrated research chips provide cache-coherent shared memory across different-ISA CPUs. The machine now behaves like a shared-memory multiprocessor that happens to have mismatched cores. Yet the shared-nothing operating systems ignore this new shared memory entirely, leaving a large suite of possible performance improvements on the table.

What people did before, and where it broke

Run a separate OS per CPU. Give each different-ISA CPU its own independent operating system and software stack. Where it broke. Applications are stranded in what looks like a distributed system, unable to share state or exploit the machine as a whole, paying coordination overhead throughout.
Use a multiple-kernel OS with message passing. Run one kernel per CPU and coordinate by exchanging messages, sharing nothing. Where it broke. Every cross-kernel operation becomes a message that must be serialized, copied, and answered, and the coordination protocols make advanced software hard to build.
Keep memory consistent with software distributed shared memory. Replicate pages between kernels and run a copy-on-write consistency protocol so a migrating app sees one address space. Where it broke. Replicating and copying pages, and messaging on every update, is expensive in both time and memory — and it entirely wastes the cache-coherent shared memory that new hardware provides.
The gap. No operating system treated cache-coherent shared memory between different-ISA CPUs as a first principle, letting the kernels share state directly instead of messaging and copying. Building that OS design, and a simulator to develop it on, is what Stramash does.

How Stramash works

From shared-nothing message passing to a shared-mostly fused kernel

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Let the kernels share state directly instead of messaging. The heart of the fused-kernel OS is a shift from shared-nothing to shared-mostly. Where a multiple-kernel OS keeps each kernel instance isolated and has them exchange messages, a fused-kernel OS lets the instances reach directly into one another’s data structures through cache-coherent shared memory, so they work as one and avoid the serialization, copying, and coordination protocols that messaging demands. Replacing round-trip request-and-reply between separate components with direct access to a shared, coherent copy of the state is the same move the serverless-metadata work covered elsewhere on this site makes, running its metadata servers over a coherent cache so they read shared state directly rather than paying a message round-trip for every lookup.

Share the physical pages rather than replicating them. Because both kernels now see the same coherent memory, Stramash-Linux keeps each kernel’s own instruction-set-specific page table but points both at the same physical pages, so an update is instantly visible to both. This lets it bypass the copy-on-write page-replication machinery that the multiple-kernel baseline relies on: a migrated thread’s page faults are handled locally without notifying the other kernel and without copying a page. Avoiding the expense of copy-on-write page duplication by deciding when a page can simply be shared in place is the same concern the copy-on-write pinning work covered elsewhere on this site addresses, reasoning carefully about when a page must be copied versus when it can be left shared, so the costly duplication is only paid when it is truly needed.

Build a fast simulator for hardware that does not exist yet. The coherent heterogeneous-ISA machines Stramash targets are only just emerging, and the FPGA and ASIC prototypes that do exist are slow to iterate on and use less-common instruction sets. So the authors fuse several ordinary single-ISA QEMU emulators together with a cache-coherence plugin, creating the first coherent, heterogeneous-ISA simulator on widely-adopted instruction sets, and develop and measure the OS in software. Standing up a fast software environment to explore and validate hardware before the silicon is available is the same strategy the pre-silicon RISC-V fuzzing work covered elsewhere on this site takes, testing processor designs in a fast simulated setting rather than waiting on real chips to run and check them.

Following a migrating thread through Stramash

An application thread moving from the x86 CPU to the Arm CPU

A trace of a cross-ISA migration under the fused-kernel design, following the paper's Stramash-Linux implementation.

Step 1 of 5

A thread starts on x86 and migrates to Arm

An application begins on the x86 kernel instance and, at a chosen migration point, moves to the Arm CPU to continue executing there, similar to offloading a phase of the computation.

Cross-ISA migration carries over the application state.

Why it works

Coherent shared memory replaces message passing. Because the different-ISA kernels can now read and write each other’s data structures directly through cache-coherent memory → they no longer serialize, copy, and exchange a message for every shared operation → so the coordination overhead that dominated multiple-kernel systems largely disappears.
Sharing pages beats replicating them. Because both kernels point their own page tables at the same physical pages → an update is instantly visible to both without copying or a software consistency protocol → so the distributed-shared-memory page replication and its messaging vanish.
Direct access removes round-trips. Because one kernel can walk the other’s page tables and lock lists itself → a page fault or lock no longer needs a request-and-reply to the origin kernel → so operations that took two round-trips become local memory accesses.
A software simulator makes unbuilt hardware explorable. Because the target coherent heterogeneous-ISA hardware is not yet available and FPGA prototypes iterate slowly → fusing fast QEMU emulators with a cache model lets the design be built and measured in software → so the OS can be developed today and validated to within about 4 percent of real machines.
The benefit depends honestly on the workload. Because the win comes from cheap direct access to shared, sometimes-remote memory → a write-intensive app that keeps data local gains a lot while a read-intensive app that thrashes a small cache into remote memory can gain little → so the paper reports both where the fused kernel helps and where the older replicating design is actually better.

What it costs (trade-offs and scope)

It is not universally faster. The paper is candid that the fused-kernel design helps in most but not all cases. For the read-intensive Conjugate Gradient benchmark on its simulated CXL 3.0 model, the older replicating multiple-kernel design can actually be faster, because Stramash’s direct access reaches into slower remote memory while replication keeps a local copy.
The benefit is workload- and hardware-dependent. Gains vary with an application’s memory access pattern (write-intensive workloads do well; read-intensive ones that evict a small cache into remote memory do poorly) and with the memory model; a larger last-level cache shrank one slowdown from 34 percent to below 1 percent and shrank one speedup from 2.1 to 1.6 times.
The results are in simulation, not on real hardware. The target coherent heterogeneous-ISA hardware does not yet exist commercially, so all measurements come from Stramash-QEMU. The authors validate it to within about 4 percent of real machines on average, but it is still a simulator, and they deliberately do not study scalability, which a software simulator cannot do efficiently.
The prototype supports two instruction sets. Stramash-Linux runs on x86-64 and Arm because it builds on the Popcorn-Linux toolchain, which supports those. Extending to other same-bit-width instruction sets is described as mostly engineering; mixing different bit-widths needs further research.
Direct sharing has requirements and security caveats. Some shared data structures need their randomized memory layout disabled for direct remote access, and letting kernels reach into each other’s memory raises security questions — the paper treats fine-grained protection of the shared region as future work.
It inherits migration’s constraints. Applications must be compiled with the migration toolchain so their state can be carried across instruction sets, so the approach targets code that can be prepared this way rather than arbitrary unmodified binaries.

The evaluation

Three operating-system designs for a cache-coherent heterogeneous-ISA machine

Property	Classic SMP OS	Multiple-kernel OS
Runs across different-ISA CPUs
Uses cache-coherent shared memory	Yes, same ISA only
Cross-kernel coordination	Not applicable
Application pages on migration	Not applicable
Performance	Cannot run	Baseline

Three operating-system designs for a cache-coherent, heterogeneous-ISA machine, as evaluated in the paper. A classic symmetric-multiprocessing OS cannot run across CPUs of different instruction sets at all. A multiple-kernel OS (Popcorn-Linux) runs one kernel per CPU, coordinates by passing messages, and replicates pages with copy-on-write to keep memory consistent. The fused-kernel OS (Stramash-Linux) instead treats cache-coherent shared memory as a first principle: the kernels reach into each other's data structures directly, share the same physical pages, and walk each other's page tables, nearly eliminating messages. On the NPB benchmarks in the Stramash-QEMU simulator, this cuts inter-kernel messages by roughly 99.8 to 99.99 percent (for example, Integer Sort from 207124 messages to 22) and yields up to a 2.1 times speedup over the multiple-kernel baseline. The simulator itself is validated to within about 4 percent of real machines on average. Crucially, the paper reports the design is not universally better — for the read-intensive Conjugate Gradient benchmark on its CXL 3.0 model, the older replicating design can win — so the gain depends on the workload's access pattern and the memory model.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

Stramash is a clear-eyed operating-systems contribution that follows a hardware trend to its software conclusion. As cache-coherent interconnects like CXL start to link CPUs of different instruction sets, a heterogeneous-ISA machine stops being a loose collection of computers and starts behaving like one shared-memory multiprocessor with mismatched cores — and the operating systems built for the old world, which pass messages and replicate pages while sharing nothing, leave that new capability unused. The fused-kernel OS embraces the shared memory as a first principle, letting the per-CPU kernels read and write each other’s data structures directly, share the same physical pages instead of replicating them, and walk each other’s page tables, which nearly eliminates inter-kernel messaging. The prototype, Stramash-Linux, is a real Linux-based system, and because the target hardware does not exist yet, the authors also built Stramash-QEMU, the first coherent heterogeneous-ISA simulator, and validated it to within about 4 percent of real machines. What keeps this honest is the evaluation. The headline is a speedup of up to 2.1 times over the state-of-the-art multiple-kernel OS, with messages and replicated pages cut by orders of magnitude, but the paper openly reports that the fused design is not always the winner: for a read-intensive workload on a CXL-like model, directly reaching into remote memory can be slower than keeping a replicated local copy, so the older design wins there. The results are in simulation, cover two instruction sets, and set scalability, fine-grained security, and mixed-bit-width ISAs aside as future work. The durable idea is that when the hardware changes what is cheap — here, making shared memory coherent across instruction sets — the operating system should be redesigned around the new cost model rather than carried over unchanged.

Background

Why coherent memory rewards software that stops copying and messaging

It helps to place Stramash in a wider pattern. Again and again, when hardware makes a resource cheaper or more directly shareable, the biggest wins come from software that stops doing the expensive thing the old hardware forced — the extra copy, the extra message, the extra round-trip. Here the old cost was treating a coherent machine as a distributed one: replicating pages, running a software consistency protocol, and messaging on every shared operation. The fused-kernel OS removes almost all of it by sharing state in place. The warehouse-scale memory-traffic work covered elsewhere on this site pushes the same idea at a different layer, eliminating the needless data movement — writebacks and fetches that carry nothing new — so scarce memory bandwidth is spent only on transfers that matter. The common thread is that data movement, whether a replicated page, a coordination message, or a redundant cache writeback, is often pure overhead that a system can be redesigned to avoid once the hardware permits. And there is a second lesson in Stramash worth carrying forward: when the hardware you are designing for does not yet exist, building a fast, validated simulator can be the difference between studying a design on paper and actually measuring it. As coherent interconnects reshape how machines are wired together, the systems that thrive will be the ones rebuilt to share directly rather than to copy and coordinate as if still apart.

If you remember only three things

1. A heterogeneous-ISA machine mixes CPUs of different instruction sets, such as x86 and Arm, in one system. New coherent interconnects like CXL now give those CPUs cache-coherent shared memory, so the machine behaves like a shared-memory multiprocessor with mismatched cores. But the operating systems built for such machines — multiple-kernel OSes like Popcorn-Linux — were designed for loosely-coupled hardware with no shared memory: they run one kernel per CPU, coordinate by passing messages while sharing nothing, and keep a migrating application's memory consistent by replicating pages in software with copy-on-write. This messaging and copying is slow and complex, and it ignores the coherent shared memory the new hardware provides.
2. The fused-kernel OS goes beyond the multiple-kernel design from shared-nothing to shared-mostly: the per-CPU kernel instances reach directly into each other's data structures through cache-coherent shared memory, nearly eliminating inter-kernel messages. The prototype, Stramash-Linux (built on Linux atop the Popcorn toolchain, for x86-64 and Arm), keeps each kernel's own instruction-set-specific page tables but points them at the same physical pages, aligns the kernels' address spaces so one can address the other's memory, lets one kernel walk the other's page tables and lock lists directly, and handles a migrated thread's page faults locally — bypassing the copy-on-write page replication the baseline relies on. Because the target hardware does not exist yet, the authors also built Stramash-QEMU, the first coherent heterogeneous-ISA simulator, by fusing single-ISA QEMU emulators with a cache-coherence plugin.
3. On the NPB benchmarks in simulation, Stramash-Linux cuts inter-kernel messages by roughly 99.8 to 99.99 percent (for example, Integer Sort from 207124 messages to 22) and runs up to 2.1 times faster than the multiple-kernel baseline, while Stramash-QEMU is validated to within about 4 percent of real machines on average. The paper is honest about the trade-offs: the fused design is not universally faster — for the read-intensive Conjugate Gradient benchmark on a CXL 3.0 model, the older replicating design can win — so gains depend on the workload's access pattern and the memory model, the results are in simulation, the prototype covers two instruction sets, and scalability, fine-grained security, and mixed-bit-width ISAs are future work. The durable lesson is that when hardware changes what is cheap, the operating system should be redesigned around the new cost model rather than carried over unchanged.

Check yourself

Check your understanding

1. What is a heterogeneous-ISA platform, and why has its software been a problem?
2. How does a fused-kernel OS differ from the multiple-kernel OS it builds on?
3. Why did the authors build Stramash-QEMU, and what does it do?
4. The paper reports that the fused-kernel design is not always faster. What does this show?

Sources

The paper is “Stramash: A Fused-Kernel Operating System For Cache-Coherent, Heterogeneous-ISA Platforms” by Tong Xing, Cong Xiong, Tianrui Wei, April Sanchez, Binoy Ravindran, Jonathan Balkind, and Antonio Barbalace, published at ASPLOS 2025, DOI 10.1145/3676641.3716275. The authors are at the University of Edinburgh, Imperial College London, UC Santa Barbara, and Virginia Tech.

This explainer is based on the full text of the paper, obtained as the open-access (CC-BY) author copy deposited in Virginia Tech’s VTechWorks repository. The problem framing (heterogeneous-ISA platforms, their historically loosely-coupled hardware without cache coherence, the per-CPU separate software stacks, and the emergence of coherent interconnects like CXL, OpenCAPI, and CCIX), the multiple-kernel OS baseline (Popcorn-Linux, shared-nothing message passing, and software distributed shared memory with page replication and copy-on-write), the fused-kernel OS design (the shared-mostly principle, local versus global OS services, the accessor-function and common-format service architectures), the Stramash-Linux implementation (built on Linux 5.2.12 with roughly 9200 lines atop Popcorn, the global memory allocator, the fused virtual address space and fused kernel virtual address space, the software remote page-table and VMA walkers, the page-fault handler that bypasses copy-on-write, and cross-ISA locking with the Stramash page-table lock, CAS-based spinlocks, and futex handling), the Stramash-QEMU simulator (fusing single-ISA QEMU instances with a cache plugin, the Separated, Shared, and Fully Shared memory models, and cross-ISA interrupts), and the evaluation results (the roughly 4 percent average simulator validation error, the message and replicated-page reductions of 99.8 to 99.99 percent, the up to 2.1 times speedup over Popcorn-Linux, the read-versus-write-intensive analysis of Conjugate Gradient versus Integer Sort, the cache-size sensitivity, and the honest finding that the replicating design can win for Conjugate Gradient on the CXL 3.0 model) are all drawn directly from the paper. The Stramash-Linux and Stramash-QEMU source code is released by the authors. Background explanations of instruction sets, operating-system kernels, cache coherence, page tables, and distributed shared memory are standard and provided to make the paper’s contribution understandable.