Efficient and scalable core multiplexing with M³v

In an OS where every core is physically isolated and one central core does all the context switching, that central core becomes the bottleneck — M³v lets each core switch itself, by virtualising the communication hardware so a core can multiplex without gaining the power to rewire the machine.

Why this problem is worth solving

M³ is an operating system built on the premise that isolation should come from physical separation rather than from software checks. It runs on a

The cost of that design is that a tile holds exactly one application. If it goes idle, the hardware sits unused, which is untenable once you want to run more applications than you have tiles. The obvious fix is to multiplex — but a

What people did before, and where it broke

• One application per tile, no multiplexing. The original M³ design. Maximum isolation, and tiles go idle whenever their application does.
• Remote context switching from a central controller. M³x has the controller decide scheduling and drive save and restore on every tile from a distance. Safe, because the endpoint manipulation stays in the trusted component, and a serialisation point on the path of every switch.
• Fall back to a slow path when the recipient is not running. M³x keeps direct communication when the destination happens to be scheduled, but otherwise routes the message through the controller, which must schedule the recipient first. Two applications sharing a tile can only ever reach each other this way.
• Let the core manage its own endpoints. The efficient option, and it hands any core the ability to fabricate arbitrary channels — that is, to take over the machine. Not available.
• Have local software mediate every access to the communication unit. The authors tried this first. It degraded all communication by an order of magnitude, because the mediator ended up in the path of ordinary sends and receives too.
• Build the equivalent on conventional hardware. The paper sketches it honestly: a multikernel across single-core sockets, a shared network card with hardware virtualisation, one socket owning the physical function and handing out virtual functions tagged per channel, plus an input-output memory management unit only that socket may program. Roughly the same properties, assembled from existing parts.
• The gap. Multiplexing required either trusting every core with a system-wide capability or funnelling every switch through one core. Nobody had made the switch cheap without also making it dangerous.

How M³v makes a core safe to switch itself

The idea

The dangerous operation was saving the state, so do not save it. The reasoning here is unusually clean. A core cannot be allowed to save and restore its own communication endpoints, because restoring an endpoint is indistinguishable from creating one, and creating one arbitrarily means owning the machine. Every previous design took this as a constraint on where the switch happens. M³v instead removes the operation: endpoints are never saved or restored at all. They simply stay in the hardware, each tagged with the identity of the activity that owns it, and a register only the local multiplexer can write says which activity is currently running. A context switch changes that number. Attempts to use another activity’s endpoint fail with a deliberately uninformative error, so an application cannot even probe what else exists. This is the same style of move as deciding where an isolation boundary should fall and then making the hardware enforce it there, rather than accepting whatever boundary the hardware happens to offer, covered elsewhere on this site.

Software must not be in the fast path, and they learned that the hard way. The first design had the local multiplexer mediate every access to the transfer unit — the natural approach, and the one that keeps all policy in software. It cost an order of magnitude on all communication, not just on switches, because the mediator now sat between every application and every send. The redesign concluded that activities have to reach the hardware directly, which is exactly why the ownership tag exists: it lets the hardware do the check that software was doing, at no cost. The same discipline shows up in the memory protection, where the unit translates virtual addresses itself through a small software-loaded lookaside buffer, and the design deliberately restricts each transfer to a single page and declines to inject interrupts on a miss, so a single check before each command suffices. Concluding that a per-tile communication engine must be usable without software mediation, and pushing the necessary checks into the hardware that was already in the path, is the same conclusion reached by work attaching accelerators to ordinary shared-memory queues, covered elsewhere on this site.

A centralised operation is affordable only if it is rare. What makes the M³x design fail is not that remote context switching is slow in itself but that it is on the path of something frequent. The measurements make this vivid: under a switch-heavy workload, M³x’s throughput barely rises as tiles are added and it does not run reliably past eight, because one single-threaded controller is performing every switch in the machine. M³v moves the frequent operation local and leaves the rare one — establishing a channel, starting an activity — with the controller, and scaling becomes almost linear to twelve tiles. The authors are candid that a single controller will eventually bottleneck anyway at large tile counts, and point to prior work running several controller instances. Judging a centralised mechanism by the product of its cost and how often it runs, rather than by its cost alone, is the same analysis that justifies an expensive but rare privilege-domain switch inside a confidential virtual machine, covered elsewhere on this site.

Following a context switch on one tile

That shared flag matters: if nothing else were ready, polling would be cheaper than blocking. The paper notes that the current implementation polls in that case and leaves a more energy-efficient answer to future work.

Why it works

• Switching costs one register write. No endpoint state moves during a switch → the expensive and dangerous part of the operation disappears → and the core can perform it without any capability it should not have.
• The hardware performs the check that software was performing. Endpoints carry an owner tag compared against a privileged register → applications reach the transfer unit directly with no mediator → recovering the order of magnitude the first, software-mediated design lost.
• Scaling follows from removing the shared component. Context switches no longer touch the controller → throughput rises almost linearly to twelve tiles where the previous design flattened immediately → and the controller retains only infrequent work like opening channels.
• The virtualisation is nearly free in silicon. A third interface plus four registers → the transfer unit grows 6% in logic → and the whole unit is 10.6% of an out-of-order core’s lookup tables with essentially no block memory.
• Uniform delivery removes the slow path. Every message goes through the hardware regardless of placement → a message for a non-running activity raises a local interrupt instead of detouring through the controller → so two activities sharing a tile can finally communicate at ordinary cost.

What it costs (trade-offs and scope)

• Each step of this design is less secure than the one before, and the paper says so. Section 5 is an argument against the paper’s own contribution. M³ isolates physically; M³x weakens that by letting activities share a tile; M³v weakens it further by keeping the multiplexer’s metadata on the same tile as potentially malicious applications, rather than physically isolated in the controller. Compromising the controller or any transfer unit takes the whole machine in every variant.
• Complexity moves from one core into all of them. M³x put multiplexing in software on a single tile. M³v puts it in hardware on every tile, plus a small kernel on each. More attack surface, replicated.
• Accelerators are left behind. M³v multiplexes general-purpose cores only. Accelerator tiles keep the unvirtualised unit and the remote mechanism, and multiplexing them is future work.
• Tile-local communication is not fast. Insisting that every message go through the hardware makes a tile-local round trip cost two interrupts and two scheduling decisions — roughly two Linux context switches. The authors note a microkernel-style local path would be faster and chose against it to keep placement transparent.
• The controller still bottlenecks eventually. One instance handles every channel setup and activity creation. Prior work shows multiple instances scale to hundreds of tiles; that is not implemented here.
• No cross-tile cache coherence, which distorts the Linux comparison. Linux cannot span non-coherent tiles, so it runs on one tile only, and the authors state plainly that the multi-tile M³v results are not comparable to it — only the single-tile shared configuration is.
• The network results are compromised by the platform. TCP was unusable on the FPGA because of packet drops and synchronisation problems between an 80 MHz soft core and a desktop processor, so the macrobenchmarks send over UDP and ignore losses.

The evaluation

Check yourself

Sources

The paper is “Efficient and Scalable Core Multiplexing with M³v” by Nils Asmussen, Sebastian Haas, Carsten Weinhold, Till Miemietz and Michael Roitzsch, published at ASPLOS 2022, pages 452-466, DOI 10.1145/3503222.3507741. The authors are at the Barkhausen Institut in Dresden, and the system, the simulation model and the hardware implementation are open-sourced.

This explainer is written from the full text, read from the open-access deposit on Zenodo, which is licensed CC BY. All figures above come from the paper’s own sections 3 through 6: the order-of-magnitude cost of the rejected software-mediated design, the 6% logic increase and four additional registers from virtualising the transfer unit, the unit occupying 10.6% and 32.6% of the out-of-order and in-order cores’ lookup tables, the controller at 11.5 thousand lines of Rust with 900 unsafe and TileMux at 1.7 thousand with 50 unsafe against NOVA’s roughly 9 thousand lines, the throughput of 45 and 49 runs per second for M³x against 84 and 111 for M³v on a single tile, the scaling behaviour to twelve tiles, the approximately 5 thousand cycles for a tile-local remote procedure call against roughly 27 thousand reported for M³x, and the 3.6% sharing overhead in the voice assistant macrobenchmark. Two measurement caveats are stated by the authors and repeated here: the M³x comparison uses simulated 3 GHz out-of-order processors rather than the FPGA, and the multi-tile results cannot be compared against Linux because Linux cannot run across non-coherent tiles.