Cloning a running process onto another machine normally means flattening it into a file and rebuilding it on the far side — when the two machines share memory, you can copy the kernel's own structures across as they are, fix up their pointers, and hand the child the parent's page tables directly.
The argument in one glance
-
Problem
Cloning a process onto another machine is expensive
Serverless platforms want to spin up a new instance of a function that is already running elsewhere. Doing that across machines means capturing everything the process is — its memory, its page tables, its open files, its registers — and reconstructing it on the far side.
-
What people did
Flatten it into a file and rebuild it
Checkpoint and Restore in Userspace serialises the whole thing with Protocol Buffers, writes it out, ships it, and deserialises it on the target. It works and it is what production uses. On a language model function, the restore phase alone takes 2.7 times as long as a local fork plus running the task, and consumes 42 times the memory.
-
Why it fell short
Or keep it on the parent and fetch lazily, which ties the child to the parent
Mitosis avoids serialising the memory by leaving it on the parent node and pulling pages over remote direct memory access as the child faults. Faster, but every page the child touches gets copied into its own local memory — 24 times a local fork — and the parent cannot exit while children remain.
-
This paper's idea
If the machines share memory, stop flattening anything
Copy the structures the kernel already built into shared memory as they are, rewrite their internal pointers to address that shared memory, and let the child attach the page table leaves of the parent directly. Read-only state then exists once for the whole cluster, and only writes pull anything local.
Why this problem is worth solving
Every mechanism for moving a running process to another machine was designed under one assumption: the two machines cannot see each other’s memory. That assumption is what forces Serialization is the process of flattening in-memory data structures into a linear stream of bytes that can be written to a file or sent over a network, and which must then be rebuilt into structures on the receiving side.
Compute Express Link, or CXL, is an interconnect that lets a processor reach memory attached elsewhere using ordinary load and store instructions, and whose newer revisions allow several machines in a rack to share the same memory coherently.
What people did before, and where it broke
- Start a fresh container. The default cold start. Correct, general, and slow enough that the entire serverless research literature exists to avoid it.
- Local fork from a warm parent. Cheap, because the child shares the parent’s pages copy-on-write, but confined to one machine — which is exactly the constraint a cluster autoscaler needs to escape.
- Checkpoint and Restore in Userspace. Serialise the process — virtual memory areas, page tables, open files, namespaces, registers and every memory page — into files with Protocol Buffers, then deserialise on the target. Caching those files in shared memory helps, but the restore phase for a language model function still takes 2.7 times as long as a local fork and its execution, at 42 times the local memory, because parent and child share nothing.
- Mitosis over remote direct memory access. Make an immutable shadow copy of the parent locally, serialise only the kernel state, ship that, and let the child fault pages across the network lazily. Much better, and still 24 times the local memory of a local fork, with page faults consuming 42% and 54% of total execution on the two largest functions tested.
- Single-node checkpoint optimisations. Techniques that speed up checkpointing within one machine by exploiting multiple virtual address spaces. Useful, and confined to one node.
- The gap. One approach decouples the checkpoint from its creator but pays to rebuild it; the other skips the rebuild but chains the checkpoint to the node and the process that made it. Both were designed for memory that cannot be shared, and neither exploits memory that can.
How CXLfork avoids rebuilding anything
From serialise-and-rebuild to copy-and-rebase
-
-
new
-
new
-
new
-
new
-
new
The idea
Serialisation was a workaround for a constraint that no longer holds. The framing that makes this paper worth reading is that existing remote fork designs are not badly engineered — they are correctly engineered for machines whose memory is disconnected. Once several nodes can address the same memory coherently, flattening a page table into Protocol Buffers so the other side can rebuild it is doing work that the interconnect has made unnecessary. So CXLfork copies the kernel’s structures into shared memory verbatim and does the one thing that actually needs doing: rewriting their internal pointers into offsets relative to the CXL device, so that a different kernel instance can map the region and follow them. Everything that is genuinely non-portable — open files, sockets, namespaces, all of which point into per-node kernel state — is still serialised, but that is a small fraction. Revisiting an interface whose design encodes an assumption a new interconnect has invalidated is exactly the exercise performed when byte-addressable storage forced a rethink of what a file system should do, covered elsewhere on this site.
Reconstruct the top of the tree and inherit the bottom. The restore path contains the paper’s sharpest engineering idea. A page table is a tree, and the leaves are the enormous part — one entry per page of the process. Conventional restore rebuilds the whole thing. CXLfork allocates and initialises only the upper levels locally, then attaches the checkpointed last-level entries directly, because those entries already contain CXL physical addresses and are already marked read-only. Reconstruction becomes nearly constant time regardless of how large the process is. Keeping those shared leaves immutable takes a small trick: an unused bit in the page table entry records any attempt by the kernel to modify them, and on such an attempt the whole leaf is lazily copied. Reusing the substantial pieces that have already been built rather than reconstructing them from a description is the same instinct behind reusing container layers across function instances instead of rebuilding each sandbox from nothing, covered elsewhere on this site.
Read-only state should exist once in the rack. The consequence of restoring by attachment rather than copying is that every clone in the cluster points at the same physical pages in shared memory for everything it has not written. Each node’s own hardware caches then hold whatever that node is actually touching, so the sharing costs nothing in software. Only a write pulls a page into local memory, and the paper measures that fault at 2.5 microseconds against under 1 for an ordinary local allocation — roughly 1.3 microseconds of data movement and 500 nanoseconds of keeping translation caches consistent. To reduce even that, CXLfork prefetches pages the parent had marked dirty, on the observed grounds that over 95% of pages written by the parent are also written by its children. The result is that a cloned function uses 13% of the local memory a cold start would, which matters because memory pressure, not scheduling cleverness, is what actually binds a serverless cluster during a load spike, covered elsewhere on this site.
Following one clone across the fabric
Scaling a serverless function onto a second node
A function has been running long enough for its just-in-time compiler to settle, and a load spike requires another instance elsewhere in the rack.
Step 1 of 5
Checkpoint after the sixteenth invocation
The autoscaler waits until the function has been called enough times for its runtime to reach a steady state, then checkpoints it.
Checkpointing earlier captures unoptimised code. The checkpoint also records the access and dirty bits from the parent page tables, which later drive how aggressively pages are pulled local.
Why it works
- Nothing portable needs converting. Kernel structures move as memory copies and only their pointers are rewritten → the deserialisation that dominates conventional restore disappears → leaving CXLfork on average just 14% slower than a local fork on the same machine.
- Restore cost stops scaling with process size. Only the upper page table levels are built; the leaves are attached → reconstruction is near constant time no matter how much memory the process maps → which is why the advantage grows on the largest functions.
- Read-only state is shared rather than copied. Clones point at the same physical pages and let their own caches do the rest → local memory falls 87% against the serialising baseline and 61% against the lazy-paging one → and a cloned function needs only 13% of what a cold start would.
- Copy-on-write beats copy-on-access here. Fetching only on writes rather than on any access → avoids copying read-only data that could have stayed shared → though the paper shows the opposite choice wins for working sets too large for the caches, which is why the policy is selectable.
- The parent’s access history predicts the child’s. Access and dirty bits are captured in the checkpoint → over 95% of pages the parent wrote are also written by children → so prefetching exactly those pages removes most copy-on-write faults without pulling in data that would have stayed shared.
What it costs (trade-offs and scope)
- There is only one machine in the evaluation. No CXL hardware exists yet that can share memory between nodes, so the two-node cluster is two virtual machines on the two sockets of a single Sapphire Rapids server, with an FPGA-based CXL device attached as a memory-only NUMA node. The authors say this estimates remote fork overheads correctly except for the network costs a real distributed system would add — a fair claim, and a real limitation.
- The hardware is CXL 1.1, not the 3.0 the design assumes. Cross-node coherent sharing is the capability the whole paper is built around and it is not yet available; the comparison against Mitosis had to be run within a single node because CXL cannot currently serve inter-node faults at all.
- Shared memory is slower than local memory. The round trip measured is 391 nanoseconds. For functions whose working sets fit in the processor caches this does not matter; for the two largest tested it does, which is why three tiering policies exist rather than one.
- The tiering policies genuinely trade against each other. Migrating on any access improves warm execution by 11% but worsens cold execution by 14% and inflates the child’s memory footprint by 250%. There is no setting that wins everywhere.
- One baseline comparison is not like for like. The serialising baseline cannot use ghost containers, because it restores from a file system, so it pays container setup that the other two avoid.
- Scaling to many nodes is untested and anticipated to be hard. The authors expect CXL bandwidth to become the bottleneck in a large cluster, and note their tiering policies are driven by latency rather than bandwidth and would need extending.
The evaluation
| Property | Metric | Existing remote fork | CXLfork |
|---|---|---|---|
| Cold function execution time | |||
| Local memory per cloned function | baseline | ||
| Versus a full cold start | baseline | ||
| Copy-on-write fault cost | under 1 microsecond for an ordinary local page fault | ||
| End-to-end tail latency at 25% local memory | baseline |
Check yourself
Check your understanding
1. Why do existing remote fork mechanisms serialise process state at all?
2. What does rebasing accomplish in CXLfork?
3. Why is restoring by attaching page table leaves so much cheaper than rebuilding them?
4. Why does CXLfork migrate pages on write rather than on any access, and when is that wrong?
Sources
The paper is “CXLfork: Fast Remote Fork over CXL Fabrics” by Chloe Alverti, Stratos Psomadakis, Burak Ocalan, Shashwat Jaiswal, Tianyin Xu and Josep Torrellas, published at ASPLOS 2025 Volume 2, pages 210-226, DOI 10.1145/3676641.3715988. The authors are at the University of Illinois Urbana-Champaign, and the paper is licensed CC BY.
This explainer is written from the full text, read from the author-hosted copy on Tianyin Xu’s page. All figures above come from the paper’s own sections 2, 4, 6 and 7: the 2.7 times restore penalty and 42 times memory cost of the serialising baseline and the 24 times memory cost of the lazy-paging one, the 391 nanosecond CXL round trip, the 2.5 microsecond copy-on-write fault with its 1.3 microsecond and 500 nanosecond components, the observation that over 95% of pages written by a parent are also written by its children, the checkpoint threshold of sixteen invocations and 512 kilobyte ghost containers, the average results of 14% slower than local fork, 2.26 and 1.40 times faster than the two baselines, 87% and 61% less local memory, 13% of a cold start’s memory with 11 times faster execution, the tiering trade-offs of 11% warm improvement against 14% cold penalty and 250% memory increase, and the roughly 16 times tail latency reduction at 25% local memory. Readers should note the evaluation platform: a single dual-socket server with two virtual machines standing in for two nodes, using CXL 1.1 hardware, since the cross-node coherent sharing the design targets requires CXL 3.0 and does not yet exist in silicon.