Copy-on-Pin: The Missing Piece for Correct Copy-on-Write

Operating systems save memory by letting many programs share the same physical page until one writes, a trick called copy-on-write, but for twenty years this has collided with page pinning — a separate mechanism that lets the kernel and hardware touch a page directly and refuses to let it move — producing a long trail of security holes, memory corruption, and stale-data bugs that were patched over and over without fixing the real cause; this work from TU Munich, Red Hat, and VMware pins down the true rule, that a shared page and a pinned page must never be the same page, designs a scheme called Copy-on-Pin to enforce it, and ships a relaxed version into Linux 5.19 that is correct, simpler, and even slightly faster.

The argument in one glance

Problem

Copy-on-write and page pinning have clashed for 20 years

Operating systems share physical memory pages between programs and copy them only when written, a technique called copy-on-write. They also pin some pages so the kernel and hardware can access them directly and they cannot move. These two mechanisms interact badly, and for two decades that clash has produced security holes, memory corruption, and stale-data bugs.
What people did

Patch each reported bug as it is found

Each time a copy-on-write and pinning bug surfaced, kernel developers fixed that specific symptom. Different operating systems patched their own versions in their own ways.
Why it fell short

Nobody understood the real cause, so fixes stayed broken

Because the underlying reason was never stated precisely, the fixes were incomplete and inconsistent, and new bugs kept appearing. This paper surveys the old bugs and finds brand-new ones in Linux, FreeBSD, and NetBSD, showing the problem was never truly solved. The hidden fact is that a pinned page holds a reference the system cannot write-protect or move, which is exactly what copy-on-write assumes it always can.
This paper's idea

Enforce one invariant: shared and pinned pages never overlap

The paper derives the real rule: a copy-on-write-shared page and a pinned page must never be the same physical page. It introduces the Copy-on-Pin scheme to enforce this, and a relaxed version for Linux that uses a single conservative flag per page instead of precise tracking. The result is correct, adds no per-page memory, does not increase complexity, and is slightly faster. It was integrated into Linux 5.19.

Why this problem is worth solving

To see the clash, you first need the two mechanisms clearly in mind.

Copy-on-write is a memory-saving trick as old as the 1960s. When two programs would otherwise hold identical copies of a page, the operating system instead keeps one physical page and maps it into both, marked read-only. Reads work normally. A write is trapped by the hardware; the system then makes a private copy of the page, points the writer at the copy, and lets the write proceed. So the copy is deferred until it is truly needed, and often avoided entirely. The most famous use is the fork system call, which creates a child process as a duplicate of its parent: rather than physically copying all the parent’s memory, the system shares it copy-on-write, so a page is only copied if the parent or child later writes to it. Page pinning is a different mechanism. Sometimes the operating system or a hardware device must access a page directly and reliably — the kernel through a special always-present mapping, or a disk or network card doing a direct transfer straight into or out of an application’s buffer, without a slow intermediate copy. To make that safe, the system pins the page: it promises the page will not move or be paged out until the transfer finishes. A pinned page therefore has a reference that, unlike an ordinary program pointer, cannot be write-protected and cannot be remapped.

That last property is exactly where the two collide.

Copy-on-write rests on a hidden assumption: that the system can always write-protect every reference to a shared page and, when needed, remap a reference to a fresh copy. A pinned reference breaks both halves of that assumption — it cannot be write-protected, and it cannot be moved. When a page is both shared and pinned, the system’s copy-on-write logic can make a wrong decision, and the consequences are serious. Consider a real bug class the paper describes. Two programs share a page copy-on-write. One of them starts a direct disk transfer out of that page, which pins it, and then unmaps its own view of the page before the transfer finishes. Now the other program writes to the page. The system looks and sees only one remaining mapping, concludes the page is no longer shared, and lets that write happen in place, on the very page the disk is still reading. The writer’s private data is now streamed out to the other program’s file. This is a genuine security issue, and it is only one of a family. The authors find five distinct ways copy-on-write and pinning interact wrongly, and they uncover previously unknown instances in Linux, FreeBSD, and NetBSD — some of the older bugs having never been correctly fixed at all. The bugs are timing-dependent and workload-dependent, which is exactly why they were so hard to pin down and kept coming back.

What people did before, and where it broke

Patch each symptom (all three operating systems). Fix each reported copy-on-write and pinning bug as a one-off, in each kernel’s own way. Where it broke. Without naming the real cause, the fixes were inconsistent and incomplete; some bugs were never truly fixed, and new ones kept surfacing because the same underlying clash remained.
Track sharing precisely by counting mappings (Linux 5.8 and earlier). Decide whether a page is shared by counting how many mappings and references point to it. Where it broke. This precise accounting was complex and error-prone, caused its own correctness and security bugs, and was eventually abandoned.
Loosen the sharing check for simplicity (Linux 5.9 to 5.18). Replace precise counting with a simpler, looser rule for when to copy. Where it broke. The looser rule was incorrect: under contention it could mistake a pinned page for a shared one and copy it, causing memory corruption.
The gap. No one had derived the actual correctness condition relating copy-on-write and pinning, nor an implementation that was both correct and simple. Stating that condition as a single invariant — a shared page and a pinned page must never coincide — and enforcing it cheaply is what Copy-on-Pin provides.

How Copy-on-Pin works

From a bug-prone copy-on-write to an enforced invariant

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Find the hidden assumption and turn it into one stated invariant. For twenty years these bugs were treated as separate accidents, each patched on its own. The paper’s first move is to name what copy-on-write silently assumes — that every reference to a shared page can be write-protected and remapped — and to show that a pinned reference breaks exactly that. From this it derives a single rule that makes all five bug classes impossible at once: a shared page and a pinned page must never be the same page. Replacing a fog of ad-hoc fixes with one precisely stated property, and enforcing that property instead of chasing symptoms, is the same discipline the formal graphics-processor memory-model work covered elsewhere on this site applies, writing the informal rules of a murky, bug-prone area as an exact specification so that violations become something you can systematically rule out rather than rediscover one crash at a time.

Do not track precisely — a conservative flag safe on one side is enough. Knowing exactly how many programs share each page is complex and, as the kernel’s own history shows, a rich source of bugs. The key realization is that correctness does not need precision. The system only needs to never call a shared page unshared, and never call a pinned page shared. A single per-page flag that is deliberately cautious — it may say a page might be shared when it actually is not — meets that bar. The occasional needless copy that results costs a little performance but never correctness. Accepting a small, bounded amount of extra work in exchange for a clean, robust safety guarantee is the same trade the fixed-length sensor-message defense covered elsewhere on this site makes, padding every message to one uniform size and paying that overhead so that a dangerous leak is closed for good.

Store the flag for free, and let it double as a memory of past work. The relaxed design adds no memory: it reuses a spare bit in each page’s existing metadata rather than adding a counter, which would have grown the metadata past its cache-line size and cost more than one percent of system memory. And because the flag records whether a page is shared, the write-fault handler no longer has to recompute that each time — it reads one bit. Caching the answer to a repeated question so the common path avoids redoing the work is the same shortcut the direct page-table indexing work covered elsewhere on this site takes, jumping straight to the needed result instead of walking through the same lookup on every access.

Following the classic bug, and how COP stops it

A shared, pinned page that used to leak private data

Tracing the bug and its fix shows why the invariant matters.

Step 1 of 5

Two programs share a page

Programs P1 and P2 share one physical page copy-on-write, both mapped read-only. So far this is normal memory saving: one page, two read-only views.

This commonly arises after a fork, where parent and child share pages until one writes.

Why it works

Pinning breaks copy-on-write’s core assumption. Because a pinned reference cannot be write-protected or moved → but copy-on-write depends on being able to write-protect and remap every reference → so a pinned page silently violates what copy-on-write needs, which is the hidden root of all the bugs.
One invariant replaces a pile of special cases. Because all five bug classes reduce to a pinned page and a shared page being the same page → forbidding that single overlap → so enforcing one rule fixes the whole family at once instead of patching symptoms one by one.
Conservative flags are enough for correctness. Because the system only needs to never call a shared page unshared and never call a pinned page shared → a one-sided flag that errs on the cautious side suffices → so the precise, error-prone tracking of every mapping becomes unnecessary.
The flag costs no extra memory. Because it reuses a spare bit in the page’s existing metadata rather than adding a counter → the metadata stays within its cache-line-aligned size → so the guarantee is gained without the one-and-a-half percent memory loss a per-page pin counter would have caused.
Remembering the answer makes the common case faster. Because the flag records whether a page is shared so the write-fault handler need not recompute it → most write faults resolve by reading a single bit → so the correct scheme also runs slightly faster than the buggy one it replaced.

What it costs (trade-offs and scope)

A conservative flag sometimes copies needlessly. Because the flag errs on the side of “might be shared,” the system occasionally makes a copy that was not strictly required. This never harms correctness, but it can cause small, transient performance and memory overheads in corner cases.
The flag must be preserved everywhere, which is real engineering. Keeping the exclusive flag correct as pages are swapped out, migrated, or split from large pages into small ones requires care in many code paths, including encoding it into the special entries left behind for swapped and migrated pages.
The implementation targets anonymous memory. The upstreamed Linux work covers anonymous memory — memory not backed by a file — because it is the most common and the most bug-prone case. Other kinds of mappings are outside the shipped scope.
Read-only pinning still forces a copy. A page pinned even for read-only access must be separated from any sharing first, so read-only pinning of a shared page still triggers a copy, handled through the existing write-fault path.
The performance win is marginal, not the point. The scheme is slightly faster on write faults and identical on real application benchmarks; its value is correctness and simplicity, not speed. On workloads that rarely fork, deduplicate, or swap, none of the schemes differ noticeably.
It formalizes and fixes, but assumes the surrounding model. The invariant is derived under the assumptions of mainstream operating system designs. A fundamentally different memory model could change what “shared” and “pinned” mean and would need its own analysis.

The evaluation

Three ways Linux has handled copy-on-write with pinning

Property	Precise (Linux 5.8 and before)	Loose (Linux 5.9 to 5.18)
Correct with pinning
How it decides sharing
Extra memory per page	Larger metadata	None
Implementation complexity		Low
Write-fault time	1116 nanoseconds	1101 nanoseconds

Three generations of Linux copy-on-write handling, compared on an x86-64 server. The precise scheme counted mappings and was complex and buggy, so it was abandoned. The loose scheme that replaced it was simpler but incorrect, able to mistake a pinned page for a shared one and corrupt memory. Copy-on-Pin, integrated into Linux 5.19 in about 747 added and 340 removed lines for anonymous memory, enforces the invariant that shared and pinned pages never coincide using a single conservative per-page flag that adds no memory. It is correct, no more complex, and marginally faster on write faults (1092 versus 1101 and 1116 nanoseconds), while a full application benchmark suite shows all three within about 0.2 percent, meaning no practical performance cost. Along the way the authors found previously unknown bugs in Linux, FreeBSD, and NetBSD.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This is a correctness paper, and a satisfying one: it takes a class of bugs that plagued three operating systems for two decades and shows they all stem from one unstated assumption, then fixes them with a rule simple enough to ship. The contribution is as much conceptual as it is code. By naming the invariant — a copy-on-write-shared page and a pinned page must never be the same page — the authors convert a fog of timing-dependent, hard-to-reproduce bugs into a single property that can be enforced. The relaxed Linux design is elegant precisely because it refuses to track sharing exactly: a one-sided flag that is always safe on the cautious side is enough for correctness, adds no memory by reusing a spare metadata bit, and even speeds up the common write fault by remembering whether a page is shared. It is upstream in Linux 5.19. Read it with its scope. The shipped work targets anonymous memory, the most common and most problematic kind; the conservative flag occasionally triggers a needless copy; and keeping the flag correct across swapping, migration, and large-page splits is genuine engineering the paper handles case by case. The performance results are a reassurance, not a headline: slightly faster on micro-benchmarks and indistinguishable on real applications. The lasting lesson is that a bug that resists fixing for twenty years is often a sign that the real invariant was never written down.

Background

Why naming the invariant beats patching the symptoms

It helps to place Copy-on-Pin in a broader pattern. Many long-lived correctness problems in systems are not hard because the fix is complex, but because nobody has stated precisely what must always be true. Once the invariant is named, the fix can be small and the whole class of bugs can be closed instead of whacked one at a time. The dynamic race-detection work covered elsewhere on this site follows the same spirit for concurrency, building a principled structure that maintains a correctness-critical ordering between events efficiently, where earlier ad-hoc approaches had struggled. In both cases the durable contribution is not a clever optimization but a clear statement of the property the system must preserve, plus a mechanism that preserves it cheaply. The general point is that when the same kind of bug keeps returning across systems and across years, the productive question is not “what patch stops this crash” but “what invariant were we silently violating.” Copy-on-Pin answers that question for copy-on-write, and the answer turned out to be a single sentence about two states never overlapping.

If you remember only three things

1. Operating systems save memory with copy-on-write (share one physical page read-only, copy it only on a write) and also pin pages so the kernel and hardware can access them directly and they cannot move. For twenty years these clashed, causing security leaks, memory corruption, and private writes escaping to disk, because a pinned page holds a reference the system cannot write-protect or move — exactly what copy-on-write assumes it always can. The bugs were patched one at a time without naming the real cause, so they kept returning; the authors even found new ones in Linux, FreeBSD, and NetBSD.
2. The paper derives the missing invariant: a copy-on-write-shared page and a pinned page must never be the same physical page, even for read-only pinning. Its Copy-on-Pin scheme enforces this by unsharing a page before it is pinned and refusing to share a pinned page. The relaxed Linux design needs no precise tracking: a single conservative per-page "exclusive" flag (set means definitely not shared, clear means maybe shared) is always safe on the cautious side. It reuses a spare metadata bit so it adds no memory, and is carried correctly through swapping, migration, and large-page splits.
3. Integrated into Linux 5.19 in about 747 added and 340 removed lines for anonymous memory, Copy-on-Pin is correct, no more complex than before, and slightly faster on write faults (1092 versus 1101 and 1116 nanoseconds) because the flag memoizes whether a page is shared, with no measurable effect on real application benchmarks. Its limits: the conservative flag sometimes copies needlessly, preserving the flag across every page transition is real engineering, the shipped work covers anonymous memory, and read-only pinning of a shared page still forces a copy. The durable lesson is that a bug resisting fixes for decades usually means the real invariant was never written down.

Check yourself

Check your understanding

1. Why do copy-on-write and page pinning conflict in the first place?
2. What single invariant does the paper identify as the fix, and why is that powerful?
3. The relaxed Linux design uses one conservative "exclusive" flag instead of precisely tracking sharing. Why is a conservative flag enough?
4. The relaxed design turned out to be slightly faster on write faults than the schemes it replaced. Why?

Sources

The paper is “Copy-on-Pin: The Missing Piece for Correct Copy-on-Write” by David Hildenbrand, Martin Schulz, and Nadav Amit, published at ASPLOS 2023, DOI 10.1145/3575693.3575716, where it received a Distinguished Artifact Award. The authors are at the Technical University of Munich and Red Hat, and at VMware Research. The relaxed Copy-on-Pin solution was integrated into Linux 5.19.

This explainer is written from the full text of the paper, obtained from co-author Nadav Amit’s academic website, which matches the ASPLOS 2023 proceedings and is distributed under the ACM open-access terms. Everything above comes from the paper’s own sections: the introduction and background on copy-on-write (its write-protect-and-trap mechanism, the fork optimization, and copy avoidance when a page is no longer shared) and on page pinning (non-remappable references for kernel linear mappings and device direct memory access, why an input-output memory management unit does not remove the need, and the read-only versus read-write protection of pinning references); the survey of pinning-related copy-on-write bugs and the newly discovered issues in Linux 5.17, FreeBSD 13.0, and NetBSD 9.2, including the missed-unsharing data-leak scenario; the five classes of wrong interaction (wrong sharing, wrong unsharing, missed unsharing, impossible unsharing, and unnecessary copies) and the derived mutual-exclusivity condition; the Copy-on-Pin scheme with its maybe-shared and maybe-pinned primitives and the three handling rules; the relaxed Copy-on-Pin Linux design with the single exclusive flag, its setting and clearing, the reuse of Linux’s imprecise pin marking (a large constant added to the reference counter) restricted to sharing decisions, the 1.5 percent memory cost that a per-page counter would have imposed, and the handling of swapping, migration, transparent huge pages, and read-only pinning; the implementation of about 747 added and 340 removed lines for anonymous memory upstreamed into Linux 5.19; and the evaluation comparing the precise, loose, and relaxed schemes, with the write-fault micro-benchmark numbers (1092, 1101, and 1116 nanoseconds) and the SPECrate2017 macro-benchmark showing all schemes within about 0.2 percent.