TrackFM: Far-out Compiler Support for a Far Memory World

When a program needs more memory than one machine has, one option is far memory — using a remote server's RAM as a slow extra tier — but today you must choose between two bad options — let the OS swap pages transparently and eat the page-fault cost, or rewrite your app around special remote data structures for speed; this work from Illinois Tech and Northwestern builds TrackFM, a compiler that takes unmodified C++ and automatically inserts lightweight checks so any heap object can live remotely, giving you the OS approach's zero-effort transparency and nearly the hand-tuned library's speed just by recompiling.

The argument in one glance

Problem

Programs that outgrow one machine can borrow memory from another

A program often needs more memory than its machine has, but buying machines with huge memory is expensive and mostly wasted — studies of warehouse-scale systems find applications actively touch only about a third of their pages, and access most of the rest rarely. That rarely-touched memory is a chance for a cheaper, slower tier. Far memory (also called remote or disaggregated memory) is exactly that: the DRAM of a remote server, reached over a fast network, used as an extra tier below local RAM. It lets a memory-hungry program spill across machines instead of demanding one giant machine, cutting cost and avoiding crashes from running out of memory.
What people did

Use it via the kernel for transparency, or a library for speed

Software far memory comes in two flavors. The kernel approach modifies the operating system's paging so that cold pages are swapped out to the remote server automatically; the exemplar, Fastswap, does this over a fast network. Its virtue is transparency: any unmodified program benefits for free. The library approach, exemplified by AIFM, instead gives developers remote-aware data structures — a remote array, a remote hash map — that fetch and cache remote data at a granularity suited to the application, with prefetching and background eviction, achieving high performance.
Why it fell short

Each approach sacrifices exactly what the other keeps

The kernel approach is transparent but slow: every access to remote data is a hardware page fault, an expensive trap, and because the OS moves data a whole page at a time, it hauls far more across the network than a small object needs — a waste called I/O amplification. The library approach is fast but not transparent: the programmer must rewrite the application around the library's remote data structures, and if the library lacks the structure they need, they must build it themselves. Transparency and performance seem to be a forced choice.
This paper's idea

Let the compiler make any heap object remotable, automatically

TrackFM proposes a third way: the compiler. You recompile your unmodified C or C++ program, and the compiler automatically makes any heap-allocated object able to live in far memory, inserting a lightweight check — a guard — before each access that pulls the object local first if needed. It reuses the fast library runtime underneath as a backend, so it gets that performance, and it needs no source changes, so it keeps the kernel approach's transparency. The compiler's knowledge of the program lets it strip most of the guard cost away.

Why this problem is worth solving

The heart of the difficulty is a semantic gap: the compiler sees memory accesses, but far memory wants to work with objects, and bridging the two without the programmer is the whole trick.

Far memory has to decide what unit of data to move between local and remote. The kernel approach uses the hardware page — typically four kilobytes — because that is the unit the paging hardware already manages, which is why it is transparent: the OS just marks a page absent and traps on access. But a four-kilobyte page is a terrible fit for a program that stores, say, tiny key-value pairs, because touching one pair drags a whole page across the network, most of it unwanted, and every such trap is an expensive page fault. The library approach fixes this by working at the granularity of objects — contiguous chunks the developer chooses, sized to the data structure, a small object for a linked-list node, a bigger one for an array. That granularity is what buys performance, but it requires the developer to declare which data is remotable and rewrite around the library’s types. So the two knobs, transparency and object-appropriate granularity, seem to pull against each other: you get one by giving up the other. What a compiler could do, in principle, is recover the object structure automatically from the code — figuring out which allocations should be remotable objects and at what granularity — and instrument the program to manage them, giving object-level performance without the programmer lifting a finger. That is the possibility that makes a compiler-based approach worth pursuing.

The reason it is hard, and why a naive compiler pass would be hopeless, is that instrumenting every memory access is enormously expensive.

To make a pointer possibly-remote, the compiler must ensure the data is local before any access to it, so it inserts a guard: a small snippet of code that checks whether the object is present locally and, if not, fetches it. The problem is the sheer number of accesses. Unlike the kernel approach, which pays only on a page fault, and unlike the library approach, which pays only through its smart-pointer types, a compiler that guards every heap load and store pays something on every single one — even when the object is already local. When the authors instrument a benchmark with a nine-gigabyte working set, the transformation produces up to fifty-six million slow-path guards and roughly ten billion fast-path guards. Paying even a handful of instructions ten billion times would seem to make the approach hopelessly slow, an insurmountable barrier. The saving grace is that real programs are regular: they touch memory in loops, sweeping sequentially through arrays, so most of those guards are redundant — once you have localized an object, the next several accesses that fall within it need no check at all. A compiler, unlike the kernel, can see that loop structure statically and prove which guards are unnecessary. Turning the naive, ruinous per-access cost into a cost paid only at object boundaries is what makes the compiler approach not just transparent but fast — and that is the crux of the design.

What people did before, and where it broke

Swap pages to remote memory in the kernel (Fastswap). Modify the OS paging subsystem so cold pages go to a remote server over a fast network. Where it broke. Every remote access is a hardware page fault, an expensive trap, and moving data a full page at a time causes I/O amplification — transferring far more than a small object needs, sometimes tens of times the working set.
Give programmers remote-aware data structures (AIFM). Provide a remote array, remote hash map, and the like, fetching and caching at object granularity with prefetching and eviction. Where it broke. The application must be rewritten around these types, sacrificing transparency, and if the library lacks a needed structure, the developer has to design their own.
Add specialized hardware for finer-grained faults. Build hardware that reduces the granularity of memory faults below a page. Where it broke. This capability exists only in research prototypes and is not available in deployed systems, so it cannot be relied on today.
Hoist whole collections local at loop entry (classic distributed-shared-memory compilers). When a loop iterates over an array, bring the entire array local before the loop. Where it broke. A single collection may span many objects in different local-or-remote states, or may not fit in local memory at all, so the whole-collection hoist does not apply — leaving every access in the loop to be guarded.
The gap. Every prior approach forced a choice — transparency without performance, performance without transparency, unavailable hardware, or an optimization that modern object-granular far memory breaks. What was missing was a way to recover object structure and loop regularity from unmodified code and instrument it cheaply, getting both transparency and near-library performance — which is what practical far memory needs.

How TrackFM works

Making unmodified heap data remotable at compile time

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Recover, in the compiler, the object structure the programmer would otherwise have to declare. The library approach is fast because the developer tells it which data is remotable and at what granularity; the kernel approach is transparent because it needs no such declaration but pays with page-granularity faults. TrackFM’s insight is that the compiler can recover the missing structure itself — deciding that heap allocations become remotable objects, distinguishing them from stack and global data by marking their pointers, and drawing object boundaries around contiguous allocations — so it gets object-level behavior with no programmer input. Using compiler analysis to recover the semantic information a lower-level view lacks, and then acting on it, is exactly what the recommendation-model compiler covered elsewhere on this site does, reconstructing the tensor-shape structure the raw graph hides so it can optimize what would otherwise be opaque.

Refuse the forced choice between transparency and performance. Prior far-memory work treated transparency and speed as a tradeoff, each approach conceding one to win the other. TrackFM’s stance is that this dichotomy is an artifact of where the work is done — in the kernel or in hand-written libraries — and that moving it into the compiler dissolves it, delivering the transparency of automatic transformation and, by reusing the library’s runtime and stripping guard costs, nearly its performance. Rejecting a supposedly-fundamental either/or by relocating the problem to a layer that can satisfy both goals at once is the same move the weak-memory binary-translation work covered elsewhere on this site makes, achieving correctness and performance together where prior tools had traded one against the other.

Pay the instrumentation cost only where it is not redundant, guided by a model. Guarding every access would be fatal, but most guards in a loop are wasted once an object is local, and a few are large-element loops where the optimization backfires. TrackFM uses the compiler’s view of loop induction variables to eliminate the redundant guards and a cost model over object density to apply the optimization only where it actually helps. Recovering statically which work can be skipped, and using a cost model to decide when a transformation is worth applying, is the same discipline the dynamic-neural-network compiler covered elsewhere on this site follows, determining what can be settled ahead of time and pruning the work that a specific case renders unnecessary.

Following a summing loop through TrackFM

Summing a large array that lives partly in far memory

Tracing a loop over a remote array shows guards and loop chunking at work.

Step 1 of 5

An unmodified program sums a huge array

A plain C loop adds up the elements of an array too large to fit in local memory. The developer writes nothing special — no remote types, no annotations — and recompiles the program with TrackFM to run against a remote memory server.

In the library approach this array would have to be declared a remote array type by hand.

Why it works

Flagging heap pointers in a spare address bit tells remotable from local without annotations. A pointer is just an address, so nothing distinguishes a remotable heap pointer from a stack or global one → but a custom allocator sets a normally-invalid high bit on managed pointers → so the compiler guards exactly the heap accesses and leaves all other memory untouched.
The common guard is cheap, and even the slow one beats a page fault. Guarding every heap access sounds ruinous → but the fast path adds only about twenty cycles over a local access, and the slow path costs less than the kernel’s page fault (which is nearly three times as expensive when data is local) → so transparent far memory is cheaper than the kernel approach, not more expensive.
Loop chunking removes the redundant guards that dominate the cost. A loop over an array re-checks the same local object on every element → but induction-variable analysis proves sequential accesses stay within one object, moving the guard to object boundaries → so billions of per-element guards vanish, dropping one benchmark loop from 1.6 billion to zero.
A cost model applies chunking only where it pays. Chunking hurts when elements are large or loops are short, since there were few redundant guards to remove → but a model over object density decides when to transform → so a benchmark that would suffer a fourfold slowdown from blind chunking instead speeds up 2.5 times.
Sub-page objects cut the I/O amplification that hobbles the kernel. Swapping whole pages hauls four kilobytes even for a tiny object → but choosing an object size smaller than a page moves only what is needed → so a hash map that the kernel amplifies 43 times over its working set is amplified only about 2 times, running 12 times faster.

What it costs (trade-offs and scope)

You pay the guards even when data is local. Unlike the kernel and library approaches, which pay only on a fault or through smart-pointer types, TrackFM checks every heap access, so only regular, loop-heavy programs amortize it — a deeply nested, tight-loop benchmark that confounds the loop analysis injects a huge number of guards and loses to the kernel approach.
One object size for the whole program. TrackFM currently picks a single object size at compile time, without the per-data-structure tuning the library approach allows; the authors note the small search space makes autotuning feasible, but it is not yet done.
It needs the source, and external libraries are a problem. The program must be available as source or compiler bitcode, and libraries it calls must either be recompiled through TrackFM or replaced with pre-transformed versions, because an untransformed library could receive a managed pointer it does not know how to handle.
Only heap allocations are remotable, and complex heaps are out of scope. Stack and global data stay local, custom allocators and memory-mapped setups are unsupported, so programs relying on those cannot be transformed as-is.
The loop analysis has gaps. The underlying framework does not handle some C++ features like exception handling, so the authors ported a C++ dataframe library to C to evaluate one workload; more general loop analysis is left to future work.
It approaches the library, not beats it, and is one system on one setup. TrackFM comes within ten percent of the hand-tuned library rather than surpassing it — its win is transparency at near-library speed and up to twice the kernel approach — and results come from one cluster with a specific network runtime, where the kernel baseline’s faster network layer can win under low concurrency.

The evaluation

Three ways to give a program far memory

Property	Kernel (Fastswap)	Library (AIFM)
Programmer effort
Data-movement granularity
Where the overhead is
I/O amplification	High (tens of times the working set)	Low
Performance

Three ways to give a program far memory, evaluated on a CloudLab cluster across microbenchmarks and applications (STREAM, a hash map, an analytics dataframe, memcached, and NAS). TrackFM keeps the kernel approach's transparency — the app is merely recompiled — while outperforming it by up to 2 times and coming within 10 percent of the hand-tuned library approach, because compiler-inserted guards made cheap by loop chunking, a flat object-state table, sub-page objects, and prefetching recover object-level far memory from unmodified code.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

TrackFM is a clean resolution of a real dilemma: far memory made you choose between the kernel approach’s zero-effort transparency and the library approach’s object-granular speed, and TrackFM argues the compiler can give both. You recompile unmodified C or C++, and the compiler makes every heap object remotable by inserting a guard before each access, flagging managed pointers with a spare address bit, caching object state in a flat table, and — crucially — using induction-variable analysis plus a cost model to chunk loops so billions of redundant guards disappear, all atop the fast library runtime for prefetching and eviction. The payoff is strong: up to twice the kernel approach’s speed with the same transparency, and within ten percent of the hand-tuned library. Read it with its scope. It pays a guard on every heap access even when local, so irregular, deeply-nested code can lose to the kernel; it picks one object size for the whole program rather than tuning per structure; it needs source and struggles with external libraries, custom allocators, memory-mapped memory, and some C++ features like exceptions; and it approaches the library rather than beating it, on one cluster with one network runtime. The durable lesson is that a tradeoff presented as fundamental — transparency versus performance — is often just a consequence of which layer does the work, and moving it into the compiler can dissolve it.

Background

Why the memory hierarchy keeps stretching across machines

It helps to see TrackFM against a broad trend in data centers: memory is increasingly disaggregated, pooled and shared across machines rather than bolted rigidly to each processor. The pressure is economic and physical — provisioning every machine for its peak memory wastes enormous capacity that sits idle, and a program’s working set rarely matches the box it happens to run on, so letting memory be borrowed from a neighbor over a fast network raises utilization and cuts cost. But disaggregation always reintroduces the same fundamental problem in a new form: the borrowed resource is far away, and moving data to and from it is the dominant cost, so the whole game becomes moving as little as possible and hiding the latency of what you must move. The in-storage serverless-acceleration work covered elsewhere on this site attacks the same tension from the storage side, moving computation to where disaggregated data already lives rather than hauling the data across the network. TrackFM attacks it from the memory side, shrinking the transfer granularity below a page and prefetching to hide the remote latency. The broad lesson is that as the memory hierarchy stretches across the network, the systems that win are the ones that treat data movement as the scarce resource — minimizing what crosses the wire and overlapping it with useful work — whether they do that in hardware, the kernel, a library, or, as here, the compiler.

If you remember only three things

1. Far memory lets a program borrow a remote server's DRAM as a cheap slow tier below local RAM, valuable because applications touch most of their pages rarely. But the two software approaches force a tradeoff: the kernel approach (Fastswap) swaps pages to remote transparently for any unmodified program but pays an expensive hardware page fault on every remote access and moves a full page at a time, causing heavy I/O amplification; the library approach (AIFM) fetches at object granularity with prefetching and eviction for high performance but requires the programmer to rewrite the application around remote-aware data structures. Transparency and performance seem mutually exclusive.
2. TrackFM introduces a compiler-based third way that keeps both: recompile unmodified C/C++, and an LLVM/NOELLE pipeline makes any heap object remotable. It flags managed pointers by setting a normally-invalid high address bit (via a custom allocator), inserts a guard before every heap load/store that localizes the object first (skipping stack/global via program analysis), caches object local-or-remote state in a flat table for cheap fast-path checks, and — the key optimization — uses induction-variable analysis with a cost model over object density to chunk loops, guarding only at object boundaries so billions of redundant per-element guards disappear (one loop drops from 1.6 billion fast-path guards to zero). It reuses AIFM's runtime as a backend for prefetching and eviction, and picks a sub-page object size to cut I/O amplification.
3. On a CloudLab cluster across STREAM, a hash map, analytics, memcached, and NAS, TrackFM outperforms the kernel approach by up to 2 times with the same transparency and comes within 10 percent of the hand-tuned library — for example cutting a hash map's I/O amplification from 43 times the working set to about 2 and running 12 times faster than the kernel. Its limits: guards are paid even for local data, so irregular deeply-nested loops (NAS FT) can lose to the kernel; it uses one whole-program object size (autotunable but hand-chosen); it needs source and struggles with external libraries, custom allocators, memory-mapped setups, and some C++ features like exceptions; and it approaches rather than beats the library, evaluated on one setup where the kernel's RDMA backend can win under low concurrency.

Check yourself

Check your understanding

1. What tradeoff did the two existing software approaches to far memory force?
2. How does TrackFM make an unmodified program use far memory, and what is its main source of overhead?
3. How does TrackFM avoid the ruinous cost of guarding every access in a loop?
4. Which statement best reflects the scope and limits of TrackFM?

Sources

The paper is “TrackFM: Far-out Compiler Support for a Far Memory World” by Brian R. Tauro, Brian Suchy, Simone Campanoni, Peter Dinda, and Kyle C. Hale, published at ASPLOS 2024, DOI 10.1145/3617232.3624856. The authors are at the Illinois Institute of Technology and Northwestern University, and TrackFM is available open-source.

This explainer is written from the full 19-page text. Everything above comes from the paper’s own sections: the introduction contrasting kernel-based far memory (Fastswap, transparent but paying page faults and I/O amplification) with library-based far memory (AIFM, fast at object granularity but requiring application rewrites) and proposing the compiler-based third approach; the TrackFM design bridging the semantic gap between object-level far memory and memory-access-level code, remoting heap allocations by overloading x86 non-canonical address bits (the 60th bit) via a custom allocator, and choosing a single sub-page object size; the implementation of the far-memory pointer transformation, the NOELLE-based analysis and transformation passes (runtime initialization, pointer guards over heap loads and stores, loop chunking, and libc transformation), the abstract data structure and object pool, the object state table caching metadata for cheap fast-path guards, and the three-part guard (custody check, 14-instruction fast path, at-least-144-instruction slow path) with its evacuator-barrier safety; the loop chunking optimization with induction-variable analysis and the object-density cost model with its roughly 730-element crossover; and the evaluation on CloudLab measuring guard overheads (fast-path plus about 21 cycles, slow-path guards cheaper than Fastswap page faults), loop chunking (STREAM Sum from about 1.6 billion guards to zero, k-means avoiding a fourfold slowdown for a 2.5-times speedup), prefetching (up to about 5 times), I/O amplification (hash map from 43 times to 2.3 times the working set, 12 times faster than Fastswap; memcached 66 times versus 15 times), and application benchmarks (analytics within 10 percent of AIFM, memcached 1.3 to 1.7 times over Fastswap, NAS with the FT outlier), concluding TrackFM beats the kernel approach by up to 2 times while matching the library approach within 10 percent.