Reconfigurable Virtual Memory for FPGA-Driven I/O

Cloud FPGAs are rented by the hour, but the way an accelerator reaches its data is still wired to the specific board it was compiled for, so existing FPGA operating systems either hide the hardware behind an abstraction so thick that performance collapses or expose it so thinly that every application must build its own storage stack; FSRF (UT Austin) takes the middle path — map a host file straight into the FPGA's own virtual address space, let the accelerator read it through a plain AXI4 port and fault in its own pages — and then uses the one thing an FPGA has that a CPU does not, reconfigurability, to compile the address-translation hardware itself to fit the application, reaching up to 64x the throughput of Coyote and 2.3x that of AmorphOS while staying within 10% of raw physical addressing.

The argument in one glance

Problem

The chip is portable; the way it reaches its data is not

Amazon, Microsoft, Alibaba, Baidu, Huawei and Tencent all rent FPGAs by the hour, and an FPGA can beat a CPU by orders of magnitude on the workloads it suits. But an accelerator is useless until data reaches it, and the vendors expose that path as raw wires: this many DDR4 controllers, that PCIe endpoint, this network port, that flash interface. Every board is different, so an accelerator written against one platform will not build against another, and every developer ends up writing a private data-retrieval stack before writing any of the logic they actually care about.
What people did

Two families of FPGA operating system, pulling in opposite directions

One family hides the hardware completely at the language level. Cascade and Synergy let hardware code call something that looks like the C standard library — open a file, read from it — by turning the design into a state machine that can be paused mid-clock-tick so the illusion holds. The other family multiplexes the real interfaces and hands each tenant a port. AmorphOS gives every application a segment of FPGA memory; Coyote gives it a small software-managed translation cache and virtual addresses over host memory, network and local DRAM. Both are real, open, and running.
Why it fell short

One is too slow to use, the other too raw to be portable, and neither knows about files

The language-level systems produce a familiar programming model but make it very hard for the runtime to overlap input and output with execution, so the performance suffers badly. The interface-multiplexing systems are fast but push the work back onto the developer: choose the data placement, size the translation cache, build the storage stack, and accept that connecting a handful of applications to a handful of interfaces needs a wire for every pair. Neither one supports the file system. And the fixed choices hurt: a small on-chip translation cache simply cannot cover a working set of gigabytes, so any application with a large random access pattern falls off a cliff, and neither system can let an application use more memory than the board physically has.
This paper's idea

Map the file into the accelerator, then compile the memory system to fit the accelerator

FSRF, the File System for Reconfigurable Fabrics, does two things at once. First it makes the file the interface: the host maps a file into the FPGA virtual address space, and the accelerator reads it through an ordinary industry-standard AXI4 port, faulting its own pages in on demand rather than waiting for the host to push data ahead of time. Second, and this is the part only an FPGA makes possible, it treats the address-translation hardware as something to be compiled per application. An accelerator with a huge random working set gets a translation cache that lives in the board DRAM and covers everything; one with high locality gets a small fast on-chip cache with page sizes and associativity chosen at compile time. The abstraction stays the same while the implementation underneath it is rebuilt to suit the tenant.

Why this problem is worth solving

Specialisation is what is left now that transistors have stopped getting cheaper, and building a new fixed-function chip for every workload is not affordable.

A chip whose logic is not fixed at manufacture but described by a configuration file, so the same physical device can be turned into a video codec in the morning and a database filter in the afternoon. That flexibility is what makes a cloud provider able to buy one kind of part and still satisfy customers who want very different accelerators, and it is also why the data path is such a problem — the logic is portable in principle, but the wires it must talk to are not.

is the compromise the industry reached, and cloud providers have bet on it heavily. The bet only pays off if the software layer around the chip is good enough that ordinary developers can use it, which today it is not.

The specific thing that breaks is address translation.

The small hardware cache that remembers where recently used virtual addresses actually live in physical memory, so the processor does not have to walk a page table on every access. Its usefulness is measured by reach — how much memory the entries it holds can cover between them — which is why systems use huge pages when they can, because one entry then covers more ground. The trade is that a bigger page is also a coarser unit for the operating system to move, protect and evict.

was designed for CPUs, and the paper argues carefully that FPGA applications do not access memory the way CPUs do. A CPU has caches that coalesce small accesses and prefetch automatically, so it hits the same translation entry over and over. An FPGA application builds its own buffering and asks for exactly as much of a page as it needs, once, then moves on — so the entry is likely never reused, and the latency of translating can be hidden behind the deep pipelining the application already has. That difference is what makes a translation cache in slow off-chip memory a sensible idea here and an absurd one on a CPU.

What people did before, and where it broke

Multiplex the raw interfaces and give each tenant a port. AmorphOS and Coyote both do a version of this, and it keeps the hardware cost low while giving real multi-tenancy. Where it broke. It is not an abstraction, it is a subdivision. The application still implements its own storage stack, still decides where data lives, still sizes its own translation cache, and connecting several applications to several interfaces at full bandwidth needs a separate connection for every pair. Move to a board with different peripherals and the design does not build.
Abstract the interface away at the language level. Cascade and Synergy convert hardware code into a state machine that can stall at sub-clock-tick granularity, so unsynthesizable operations appear to happen instantly and the developer gets calls that behave like opening and reading a file. Where it broke. Creating the illusion that input and output are instantaneous makes it structurally hard for the runtime to overlap them with computation, and the resulting performance is poor. The developer also has very little control over recovering it.
Pin host memory and share a virtual address space across the link. OPTIMUS does this on Intel HARP, where the FPGA sits beside the CPU and reaches DRAM through it, configuring the input-output memory management unit so translation works across the link. Where it broke. Pinning is the whole trick, and pinning is exactly what stops the operating system from moving those pages anywhere. There is no demand paging, no transparent oversubscription, and because every access is remote there is nothing local to stripe or batch, so those policies do not exist either.
Give the FPGA a CPU-shaped memory hierarchy. The natural instinct is to copy what works: segments, or a small on-chip translation cache with a couple of page sizes. Where it broke. On-chip memory is the scarcest resource on an FPGA, so the cache stays small, and a small cache forces a choice between reach and granularity. Pick huge pages and you get reach but the operating system can only move memory in enormous chunks; pick small pages and the working set no longer fits. The paper measures a case where scaling the huge-page cache by one to two orders of magnitude would be needed for it to matter, which is not available.
The gap. Nobody had built an FPGA I/O layer that is portable, supports the file system, allows the board memory to be oversubscribed, and still performs close to raw physical addressing — and nobody had asked whether the translation hardware itself should be a per-application design decision rather than a fixed part of the platform.

How FSRF gets a file into an accelerator

From a host-pushed buffer to a demand-paged, memory-mapped file

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The interface the accelerator sees never changes. What changes is which translation hardware was compiled in, and which policies the host runs behind it.

The idea

Make the file the interface, and let the accelerator pull rather than be pushed. Everything else follows from this one choice. If the accelerator addresses a file through virtual memory, then the host no longer has to know in advance what data the accelerator will want, which means the dataset no longer has to be staged into board DRAM before the run, which in turn means the dataset can be larger than the board. It also means the operating system keeps the freedom to move those pages around — the pinning that other systems require to make host memory reachable is precisely what freezes the memory hierarchy in place. And because translation happens on the chip, nearly every access completes without a system call, so the abstraction is paid for once at fault time rather than on every read. An accelerator that fetches its own data instead of waiting to be fed is the same inversion made by the storage-access work for GPUs covered elsewhere on this site, where letting the accelerator initiate the request removed the host from the critical path entirely.

Treat the address-translation hardware as part of the application rather than part of the platform. This is the argument that makes the paper an FPGA paper rather than a general accelerator paper. On a CPU the memory management unit is fixed silicon and everyone lives with it. On an FPGA it is just more logic, so it can be different for every tenant. An application with high locality gets a small on-chip translation cache, where the number of entries, the page sizes and the associativity are all compile-time parameters. An application with a large random working set gets something no CPU would ever build: a translation cache that lives in the board DRAM, giving effectively unlimited reach at the price of a translation latency that this class of workload can hide anyway, plus a slice of the very bandwidth it is trying to use. Refusing to pick one point on the reach-against-granularity curve, and instead making the curve itself a compile-time choice, is the same escape from a fixed trade that the work on getting large translation reach out of small pages covered elsewhere on this site makes, arriving at it from the opposite direction.

Let the operating system decide how data is spread across the memory channels, because the application cannot see the other tenants. Splitting a dataset across all four memory channels is the obvious way to get bandwidth, and it is what both prior systems do. It is also the wrong thing to do when four applications are sharing the board, because now every application collides with every other on every channel. FSRF instead partitions channels among tenants when it can divide them evenly, and falls back to controlling the virtual-to-physical mapping in software when it cannot — when applications differ enough in footprint, bandwidth appetite or access pattern that an even split would be unfair. The measured result is not just higher throughput but far steadier throughput. Deciding how to stripe data across independent devices, and discovering that the naive full-width answer creates interference the individual client cannot see or fix, is the same lesson as the disaggregated storage work covered elsewhere on this site.

Walking one access through FSRF

What happens when an accelerator reads a byte it has never touched

One application, using the DRAM-backed translation cache, reading from a mapped file whose data is still on the host SSD.

Step 1 of 5

The accelerator issues a plain bus read

The application logic puts a virtual address on its AXI4 read channel and asks for some number of words. It does not know whether that address is backed by board DRAM, host memory, or a file it has never seen. Write requests are handled the same way, except the unit waits until all the write data has been buffered first.

Buffering the write data before translating is a small but important detail. If translation happened first, the physical access could begin and then stall waiting for data that had not arrived, holding a memory controller hostage. Draining first means every translated transaction can run to completion.

Why it works

The accelerator can hide translation latency in a way a CPU cannot. FPGA applications build their own buffering and prefetch far ahead of use → an extra memory access to translate disappears into a pipeline that was already deep → putting the translation cache in slow board DRAM buys effectively unlimited reach for a cost this workload class does not feel.
Files, not pinned host pages, back the addresses. Nothing has to be locked in place for the accelerator to reach it → the operating system stays free to fetch, evict and write back while the accelerator runs → demand paging and oversubscription become possible, which is what lets a 32 GiB dataset run on a board with far less memory.
Translation on the chip removes the host from the common path. Only misses cross to software → the system-call and driver costs are paid per fault instead of per access → the measured throughput lands at 90% of raw physical addressing across the suite, and 98% on the streaming workloads.
The interface is fixed while the implementation is not. The accelerator always sees the same standard bus → the platform can swap a small on-chip translation cache for a DRAM-backed one, change page sizes, or restripe memory without the application changing → one design gets a memory system tuned to it rather than a compromise tuned to the average of everyone.
Someone with a global view assigns the memory channels. The operating system can see all the tenants while each application cannot → it partitions channels rather than letting everyone stripe across all of them → interference collapses, and throughput under a mixed workload stays close to what each application gets alone.

What it costs (trade-offs and scope)

The comparisons are reimplementations, not the real systems. Coyote and AmorphOS baselines were rebuilt inside FSRF on Amazon F1 — Coyote as a port of its translation cache with its default geometry and its striping policy, AmorphOS as segment translation with its fixed-size request interface. The authors say directly that this is not an end-to-end replica of Coyote and argue it captures the key aspects. It is a fair-minded approach, and it is still not the same as running the original.
The Coyote translation cache was deliberately not resized. Coyote allows it, and FSRF chose not to, on the grounds that on-chip memory limits mean growing the small-page cache would not help and the huge-page cache would need to be one to two orders of magnitude larger to matter. That reasoning is stated and plausible, but it does mean the headline gap against Coyote is measured against a configuration Coyote users could in principle change.
The physical-addressing upper bound is only measured on hot data. That is unavoidable, since demand paging and oversubscription need virtual memory to exist at all, but it means the 90% figure is a comparison in the regime most favourable to having no translation.
Two of the oversubscription experiments are simulated by shrinking the board. Triangle counting and the random-access benchmark run too long on a 32 GiB dataset, so those runs use the 2 GiB dataset with the application artificially limited to 1920 MiB of board DRAM instead.
Where the datasets are small, the advantage largely vanishes. On cold and warm small datasets FSRF is ahead of Coyote and AmorphOS by geometric means of only 5% and 15%, because startup and system initiation costs dominate everything else. The large wins are on large data.
Eviction is first-in-first-out and the fault path is a polling host thread. Both are the simplest thing that works. The paper does not claim otherwise, but a page-replacement policy that ignores reuse is leaving something on the table in exactly the oversubscribed case the system is proudest of.
Reconfiguring the memory system means rebuilding the bitstream. Choosing between the on-chip and DRAM-backed translation caches requires that both have been compiled, and FPGA compilation is slow. The paper measures the benefit of specialising the hardware, not the cost of producing each specialised version.
One platform, four tenants. Everything is measured on Amazon F1 with four concurrent applications, and one of that board four memory channels is about 6% slower than the others, which shows up in the fairness numbers.

The evaluation

Three ways to give an FPGA tenant access to its data

Property	AmorphOS (segments)	Coyote (on-chip translation cache)	FSRF (mapped files, reconfigurable translation)
What the accelerator is given
Address translation
Using more data than the board holds
Measured throughput on hot data
Fairness under a mixed workload

Figures are from the paper's runs of four concurrent workloads on Amazon F1, across cold, warm and hot data and small, large and oversubscribed datasets.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The headline numbers span a very wide range, and the range is the point rather than the maximum. Against Coyote the geometric mean is a 75% improvement while the best single case is 64x, and the enormous figures — the 505% geometric mean on large non-streaming hot data, or the roughly 6,100% on the oversubscribed PageRank workload — come from cases where the competing design has essentially run out of translation reach, not from cases where FSRF is broadly faster. On small datasets the advantage nearly disappears. Both baselines are careful reimplementations inside FSRF rather than the original systems, the Coyote translation cache is left at its default geometry by a stated but debatable argument, and two of the oversubscription experiments shrink the board instead of growing the dataset. The area cost is genuinely small — the interconnect, four translation units of each kind and the transfer engine together take about 3.5% of the chip lookup tables and 3% of its block memories, less than a single vendor transfer engine — but the cost of compiling a separate bitstream per translation configuration is not measured here at all.

Background

Why this sits in the FPGA operating system conversation

Almost every published FPGA operating system has concentrated on dividing up the fabric — how to let several tenants occupy one chip safely, how to relocate a design, how to preempt one. This paper takes that as given and asks about everything outside the fabric: the memory, the storage, the network, the file system. Its claim is that the reconfigurability that FPGA research usually spends on sharing logic should also be spent on the system software itself, so that the memory management unit is compiled to fit the tenant rather than fixed by the platform. That argument only lands if putting a new design on the chip is cheap enough to do routinely, which is why it belongs next to the work covered elsewhere on this site on making FPGA compilation fast enough to fit into a normal development cycle — the two halves of the same bet that reconfigurability is a resource the system layer should be spending, not just the application layer.

If you remember only three things

1. FSRF makes a host file directly addressable from an FPGA accelerator through ordinary virtual memory and a standard bus port, so the accelerator faults in its own data instead of waiting to be fed — which is what makes portability, demand paging and oversubscribing board memory possible at once.
2. The distinctive move is treating the address-translation hardware as a per-application design decision: a small on-chip cache for locality, or a translation cache living in board DRAM for huge random working sets, because FPGA applications pipeline deeply enough to hide the latency that idea would cost a CPU.
3. It lands within about 10% of raw physical addressing overall while beating the state of the art by wide but very uneven margins — the largest gaps appear precisely where a fixed on-chip translation cache runs out of reach, and nearly vanish on small datasets.

Check yourself

Check your understanding

1. Why is putting the translation cache in the FPGA board DRAM a reasonable idea here, when it would be absurd on a CPU?
2. What does backing FPGA virtual addresses with files, rather than with pinned host pages, actually buy?
3. Why does FSRF partition memory channels among tenants instead of striping every dataset across all of them?
4. How should the reported speedups of up to 64x over Coyote be read?

Sources

The paper is “Reconfigurable Virtual Memory for FPGA-Driven I/O” by Joshua Landgraf, Matthew Giordano, Esther Yoon and Christopher J. Rossbach, published at ASPLOS 2023 Volume 3, pages 556-571, DOI 10.1145/3582016.3582048. The authors are at The University of Texas at Austin, with Rossbach also at Katana Graph, and the paper carries an artifact appendix. It is open access under a Creative Commons licence.

This explainer is written from the full text, read through a proxy of the ACM Digital Library copy. Everything above comes from the paper’s own sections 3 through 6: the motivation argument that FPGA applications do not access memory the way CPUs and GPUs do because they build their own buffering and prefetching; the mapping interface with its open, map, unmap, synchronise and advise calls and its streaming, random, reuse and low-use hints; the compile-time interface that sets the number of translation caches and their page sizes, capacities and associativities; the DRAM-backed translation cache with eight-way associativity, 4 KiB pages, a 48-bit virtual and 36-bit physical address space and a pipeline that translates once per cycle; the first-in-first-out eviction policy and the host fault-handler thread; the custom interconnect that works around the vendor restriction on concurrent same-identifier transactions; the transfer engine with its huge-page host buffers and adaptive batch size up to 2 MiB; the thirteen workloads with their dataset sizes, reuse factors and read fractions; the results at 98% and 82% of physical addressing with geometric means of 42% and 505% over Coyote on hot data and 5%, 15%, 89% and 264% in the cold and warm cases; the scaling and fairness measurements with their 1.1% runtime deviation and the note that one memory channel is about 6% slower; the mixed-workload figures of 66% against 39% for Coyote and 74% against 27% for AmorphOS; and the resource table showing about 3.5% of lookup tables and 3% of block memories.