A Software Caching Runtime for Embedded NVRAM Systems

A new kind of memory called FRAM lets tiny battery-powered devices keep both their code and their data in one dense, non-volatile chip, but that chip is slow — it runs at a third of the processor's speed — and since fetching instructions is the bulk of what the processor does, everything crawls; this paper (Virginia Tech) notices that these devices leave their small, fast SRAM barely used, and builds SwapRAM, a compiler-plus-runtime that quietly copies hot functions into that SRAM as the program runs, speeding execution up to 46 percent and cutting energy up to 36 percent with no changes to the application.

The argument in one glance

Problem

The dense non-volatile memory that makes tiny devices possible is also slow

Ultra-low-power devices — sensors, wearables, implantables, batteryless nodes — live under tight limits on power, size, and cost. A new class of non-volatile memory, especially Ferroelectric RAM (FRAM), is a boon for them: it keeps its contents without power, and it is far denser than SRAM, about one transistor per bit versus six. That density lets a device store both its code and its data in one FRAM chip as a single unified memory (a typical part has 128 kilobytes of FRAM but only 2 kilobytes of SRAM), enabling more sophisticated software than the SRAM alone could hold, and its non-volatility lets the device switch off its volatile memory entirely during hibernation to save leakage. The catch is speed. FRAM is much slower and more energy-hungry to access than SRAM: on a commercial part it tops out at 8 megahertz while the processor core runs at 24, so the chip must insert wait states that throttle the whole processor whenever it touches FRAM. And because instruction fetches are more than 75 percent of all memory accesses, running code out of FRAM means the processor spends most of its time waiting on slow memory.
What people did

Add a small hardware read cache in the memory controller

The standard mitigation is what these chips already ship with: a tiny hardware cache in the FRAM memory controller that holds recently-fetched code so repeated accesses avoid the wait states. It is dedicated circuitry, fast, and automatic — the classic way to hide slow memory behind a small fast buffer.
Why it fell short

The fixed tiny cache is too small, and unified memory makes it worse

That hardware cache is minuscule and unchangeable — one part pairs a 32-byte cache with 128 kilobytes of FRAM — because the SRAM it would need to grow is scarce and the designers reserve almost all of it for program memory. Its size is fixed at chip-design time and the programmer cannot change it. Worse, the whole appeal of FRAM is unified memory, storing code and data together, and that is exactly where the small cache fails: a single instruction can dispatch simultaneous accesses to distant code and data addresses in FRAM, and those alternating, low-locality accesses thrash the tiny cache, so it barely helps the primary use case. Meanwhile, on a unified-memory device the fast SRAM is left largely unused, because the bulk data has been mapped into the roomy FRAM. So the system pays FRAM's speed penalty on nearly every instruction while its one piece of fast memory sits idle, and there is no hardware knob to fix it.
This paper's idea

Turn the idle SRAM into a software-managed instruction cache

SwapRAM repurposes that underused SRAM as an instruction cache, but implemented entirely in software since the hardware cannot be changed. It observes that when you must split code and data, code belongs in the fast SRAM because instruction fetches dominate. A compile-time pass makes each function runtime-relocatable and redirects its call sites; a runtime then transparently copies functions into SRAM the first time they are called, rewrites the call to point at the fast SRAM copy, and runs subsequent calls at full speed with no overhead. It keeps metadata to retain hot functions and evict cold ones, and it exploits application-level knowledge a hardware cache never sees — function-level locality and the call stack — to decide what to cache. The result runs up to 46 percent faster and uses up to 36 percent less energy, with the application untouched.

Why this problem is worth solving

The value of these new memories is real, but it comes wrapped in a performance tax that lands on the most common operation.

Ferroelectric RAM and its relatives blend the non-volatility and density of flash with much of the read-write performance of SRAM, which unlocks a genuinely new design space for ultra-low-power computing. Because FRAM is dense and byte-addressable, a device can use it as a single unified memory holding both program code and data, giving software far more room than the tiny on-chip SRAM would; and because it is non-volatile, the device can power down its volatile SRAM during the long idle stretches typical of battery and batteryless deployments, saving the leakage that would otherwise drain it. These are exactly the properties that make sophisticated sensing and on-device inference feasible on parts that must last years on a coin cell. But current FRAM pays for this with access speed and energy: reads and writes are slower and costlier than SRAM, and the memory runs at a fraction of the core clock, forcing the processor to stall with wait states whenever it reads FRAM, and even so a FRAM part can draw more than twice the power of a comparable flash device during execution. Since the whole point is to keep code in FRAM, and code fetches dominate execution, that penalty is paid almost constantly.

The obvious fix is caching, but on these systems the usual caching assumptions break.

Hardware caches hide slow memory by keeping recent accesses in a small fast buffer, and commercial FRAM parts include one — but it is fixed at chip-design time and tiny, because the SRAM it would consume is scarce and reserved for program storage, so the programmer cannot enlarge it. And in the unified-memory case it is defeated by contention: a single instruction touching both code and data dispatches simultaneous accesses to distant FRAM addresses whose poor locality thrashes the small cache. One might instead borrow software instruction caches built for older systems that stored code in slow external DRAM — but those relied on the long latency of a DRAM cache miss to hide their own bookkeeping overhead behind the stall. On a modern integrated NVRAM part the code memory is on the same die, so a miss costs only a few cycles, which exposes rather than hides the software cache’s overhead, and the scarce SRAM cannot hold much cache metadata. Those fine-grain software caches therefore either slow these systems down or fail to run at all. The open question is whether an instruction cache that is both efficient and effective can be built for such resource-constrained low-power systems purely in software.

What people did before, and where it broke

Accept FRAM’s speed and run code from it. Store code in the dense non-volatile memory and pay the access penalty. Where it broke. Instruction fetches are more than 75 percent of memory accesses, so the wait states throttle almost the entire execution, costing steep time and energy.
The built-in hardware read cache. Use the tiny cache in the FRAM memory controller to hold recent code. Where it broke. It is fixed and minuscule (32 bytes against 128 kilobytes of FRAM), cannot be resized, and is thrashed by the low-locality alternating code and data accesses of unified memory.
Use a larger SRAM part instead. Switch to a device with enough SRAM to hold the program. Where it broke. It increases size, cost, and complexity and forgoes FRAM’s density and non-volatility, which for many size-, weight-, and power-constrained devices means abandoning the design.
Prior software instruction caches (built for DRAM code storage). Instrument basic blocks to predict fetches and overlap cache bookkeeping with slow DRAM-miss stalls. Where it broke. On integrated NVRAM a miss is only a few cycles, so there is no long stall to hide the fine-grain overhead behind, and the scarce SRAM cannot hold the metadata — so these approaches degrade performance or fail entirely.
The gap. No approach turned the idle SRAM of a unified-memory NVRAM device into an effective instruction cache purely in software, lightweight enough to run on such constrained parts. Doing that is what SwapRAM contributes.

How SwapRAM works

Copying hot code into fast SRAM, just in time, entirely in software

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Put the hot code where it runs fast, and let compile-time knowledge decide what is hot. SwapRAM’s foundation is the measured, slightly counterintuitive fact that on a unified-memory device where code and data must be split across a fast and a slow memory, code should go in the fast one, because instruction fetches so dominate that keeping them cheap matters more than keeping data cheap. From there the task is to arrange for the hottest code to reside in the small fast SRAM as much of the time as possible — a placement problem, solved not by a hardware cache reacting to addresses but by software that understands the program. Deciding where a program’s code should live so that the frequently-executed parts are the cheapest to fetch, using knowledge of the program’s structure rather than raw address traffic, is the same profile-informed code-placement strategy the work on profile-guided relinking for warehouse-scale applications covered elsewhere on this site pursues, laying out code so the hot paths land in the fast part of the memory hierarchy.

Move the working set into the fast tier transparently, and keep it there. Concretely, SwapRAM treats the SRAM as a fast tier and the FRAM as a slow one, and migrates whole functions into the fast tier just in time, the first time they are called, rewriting the call site so future calls hit the fast copy directly with no ongoing cost. It maintains metadata to retain the hot working set and evict cold functions only when space is needed, and it never disturbs code that is currently executing on the call stack. All of this happens without any change to the application, which simply runs faster. Transparently relocating the frequently-used working set into a faster memory tier and holding it there, so most accesses land in fast memory while the application is none the wiser, is the same hot-in-fast-tier discipline the work on transparent page placement for tiered memory covered elsewhere on this site is built on, promoting what is hot into the fast tier behind the program’s back.

When the memory technology changes, the cache has to be reinvented, not reused. The deeper point is that a new memory technology does not slot into the old caching machinery. The fine-grain software caches built for slow external DRAM assumed long miss stalls to hide their overhead behind; integrated NVRAM, with its few-cycle miss, removes that cover and demands a fundamentally lighter design — coarse function-level granularity, call-site rewriting that leaves cached code overhead-free, and a policy that sometimes declines to cache at all. SwapRAM is a caching layer redesigned around the specific latencies and scarcities of embedded NVRAM. Rethinking how a cache should be structured because an emerging, faster memory technology has invalidated the assumptions the old cache design rested on is the same reckoning the work on re-architecting caches for emerging fast storage devices covered elsewhere on this site undertakes, redrawing the cache to match a memory whose performance profile the previous design never anticipated.

Following a hot function through SwapRAM

Caching a frequently-called function into SRAM

A trace, following the paper's design, of how SwapRAM moves a hot function into fast SRAM and runs it there.

Step 1 of 5

A function is called for the first time

At compile time, SwapRAM replaced the direct call with an indirect call through a redirect entry that points at the cache miss handler, and added a marker naming the function. On the first call, execution enters the miss handler instead of running the function from FRAM.

It first saves the argument registers, since it works below the level of function signatures.

Why it works

Instruction fetches dominate, so caching code buys the most. More than 75 percent of memory accesses are instruction fetches → but those are exactly what runs slowly from FRAM → so moving code into fast SRAM removes the bottleneck on the great majority of accesses, which data caching could not.
Function granularity balances prediction against overhead. Basic-block instrumentation is accurate but so frequent it swamps execution with bookkeeping → but function calls are far less frequent yet still capture a whole span of adjacent hot instructions → so caching per function predicts execution well while keeping instrumentation cheap enough to run on tiny parts.
Rewriting the call site makes cached code free. A software cache that checked residency on every call would add overhead to hot code → but SwapRAM rewrites the call to point directly at the SRAM copy and removes the redirection → so once cached, a function runs with no per-call caching cost, amortizing the one-time copy.
Application-level knowledge beats address-level reaction. A hardware cache sees only addresses and reacts after the fact → but software sees function-level locality and the live call stack → so SwapRAM can predict and retain what will be needed, and prefetch code far ahead, in ways address-only hardware cannot.
Being allowed to not cache avoids counterproductive work. In a flat memory, code can execute from FRAM directly, so caching is optional → but caching cold code can evict hotter code or cost more to transfer than it saves → so declining to cache in those cases keeps the fast SRAM holding what actually matters.

What it costs (trade-offs and scope)

Correctness forbids evicting on-stack code, which can stall. To never crash on return, SwapRAM refuses to evict a function that is currently active, and if a miss would require it, aborts the caching and runs that function from FRAM; a pathological pattern where a function repeatedly cannot evict its own caller invokes the miss handler each time and degrades performance.
Function granularity has limits. Caching whole functions works because calls are infrequent, but a function too large for the small SRAM cannot be cached effectively, and the benefit depends on programs whose hot code is organized into functions that fit and are called enough to amortize the copy.
The software cache spends cycles managing itself. Unlike a hardware cache’s dedicated circuitry, loading code in software means copying, updating metadata, and redirecting control flow; the net win depends on amortizing that overhead over many fast SRAM executions, so cold or rarely-called code may not be worth caching.
It needs toolchain integration and has edge cases. SwapRAM adds a compile-time assembly instrumentation pass and a runtime library, requires special handling for jump tables, offers a blacklist for timing-critical or rarely-run functions, and targets C-based embedded software.
It applies where SRAM is genuinely idle, and is evaluated on one platform. The approach depends on the unified-memory setting leaving SRAM underused; a system already using its SRAM fully as main memory has none to repurpose. Results are from one commercial FRAM microcontroller and a subset of one benchmark suite that fits on it.

The evaluation

Keeping code fast on a unified-memory FRAM device

Property	Run code from FRAM	Prior software cache
Where hot code executes
Caching granularity
Fits resource-constrained parts	n/a
Uses application knowledge
Measured result vs FRAM baseline	Baseline	Worse or infeasible

From the paper's evaluation on a commercial FRAM microcontroller running nine MiBench2 embedded benchmarks. SwapRAM eliminates 65 percent of FRAM accesses on average by shifting execution to SRAM, improving execution speed by an average 26 percent (up to 46 percent) and reducing energy by an average 24 percent (up to 36 percent) versus the FRAM baseline with its hardware cache, on constrained platforms where prior software caches degrade performance or fail to run.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The contribution is a practical, fully-software instruction cache tailored to the real constraints of embedded NVRAM. SwapRAM starts from two observations — that on unified-memory FRAM devices the fast SRAM is idle and that code, not data, is what belongs in fast memory — and builds a compile-time pass plus runtime that makes functions relocatable, copies hot functions into SRAM on first call, rewrites call sites so cached code runs overhead-free, and evicts cold functions while protecting on-stack ones, all guided by application-level knowledge a hardware cache cannot use. On a commercial FRAM part it cuts 65 percent of FRAM accesses and runs up to 46 percent faster at up to 36 percent less energy, transparently. Read it within scope. Correctness forbids evicting active functions, with a pathological stall case; function granularity limits which programs benefit and excludes functions too big for SRAM; the software management costs cycles that only pay off when amortized over hot code; it needs toolchain integration and special handling for jump tables; and it applies only where SRAM is genuinely underused, evaluated on one platform and benchmark subset. Within those bounds it is a strong example of recovering performance from a new memory technology by rethinking, in software, where code should live.

Background

Why ultra-low-power devices push memory this hard

It helps to see the pressure this work operates under. The frontier of embedded computing is devices that must do more and more — capture images, monitor environments, run inference — while shrinking to fit inside a wearable, an implant, or a sensor that runs for years on a tiny battery or harvests its own power. Every microjoule and every cycle counts, and memory is often the binding constraint: there is never enough fast SRAM, and the alternatives each carry a penalty in energy, endurance, or speed. That is why emerging non-volatile memories are so eagerly adopted despite their rough edges, and why squeezing efficiency out of a fixed, austere hardware budget is a recurring theme for this class of device, where a clever software technique can be the difference between an application being feasible and being impossible. Wringing capability out of severely power- and resource-constrained sensing hardware, rather than reaching for a bigger chip, is the same discipline the work on protecting adaptive sampling on low-power sensors covered elsewhere on this site operates under, treating the harsh energy and memory budget as the fixed reality to design within. SwapRAM answers that pressure by making the cheap, fast memory a device already has work far harder than its hardware alone allows.

If you remember only three things

1. Emerging non-volatile memory like FRAM lets ultra-low-power devices hold both code and data in one dense, non-volatile chip and power down their SRAM to save energy, but FRAM is far slower and more energy-hungry than SRAM — it runs at a fraction of the core clock, forcing wait states — and since more than 75 percent of memory accesses are instruction fetches, running code from FRAM makes the whole system slow, while the tiny fixed hardware cache these chips ship with is thrashed by unified-memory code-and-data contention and the fast SRAM sits underused.
2. SwapRAM repurposes that idle SRAM as a software instruction cache, starting from the measured fact that when code and data must be split, code belongs in fast memory because fetches dominate: a compile-time assembly pass makes each function relocatable and redirects its call sites through a table pointing at a miss handler, and a runtime copies a function into SRAM the first time it is called, fixes up its branches, rewrites the call site to point at the fast copy so later calls run at full speed with no overhead, and evicts cold functions while a per-function active counter prevents evicting any function on the call stack (aborting to run from FRAM if needed for correctness).
3. Caching at function granularity keeps instrumentation cheap while capturing locality, and the runtime exploits application-level knowledge — function-level locality and call-stack monitoring — that a hardware cache cannot see, even declining to cache cold code when it would evict hotter code or cost more than it saves; on a commercial FRAM microcontroller across nine MiBench2 benchmarks it eliminates 65 percent of FRAM accesses, running up to 46 percent faster (average 26 percent) at up to 36 percent less energy (average 24 percent), transparently, on constrained parts where prior basic-block software caches degrade or fail — with the caveats that on-stack functions cannot be evicted, oversized functions cannot be cached, management costs cycles, and it needs idle SRAM and toolchain integration.

Check yourself

Check your understanding

1. Why does storing code in FRAM make an ultra-low-power device slow, despite FRAM's advantages?
2. Why can the built-in hardware cache and prior software caches not solve this?
3. How does SwapRAM cache code in software, and why is cached code overhead-free?
4. What did SwapRAM achieve, and what are its limits?

Sources

The paper is “A Software Caching Runtime for Embedded NVRAM Systems” by Harrison Williams and Matthew Hicks (Virginia Tech), published at ASPLOS 2024 (Volume 4), DOI 10.1145/3622781.3674183. It is released under a Creative Commons Attribution license and the authors open-source their design and data.

This explainer is written from the full text of the authors’ open-access version. Everything above comes from the paper’s own sections: the introduction and background on ultra-low-power embedded systems and emerging non-volatile RAMs, FRAM’s density (about one transistor per bit versus six for SRAM) and non-volatility advantages and its use as unified memory (a representative part with 2 kilobytes of SRAM and 128 kilobytes of FRAM), and FRAM’s performance penalty (an 8-megahertz maximum access frequency against a 24-megahertz core, forcing wait states, and more than twice the power of comparable flash); the observation that more than 75 percent of memory accesses are instruction fetches and that unified memory’s alternating code and data accesses thrash the tiny fixed hardware cache (32 bytes for 128 kilobytes of FRAM); the finding, illustrated by measuring an arithmetic benchmark on an MSP430FR2355 with code and data in different memories, that when separated, code is best placed in SRAM; the limitations of prior DRAM-oriented software instruction caches on integrated NVRAM (a few-cycle miss exposing fine-grain overhead, and scarce SRAM for metadata); the SwapRAM design — a compile-time assembly pass rendering functions runtime-relocatable and replacing direct calls with indirect calls through a redirect table plus a funcId marker, and a runtime with a cache miss handler that copies functions into SRAM, fixes up branch targets, rewrites call sites, and later bypasses the handler; the choice of function-level granularity over basic-block instrumentation, the eviction mechanism and the per-function active counter guaranteeing call-stack integrity (aborting to execute from NVRAM when necessary), and the use of function-level locality and call-stack monitoring plus the option to decline caching; and the evaluation on a commercial FRAM microcontroller across nine MiBench2 benchmarks reporting a 65 percent average reduction in FRAM accesses, an average 26 percent (up to 46 percent) execution-speed improvement, and an average 24 percent (up to 36 percent) energy reduction versus the FRAM baseline, on platforms where prior software caches degrade or fail.