Performance Predictability in Heterogeneous Memory

When you move a workload's data from fast local DRAM onto slower CXL-attached memory, some workloads barely slow down while others crater — and the metrics people use (access frequency, latency, stall cycles) predict the slowdown poorly, so operators either over-buy DRAM or migrate data reactively after performance already tanked; this paper (Virginia Tech and Microsoft — Jinshu Liu, Daniel Berger, Huaicheng Li, and colleagues) presents CAMP, which reads a dozen CPU counters during a normal DRAM run and, from the CPU's internal pressure points, predicts the CXL slowdown in advance (91 to 97 percent accurate) and computes the best DRAM-to-CXL split before committing to it.

The argument in one glance

Problem

You cannot tell in advance how much CXL memory will slow a workload

Modern servers increasingly pair fast local DRAM with larger, slower memory tiers — remote NUMA memory, or memory attached over Compute Express Link (CXL), which lets memory be disaggregated and pooled across machines to expand capacity cost-effectively. The catch is that moving a workload's data onto the slower tier affects performance very unevenly: some workloads barely notice the extra latency, while others suffer severe pipeline stalls, waste scarce DRAM bandwidth, and blow through their Service Level Objectives. To manage such systems well, you need predictive visibility — the ability to know, before you make a placement decision, exactly how much it will slow a given workload down. But that is precisely what is missing. The metrics people use to guide placement — how frequently a page is accessed, average memory latency, memory bandwidth, or raw CPU stall cycles — correlate only weakly with the actual slowdown a workload experiences on CXL. Because they cannot forecast the impact, operators are left with bad options: over-provision DRAM to be safe (wasting the capacity savings CXL was supposed to deliver), or run reactive migration loops that only fix placement after performance has already degraded. What is needed is a way to predict CXL-induced slowdown accurately, in advance.
What people did

Guide placement with access frequency, latency, or stall cycles

Existing systems lean on runtime signals to decide what belongs in fast memory. Some rank pages by access frequency or last-level-cache misses; some monitor average memory latency or bandwidth; some measure CPU stall cycles directly. Profiling tools flag architectural events that correlate with performance, and reactive tiering systems migrate pages once they detect hotness or a degradation. A related tool, Melody, can even decompose a slowdown into components — but it does so as an attribution of the past, requiring the workload to actually run on both DRAM and CXL to explain what already happened.
Why it fell short

These signals describe the past; they do not predict the future

Each of these signals is either a weak proxy or backward-looking. Access frequency does not capture how much an access actually stalls the CPU, because memory-level parallelism can hide the latency of some accesses entirely while serialized ones expose all of it. Average latency is a poor proxy too: two workloads at the same measured latency can incur wildly different stalls depending on instruction dependencies, how many instructions can be in flight, and how outstanding misses overlap. Bandwidth is imprecise, and neither latency nor bandwidth captures pressure on the CPU's internal data buffers. Even raw stall cycles, though they quantify current performance loss, do not predict: stalls measured on DRAM do not scale linearly to CXL's higher latency, because the scaling depends on overlap, dependency structure, and prefetch behavior that themselves change on the slower tier. Attribution tools like Melody explain a slowdown only after running on both tiers, and reactive tiering only acts after the damage is done. So the field could describe degradation, or attribute it in hindsight, but could not answer the forward-looking question an operator actually needs: given this workload, how much will it slow down on CXL, or at a given DRAM-to-CXL mix, before I put it there? What was missing was a predictive, causal model of slowdown built from signals you can read during an ordinary run.
This paper's idea

Read the CPU's pressure points on DRAM to forecast the CXL slowdown

CAMP (Causal Analytical Memory Prediction) turns backward-looking attribution into forward-looking prediction. Its central insight is that a workload's slowdown on slower memory is not an opaque property of the memory device — it is a predictable consequence of specific microarchitectural pressure points inside the CPU. The structures that bottleneck when memory gets slower — the Line Fill Buffers that track outstanding cache misses, the Store Buffers that absorb writes, and the limit on how much memory-level parallelism can overlap — leave distinct "fingerprints" that are visible even while the workload runs entirely on fast DRAM. Those three pressure points correspond to three orthogonal, additive sources of slowdown: demand reads, cache and prefetching, and stores. CAMP reads at most 12 Performance Monitoring Unit (PMU) counters during a run and analytically decomposes the predicted slowdown into those three components, so for most (non-bandwidth-bound) workloads it can forecast the CXL slowdown from a single DRAM-only execution — no need to actually run on CXL first. Because the models capture fundamental pipeline behavior, they also generalize to interleaved DRAM-CXL: CAMP derives a closed-form model that synthesizes the whole performance curve across every DRAM-to-CXL ratio from at most two profiling runs. Across 265 workloads on NUMA and three CXL devices, CAMP predicts slowdown with 91 to 97 percent accuracy within 10 percent absolute error, and it powers practical policies — "Best-shot" interleaving that analytically picks the optimal ratio, and colocation placement — improving performance by up to 21 and 23 percent over existing tiering and colocation approaches.

Why this problem is worth solving

The entire economic case for CXL memory rests on being able to place data on the slow tier without unpredictable performance loss, so a good slowdown predictor unlocks the technology.

CXL exists to let datacenters add memory capacity cheaply by pooling and disaggregating it, but that only pays off if you can safely put real data on the slower tier. The blocker is uncertainty: because the same CXL memory slows different workloads by wildly different amounts, an operator who cannot predict the impact must assume the worst. In practice that means over-provisioning expensive DRAM to protect latency-sensitive workloads — which throws away the capacity savings CXL was meant to provide — or accepting reactive migration systems that let performance degrade first and scramble to fix placement afterward, wasting resources and still violating Service Level Objectives in the meantime. A predictor that says, before you act, exactly how much a workload will slow down at a given placement changes the game: you can put data where it is cheapest without gambling on performance, size DRAM to actual need rather than to fear, and choose placements analytically instead of by trial and error. Because heterogeneous memory is becoming standard in cloud servers, and because the cost of memory is a major fraction of datacenter spending, a tool that makes CXL performance predictable acts directly on one of the largest levers in modern infrastructure economics.

The key realization is that slowdown is caused by identifiable CPU structures whose stress is already observable before you ever touch the slow tier.

The reason prior metrics fail is that they treat slowdown as a black box — a single number attached to a memory device — and try to correlate it with a single proxy like latency or access count. CAMP’s insight is that slowdown is mechanistic: when memory gets slower, specific hardware structures inside the CPU become the bottleneck, and each stresses in a way that is measurable in advance. Outstanding cache misses pile up in Line Fill Buffers; writes back up in Store Buffers; and the processor can only overlap so many in-flight memory operations before it stalls waiting to retire instructions. Crucially, how close a workload already is to saturating each of these structures is visible while it runs on fast DRAM — the fingerprints are there before the latency increases. And because these three pressure points act on independent parts of the pipeline, their contributions to slowdown are orthogonal and simply add up: total slowdown equals the demand-read part plus the cache part plus the store part. That decomposition is what makes prediction, rather than mere correlation, achievable: instead of guessing from one proxy, CAMP measures how much each causal mechanism will amplify under higher latency and sums them. This is also why the model generalizes — the same pipeline physics governs pure CXL and any interleaved mix, so one set of measurements extrapolates across configurations that a reactive metric could never anticipate.

What people did before, and where it broke

Rank pages by access frequency or LLC misses. Assume the most-accessed data is the most placement-sensitive. Where it broke. Frequency ignores whether accesses actually stall the CPU — memory-level parallelism hides some latency and exposes other — so it predicts slowdown weakly.
Monitor average latency or bandwidth. Use observed memory latency or bandwidth as the placement signal. Where it broke. Two workloads at the same latency can suffer very different stalls depending on dependencies and overlap, and neither latency nor bandwidth captures CPU buffer pressure.
Measure raw stall cycles. Read CPU stall counters to quantify current performance loss. Where it broke. Stalls on DRAM do not scale linearly to CXL’s higher latency, and moving to a slower tier changes prefetch and overlap behavior, so stalls describe the present but do not predict the future.
Attribute slowdown after the fact (Melody) or migrate reactively (TPP, Memtis, Nomad). Decompose degradation once observed, or move pages after detecting it. Where it broke. Attribution needs runs on both tiers to explain the past, and reactive migration acts only after performance has already dropped.
The gap. No approach predicted, before placement and from an ordinary run, how much a workload would slow down on CXL or at a given DRAM-to-CXL ratio — because none had a causal, decomposed model of the microarchitectural sources of slowdown.

How CAMP works

From reactive, weakly-correlated metrics to a causal slowdown forecast

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Measure the true performance cost from CPU counters, not a weak proxy. CAMP replaces frequency, latency, and bandwidth with a signal grounded in what actually causes slowdown — how hard specific CPU structures are being pressed — read directly from hardware performance counters. Quantifying a page or workload’s genuine performance impact from CPU stall behavior via microarchitectural counters, rather than trusting an easy-to-measure but weakly-correlated proxy, is the same principle the sibling work on criticality-first tiered memory placement (PACT) covered elsewhere on this site is built on, deriving true cost from what the processor actually experiences. CAMP applies this to forecasting slowdown, using the counters to model the three causal pressure points instead of inferring importance from access counts.

Predict the outcome before acting, instead of reacting after damage. The heart of CAMP is turning attribution and reaction into a-priori prediction: from an ordinary run it forecasts how a workload will perform on CXL or at any interleaving ratio, so the system chooses correctly the first time rather than degrading and correcting. Building a forward-looking model that anticipates a decision’s effect and acts pre-emptively, rather than running a reactive control loop that fixes problems only after they appear, is the same shift the work on predicting rather than reacting for fine-grain GPU frequency control (predict, not react) covered elsewhere on this site makes, replacing feedback correction with foresight. CAMP’s closed-form models are what let it answer these what-if questions without ever having to run the workload on the slow tier to find out.

Turn the forecast into optimal placement and interleaving. CAMP’s predictions feed concrete policies — “Best-shot,” which analytically computes the optimal DRAM-to-CXL ratio, and colocation scheduling — for placing data across a fast and slow tier without trial-and-error search. Deciding how to split and place data across DRAM and CXL to get capacity at low cost without paying an unacceptable performance penalty is the same enterprise the work on transparent page placement for CXL-enabled tiered memory (TPP) covered elsewhere on this site pursues, managing the fast-slow boundary. CAMP advances it by replacing reactive migration and exhaustive ratio search with an analytical prediction that names the best placement directly.

Following a workload through CAMP

Forecasting a workload's CXL slowdown before placing it

A trace, drawn from the paper, of how CAMP predicts slowdown from a DRAM-only run and picks a placement.

Step 1 of 5

Run the workload once on fast DRAM

CAMP profiles the workload during an ordinary execution on local DRAM, recording at most 12 hardware Performance Monitoring Unit counters — the ones that reveal pressure on Line Fill Buffers, Store Buffers, and memory-level parallelism.

For most workloads no CXL run is needed; bandwidth-bound ones add one CXL run.

Why it works

Slowdown has identifiable causes, so it can be modeled not just correlated. A single proxy like latency treats slowdown as a black box → but attributing it to Line Fill Buffer, Store Buffer, and MLP pressure → so CAMP models the mechanism rather than guessing from one number.
The causes are visible on DRAM before CXL is ever touched. You would seem to need a CXL run to know the CXL slowdown → but the pressure-point fingerprints already appear during a DRAM-only run → so CAMP forecasts from a single ordinary execution.
Orthogonal components make the model additive and generalizable. Interacting effects would be hard to predict → but demand-read, cache, and store slowdowns act on independent pipeline parts → so total slowdown is their sum and the model transfers across configurations.
A closed-form interleaving model covers every ratio cheaply. Testing each DRAM-to-CXL mix would need exhaustive runs → but deriving the performance curve analytically from at most two runs → so any interleaving ratio can be evaluated without searching.
Prediction enables acting right the first time. Reactive migration degrades performance before correcting it → but forecasting the impact before placement → so Best-shot and colocation choose the optimal placement up front.

What it costs (trade-offs and scope)

Bandwidth-bound workloads need an extra CXL run. The single-DRAM-run prediction covers non-bandwidth-bound workloads; bandwidth-bound ones require an additional CXL profiling run, so the “one run” convenience does not hold universally.
It depends on the right hardware counters and microarchitecture. CAMP reads up to 12 PMU counters tied to Line Fill Buffers, Store Buffers, and MLP; the approach assumes those counters exist and behave consistently on the target CPU.
It is a model, so predictions carry error. Accuracy is 91 to 97 percent within 10 percent absolute error — excellent, but not exact; some workloads and configurations will fall outside the tight band, and policies built on it inherit that residual error.
It targets placement and interleaving, not the workload itself. CAMP predicts and optimizes where data lives across DRAM and CXL; it does not change the application’s memory behavior, so its benefit is bounded by how much smarter placement can help.
Reported figures are on specific devices and baselines. The accuracy and the up-to-21-and-23-percent gains come from 265 workloads on NUMA and three CXL devices against particular tiering and colocation approaches; results on other hardware or workloads may differ.

The comparison

Deciding how to place data across DRAM and CXL

Property	Frequency / latency metrics	Attribution and reactive tiering	CAMP
What it produces
Correlation with real slowdown
Runs needed to predict CXL
Handles interleaving ratios
Policy outcome

A comparison of ways to decide data placement across DRAM and CXL. Frequency and latency metrics correlate weakly with actual slowdown; attribution tools and reactive tiering explain or fix degradation only after it happens. CAMP reads a dozen CPU counters during an ordinary DRAM run, decomposes slowdown into three causal microarchitectural components, and predicts CXL slowdown (and the whole interleaving curve) in advance with 91 to 97 percent accuracy — driving Best-shot placement that beats prior tiering and colocation by up to 21 and 23 percent. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ copy hosted at Virginia Tech). The described system is the paper’s: CAMP (Causal Analytical Memory Prediction) is a framework for predicting CXL-induced slowdown in heterogeneous DRAM-plus-CXL memory. Its insight is that slowdown is a predictable consequence of microarchitectural pressure points — Line Fill Buffers (outstanding cache misses), Store Buffers (write backpressure), and memory-level-parallelism-limited retirement — whose fingerprints are visible during a DRAM run. CAMP uses at most 12 Performance Monitoring Unit counters to analytically decompose slowdown into three orthogonal components (demand reads, cache/prefetching, and stores), predicting CXL slowdown for non-bandwidth-bound workloads from a single DRAM-only execution (bandwidth-bound workloads also need a CXL run), and derives a closed-form model for software-based weighted interleaving that synthesizes performance across DRAM-to-CXL ratios from at most two profiling runs. Reported accuracy is 97 percent under NUMA and 91 to 96 percent under CXL, with most predictions within 10 percent absolute error, across 265 workloads on NUMA and three CXL devices; the paper reports a Pearson correlation of 0.97 versus 0.37 to 0.88 for prior metrics. CAMP enables policies including “Best-shot” predictive interleaving and colocated workload placement, improving performance by up to 21 percent and 23 percent over existing tiering and colocation approaches. CAMP is open-source. Background on CXL, heterogeneous memory, memory-level parallelism, and CPU buffers (Line Fill and Store Buffers) is standard context. The step-by-step walkthrough is an illustration of the described mechanism. Exact model derivations, counter choices, and the full evaluation should be read from the paper.

Background

Why heterogeneous memory keeps demanding better ways to reason about placement

It helps to see CAMP within the broader arc of heterogeneous and disaggregated memory. As DRAM cost and capacity pressures push datacenters to add slower tiers — NUMA, persistent memory, and now CXL pools shared across a rack — the central recurring question becomes: what data belongs where, and what will it cost in performance? The field has cycled through answers, each a signal to drive placement: recency, frequency, latency, stall cycles. The recurring lesson is that easy-to-measure proxies stop tracking real end-to-end performance once modern out-of-order CPUs enter the picture, because the same access can be free or crippling depending on what the pipeline is doing around it. That pushes the frontier from measuring proxies toward modeling the actual microarchitectural mechanism of slowdown, and from reacting to degradation toward predicting it before it happens. CAMP is a clean instance of both moves at once. The broader platform it serves — making CXL-based, poolable memory practical to deploy in the cloud so that capacity can be added cheaply — is the same terrain the work on CXL-based memory pooling systems for cloud platforms (Pond) covered elsewhere on this site occupies, turning disaggregated memory into a usable resource. CAMP’s particular contribution is to make that resource’s performance predictable: by reading the CPU’s pressure points during an ordinary run and decomposing slowdown into causal, additive parts, it forecasts the impact of a placement before you commit to it, and turns that foresight into measurably better tiering, interleaving, and colocation.

If you remember only three things

1. Heterogeneous memory pairs fast local DRAM with slower tiers like NUMA and CXL (which disaggregates and pools memory to add capacity cheaply), but moving data onto the slow tier slows different workloads by wildly different amounts. The problem is that the signals systems use to guide placement — access frequency, average latency, bandwidth, raw stall cycles — correlate only weakly with the actual CXL-induced slowdown (Pearson 0.37 to 0.88 in the paper). They are weak proxies or backward-looking: frequency ignores whether accesses stall the CPU, equal latencies can cause very different stalls, stall cycles measured on DRAM do not scale linearly to CXL, and attribution tools or reactive tiering only explain or fix degradation after it has happened. So operators cannot tell in advance how a placement will perform, forcing them to over-provision DRAM or migrate reactively.
2. CAMP (Causal Analytical Memory Prediction) turns attribution into prediction. Its insight is that CXL slowdown is not an opaque device property but a predictable consequence of specific CPU pressure points — Line Fill Buffers holding outstanding misses, Store Buffers absorbing writes, and the limit on overlapping in-flight accesses — whose fingerprints are visible even while the workload runs on fast DRAM. CAMP reads at most 12 hardware performance counters during a run and analytically decomposes the predicted slowdown into three orthogonal, additive components (demand reads, cache/prefetching, stores), so for most workloads it forecasts the CXL slowdown from a single DRAM-only execution, and a closed-form model extends this to the full performance curve across every DRAM-to-CXL interleaving ratio from at most two runs.
3. Across 265 workloads on NUMA and three CXL devices, CAMP predicts slowdown with 91 to 97 percent accuracy within 10 percent absolute error (Pearson 0.97, versus 0.37 to 0.88 for prior metrics), and it powers practical policies: "Best-shot" interleaving that analytically picks the optimal DRAM-to-CXL ratio, and colocation placement — improving performance by up to 21 and 23 percent over existing tiering and colocation approaches. The trade-offs: bandwidth-bound workloads need an extra CXL run, it depends on the right hardware counters and microarchitecture, its predictions carry residual error, and it optimizes placement rather than the workload itself. This explainer is based on the paper's full text; the model derivations and evaluation should be confirmed in the paper, which is open-source.

Check yourself

Check your understanding

1. Why do existing metrics fail to tell operators how much CXL memory will slow a workload?
2. What is CAMP's central insight about where CXL slowdown comes from?
3. How does CAMP predict CXL slowdown, and what makes it able to forecast rather than just describe?
4. What does CAMP achieve, and what are the limits of the approach?

Sources

The paper is “Performance Predictability in Heterogeneous Memory” by Jinshu Liu, Hanchen Xu, Daniel S. Berger, Marcos K. Aguilera, and Huaicheng Li (Virginia Tech, Microsoft and the University of Washington, and NVIDIA), published at ASPLOS 2026, DOI 10.1145/3779212.3790201. The authors host a copy at people.cs.vt.edu, and CAMP is open-source at github.com/MoatLab/CAMP.

This explainer is written from the paper’s full text (the authors’ hosted copy). CAMP (Causal Analytical Memory Prediction) predicts CXL-induced slowdown in heterogeneous DRAM-plus-CXL memory by treating slowdown as a predictable consequence of microarchitectural pressure points — Line Fill Buffers, Store Buffers, and memory-level-parallelism limits — whose fingerprints appear during a DRAM run. Using at most 12 performance counters, it decomposes slowdown into three orthogonal components (demand reads, cache/prefetching, stores), predicts CXL slowdown for most workloads from a single DRAM-only run, and derives a closed-form interleaving model across DRAM-to-CXL ratios from at most two runs. Across 265 workloads on NUMA and three CXL devices, the paper reports 91 to 97 percent accuracy within 10 percent error (Pearson 0.97), and policies (Best-shot interleaving and colocation) improving performance by up to 21 and 23 percent. Background on CXL, heterogeneous memory, and CPU buffers is standard context. The numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.