Systematic CXL Memory Characterization and Performance Analysis at Scale

CXL memory expanders are shipping and everyone models them as "slower DRAM" — Melody measures 265 workloads on 4 real devices and finds the model wrong, exposing microsecond tail latencies under LIGHT load (local DRAM never wobbles), wild vendor heterogeneity, a prefetcher cascade that silently converts latency into cache stalls, and a CXL+NUMA combination that turns a 5%-slowdown workload into a 2.9× one — while its Spa method turns 9 stall counters into a sub-5%-error breakdown of exactly where each slowdown comes from.

Why this problem is worth solving

The assumption deserves scrutiny because CXL changes what kind of thing memory is. Local DRAM sits behind an integrated memory controller (iMC) that CPU vendors have tuned for decades, on a deterministic DDR bus. CXL memory sits behind (1) a transaction and link layer with queuing, CRC, retries, and flow-control back-pressure, and (2) a third-party controller from a young ecosystem — ASICs from several vendors, sometimes FPGAs. Whether the resulting tier behaves like “NUMA, but a bit further” or like something qualitatively new determines how operating systems should tier pages, how schedulers should place workloads, and whether latency-sensitive services can use CXL at all. Melody’s answer — measured, not modeled — is that the difference is real: stability, not just speed, separates CXL from everything the memory hierarchy has held before.

What people did before, and where it broke

• Coarse characterization. Prior CXL studies (on early hardware) measured average latency and bandwidth on small workload sets. Useful, but they inherited DRAM’s implicit contract — that latency is stable — without checking it. No study had reported CXL tail latencies at all.
• Heuristic slowdown analysis. To explain why workloads slow down on far memory, research leaned on system-level metrics (LLC misses, memory-boundedness) or learned models combining many counters. These are inaccurate and uninterpretable — an LLC miss rate says nothing about whether the damage comes from demand reads, store-buffer pressure, or broken prefetching.
• Top-down analysis (TMA) doesn’t diff. Intel’s standard methodology attributes pipeline inefficiency within ONE run. CXL analysis needs the difference between two runs (local vs CXL) attributed to components — a question TMA wasn’t built to answer, since its metrics capture hardware contention, not workload-level slowdown.
• Black-box controllers. CXL devices expose no internal performance counters (a Performance Monitoring Unit arrives only with CXL 3.0), so any explanation must be reconstructed from the CPU side.

What Melody measured, and how Spa explains it

The idea

Tail latency is the missing dimension of CXL characterization. Melody’s headline empirical result: local DRAM and NUMA hold p99.9-minus-median latency to 45–61 ns even at 90% bandwidth utilization — decades of iMC engineering made stability an invisible guarantee. Three of four CXL devices break it. CXL-B and CXL-C show 160 ns tail gaps (+50% over median) under light, unsaturated load; at p99.99+ every device exceeds 700 ns and two reach 1 µs; the FPGA device spikes to 3 µs. Average latency begins climbing at 50–86% utilization on CXL versus 90–95% locally, mixed read/write traffic makes tails worse, and prefetchers don’t hide them. The causes are structural, not incidental: a non-deterministic PCIe-based protocol (flow-control back-pressure, transaction-layer queuing) multiplied by young third-party controllers that lack the iMC’s generation-polished scheduling. “Slower DRAM” is therefore the wrong model — CXL is less predictable DRAM, and unpredictability, not the extra 100 ns, produces the study’s most violent anomaly: a workload with under 5% slowdown on every CXL device collapses to 2.9× under CXL-behind-NUMA, driven purely by an 800 ns tail that emerges at p98.

To explain a slowdown, difference two runs — the stalls tell you everything. Spa’s insight is that the right unit of analysis is neither one system’s profile (TMA) nor proxy metrics (LLC misses) but the differential: the same instructions retire in both runs, so extra cycles on CXL are extra stall cycles, and frontend stalls barely change (instruction caches don’t feel CXL). What remains — backend memory stalls — is countable per-component because Intel’s bound-on-loads/stores and L1/L2/L3-miss stall counters partition exclusively. Nine counters thus yield an exact, interpretable decomposition of slowdown into store-buffer pressure, per-cache-level damage, and true demand-read latency — no models, no learning, under 5% error. The decomposition immediately pays off: identical-looking slowdowns have wildly different anatomies (605.mcf is nearly all LLC demand misses; 603.bwaves is nearly all prefetch damage), so the right fix differs per workload.

The prefetcher cascade, finally mechanized. “Prefetchers are less effective under CXL” was folklore; Spa turns it into a mechanism with a smoking-gun correlation. CXL’s latency delays L2 prefetches — they arrive late or pending, so L2 prefetch coverage drops (2–38% fewer L2-prefetch LLC misses). L1 prefetches that would have hit freshly-prefetched L2 lines now bypass to CXL directly — and the increase in L1-prefetch LLC misses matches the L2 decrease almost exactly (y = x, Pearson 0.99). Delayed L1 hits then surface as cache-level stalls. Turn the prefetchers off and the “cache slowdown” vanishes into the DRAM category — proving the attribution — but performance gets worse (bwaves −50%), so the answer isn’t disabling prefetch; it’s recognizing that timeliness, not accuracy, is what long latency steals.

Spa in action: from mystery slowdown to two guilty objects

the semantic gap: architecture events vs workload behavior
prior tools: heuristics, learned models, low interpretability

Why it works

• Retired instructions are identical across memory backends → every extra CXL cycle is a stall cycle, and frontend deltas are negligible → slowdown ≈ Δ backend-memory stalls, a claim validated within 5% for over 95% of 265 workloads rather than assumed.
• Intel’s stall counters partition exclusively by level → simple subtraction attributes the delta to store buffer, L1, L2, LLC, or DRAM → attribution is exact and interpretable where miss-rate heuristics and learned models are neither; 9 counters make it deployable in production profilers.
• Pointer-chase at cacheline granularity with per-request logging (MIO) → distributions, not averages → the tails local DRAM never shows become visible, measurable, and traceable into application-level latency (Redis).
• Prefetch damage hides inside “cache stalls” → the disable-and-reattribute experiment (stalls migrate to DRAM category) plus the y = x correlation between L2-coverage loss and L1-bypass gain pin the mechanism → timeliness, not prediction accuracy, is what CXL latency breaks.
• Measuring across 4 devices × 7 latencies × 5 CPUs separates what is intrinsic to CXL (tails, prefetcher cascade, super-linear latency sensitivity) from vendor artifacts (the FPGA device’s half-duplex behavior, per-device read/write sweet spots) → conclusions generalize where single-device studies could not.

What it costs — and what to keep in mind

• CPU-side vision only. The CXL controller stays a black box; tail-latency causes are triangulated (protocol non-determinism, immature scheduling) rather than observed — thermal effects were tested to 70 °C and not confirmed. CXL 3.0’s performance-monitoring unit may eventually open the box.
• Intel-specific plumbing. Spa’s decomposition leans on Intel’s exclusive stall-counter hierarchy; porting to AMD/ARM requires equivalent counters. Devices are CXL 1.1 type-3 expanders; switches (600 ns territory) appear only via published data.
• Differential means two runs. Spa diagnoses by comparison, so it needs a local-DRAM reference execution — natural for offline analysis and tiering studies, less so for one-shot production triage (the tech-report prediction models address this).
• A snapshot of a young ecosystem. The measured heterogeneity — an FPGA device that can’t use the full-duplex link, ASICs with 3µs spikes — is partly infancy; CXL-D already nearly matches NUMA, and dual-device interleaving closes the bandwidth gap. The specific numbers will age; the methodology and the tail-latency lens are the durable contributions.
• Worst-case framing. All-on-CXL placement (no tiering or interleaving in the main results) deliberately bounds the damage; production deployments with sensible tiering will sit between the local and CXL curves.

The evaluation

The paper is an evaluation; its shape is the findings. Device level: not all CXL is created equal — 214–394 ns averages, 21–59 GB/s peaks, full-duplex sweet spots at different read/write ratios per device, and tails that separate CXL from every iMC-managed memory. Workload level: many workloads tolerate CXL (43% suffer under 5% slowdown on the best device; even at an emulated 410 ns, 16% stay under 10%), a 7% tail suffers 1.5–5.8× (bandwidth-starved on 8-lane devices), and slowdown grows super-linearly with latency. Architecture level: Spa’s under-5%-error attribution, the prefetcher cascade, and the CXL+NUMA tail anomaly.

Check yourself

Sources

• ACM DL (open access): Systematic CXL Memory Characterization and Performance Analysis at Scale — ASPLOS 2025
• Melody artifacts on GitHub