Rethinking Java Performance Analysis

Stephen M. Blackburn, Zixian Cai, Rui Chen, Xi Yang, John Zhang and John N. Zigman

Systems research advances by measuring what's faster — but that only works if we measure the right things with representative workloads, and this paper (from the team behind the DaCapo Java benchmarks) argues our field's measurement rigor keeps falling behind its ideas, with an alarming example — modern Java garbage collectors built to cut latency actually burn MORE total CPU than 20-year-old ones (15% of all cycles even at best, up to 17x in tight memory), and almost nobody noticed, because the community measured proxies (GC pause times, wall-clock time, one heap size) instead of the real quantities (user-visible latency, total CPU across all cores, the whole memory-versus-time tradeoff); the fix is DaCapo Chopin, a 14-years-in-the-making overhaul of the benchmark suite with 22 workloads (8 brand-new, 9 latency-sensitive real frameworks), 47 statistics, and new methods that directly measure user latency and the true cost of garbage collection — plus recommendations to keep a fast-moving field from being misdirected by unsound measurement.

The argument in one glance

Problem

Our field runs on measurement — and its measurement foundations keep falling behind

Systems research advances by PERFORMANCE ANALYSIS: measuring which design is faster or more efficient, which is how we justify claims and decide what to build. Sound performance analysis rests on two foundations — representative BENCHMARKS (realistic workloads to run) and principled METHODOLOGIES (how you measure and report). Both are hard: good benchmarks are expensive to create and maintain, methodologies are hard to develop and easy to get subtly wrong, and computer systems research moves fast. So the foundations lag behind the field. And when they lag, the consequence is not just noise — an unsound measurement can misdirect an entire field, encouraging everyone to chase ideas that should have been abandoned and to abandon ideas that were actually promising.
What people did

Use the standard practice: familiar benchmarks and familiar metrics

The usual approach is to grab an established benchmark suite, run it, and report a headline number — typically wall-clock time. For garbage collection specifically, a long-standing convention is to report GC PAUSE TIMES (how long the collector stops the application) as a stand-in for how responsive the system feels to a user, and to evaluate at a convenient heap size. These conventions are widely used, easy to apply, and have been around for years.
Why it fell short

The standard metrics measure the wrong things, and hide real regressions

The trouble is that these conventions measure PROXIES, not the real quantities — and the gap can hide serious problems. The paper's alarming case study measures five production Java garbage collectors: even in the most favorable conditions, garbage collection consumes 15% of CPU cycles, and in tight memory the overhead reaches 17x. Worse, the newer collectors designed to reduce latency (Shenandoah in 2014, ZGC in 2018) actually have HIGHER total overhead than collectors from 1998 to 2009 — a regression that has "largely gone unnoticed." How? Because the field measured the wrong things: GC pause times instead of user-experienced latency (twenty years ago Cheng and Blelloch showed several short pauses can be as bad as one long one, yet the proxy persists); wall-clock time instead of total CPU cost (a collector that hides on idle cores looks free but is not); a single heap size instead of the whole time-space tradeoff; and ad-hoc micro-benchmarks instead of representative workloads. As the authors put it, "we don't improve what we don't measure."
This paper's idea

Rebuild the benchmarks and the methodology to measure the real quantities

This paper (Australian National University and Google, with industry — Blackburn, Cai, Chen, Yang, Zhang, Zigman) responds with DaCapo Chopin, a complete overhaul of the DaCapo Java benchmark suite plus a set of methodological fixes, all aimed at measuring what actually matters. The suite is fourteen years of work and roughly 16 million lines of code: 22 real-world workloads (8 entirely new, all others refreshed), including 9 LATENCY-SENSITIVE workloads built on real frameworks (a game engine, Spring, Cassandra, Kafka, and more) that DIRECTLY measure user-experienced latency, with minimum heap sizes from 5 MB to 20 GB, plus 47 statistics characterizing each workload and Principal Component Analysis demonstrating the suite is genuinely diverse. Alongside it come methodological recommendations that answer the four failures: sweep a RANGE of heap sizes to expose garbage collection's time-space tradeoff; report USER-experienced latency (with new "simple" and "metered" latency metrics), not GC-pause proxies; report BOTH wall-clock and total CPU overhead; and use a Lower-Bound-Overhead method to measure garbage collection's diffuse, hard-to-attribute real cost. The Java example is concrete, but the lesson generalizes: keep a fast-moving field's measurement rigor from lagging behind its ideas, so it identifies profitable directions and quickly discards unprofitable ones.

Why this problem is worth solving

An entire research field can be quietly steered wrong by how it measures, and the cost of that is enormous but invisible — years spent optimizing the wrong thing, promising ideas dropped because a flawed metric made them look bad. Understanding what sound measurement requires comes first. A

PERFORMANCE ANALYSIS is the empirical practice at the heart of systems research: measuring how a system performs so you can compare designs and justify claims about which is better. It has two foundations. Representative BENCHMARKS are the workloads you run — they must resemble real applications, because a benchmark that is not representative (a tiny synthetic loop, an outdated program) can make a design look good or bad in ways that will not hold in practice. Principled METHODOLOGIES are how you measure and report — which quantity you measure, under what conditions, with what statistical care, and how you present it. Both are hard to get right and expensive to maintain, and a field that moves fast keeps outrunning them. The stakes are high because the whole edifice of “which design is faster” rests on these foundations: an unsound measurement does not merely mislead one paper, it can misdirect a community

is the setting. The paper’s argument is that the community’s measurement rigor has fallen behind, and its case study shows the damage.

The concrete failure the paper diagnoses is that the field measured convenient stand-ins instead of the quantities that matter. A

garbage collection has a fundamental TIME-SPACE TRADEOFF: with less memory it must collect more often, costing more CPU time, so its overhead depends on how much memory it is given — and evaluating at a single heap size, as is common, hides this entirely. Beyond that, the community relied on PROXIES for the things that are genuinely hard to measure. GC PAUSE TIME (how long the collector halts the application) was used as a proxy for user-experienced LATENCY, even though many short pauses can hurt responsiveness as much as one long pause, so the proxy misses the real user experience. WALL-CLOCK time was reported instead of TOTAL CPU cost, so a concurrent collector that does its work on otherwise-idle cores looks nearly free, when in a multi-tenant setting those cores have a real opportunity cost. And GC’s overhead is finely woven into the application (allocation cost, read and write barriers, cache-locality effects), making it hard to attribute at all. Each of these gaps let real overheads and regressions go unnoticed

is the crux. The problem is not any one engineer’s mistake; it is a collective methodological inattention, and fixing it means measuring the true quantities.

What people did before, and where it broke

GC pause times as a latency proxy. Report how long the collector stops the application, and treat that as responsiveness. Easy to measure, but Cheng and Blelloch showed two decades ago that many short pauses can be as bad as one long one, so it does not reflect what a user actually experiences.
Wall-clock time as the cost metric. Report elapsed time. But it hides total computational cost: a concurrent collector using idle cores looks free on wall-clock while consuming real CPU that has an opportunity cost on any shared machine.
Single-heap-size evaluation. Measure at one convenient memory size. This hides garbage collection’s time-space tradeoff, so a collector’s overhead at scarce memory — often much larger — never appears.
Ad-hoc and micro-benchmarks (Whetstone, gcbench, hand-picked loops). Cheap and easy, but far from representative of real applications, so results may not transfer to practice. Even good older suites age as the ecosystem evolves.
The gap. No shared foundation combined representative, diverse, modern workloads with methodologies that measure user-experienced latency, total CPU cost, and the time-space tradeoff — so a real, field-wide regression could hide in plain sight. That is the gap DaCapo Chopin fills.

How to measure Java performance soundly

Measure the real quantities: user latency, total CPU, the tradeoff, the diffuse cost, on representative workloads

The stakes: measurement is the foundation

Systems research depends on representative benchmarks and principled methodologies, and when they lag the field is misdirected. A case study measuring five production Java garbage collectors finds large overheads and a regression in newer collectors that had gone largely unnoticed — because the field measured proxies.
↓ →
Measure user-experienced latency, not GC pauses new

DaCapo Chopin times every user-visible event (a frame render, a client request) and reports the full latency distribution as percentiles or CDFs. It defines SIMPLE latency (time each event directly) and METERED latency (which models request queuing, so a pause delays not only the running request but the whole backlog behind it) — capturing what users feel, not a pause-time stand-in.
↓ →
Measure total CPU, not just wall-clock time new

It reports both wall-clock overhead AND total CPU overhead (task clock, summing all threads). A concurrent collector that hides on idle cores has low wall-clock overhead but real total-CPU cost — an opportunity cost on any shared or multi-tenant machine — which only the task-clock measure reveals.
↓ →
Expose the time-space tradeoff by sweeping heap sizes new

Because garbage collection trades memory for CPU time, DaCapo Chopin evaluates across a range of heap sizes (up to 6x the minimum heap in which each workload runs), expressed as multiples of that minimum, rather than at one cherry-picked size — so the collector's behavior under both scarce and generous memory is visible.
↓ →
Measure the diffuse real cost with a principled lower bound new

Garbage collection costs are woven into the application (barriers, locality) and hard to attribute. The Lower-Bound-Overhead method distills a conservative approximation of an ideal zero-cost collector as a baseline; since the baseline overestimates the ideal, the measured overhead is a guaranteed UNDERESTIMATE — a transparent lower bound that still captures the hard-to-attribute costs.
↓ →
Use diverse, representative workloads, and prove diversity new

The suite is 22 real-world workloads (8 new, 9 latency-sensitive frameworks) with 47 characterizing statistics, and Principal Component Analysis shows the workloads are well-distributed rather than redundant — so the suite genuinely covers a range of behaviors. More than half were proposed or co-developed with industrial users.

DaCapo Chopin evaluated on OpenJDK 21 across five production garbage collectors and 22 workloads, on x86 (AMD Zen4) and ARM, with 95% confidence intervals over ten invocations. A methodology and benchmark-suite contribution, with lessons the authors argue generalize beyond Java.

The idea

Measure the real quantity, not a convenient proxy. The paper’s spine is a single discipline applied over and over: the field measured GC pause time when it meant user latency, wall-clock time when it meant total computational cost, and one heap size when the truth is a tradeoff across all of them — and each proxy hid a real effect. DaCapo Chopin replaces each proxy with the quantity that actually matters: it times user-visible events directly, reports total CPU as well as wall-clock, and sweeps the heap-size range. Refusing a convenient proxy for the thing you actually care about is the most-repeated discipline across the site: the I/O-scheduler page insists on true device occupancy over a queue-length proxy, the Go-concurrency fuzzer guides on channel state over branch coverage, the GPU value-profiler measures values over counters, and the datacenter-power page validates gains with a real throughput proxy rather than assuming them. This paper is that discipline turned into a manifesto for a whole field, with a case study of what the proxies cost: a years-long, unnoticed regression.

You cannot improve what you do not measure — so a new lens reveals the hidden problem. The regression in modern garbage collectors was not subtle once measured; it was invisible only because no one was measuring total overhead across the tradeoff on representative workloads. Building the instrument that surfaces a previously-invisible problem, and thereby making it fixable, is the same new-observability-lens move the GPU value-profiler page makes when it exposes value-level waste that hardware counters could never show. And it pairs with the security-cluster lesson that what you fail to test, you fail to defend: just as a Rowhammer defense is only sound if it is analyzed against the worst case, a design is only as good as the metric that would reveal its flaws. The corollary is cultural: the tools and benchmarks a community shares determine which problems it can even see, so investing in better measurement is investing in the field’s ability to make progress at all.

The benchmark and the methodology ARE the contribution. Most papers contribute an algorithm or a system; this one contributes shared infrastructure and rigor — a diverse, representative, maintained benchmark suite and a set of methodologies — because that is what a field needs and what keeps eroding. Treating the measurement apparatus and the demand for realism as first-class is the same measure-in-the-real-conditions discipline the site’s pages on interconnect contention and cloud side-channels insist on when they refuse idealized numbers, and the same experience-and-infrastructure spirit as the datacenter-scale deployment pages where the operational reality, not a new trick, is the point. The LBO method is emblematic: because GC’s true cost is diffuse and cannot be cleanly isolated, the contribution is a principled, transparent way to bound it — a conservative lower bound — rather than a false claim of exact attribution. Rigor about what you can and cannot honestly measure is itself the result.

The regression that went unnoticed

What proper measurement reveals

Measuring five OpenJDK 21 production garbage collectors with DaCapo Chopin. Numbers from the paper.

Step 1 of 5

Measure the collectors properly

The study evaluates five production Java garbage collectors (Serial, Parallel, G1, Shenandoah, ZGC) across a range of heap sizes on 22 real-world workloads, using the Lower-Bound-Overhead method to capture the total, hard-to-attribute cost of collection — not just its stop-the-world pauses.

the collectors span 1998 (Serial) to 2018 (ZGC), so a fair method should show how designs improved over time
the evaluation reports both wall-clock and total CPU overhead at heap sizes from one to six times the minimum

Why it works

The field measured GC pause time as a proxy for latency, but many short pauses hurt as much as one long one → measuring user-visible events directly (simple and metered latency) → responsiveness is judged by what users feel, not a stand-in that misreads it.
Wall-clock time hides work done on idle cores → reporting total CPU (task clock) as well → a concurrent collector’s real computational cost, and its opportunity cost on shared machines, becomes visible.
Garbage collection trades memory for CPU time, so one heap size hides the tradeoff → sweeping a range of heap sizes expressed as multiples of the minimum → overhead under both scarce and generous memory is exposed.
GC cost is diffuse and hard to attribute exactly → distilling a conservative ideal-collector baseline and reporting the overhead as a lower bound → the hard-to-attribute costs are captured honestly, without overclaiming precision.
Ad-hoc micro-benchmarks are not representative → 22 real-world workloads with 47 statistics and PCA-verified diversity → results reflect real application behavior and cover a genuine range, so a field-wide effect cannot hide.

What it costs (trade-offs and scope)

It is Java- and OpenJDK-21-specific in the concrete. The suite, metrics, and GC case study target Java on a particular runtime; the authors argue the lessons generalize to other garbage-collected languages and to performance analysis broadly, but the benchmarks and numbers are a snapshot that will age.
Representative benchmarks are a Sisyphean, never-finished cost. DaCapo Chopin took fourteen years and depends on ongoing community input to stay representative; the paper is explicit that no suite or methodology can ever be complete, and there is no guarantee it stays current as the ecosystem evolves.
The metrics carry deliberate judgment. The per-workload statistics are relative ranks in a fixed hardware and software context, not absolute measures; the LBO result is a conservative lower bound (an intentional underestimate); and metered latency uses a synthetic queuing model with a smoothing-window knob (they pick 100 ms as a middle ground) — so the numbers depend on these choices.
The recommendations add real experimental cost. Sweeping heap sizes, crossing collectors, reporting both wall-clock and task clock, plotting distributions, and running many invocations is far more work than a single wall-clock number — part of why it was skipped. The paper argues the rigor is worth it, but it is not free.
It is a critique of collective practice, not a new algorithm. The paper does not propose a better garbage collector or blame the engineers who built the ones it measures; its result is exposing a measurement gap and providing tools, so the payoff is indirect — better future research rather than a direct speedup.

The evaluation

DaCapo Chopin on OpenJDK 21, evaluating five production garbage collectors (Serial, Parallel, G1, Shenandoah, ZGC) across heap sizes from one to six times the minimum, on 22 real-world workloads (including 9 latency-sensitive frameworks like Cassandra, Kafka, and Spring), on x86 (AMD Zen4) and ARM, with 95% confidence intervals over ten invocations, using Lower-Bound-Overhead, wall-clock and task-clock measures, direct latency metrics, and Principal Component Analysis for diversity.

Measuring garbage-collector performance

Property	Common practice (proxy)	What the proxy hides	DaCapo Chopin (true metric)
Responsiveness
Cost
Memory
GC overhead
Workloads

Common practice measures convenient proxies — GC pause time, wall-clock, one heap size, micro-benchmarks — each of which hides a real effect, which is how a field-wide regression in modern collectors went unnoticed. DaCapo Chopin measures the true quantities: user-experienced latency, total CPU, the whole time-space tradeoff, and a principled lower bound on GC cost, on 22 representative, diversity-verified workloads.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

Read the scope precisely. The concrete artifact is Java- and OpenJDK-21-specific — the suite, the metrics, and the garbage-collector case study target one language and one runtime snapshot that will age — though the authors argue the methodological lessons generalize to other garbage-collected languages and to performance analysis broadly. Building representative benchmarks is the Sisyphean cost the paper itself names: DaCapo Chopin took fourteen years, relies on ongoing community input for representativeness, and can never be complete. The metrics carry deliberate judgment — the per-workload statistics are relative ranks in a fixed context, the LBO result is a conservative lower bound (an intentional underestimate), and metered latency uses a synthetic queuing model with a smoothing-window knob — so numbers depend on those choices. The recommendations add real experimental cost (sweeping heaps, crossing collectors, both wall-clock and task clock, full distributions, many invocations), which is part of why they were skipped. And it is a critique of collective practice, not a new algorithm or a better collector — its payoff is indirect, better future research rather than a direct speedup. Within that scope the lessons are sharp and general: measure the real quantity, not a convenient proxy; you cannot improve what you do not measure, so build the lens that reveals the hidden problem; expose fundamental tradeoffs by sweeping the range and report full distributions, not single numbers; bound a diffuse cost honestly rather than overclaim precision; and treat representative benchmarks and sound methodology as first-class, field-sustaining contributions.

Background

Measure the true quantity, build the lens, and treat methodology as the contribution

This paper sits in the site’s performance-analysis and methodology cluster and rhymes, at a meta level, with much of the site. Its spine is the most-repeated discipline here: measure the real quantity, not a convenient proxy — GC pause time is not user latency, wall-clock is not total CPU, one heap size is not the tradeoff — the same refuse-the-proxy move the I/O-scheduler page makes with true device occupancy over a queue-length stand-in, the Go-concurrency fuzzer with channel state over branch coverage, the GPU value-profiler with values over hardware counters, and the datacenter-power page with a measured throughput proxy over assumed gains. It is, in effect, a manifesto for that discipline across a whole field. Its case study embodies you-cannot-improve-what-you-do-not-measure: a real regression stayed invisible until the right instrument surfaced it, the same new-observability-lens move the value-profiler makes, and the same what-you-fail-to-test-you-fail-to-defend lesson the Rowhammer-defense page draws when it insists on worst-case analysis. Its LBO method meets a diffuse, hard-to-attribute cost with a principled conservative bound rather than a false claim of exact attribution, the same honest-bound discipline the memory-defense and consensus pages use. Its insistence on sweeping the time-space tradeoff and reporting full distributions rather than single numbers echoes the tradeoff-ladder and tail-latency discipline the FPGA-compilation, real-time-scheduling, and attention-overlap pages rely on. And its deepest move is to treat the benchmark suite and the methodology as the contribution — the measure-in-the-real-conditions, infrastructure-and-experience spirit of the contention and cloud-deployment pages. Together they make this a case study in keeping a field’s measurement honest, so its ideas are judged by what is true.

If you remember only three things

1. Systems research advances by performance analysis, which rests on representative benchmarks and principled methodologies — both hard to build and easy to let lag behind a fast-moving field. When they lag, unsound measurement misdirects the field. The paper's case study makes this vivid: measuring five production Java garbage collectors reveals GC consumes 15% of CPU even at best and up to 17x in tight memory, and — the punchline — newer collectors built to cut latency have HIGHER total overhead than 20-year-old ones, a regression that went largely unnoticed because the community measured proxies (GC pause times, wall-clock, one heap size) instead of the real quantities.
2. DaCapo Chopin is the response: a fourteen-year, roughly 16-million-line overhaul of the DaCapo Java suite with 22 representative workloads (8 new, 9 latency-sensitive real frameworks), 47 characterizing statistics, and PCA-verified diversity. It comes with methodological fixes that measure what matters — direct USER-experienced latency (new "simple" and "metered" metrics, the latter modeling request queuing) instead of GC-pause proxies; BOTH wall-clock and total CPU (task clock); a sweep of heap sizes to expose the time-space tradeoff; and a Lower-Bound-Overhead method that honestly bounds garbage collection's diffuse, hard-to-attribute cost.
3. The Java example is concrete, but the lessons generalize to any fast-moving field. The transferable habit is: measure the real quantity, not a convenient proxy, because you cannot improve what you do not measure and a flawed metric can hide a field-wide regression; build the observability lens that surfaces the hidden problem; expose fundamental tradeoffs by sweeping the range and report full distributions rather than single numbers; bound a diffuse cost honestly rather than overclaim exact attribution; and treat representative benchmarks and sound methodology as first-class contributions that keep a field's ideas judged by what is true.

Check yourself

Check your understanding

1. The paper finds that newer Java garbage collectors, designed to cut latency, actually have HIGHER total overhead than 20-year-old ones — yet this went unnoticed for years. Predict the root cause.
2. DaCapo Chopin reports both wall-clock time AND total CPU (task clock), rather than wall-clock alone. Predict why the distinction matters, especially for concurrent garbage collectors.
3. The paper argues GC pause time should not be used as a proxy for user-experienced latency — a point made two decades ago yet still ignored. Predict why pause time is a poor proxy, and what DaCapo Chopin measures instead.
4. The main deliverable of this paper is a benchmark suite and a set of methodologies, not a new algorithm or a faster system. Predict why that kind of contribution matters for a research field.

Sources

Rethinking Java Performance Analysis (author PDF with appendix) — ASPLOS 2025 (full text read).
ACM DL: Rethinking Java Performance Analysis (DOI 10.1145/3669940.3707217) — published version.
DaCapo Chopin benchmark suite — the open-source suite and statistics.