A Journey of a 1, 000 Kernels Begins with a Single Step: A Retrospective of Deep Learning on GPUs

Michael Davies, Ian McDougall, Selvaraj Anandaraj, Deep Machchhar, Rithik Jain and Karthikeyan Sankaralingam

For a decade the story of deep-learning hardware was simple — accelerate the big matrix multiply, and the GPU flies — but this measurement study from Wisconsin profiles ten real AI applications, from physics simulators to GPT-3, across three GPU generations and finds the story has changed — matrix multiplies are now often a minority of runtime, generational gains have stalled, the chip sits mostly idle, and thousands of small, irregular, memory-bound kernels dominate; the authors argue the era of just accelerating matrix multiply is over, and the next gains must come from orchestrating compute, data movement, and dependencies together, worth a projected 2.3 times.

The argument in one glance

Problem

For years, deep-learning hardware advanced by accelerating one thing

The rise of deep learning was powered by a beautifully simple hardware story. Neural networks spend enormous effort on dense matrix multiplies, so chip makers built dedicated hardware — most famously NVIDIA's TensorCore — to do exactly that operation fast, and each GPU generation delivered big, transparent speedups. The community's standard benchmark, MLPerf, is dominated by the convolution-heavy vision models that reward this focus. It looked like the path forward was clear: keep making the matrix multiply faster.
What people did

Judge the hardware with GEMM microbenchmarks and MLPerf

To understand how well a GPU handles AI, researchers lean on two things: benchmark the matrix-multiply kernels in isolation, and run MLPerf, the standard suite. Academic studies typically profile a few convolution or matrix-multiply layers. If the matrix multiply is the heart of deep learning, then measuring it, and the GEMM-heavy models MLPerf collects, should tell you how the hardware is doing.
Why it fell short

That view is a sliver of real, diverse AI

But MLPerf is overwhelmingly convolution-heavy, capturing only a narrow slice of what AI actually does, and microbenchmarks of one kernel say nothing about how a whole application behaves. Real deep learning now spans physics simulators, reinforcement learning, view synthesis, tabular models, speech, and giant language models, each a sprawling mix of thousands of different kernels. Nobody had measured a broad set of these end to end, across GPU generations, to see how the hardware-software machine is really performing — or where it is running out of room.
This paper's idea

Profile ten real applications across three GPU generations, top to bottom

The paper runs ten influential, diverse AI applications — from a physics simulator to GPT-3 — on three generations of NVIDIA GPUs, and instruments every layer at once: the framework operators, the device kernels, and the hardware performance counters. From this longitudinal data it draws a set of insights about how the AI hardware revolution actually played out, and reaches a striking conclusion — the matrix multiply is no longer the whole story, generational gains are stalling, the chip is mostly idle, and the future must come from orchestrating a broad, messy tail of operations instead.

Why this problem is worth solving

The operation this whole retrospective revolves around is the one that made GPUs the engine of the AI boom.

GEMM, for general matrix multiply, is the workhorse of deep learning: the dense multiplication of two matrices that implements the fully-connected and convolutional layers where neural networks do most of their arithmetic. It is attractive to accelerate because it is regular and compute-heavy — it does a huge number of multiply-adds per byte of data it reads, a property called high arithmetic intensity, so a chip can keep its math units busy without waiting on memory. NVIDIA’s TensorCore is a hardware block built to do exactly this operation at enormous throughput, and its arrival in the V100 generation gave a large, automatic speedup to any workload dominated by matrix multiplies. For years this made accelerating GEMM the obvious lever for faster AI hardware — a single, well-understood target that repaid every bit of silicon spent on it. This paper’s central question is whether that lever still has room to pull.

What the study measures is the gap between the clean operators a programmer writes and the messy reality the hardware runs.

When a programmer writes a line of a deep-learning application, they call a high-level operator — a matrix multiply, a convolution, an addition, a reshape. Underneath, the framework and its libraries lower each operator into one or more device kernels: the actual GPU programs that run on the hardware. The gap between the two is vast. This study finds that 185 distinct operators across its applications expand into 744 distinct kernels on one GPU, because each operator often needs several specialized versions — one tuned for this tensor shape, another for that one — to run fast. And a single run may execute hundreds of thousands of these kernels. The full software stack that manages this lowering runs to hundreds of thousands of lines of code and cost NVIDIA billions of dollars to build. Understanding a real application’s performance therefore means looking not at one kernel but at the whole sprawling population of them.

What people did before, and where it broke

Benchmark the matrix multiply in isolation. Measure the throughput of GEMM kernels to gauge how well a GPU accelerates deep learning. Where it broke. A single kernel’s speed says nothing about a whole application, where matrix multiplies turn out to be a minority of runtime — from 2 to 58 percent across the studied apps.
Run MLPerf, the standard benchmark suite. Use the community’s accepted set of models to compare hardware. Where it broke. MLPerf is overwhelmingly convolution- and GEMM-heavy, with high utilization and low behaviour variation, so it flatters GEMM-centric hardware — the studied apps see only a 1.1-times A100 gain where MLPerf sees about 2 times.
Profile a few convolution or matrix-multiply layers. The common academic shortcut of studying representative layers. Where it broke. It misses the heavy tail of hundreds of other operators and the rapid, diverse execution dynamics of a real application.
Prior GPU retrospectives. Longitudinal studies of GPUs, but on non-deep-learning workloads, or a TPU-inference retrospective that lacked application and microarchitecture detail. Where it broke. None characterized deep learning’s operator, kernel, and utilization trends across GPU generations at every layer of the stack.
The gap. The whole field judged AI hardware by the one operation it had learned to accelerate and by a benchmark suite built from that same kind of model, while real deep learning had exploded into a diverse, sprawling mess of thousands of kernels no one had measured end to end. What was missing was a broad, multi-level, cross-generational study of real applications — enough to reveal not just how fast the matrix multiply runs, but where the time actually goes and where the next decade of speedups must come from.

How the study works

Measuring the AI hardware revolution, layer by layer

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Measure whole real applications, not the sliver everyone already optimizes. The study’s foundational choice is to refuse the convenient view. Instead of benchmarking the matrix multiply on its own, or trusting MLPerf’s convolution-heavy models, it runs ten genuinely diverse applications end to end and profiles every layer of the stack, from the operators a programmer writes down to the hardware counters. That honesty is what overturns the received wisdom: seen whole, the matrix multiply is often a minority of runtime, and the community had been optimizing the part it could see rather than the part that dominates. Insisting that the right way to understand a system’s real cost is to measure the actual whole workload rather than a convenient, unrepresentative proxy for it is the same discipline the virtual-memory-simulation work covered elsewhere on this site is built on, which likewise argues that a fixed stand-in for real behaviour quietly misleads and that fidelity comes from measuring the real thing.

Recognize that the bottleneck has moved from compute to data movement and dependence. Having looked at the whole picture, the study finds the chip mostly idle even on regular workloads, and diagnoses why: alongside the matrix multiplies sit a huge population of low-compute operations — gathers, scatters, reshapes, elementwise math — that do almost no arithmetic but flood the memory system, and small or irregular matrix multiplies that cannot fill the compute units. The scarce resource is no longer raw math throughput but the movement of data and the ability to overlap opposite kinds of work. Recognizing that the true limiter of a modern workload is the memory-bound, irregular part that a compute-focused design leaves starving, rather than the dense arithmetic everyone accelerates, is the same insight the GPU graph-learning work covered elsewhere on this site acts on, redirecting effort to the memory and data-movement bottleneck that a matrix engine alone never addresses.

Point the way forward at overlapping behaviours the chip cannot currently exploit. The study’s prescription is that future gains lie in orchestration — and especially in dependence orchestration, the idea that a compute-heavy kernel leaves the memory system idle while a memory-heavy kernel leaves the compute units idle, so if the algorithmic dependencies between them could be broken, they could run in each other’s shadow and fill the chip. This is parallelism the current stack, which fuses only within a single unit, simply leaves on the table. Finding a whole axis of overlap that existing systems fail to exploit, and arguing that co-designing algorithm and architecture to capture it unlocks large gains, is the same move the transformer-training work covered elsewhere on this site makes, uncovering a dimension of parallelism the field had left unused and overlapping work that used to run serially so the expensive resources stop sitting idle.

Following the argument that the GEMM era is over

Why accelerating the matrix multiply stopped paying off

The study builds its central claim from measured numbers, step by step.

Step 1 of 5

On the P100, accelerating GEMM was a clear win

The study estimates that speeding up the matrix multiplies by ten times on the P100 — roughly what the V100's TensorCore delivered — would raise whole-application performance by 2.6 times. The actual measured V100-over-P100 gain across the applications was 2.4 times, confirming that the TensorCore was the engine of that jump.

This is why GEMM acceleration looked like a silver bullet: a single, easy target that lifted everything.

Why it works

Profiling whole applications, not isolated kernels, overturns the GEMM assumption. Everyone benchmarks the matrix multiply and MLPerf’s GEMM-heavy models → but running ten diverse real applications end to end shows the matrix multiply is often a minority of runtime, from 2 to 58 percent → so the operation the whole field optimizes turns out not to be where most of the time goes.
Tracing operators down to their kernels exposes a long, specialized tail. A handful of famous operators seem to define deep learning → but 185 operators expand into 744 device kernels, and covering ninety percent of runtime needs hundreds of matrix and non-matrix kernels, each shape-specialized → so the burden on the compiler and hardware is a vast population of code, not a few hot loops.
Watching utilization across generations shows the GEMM lever running dry. Peak matrix-multiply throughput kept rising generation to generation → but measured utilization fell, and doubling matrix-multiply speed on the A100 buys only thirty percent end to end → so the silver bullet that carried a decade of speedups no longer has room to pull.
A taxonomy of execution states shows an idle, fast-flipping chip. One might expect regular workloads to saturate the hardware → but six of ten applications spend most of their time below a quarter of any resource, switching execution state every one or two kernels → so the real losses are the chip sitting idle and thrashing between behaviours, not slow arithmetic.
Naming the new bottlenecks lets a model price the opportunity. It was unclear where post-GEMM gains could come from → but classifying the idle time into compute, data, and dependence orchestration and modelling a doubling of each shows data movement is the biggest lever → so orchestrating all three together projects a 2.3-times speedup over today’s hardware.

What it costs (trade-offs and scope)

It is a characterization study, not a design. The deliverables are insights, an open application suite, and a projection; no new accelerator is built, and the path-forward orchestrations are named and motivated rather than designed in detail.
The ten applications are a curated sample. They were chosen by citations, diversity, and code availability, so the conclusions reflect that selection; a different set could shift the picture, and the study does not claim to be exhaustive.
The most important model is the exception. GPT-3, the flagship large language model, behaves like the GEMM-regular MLPerf workloads — few kernels, high matrix-multiply utilization — so the headline that the GEMM era is over holds for the diverse tail, not for the compute-regular models that dominate today’s largest deployments.
It covers NVIDIA GPUs at a moment in time. Three generations in half-precision, mostly single-device; other accelerators like TPUs and AMD GPUs, later hardware such as Hopper, and large-scale multi-GPU training are out of scope or only discussed.
The 2.3-times figure is a projection, not a measurement. It comes from a hypothetical chip that idealizes doubling compute, data, and dependence orchestration, with a conservative fixed-window model of overlap; real hardware need not achieve it, and doubling all three is an assumption.
The claim that GEMM acceleration is over is strong and time-stamped. It rests on this sample at the A100 generation; MLPerf and GPT-3 still benefit from matrix-multiply hardware, and later chips added more of it, so the claim is directional rather than absolute.
Utilization depends on how it is defined. The headline numbers use particular definitions — for instance, compute-unit utilization as a maximum over several pipeline metrics — and other reasonable definitions would paint somewhat different pictures.

The evaluation

Ways to evaluate deep-learning hardware

Property	GEMM microbenchmark	Prior GPU retrospectives
What it measures
Application diversity	None	Limited
A100-over-V100 gain observed	Not applicable	Less (non-DL)
Where the bottleneck is	Not applicable	Not applicable
What it concludes	The matrix multiply is fast	A100 gives less

Approaches to evaluating deep-learning hardware, with this study profiling ten diverse applications plus MLPerf across the P100, V100, and A100 at the framework, kernel, and counter levels. It finds a 2.4-times geomean gain from the V100 but only 1.1 times from the A100 on the diverse apps, 185 operators expanding into 744 kernels, six of ten apps under 25 percent utilization for 60 percent of runtime, only a 30 percent end-to-end gain from doubling matrix-multiply speed on the A100, and a projected 2.3-times speedup from orchestrating compute, data, and dependence; the CaSiO suite is open source.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This is a rigorous and useful retrospective: by profiling ten genuinely diverse AI applications across three GPU generations at every layer of the stack, it shows that the matrix multiply is often a minority of runtime, that generational gains have stalled at the A100 for real apps, that the chip sits mostly idle even on regular workloads, and that the next gains must come from orchestrating compute, data movement, and dependencies — a shift it prices at a projected 2.3 times. Read it as a measurement study, with its scope. The deliverable is insight, open data, and a projection, not a built accelerator; the applications are a curated sample; and the flagship model, GPT-3, is the GEMM-regular exception to the headline, as are MLPerf’s models. It covers NVIDIA GPUs of one era in half-precision, mostly single-device, and its utilization figures depend on chosen definitions. The 2.3-times number is a model, not a measurement, and the striking “GEMM era is over” is a directional, time-stamped claim. The durable value is the reframing: stop measuring the sliver you can accelerate, look at the whole workload, and the bottleneck has moved.

Background

Why the free lunch from one fast operation had to end

It is worth stepping back to see why a study like this matters beyond its numbers. For most of the deep-learning boom, hardware progress rode on a single, fortunate fact: the workload was dominated by one regular, compute-heavy operation, the matrix multiply, which is unusually easy to build fast silicon for and unusually easy for a compiler to target. That let a chip maker pour transistors into one well-understood block and hand the whole field a transparent speedup every generation. But such a free lunch is inherently temporary — once the one operation is fast, its share of the runtime shrinks, and everything else it left unoptimized rises to become the new bottleneck, which is exactly the transition this study documents. The lesson is that the next round of gains cannot come from a single silver bullet; it has to come from co-designing the algorithm and the architecture together to attack the messy, varied, memory-bound remainder. The same conviction that future performance is won by full-stack co-design rather than one heroic component drives the resource-aware SLAM work covered elsewhere on this site, which reshapes an algorithm and its accelerator together to squeeze real-time behaviour from a constrained device. The broader pattern is that whenever a workload is carried by one dominant, easily-accelerated primitive, the interesting engineering begins the moment that primitive is no longer the bottleneck.

If you remember only three things

1. For a decade, deep-learning hardware advanced by accelerating the dense matrix multiply with dedicated blocks like NVIDIA's TensorCore, and the field judged GPUs by that operation and by MLPerf, a benchmark dominated by the convolution-heavy models that reward it; but this longitudinal study profiles ten genuinely diverse real applications — physics simulation, reinforcement learning, view synthesis, vision, tabular, speech, and GPT-3 — across the P100, V100, and A100 at the framework, kernel, and hardware-counter levels, a whole-workload view no prior work had taken.
2. The measurements overturn the received wisdom: the matrix multiply is often a minority of runtime (2 to 58 percent), 185 operators expand into 744 device kernels with a heavy tail needing hundreds to cover 90 percent of runtime, generational gains have stalled (2.4 times from the V100 but only 1.1 times from the A100 on the diverse apps), doubling matrix-multiply speed on the A100 would buy only 30 percent end to end, and six of ten apps spend 60 percent of their time under 25 percent utilization of any resource while flipping execution state every one or two kernels — so the era of just accelerating the matrix multiply is over.
3. The study argues future gains must come from three orchestration behaviours — compute (small and irregular GEMMs that cannot fill the chip), data (the flood of low-compute gather, scatter, and elementwise operations that saturate memory), and dependence (overlapping compute-heavy and memory-heavy kernels that serial dependencies keep from running together) — and an empirical model projects a 2.3-times speedup over an A100 from doubling all three, with data orchestration the biggest single lever; but it is a characterization study, not a design, its apps are a curated sample, GPT-3 and MLPerf are GEMM-regular exceptions, and the 2.3-times figure is a projection.

Check yourself

Check your understanding

1. What is the study's central, surprising finding about the matrix multiply?
2. Why does the study conclude that the GPU is mostly idle, and what causes it?
3. What does the study propose as the path forward, and how much does it project?
4. Which statement best reflects the limits of this study?

Sources

The paper is “A Journey of a 1,000 Kernels Begins with a Single Step: A Retrospective of Deep Learning on GPUs” by Michael Davies, Ian McDougall, Selvaraj Anandaraj, Deep Machchhar, Rithik Jain, and Karthikeyan Sankaralingam, published at ASPLOS 2024, pages 19 to 35, DOI 10.1145/3620665.3640367. The authors are at the University of Wisconsin-Madison, and the CaSiO application suite is openly available.

This explainer is written from the full 17-page text. Everything above comes from the paper’s own sections: the introduction framing the three-way explosion of AI domains, hardware, and software and the three research questions, with the critique of MLPerf as convolution-dominated and unrepresentative; the background on the deep-learning GPU stack, the CaSiO suite of ten applications (MeshGraphNets, MuZero, NeRF, PINN, QDTrack, Swin Transformer, TabNet, Tacotron2, WaveNet, and GPT-3) across the P100, V100, and A100, and the profiling methodology; the software-perspective analysis yielding the heavy-tail and shape-specialization insights (185 operators expanding to 744 kernels, 308 GEMM plus 137 non-GEMM kernels for 90 percent runtime coverage, and the 30-percent end-to-end gain from a 2-times A100 GEMM speedup) and the “era of GEMM acceleration is over” claim; the GEMM deep dive across 317 shapes showing generational geomean speedups of 2 and 1.6 times against 6 and 2.5 times peak, falling utilization, and arithmetic intensity only partly explaining performance; the application-informed hardware analysis with the state taxonomy, the under-utilization finding (six of ten apps below 25 percent for 60 percent of runtime), the rapid state variation, the 2.4-times V100 and 1.1-times A100 geomean speedups, and the GPT-3 and MLPerf deep dives; and the path-forward section defining compute, data, and dependence orchestration, the MeshGraphNets case study, and the empirical model projecting 24, 49, and 13 percent individual and a combined 2.3-times speedup over an A100.