Virgo: Cluster-level Matrix Unit Integration in GPUs for Scalability and Energy Efficiency

Hansung Kim, Ruohan Richard Yan, Joshua You, Tieliang Vamber Yang and Yakun Sophia Shao

Modern GPUs place a matrix unit (NVIDIA calls it a Tensor Core) inside each SIMT core to speed up the matrix multiplications at the heart of deep learning, but wiring it into the core means every operand and result must flow through the core's register file, whose limited capacity and bandwidth cap the operation size and force many small tiles — wasting energy on instruction processing and lost data reuse; this paper (UC Berkeley — Hansung Kim, Ruohan Richard Yan, Joshua You, Tieliang Vamber Yang, Yakun Sophia Shao) presents Virgo, a GPU design that instead places one large, unified matrix unit beside the cores at the cluster level, physically separated from them, so it reads operands straight from shared memory and keeps its own accumulators — cutting on-chip active power by 67.3 percent versus an Ampere-style and 24.2 percent versus a Hopper-style core-coupled design in synthesizable RTL.

The argument in one glance

Problem

GPU matrix units are trapped inside the core that feeds them

Deep learning runs on general matrix multiply, or GEMM — multiplying large grids of numbers. To speed this up, modern GPUs add a specialized matrix unit, such as an NVIDIA Tensor Core, that does many multiply-add operations at once. But these matrix units are built inside the GPU's basic execution core, the SIMT core (Single Instruction, Multiple Threads — a core that runs many threads together in lockstep). Being inside the core, the matrix unit must receive its input numbers and hand back its results through the core's register file, the small, fast on-chip storage that holds the values threads are working on. The register file has limited capacity and limited bandwidth, and that limit caps how big a single matrix operation the unit can perform. As matrix units grow more powerful, this coupling becomes the wall they cannot climb over.
What people did

Keep the matrix unit in the core, and make it bigger or feed it from shared memory

The path GPUs have taken is to keep the matrix unit coupled to the SIMT core and scale it up within that design. Older designs (Ampere-style) route both operands and results through the register file. Newer designs (Hopper-style) let the matrix unit read some operand data directly from shared memory, a software-managed on-chip scratchpad, to relieve pressure, but the unit still lives inside the SIMT core.
Why it fell short

The register file caps operation size, so tiles stay small and energy is wasted

Because the coupled matrix unit leans on the register file, the operations it can do are small — tiles as small as 16 by 8 numbers. Small tiles hurt in two ways. First, they limit data reuse: the same input values must be reloaded again and again instead of being used many times once fetched, so a lot of energy goes into moving data. Second, doing a big matrix multiply as many tiny operations means the core must fetch and process a huge number of instructions, and instruction processing itself burns substantial energy. Even the Hopper-style fix only sources operands from shared memory while keeping the unit in the core, so the register file and the core pipeline remain in the loop. The result is that it is hard to raise both compute throughput and energy efficiency at once as matrix units keep scaling.
This paper's idea

Move the matrix unit out of the core, up to the cluster, as one large shared unit

Virgo is a GPU microarchitecture that integrates a dedicated matrix unit at the level of the SIMT core cluster — the group of cores that already share a shared memory — rather than inside each core. It is the first design to place the matrix unit at the cluster level and physically separate it from the cores. This physical disaggregation changes everything downstream. Because the matrix unit no longer lives inside a core, it is no longer limited by that core's register file capacity or bandwidth, so the scalability wall disappears and it can do much larger operations. Larger operations mean the core issues far fewer instructions to drive the same work, cutting instruction-processing energy. The disaggregated unit is a single, large, unified design shared by the whole cluster, and it reads its operands directly from shared memory and keeps its accumulator (the running sum of results) in its own storage — completely bypassing the register file for both, which improves data reuse and saves energy. A dedicated data-movement engine (a DMA, or Direct Memory Access, engine) feeds it efficiently. And because the unit is separate, the SIMT cores can keep working at the same time, which suits fused deep-learning workloads. Built and measured in synthesizable RTL (a real hardware description that can be synthesized to a chip), Virgo cuts on-chip active power by 67.3 percent against an Ampere-style baseline and 24.2 percent against a Hopper-style baseline.

Why this problem is worth solving

Deep learning has made matrix multiplication the single most important operation in the data center, and GPUs win or lose on how efficiently they run it.

Training and running large neural networks, including large language models, is dominated by general matrix multiply, written GEMM — multiplying big grids of numbers together, over and over. GPUs answered this by adding specialized matrix units, such as NVIDIA Tensor Cores and AMD Matrix Cores, that perform a whole tile of multiply-add operations in one step, far more efficiently than the GPU’s general-purpose arithmetic units. As deep-learning demand exploded, vendors kept scaling these units up: each GPU generation packs more matrix throughput than the last. So the efficiency of the matrix unit — how much useful computation it delivers per unit of energy and chip area — directly sets the cost of training and serving models. But raw throughput is not the whole story. Data centers are power-limited: a chip can only draw so many watts and shed so much heat. That makes energy efficiency, not just peak speed, the real currency. A matrix-unit design that computes the same results while drawing much less power lets a data center do more work within its power budget, which is why exactly how the matrix unit is wired into the GPU has become a first-order design question.

The trouble is that the way matrix units are attached to the GPU quietly limits how far they can scale.

In today’s GPUs the matrix unit sits inside the SIMT core — the core that runs many threads in lockstep — and it is fed through that core’s register file, the small, very fast on-chip memory holding the values threads compute on. The register file has a fixed capacity and a fixed bandwidth, and a matrix operation can only be as large as what that register file can supply and absorb. That forces the unit to work in small tiles, sometimes as small as 16 by 8 numbers. Small tiles cause two distinct kinds of waste. The first is poor data reuse: in a matrix multiply, each input value should ideally be fetched once and then used in many multiply-add operations, but small tiles mean values are reloaded repeatedly, and moving data costs far more energy than the arithmetic itself. The second is instruction overhead: performing a large multiply as a long sequence of tiny tile operations means the core must fetch, decode, and issue a huge number of instructions, and that instruction processing consumes real energy that produces no arithmetic. Newer Hopper-style designs let the unit read operands straight from shared memory to ease the first problem, but the unit still lives in the core, so the register file and the core pipeline are still in the path. As long as the matrix unit is coupled to the core, growing it just pushes harder against these same limits.

What people did before, and where it broke

Core-coupled matrix unit through the register file (Ampere-style). Put the matrix unit inside the SIMT core and route both its operands and its results through the register file. Where it broke. The register file’s capacity and bandwidth cap the operation size, forcing small tiles that limit data reuse and demand many instructions, so a great deal of energy is spent moving data and processing instructions.
Core-coupled but operands from shared memory (Hopper-style). Keep the matrix unit in the core but let it read some operand data directly from shared memory instead of only the register file. Where it broke. It relieves operand pressure but the unit is still inside the SIMT core, so results still involve the core and the register file, and the core-microarchitecture scalability limit remains.
Just make the coupled unit bigger each generation. Scale up the in-core matrix unit with every GPU generation. Where it broke. A bigger unit needs proportionally more register-file capacity and bandwidth, which do not scale as cheaply, so throughput and energy efficiency cannot both improve at once.
The gap. No GPU design had removed the matrix unit from the core entirely, so none had escaped the register-file bottleneck while also cutting the instruction-processing and data-movement energy that the coupling causes.

How Virgo works

Lifting the matrix unit out of the core and up to the cluster

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Maximize data reuse by choosing where the matrix data lives. The deepest cost in matrix multiplication is not the arithmetic but moving the numbers, so the design that reuses each fetched value the most wins on energy. Virgo’s large, cluster-level unit reads operands straight from shared memory and keeps accumulators in its own storage, so values are fetched once and reused across a big operation instead of being reloaded through the register file for each small tile. Treating data movement, not computation, as the thing to minimize — and restructuring the hardware’s data path so each value is reused as much as possible — is the same principle the work on an optimized dataflow for mitigating attention bottlenecks covered elsewhere on this site is built on, reshaping how operands stream so the expensive traffic shrinks. Virgo applies that principle at the level of where the matrix unit sits and which memory feeds it.

Disaggregate a resource to escape the limit of the thing it was bolted to. The core move in Virgo is to physically separate the matrix unit from the SIMT core, so it stops inheriting the core’s register-file capacity and bandwidth limits. Once decoupled, the unit can be made large and unified and can scale on its own terms. Removing a resource from the component that constrained it, and making it a first-class shared unit that can grow independently, is the same architectural insight the work on a hardware-software co-designed disaggregated memory system covered elsewhere on this site relies on, breaking a resource free of the server it was trapped in so it can scale. Virgo brings that disaggregation idea inside the GPU, applying it to the matrix unit rather than to memory across machines.

Let the specialized unit and the general core run at once, for fused workloads. Real deep-learning layers are rarely a bare matrix multiply; they fuse the multiply with other operations like activation functions or normalization. The matrix multiply should run on the matrix unit while those other operations run on the SIMT cores, at the same time, so neither waits for the other. Because Virgo’s matrix unit is a separate instance, this concurrent execution falls out naturally, and the mapping of a fused workload can be optimized to overlap the two. Pairing a dense matrix engine with a companion unit that handles the surrounding non-matrix operators so a fused network keeps both busy is the same division of labor the work on a companion processor for emerging operators in neural networks covered elsewhere on this site pursues, splitting a layer across specialized and flexible hardware. Virgo achieves the overlap through physical disaggregation within the GPU itself.

Following a matrix multiply through Virgo

Running one fused deep-learning layer on Virgo versus a coupled design

A trace, drawn from the paper, of how Virgo executes a matrix multiply at the cluster level instead of inside the core.

Step 1 of 5

A coupled design would break the multiply into tiny tiles

On an Ampere-style GPU, the layer's matrix multiply runs on the in-core matrix unit, which reads operands and writes results through the register file. Limited register-file capacity forces small tiles (for example 16 by 8), so the multiply becomes a long sequence of tiny operations.

Each tiny tile reloads operands and costs an instruction, wasting data-movement and instruction energy.

Why it works

Disaggregation removes the register-file ceiling. The in-core matrix unit is capped by the register file it depends on → but moving the unit out to the cluster cuts that dependency → so operation size is limited by the shared memory design, not a single core’s registers, and can grow.
Coarser operations mean fewer instructions. A large operation replaces a long run of tiny tile operations → so the core fetches, decodes, and issues far fewer instructions → so the energy spent on instruction processing, which does no arithmetic, drops sharply.
Sourcing from shared memory improves data reuse. Operands are read once into shared memory and used across a big operation → so the same values are not reloaded through the register file per tile → so data-movement energy, the dominant cost, falls.
Keeping accumulators in the unit bypasses the register file both ways. The running sums stay in the matrix unit’s own storage → so results never travel through the register file either → so the register file is off the critical path entirely and consumes less power.
Physical separation enables concurrency. The matrix unit is a distinct instance from the cores → so the cores can run non-matrix operations while it computes → so a fused layer overlaps its matrix and non-matrix work instead of serializing them.

What it costs (trade-offs and scope)

It is a from-scratch microarchitecture change, not a drop-in. Virgo restructures where the matrix unit lives and adds a cluster-level unit, its own accumulator storage, and a DMA engine; this is a significant hardware redesign, not a tweak to an existing GPU, so adopting it means building a different chip.
Shared memory and its bandwidth become the new pressure point. Because operands now stream from shared memory, the design leans heavily on shared-memory capacity and bandwidth and on the DMA engine feeding the unit; the bottleneck is moved rather than made to vanish, so shared memory must be provisioned to keep the large unit fed.
A single shared unit must be scheduled across the cluster. One large matrix unit serves all the cores in a cluster, so its benefit depends on keeping it well utilized and mapping fused workloads so the cores and the unit actually overlap; workloads that cannot be mapped that way will see less of the gain.
The headline result is active power, from RTL, on specific baselines. The 67.3 percent and 24.2 percent figures are reductions in on-chip active power measured from synthesizable RTL against Ampere-style and Hopper-style baselines the authors built; they are a careful hardware evaluation, but they are power reductions on those baselines and workloads, not a universal end-to-end speedup on commercial silicon.
It targets dense GEMM-style deep learning. Virgo is built around accelerating dense matrix multiplication for deep-learning workloads; applications with very different compute patterns, or sparse and irregular ones, are outside what this design is tuned for.

The evaluation

Where the matrix unit sits, and what it costs in power

Property	Ampere-style (in-core, register file)	Hopper-style (in-core, shared-memory operands)	Virgo (cluster-level, disaggregated)
Where the matrix unit lives
Operand and result path
Operation granularity
Concurrent core plus matrix execution
On-chip active power (relative)

A comparison of matrix-unit integration, evaluated in synthesizable RTL. Virgo disaggregates the matrix unit to the cluster level, bypassing the register file for both operands and accumulators, and reduces on-chip active power by 67.3 percent against an Ampere-style and 24.2 percent against a Hopper-style core-coupled baseline.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (arXiv:2408.12073). The mechanism described is the paper’s: Virgo is a GPU microarchitecture that integrates a dedicated matrix unit at the SIMT core cluster level, physically disaggregating it from the SIMT core. This removes the scalability constraints imposed by the core microarchitecture, increases operation granularity to reduce energy from core instruction processing, enables a unified matrix unit design, and offloads both operand and accumulator accesses from the register file (reading operands from shared memory and keeping accumulators locally) to improve data reuse and energy efficiency. The disaggregation also supports efficient concurrent execution of the SIMT core and the matrix unit, optimizing the mapping for fused deep-neural-network workloads, and uses a dedicated DMA engine for matrix data movement. The matrix unit is built using a Gemmini-based systolic-array generator. Evaluations use synthesizable RTL and report 67.3 percent and 24.2 percent reductions in on-chip active power consumption versus Ampere-style and Hopper-style core-coupled baselines respectively. Exact microarchitectural parameters, the workloads, and the full set of measured results should be read from the paper.

Background

Why specialized hardware keeps pulling functions out of the general core

It helps to see Virgo as one move in a long trend: as deep learning made a handful of operations dominate computing, hardware designers have steadily pulled those operations out of general-purpose execution and into dedicated, specialized units, and then kept refining exactly how those units attach to the rest of the chip. The first step was adding matrix units to GPUs at all. The next question, which Virgo answers, is where they should live and what should feed them, because the attachment point turns out to decide how large, how efficient, and how concurrent they can be. Underneath all of this is a power wall: data centers are limited by watts and heat, so every generation the real prize is doing more computation per joule, which is why a design measured on active-power reduction matters as much as one measured on raw speed. The broader effort to co-design the whole stack — the accelerator hardware together with the workloads it runs — so the specialized unit is shaped to what deep learning actually needs is the same undertaking the work on a full-stack search technique for domain-optimized deep learning accelerators covered elsewhere on this site takes on, searching the design space for the most efficient accelerator. Virgo contributes a specific, powerful answer for GPUs: disaggregate the matrix unit to the cluster so it can grow and sip power.

If you remember only three things

1. Deep learning is dominated by matrix multiplication (GEMM), so GPUs add specialized matrix units like NVIDIA Tensor Cores. But those units are built inside the SIMT core and fed through the core's register file, whose limited capacity and bandwidth cap the operation size. That forces small tiles, which limit data reuse and require processing many instructions, so a lot of energy is wasted moving data and running the core pipeline — and throughput and energy efficiency cannot both improve as the units scale.
2. Virgo is the first GPU design to place the matrix unit at the SIMT core cluster level, physically disaggregated from the cores. Freed from any core's register file, it becomes one large, unified unit that reads operands directly from shared memory, keeps its accumulators in its own storage, and is fed by a dedicated DMA engine. This lifts the size limit, cuts instruction-processing energy through coarser operations, and improves data reuse by bypassing the register file for both operands and results.
3. Because the matrix unit is a separate instance, the SIMT cores can run non-matrix work concurrently, which suits fused deep-learning layers. Built and measured in synthesizable RTL, Virgo reduces on-chip active power by 67.3 percent versus an Ampere-style and 24.2 percent versus a Hopper-style core-coupled baseline. The trade-offs: it is a substantial microarchitecture redesign, it shifts pressure onto shared-memory bandwidth and the DMA engine, and the one shared unit must be kept well utilized to pay off.

Check yourself

Check your understanding

1. Why does coupling the matrix unit to the SIMT core limit deep-learning GPUs?
2. What is Virgo's central design change?
3. Why does disaggregating the matrix unit save energy?
4. Predict: for a fused deep-learning layer (a matrix multiply followed by an activation function), what does Virgo's separate matrix unit enable that a coupled design does not?

Sources

The paper is “Virgo: Cluster-level Matrix Unit Integration in GPUs for Scalability and Energy Efficiency” by Hansung Kim, Ruohan Richard Yan, Joshua You, Tieliang Vamber Yang, and Yakun Sophia Shao (University of California, Berkeley), published at ASPLOS 2025, DOI 10.1145/3676641.3716281. It is licensed CC BY.

This explainer is written from the paper’s full text (arXiv:2408.12073). Modern GPUs incorporate specialized matrix units such as Tensor Cores to accelerate GEMM operations central to deep learning, but existing matrix-unit designs are tightly coupled to the SIMT core, restricting operation size due to register-file capacity and bandwidth constraints; this limits scalability and makes it hard to improve compute throughput and energy efficiency simultaneously. Virgo is a GPU microarchitecture that integrates dedicated matrix units at the SIMT core cluster level. By decoupling the matrix unit from the SIMT core, Virgo eliminates the scalability constraints imposed by the core microarchitecture, increases operation granularity at the hardware level to reduce energy overhead from core instruction processing, enables a unified matrix unit design, and offloads both operand and accumulator accesses from the register file (sourcing operands from shared memory and keeping accumulators locally) to improve data reuse and energy efficiency. This disaggregation also supports efficient concurrent execution of the SIMT core and matrix unit, optimizing the mapping for fused deep-neural-network workloads, and it uses a dedicated DMA engine for matrix data movement; the matrix unit is generated using a Gemmini-based systolic-array generator. Evaluations using synthesizable RTL demonstrate that Virgo achieves 67.3 percent and 24.2 percent reductions in on-chip active power consumption compared to the baseline Ampere-style and Hopper-style core-coupled designs. No numbers beyond the paper’s own have been introduced; exact microarchitectural parameters and full results should be confirmed in the paper.