Every generation of graphics processor adds more compute units and more memory, and all of them are wired together by an on-chip network that must give every unit equal access to every piece of cache — an arrangement that is fair, simple to reason about, and increasingly ruinous, because the cost of such a network grows with the SQUARE of the number of things connected to it, so designers are forced to build a network slower than the cache it serves and then watch the cache sit half idle; this paper (Ghent, NTNU and Chinese defence research) proposes giving that up, carving the chip into partitions that each hold a few compute units, a slice of cache and a memory controller wired together by direct links, with the network used only for the rarer case of reaching another partition; the catch is that unequal access only pays if the data a unit needs is usually in its own partition, so the design only works alongside two co-designed pieces — an operating-system page placement rule that keeps data local until the partitions become unbalanced, and a compiler analysis that finds read-only shared data which hardware then replicates when a model says the bandwidth gain outweighs the wasted cache; together they give 23 percent more performance, or the same performance with 12 times less network power.
The argument in one glance
-
Problem
The network that connects everything to everything costs too much
A graphics processor gets faster by adding more compute units, more cache and more memory bandwidth every generation — one vendor went from 16 compute units to 108, and from 177 gigabytes per second of memory bandwidth to 1.5 terabytes per second, across a handful of designs. All those compute units must reach all the cache, and conventional designs do that with a crossbar network giving every unit EQUAL bandwidth to every cache slice. The problem is arithmetic. The cost of a crossbar grows with the SQUARE of the number of endpoints attached, so as the chip grows the network becomes disproportionately power-hungry and complex. Designers respond the only way they can within a power budget — by giving the network less bandwidth than the cache behind it can actually deliver. The result is a chip whose cache is throttled by the wiring in front of it.
-
What people did
Reorganize the cache so each slice serves its own compute units
Recent designs, including one leading vendor's current flagship, take a different tack. Instead of each cache slice holding one region of memory for the whole chip, each slice caches data from ANY memory region but serves only its nearby compute units. That cuts the traffic crossing the network dramatically, because a unit usually finds what it needs in the slice attached to it, and so it relieves exactly the pressure that made the crossbar expensive.
-
Why it fell short
But now the same data lives in several slices, and they must agree
Once every slice can hold a copy of any address, two slices can hold the same line, and the hardware must keep those copies consistent — the coherence problem. That machinery is genuinely complex, and worse, its cost grows as partitions are added, so the fix that relieved the network becomes the thing limiting how far the design can scale. Meanwhile the original arrangement remains stuck with its quadratic network. The measured consequence is stark: on the conventional design EVERY miss in a compute unit's private cache must cross the network, so all of them are remote, and the second design only outperforms the first by about 1 percent on average. Neither escapes the underlying situation, which is that a chip is being built as though every compute unit should be equally close to every byte.
-
This paper's idea
Give up on equal access, then co-design software to make locality real
This paper (Ghent, NTNU and Chinese defence research — Zhao, Jahre, Tang, Zhang, Eeckhout) proposes NUBA, a NON-UNIFORM BANDWIDTH ARCHITECTURE. Carve the chip into PARTITIONS, each holding a couple of compute units, a couple of cache slices and a memory controller, wired together by direct point-to-point links so the units see the cache's full bandwidth. Use the network only for the rarer case of reaching another partition. This is structurally the same move CPUs made decades ago with non-uniform memory, but for a completely different reason — a CPU accepts non-uniformity to reduce LATENCY, while a graphics processor barely cares about latency, because it hides it by switching among thousands of threads. What it cares about is BANDWIDTH. The catch is that unequal access only pays if data is usually local, which hardware cannot arrange alone, so NUBA co-designs two software pieces: a page placement rule in the driver that keeps data local until the partitions become unbalanced, and a compiler analysis that identifies read-only shared data which hardware then replicates when a model predicts the bandwidth gain beats the wasted cache capacity. Together they deliver 23 percent more performance, or equal performance with 12 times less network power.
Why this problem is worth solving
The bottleneck here is not the compute units and not the memory, but the wiring
between them, so it is worth being precise about how a graphics processor is
put together.
A modern GRAPHICS PROCESSOR
contains dozens of compute units, each with a small private cache, plus a large
shared LAST-LEVEL CACHE that is physically split into many SLICES, plus a set
of MEMORY CONTROLLERS that talk to the memory chips. Connecting all the compute
units to all the cache slices is the job of an on-chip network, usually a
CROSSBAR, which is a switch allowing any endpoint to reach any other. The
crucial property of a crossbar is that its cost — in power, area and design
complexity — grows with the SQUARE of the number of endpoints, so doubling the
chip roughly quadruples the network. That scaling is what makes the network the
limiting factor. Because a chip has a fixed power budget, architects are forced
to give the crossbar less bandwidth than the cache slices behind it could
actually supply, which means the cache is throttled by the wiring in front of
it. In the design measured here the network delivers 1.4 terabytes per second
while direct links to a local cache slice could deliver 2.8, so half the
available cache bandwidth is unreachable
The escape is to stop insisting that every compute unit be equally close to
every byte — an old idea, imported for a new reason.
A
UNIFORM BANDWIDTH ARCHITECTURE gives every compute unit the same bandwidth to
every cache slice, which is what conventional graphics processors do. A
NON-UNIFORM BANDWIDTH ARCHITECTURE deliberately makes some cache nearer than
others — grouping a few compute units, a few cache slices and a memory
controller into a PARTITION connected by direct links, with a network only
between partitions. Structurally this mirrors what CPUs did long ago with
non-uniform cache and memory architectures, but the MOTIVATION is completely
different, and the difference matters. A CPU runs few threads per core, so when
one stalls waiting for memory there is little else to run and the time lost is
LATENCY; CPUs therefore accept non-uniformity to place data physically closer
and reduce waiting. A graphics processor runs thousands of threads and simply
switches to another whenever one stalls, so latency is largely hidden and
performance is governed almost entirely by BANDWIDTH — how much data can flow
per unit time. Non-uniformity in a graphics processor is therefore not about
being closer but about being connected by fatter, cheaper links
What people did before, and where it broke
- Memory-side uniform bandwidth, the industry default. Each cache slice holds one region of the address space and a big crossbar connects everything to everything. Simple and predictable, but the crossbar’s quadratic cost forces its bandwidth below what the cache could supply, and every private cache miss must cross it.
- Compute-side uniform bandwidth, used in a current flagship. Each slice caches any address but serves nearby units, so most traffic stays off the network. It genuinely cuts network cost, but now several slices can hold the same line and must be kept coherent, which adds complexity that grows with the number of partitions and limits scaling. Measured here, it beats the default by only about 1 percent on average.
- Non-uniform designs in the CPU world. Decades of work on placing data near the core that uses it. Directly relevant in structure, but aimed at cutting latency, which is the wrong target for a machine that hides latency by multithreading — so the policies do not transfer unchanged.
- Multi-chip module GPUs and their page policies. Prior work on GPUs built from several dies considered placing pages by first-touch or round-robin. Closely related, but those operate at a much coarser granularity than partitions on a single die, and neither policy works well at this finer grain.
- The gap. Nobody had built a graphics processor that embraces unequal bandwidth for bandwidth’s own sake, and nobody had supplied the software the idea requires — because unequal access is worthless unless data actually ends up local, which is a decision made by the driver and the compiler, not the hardware. That is the gap NUBA fills.
How unequal bandwidth is made to pay off
Partition the chip, then co-design placement and replication to keep accesses local
-
A uniform network throttles the cache behind it
Connecting every compute unit to every cache slice needs a crossbar whose cost grows with the square of the endpoints, so its bandwidth is set below what the cache can deliver. Every private-cache miss must cross it, making all accesses effectively remote and leaving cache bandwidth stranded.
-
Carve the chip into partitions with direct links new
Group a couple of compute units, a couple of cache slices and a memory controller into a partition, connected by point-to-point links that expose the cache slices' full bandwidth — 2.8 terabytes per second against the network's 1.4. The network now carries only cross-partition traffic.
-
Route each request by the address it targets new
A cache slice inspects an incoming request and, if the address belongs to its own partition, queues it for the local tag and data arrays; otherwise it forwards it over the network to the owning partition, bypassing the local slice entirely. Each slice also serves remote requests arriving from the network, alternating fairly between the two queues.
-
Place pages locally, but only while balance holds new
Locality now depends on where the driver puts each page. A placement rule tracks how many pages each partition holds and computes a balance score; while the score stays above a threshold it places each new page in the requesting partition, and when imbalance grows it places pages in the emptiest partition instead until balance is restored.
-
Find shared data the compiler can prove is read-only new
Data shared by many partitions is remote for most of them however it is placed. The compiler uses data-flow analysis to identify read-only shared structures, which can be copied into several slices with no coherence machinery at all, since nothing ever writes them.
-
Replicate only when a model says it pays new
Copying shared data into every partition buys bandwidth but wastes cache capacity, and a large shared structure would thrash the cache. Hardware samples a handful of cache sets, feeds hit rates and local-access fractions into an analytical bandwidth model each epoch, and switches replication on or off accordingly.
The idea
Import the structure of an old idea, but be clear that the reason is different. Non-uniform access is not new — CPUs adopted it decades ago, and it would be easy to describe this work as bringing that to graphics processors. The authors are careful to reject that framing. A CPU runs few threads per core, so a stall on memory is dead time and the enemy is LATENCY; non-uniformity helps by putting data physically nearer. A graphics processor runs thousands of threads and switches to another the moment one stalls, so latency is largely hidden and the enemy is BANDWIDTH. The same structural choice therefore serves a different purpose, and that changes what the policies must optimize — placing data to shorten a path is not the same as placing data to widen one, which is why the CPU-derived page policies do not transfer. Recognising that a technique must be re-derived rather than ported when the substrate’s bottleneck differs is the same redesign-for-the-substrate discipline covered elsewhere on this site, where a design correct on one machine becomes wrong on another because what was scarce has changed.
Hardware can only offer locality; software has to deliver it. The most useful lesson here is architectural humility. Partitioning the chip creates the POSSIBILITY of fast local access but guarantees nothing, because whether a compute unit’s data sits in its own partition is decided by the driver when it places a memory page, and by the compiler when it decides what may be replicated. So the paper spends most of its effort on those two pieces. The placement rule is deliberately simple — prefer local, but switch to filling the emptiest partition whenever a measured balance score drops below a threshold — and it captures the best of both existing policies, behaving like the local policy on applications that share little and like the balanced policy on those that share a lot. Refusing a dichotomy by measuring which regime you are in and switching, rather than committing to one policy, is the same decide-at-runtime instinct covered elsewhere on this site in the pages that measure a crossover point and pick a path accordingly, and here as there the threshold has a measured sweet spot rather than an obvious value.
Let the compiler establish what is safe and let hardware decide what is profitable. The treatment of shared data splits one hard question into two easier ones. Shared data is remote for most partitions however it is placed, and copying it everywhere would fix that — except that copies consume cache capacity, so replicating a large structure causes thrashing. Worse, copies of writable data must be kept consistent, which is the very complexity NUBA set out to avoid. The split is elegant: the COMPILER proves, by data-flow analysis, which shared data is read-only, which sidesteps coherence entirely because nothing ever writes it; then HARDWARE, which alone can see actual hit rates, decides each epoch whether replication currently pays, using a small analytical model fed by sampling a handful of cache sets. Assigning the safety question to static analysis and the profitability question to a runtime measurement is a division of labour covered elsewhere on this site wherever a compiler guarantees a property and a runtime policy chooses whether to exploit it.
Making a memory access local
From a request that must cross the chip to one that never leaves home
How a NUBA design turns remote accesses into local ones. Numbers from the paper's simulated 64-unit design.
Step 1 of 5
What happens on a conventional design
A compute unit misses in its small private cache and must reach the shared cache. Because any address may live in any slice, the request enters the crossbar and travels to whichever slice owns that address. Every single private-cache miss does this, so by construction all of them are remote and all of them compete for one shared network.
the network supplies 1.4 terabytes per second while a directly attached cache slice could supply 2.8 the shortfall exists because crossbar cost grows with the square of the endpoints and the power budget is fixed
Why it works
- Crossbar cost grows with the square of the endpoints, so a growing chip cannot afford a network matching its cache bandwidth → the network is provisioned below the cache and every miss must cross it → cache bandwidth sits stranded behind the wiring.
- Point-to-point links inside a partition avoid that scaling entirely → a local cache slice can be reached at 2.8 rather than 1.4 terabytes per second → and since 63.9 percent of misses become local, most traffic gets the wider path.
- Most traffic no longer touches the network → so the network can be made much narrower without hurting performance → which is how a 700 gigabyte per second network matches designs carrying 5.6 terabytes per second, at 12 times less power.
- Locality depends on placement, and neither pure-local nor pure-balanced placement works for all applications → measuring a balance score and switching between them captures both regimes → giving the behaviour of each policy exactly where that policy is right.
- Shared read-only data has no coherence obligations → so it can be copied into many slices freely → and because copies cost cache capacity, a runtime model enables replication only while the bandwidth gain exceeds the extra misses.
What it costs (trade-offs and scope)
- The benefit depends entirely on the application’s sharing behaviour. Applications where most pages are touched by a single compute unit gain 30.4 percent, while heavily sharing ones gain 15.1 percent, since a substantial fraction of their accesses must be remote no matter how pages are placed.
- It requires changes across three layers to work at all. The architecture alone is not enough — a new driver placement policy and a compiler analysis are both prerequisites, which is a far higher adoption barrier than a purely hardware change.
- The placement rule needs a tuned threshold and bookkeeping. The driver must track pages per partition and the switching threshold was determined empirically to balance the two application classes, so it is a fitted constant rather than a derived one.
- Replication trades bandwidth for cache capacity, and can misjudge. The model is an approximation fed by sampling a few cache sets, and the paper is candid that one benchmark gains a dramatic increase in local accesses from replication yet barely any performance, because it is not bandwidth-bound to begin with.
- Everything is simulated, and the layout argument is qualitative. Results come from a simulator, not silicon. The authors argue manufacturing is feasible because current chips already cluster cache in the middle with slices near their memory controllers, so partitions could be formed by linking nearby units — but that is an argument, not a demonstration.
The evaluation
The design was simulated as a 64-compute-unit graphics processor with 6 megabytes of last-level cache split across 64 slices and 32 memory channels, arranged as 32 partitions of two compute units, two cache slices and one memory controller. Benchmarks were split into low-sharing and high-sharing groups by measuring how many compute units touch each memory page — in the low-sharing group more than 80 percent of pages are touched by a single unit, while in the high-sharing group some applications have more than 70 percent of pages touched by 26 to 64 units. Comparisons are against both conventional organizations at equal resources, and then at equal performance to isolate the power saving.
Connecting compute units to cache on a large GPU
| Property | Memory-side uniform | Compute-side uniform | Non-uniform bandwidth |
|---|---|---|---|
| Where a miss goes | |||
| Network cost | |||
| Coherence needed | |||
| Software required | |||
| Result |
Check yourself
Check your understanding
1. The authors say NUBA mirrors what CPUs did with non-uniform memory but for a fundamentally different reason. Predict the difference and why it matters.
2. The paper says a crossbar network's cost grows with the square of the number of endpoints. Predict what that forces designers to do, and what it costs them.
3. Partitioning the chip creates fast local access, yet the paper spends most of its effort on driver and compiler changes. Predict why the hardware alone is not enough.
4. Shared data is replicated only if the compiler proves it read-only AND a hardware model says it currently pays. Predict why the work is split that way.
Sources
- NUBA: Non-Uniform Bandwidth GPUs (ACM DL, DOI 10.1145/3575693.3575745) — ASPLOS 2023.
- Author PDF (Ghent University) — the full 16-page paper, read in full for this explainer.
- ASPLOS 2023 conference talk — the authors’ presentation of the design and results.