4.2–4.3On-Chip Interconnect & the Memory Partition Unit

4.2 On-Chip Interconnection Network

Feeding the SIMT cores takes multiple DRAM chips in parallel, reached through the memory partitionsmemory partitionA slice of last-level cache paired with a memory channel, connected to cores via the on-chip interconnect.Glossary →. Traffic is spread across them by address interleaving — NVIDIA patents describe balancing across up to six partitions at 256-byte or 1,024-byte granularity. The cores connect to the partitions over an on-chip network: crossbars in recent NVIDIA patents; AMD parts have sometimes been described using rings.

4.3 Inside a Memory Partition

Each partition (Fig 4.3) packages three tightly coupled units: a slice of the L2 cache, one or more memory access schedulers (historically “frame buffer”, FB), and a raster operation (ROP) unit — serving graphics and compute alike.

4.3.1 The L2 slices

Each partition’s L2 is two slicesL2 sliceHalf of a partition's L2: own tags + data, in-order, sectored; write-miss sectors fully covered by coalesced stores skip the DRAM read.Glossary →, each with its own tags and data, processing requests in order. Lines are 128 bytes of four 32-byte sectorssector32-byte quarter of a 128-byte cache line, matching the GDDR5 access atom; Maxwell/Pascal L1s and L2 slices are sectored.Glossary → — the GDDR5 access atom. Two throughput tricks worth noticing:

• Fully-covered write misses skip the DRAM read. GPU kernels stream results to big arrays; when a coalesced store overwrites whole sectors, the slice just allocates and writes. (Partially covering writes need byte-valid bits — or bypass the L2.) Textbook CPU caches don’t dare.
• L2 lines double as the write buffer: dirty data waits in the cache itself while the scheduler finds a good moment — no separate write-queue SRAM.

4.3.2 Atomic operations

The ROP unitROP unitRaster operations unit in each partition: alpha blending, surface compression, and the GPU's atomic/reduction operations (with a small cache to pipeline same-address atomics).Glossary → executes atomic and reduction operations (besides its graphics day job of alpha blending and surface compression). A small ROP cache pipelines back-to-back atomics to the same address — which is why atomics are a workable synchronization primitive across thread blocks.

4.3.3 The memory access scheduler

DRAM’s physics set the stage: bits live in capacitors; reading means precharging bitlines, activating a row into the bank’s row bufferrow bufferThe DRAM bank's staging row: precharge + activate move a page into it; hits in the open row are fast, switches cost dead cycles.Glossary → via sense amplifiers (refreshing the cells as a bonus), and only then streaming data. Row switches are dead time — so banks exist, and so do memory access schedulersmemory access schedulerThe partition's "frame buffer" logic reordering DRAM reads/writes (read request sorter + store tables) to hit open rows.Glossary → that reorder requests to preserve row-buffer localityrow-buffer localityConsecutive requests touching the same DRAM row of the same bank; created by one SM's access stream, destroyed by interleaving many SMs.Glossary →. Compare first-come-first-served against the scheduler (read request sorter + store tables grouping same-row reads):

arrival order: R1(b0,row A)  R2(b0,row B)  R3(b0,row A)  R4(b1,row C)  R5(b1,row C)

sorter output: {R1, R3} → row A · {R2} → row B · {R4, R5} → row C

One subtlety that will matter in §4.4: a single SM’s coalesced stream has excellent row locality — it’s the interleaving of many SMs’ streams in the interconnect that shuffles rows together and makes the scheduler’s job hard. That tension (fix it in the network? in the scheduler?) opens the research section.