4.1–4.1.1Scratchpad Memory & the L1 Data Cache

Chapter 3 built the cores; this chapter feeds them. Kernels touch memory through loads and stores against a subdivided address space: local memorylocal memoryPer-thread private memory space, used mostly for register spills and stack data; cached write-back in L1.Glossary → (per-thread — mostly register spills), global memoryglobal memoryThe address space accessible to all threads; slower and more energy-costly than shared memory; the only way threads in different CTAs communicate.Glossary → (shared data structures), and the programmer-managed shared memoryshared memoryNVIDIA's name for the per-SM scratchpad (16–64 KB), declared with __shared__; a software-controlled cache.Glossary → scratchpad. The motivation is brutal economics: DRAM bandwidth is tiny next to instruction throughput, and off-chip transfers cost orders of magnitude more energy than on-chip accesses (remember Table 1.1) — so the programmer who stages data on-chip wins twice.

The two programmer-visible hazards

Two access patterns dominate GPU memory performance, and both come down to how a warp’s 32 addresses relate to each other. For global memory, the question is whether they fall in the same cache block — coalescingmemory coalescingMerging a warp's per-thread global accesses that fall in one cache block into a single memory request; uncoalesced access = multiple requests.Glossary →:

For shared memory (one 32-bit bank per lane, one read + one write port each), the question is whether two threads hit different rows of the same bank — a bank conflictshared-memory bank conflictTwo threads targeting different locations in the same shared-memory bank in one cycle; the arbiter splits the access and replays the conflicting part.Glossary →:

Programmers avoid both; the hardware tolerates both — that tolerance is the story of Fig 4.1.

4.1.1 One SRAM, two personalities

Fermi and Kepler implement shared memory and the L1 data cache as one configurable SRAM array: partly direct-mapped (shared), partly set-associative (cache) — early NVIDIA patents even call shared memory a “Global Register File,” and its latency is register-file-like. The surrounding machinery gives the instruction pipeline a non-stalling interface: hazards become replays, never stalls. Walk all four operation types through the numbered datapath:

Details worth pinning down:

• Shared accesses bypass the tag unit (direct-mapped, constant latency), which is what lets the arbiter schedule the register-file writeback at accept time. The banks’ independent decoders are why arbitrary per-lane rows work at all.
• Global accesses are one cache block per cycle — 128-byte blocks (Fermi/Kepler), split into four 32-byte sectorssector32-byte quarter of a 128-byte cache line, matching the GDDR5 access atom; Maxwell/Pascal L1s and L2 slices are sectored.Glossary → from Maxwell/Pascal on, matching the GDDR5 access atom. Less tag overhead, DRAM-shaped requests.
• The pending request tablepending request table (PRT)The GPU L1's MSHR-like structure: merges misses to the same block and remembers which deferred access to replay when the fill returns.Glossary → is the GPU’s take on the classic MSHRMSHRMiss-status holding register: tracks an outstanding cache miss so further misses to the same line merge instead of replaying to memory.Glossary →: merge same-block misses, remember which deferred access to replay, find it again via the request’s subid when the fill returns — and the fill unit locks the line so the replayed load cannot miss twice.
• The L1 is virtually indexed and virtually taggedvirtually-tagged L1GPU L1s are virtually indexed AND tagged (unlike CPU VIPT): no per-cycle translation needed, acceptable because all warps share one address space.Glossary →. CPUs avoid VIVT because context switches force flushes — but every warp on a GPU belongs to the same application, so the GPU takes the faster, simpler option (keeping page-based virtual memory for allocation simplicity and fragmentation control).
• Writes match their space’s locality: global stores → write-through, no-allocate (kernels stream results out and exit); local-memory spills → write-back, write-allocate (they’ll be re-read). Store data stages in the write data buffer; fully-covering coalesced writes may bypass the cache entirely (invalidating stale tags on the way).
• There is no L1 coherence. SM 1 can happily read a stale copy of a location SM 2 just wrote. NVIDIA’s Kepler-era answer: only spills, stack data, and read-only global data may live in L1. (Research on coherent GPU L1s and properly defined GPU memory consistency models is ongoing — the book cites Singh et al., Ren & Lis, Alglave et al.)