4.1.2–4.1.3The L1 Texture Cache & Unified Texture/Data Cache

4.1.2 The L1 Texture Cache

Texture caching earns its place in a GPGPU book for one reason: it is the purest expression of throughput-processor cache design — and recent NVIDIA/AMD parts literally reuse it for compute. First, thirty seconds of graphics: texture mappingtexture mapping / texelApplying an image (texture) to a 3D surface; a texel is one texture sample — neighboring pixels read neighboring texels, giving caches their locality.Glossary → pastes an image onto 3D geometry (wood grain onto a table); the rendering pipeline computes which texels (texture samples) each screen pixel needs — four or eight of them for bilinear/trilinear filtering — and averages them. Adjacent pixels read adjacent texels, so there’s real locality for a cache to catch… but the textures themselves are far too big to fit on-chip, so misses are constant companions.

A CPU cache treats a miss as an exception. The texture cachetexture cacheThroughput-first L1: the tag array runs a memory-round-trip AHEAD of the data array, a fragment FIFO delays hits, a reorder buffer keeps fills in order — hits and misses see similar latency.Glossary → (Igehy et al.’s design, matching industry patents) treats it as the common case and restructures everything around it — watch how a hit and a miss take almost the same journey:

The three load-bearing ideas:

1. The tag array runs ahead of the data array. Tags update at lookup time; the data array catches up one memory-round-trip later. Both stay small — no giant miss-handling structures.
2. The fragment FIFO absorbs the latency. Hit or miss, every access parks in a FIFO sized to the round trip. Hits don’t jump the queue — hits and misses see roughly equal latency, which sounds terrible until you remember the metric is throughput, and warps are covering the latency anyway.
3. The reorder buffer keeps memory honest. The DRAM scheduler will happily service requests out of order for row locality (§4.3.3); the reorder buffer restores arrival order so the data array remains an exact time-delayed image of the tags.

4.1.3 Unified Texture and Data Cache

Maxwell- and Pascal-era NVIDIA GPUs (and AMD’s GCN) merge the L1 data cache into this structure. The trick that makes it nearly free: only data guaranteed read-only is cached — which, per the coherence discussion, is what the L1 was drifting toward anyway. With writes off the table, the texture hardware needs little more than new addressing logic (the design appears in a 2017 NVIDIA patent). GCN goes all-in: every vector memory operation flows through the texture cache.

The broader lesson the book draws: CPU intuitions (hits fast, misses rare, order preserved by stalling) simply don’t survive contact with a machine whose job is to keep ten thousand threads streaming.