4.4.6–4.4.10Research: Memory Systems II — Management, Placement, MCM-GPUs

4.4.6 Adaptive cache management

Chen et al. coordinate warp throttling with cache bypassingcache bypassingSending selected accesses around a cache (on associativity stalls, low-reuse PCs, non-all-hit warps…) to cut contention and queueing.Glossary →, detecting cache and memory-resource contention at runtime and adjusting both. The bypass mechanism borrows the CPU idea of protection distanceprotection distanceBypass technique giving each inserted line N protected accesses before it may be evicted; inserts finding no unprotected line bypass instead.Glossary →: each inserted line is untouchable for its next N accesses; a fill that finds no unprotected victim in its set bypasses instead. (Pleasant surprise: GPU workloads are fairly insensitive to the exact N.)

4.4.7 Cache prioritization: tokens

Li et al. spotted the blind spot of throttling à la CCWS: tuning warp count for L1 hit rate leaves off-chip bandwidth and L2 idle. Their split: W warps may execute, but only T ≤ W hold tokens allowing L1 allocation — the rest run as “non-polluting warps”:

4.4.8–4.4.9 Placement: pages and data structures

Two placement problems, two altitudes:

• Virtual-memory page placement (Agarwal et al.): future systems mix bandwidth-optimized and capacity-optimized DRAM, but stock OS policies ignore bandwidth non-uniformity. With capacity to spare, allocate pages randomly in proportion to bandwidth — near-optimal for bandwidth-bound kernels (uniform-access assumption). When BW-memory fills up, frequency matters: a profiling pass (modified nvcc/ptxas) plus CUDA placement hints recovers ≈90% of an oracle. Page migration: future work.
• PORPLE (Chen et al.): which memory space (global? shared? constant? texture?) for each array — a decision that’s hard for programmers and non-portable across GPU generations. Three pieces: a memory specification language describing each GPU’s spaces by their serialization conditions (bank conflicts vs. coalescing rules); the PORPLE-C source-to-source compiler emitting placement-agnostic code with guards; and a lightweight runtime placer (estimated transactions
  • reuse-distance cache model, linear cache partitioning across arrays) — with a brief CPU-side trace run when static analysis can’t see the access pattern.

4.4.10 Multi-chip-module GPUs

With Moore’s Law slowing, Arunkumar et al. scale the package instead of the die: the MCM-GPUMCM-GPUMulti-chip-module GPU: several GPU modules on one package with local caching of remote data, locality-aware CTA scheduling, first-touch pages — near-monolithic performance past reticle limits.Glossary → — several GPU modules (GPMs), each with its own SMs, L2, and DRAM, stitched into one logical GPU:

Three techniques tame the scarce inter-module links — local caching of remote data, locality-aware CTA scheduling (adjacent CTAs touch adjacent data → keep them on one module), and first-touch page allocation (pages land where first used). Result: within 10% of an equally-sized-but-unbuildable monolithic GPU, and 45% faster than the largest monolithic GPU actually constructible in the same process.

That’s the memory system. Chapter 5 closes the book with research that cuts across cores and memory.