4.4–4.4.5Research: Memory Systems I — Scheduling, Caching, Bypassing

The baseline memory system is built; now the research. Part I covers networks, caching effectiveness, and the bypass/prioritization families (§4.4.1–4.4.5); Part II continues with adaptive management through MCM-GPUs.

4.4.1 Scheduling meets the interconnect

Yuan et al. measured that a single SM emits requests with healthy row-buffer localityrow-buffer localityConsecutive requests touching the same DRAM row of the same bank; created by one SM's access stream, destroyed by interleaving many SMs.Glossary → — which the crossbar then shreds by interleaving many SMs’ streams into each partition (the tension we saw in §4.3). Their move: fix it in the network — arbitration policies that keep packets from the same SM (or with similar row-bank addresses) together — letting the DRAM scheduler stay simple. Bakhoda et al. studied the NoC itself: GPU traffic is many-to-few-to-many (many SMs, few partitions), throughput is largely insensitive to interconnect latency, and exploiting the pattern with “half-routers” cuts router area while scaling to mesh topologies as SM counts grow.

4.4.2 Does caching even help?

Simulation (Bakhoda et al.) and Fermi hardware measurements (Jia et al.) agree: some applications benefit from the L1, others lose. Obviously, kernels staging data into shared memory via L1 gain nothing — but even beyond those, hit rate alone doesn’t predict the win. The missing variable is traffic: Fermi’s unsectored L1 turns every miss into a 128-byte off-chip fetch, so bandwidth-bound kernels can slow down with caching enabled. Jia et al.’s locality taxonomy — within-warp (one load, one warp’s threads share a block), within-block (same load, different warps of a CTA), cross-instruction (different loads) — feeds a compile-time algorithm that enables caching per load instruction.

4.4.3 Prioritization and bypassingcache bypassingSending selected accesses around a cache (on associativity stalls, low-reuse PCs, non-all-hit warps…) to cut contention and queueing.Glossary →

• MRPBMRPBMemory request prioritization buffer: warp-ID-signature FIFOs reordering requests just before the L1 to restore locality after scheduling.Glossary → (Jia et al.): a stack of FIFOs in front of the L1, requests distributed by warp-ID signature, drained by fixed priority — restoring per-warp locality after scheduling has interleaved it (same goal as CCWS, §5.1.2, but at the memory-access level). Plus bypass on associativity stall: when a miss can’t allocate because the whole set is reserved for in-flight misses (intra-warp contention — one warp’s indexing pattern hammering one set), skip the L1 instead of stalling. Combined: ~4% geomean over a 16 KB 64-way baseline, more on contentious workloads. (Footnote fairness: baselines matter — Fermi’s real set-index function XORs bits, which already dampens such conflicts.)
• Arunkumar et al.: choose bypassing and cache-line size per static instruction from its memory-divergence degree and reuse-distance pattern.
• Lee & Wu: a runtime control loop per PC — monitor whether a load’s lines get reused; if not, bypass that instruction’s future accesses.

4.4.4 Inter-warp heterogeneity: MeDiC

The sharpest observation in this section (Ausavarungnirun et al.): within one kernel, warps split into all-hit / mostly-hit / balanced / mostly-miss / all-miss types — and a warp that isn’t all-hit gains little from its hits, since it waits for its slowest lane anyway. MeDiC routes accordingly:

4.4.5 Coordinated bypassing

Xie et al. close the loop with profiling: static loads are marked good / poor / moderate locality; good always caches, poor always bypasses, and moderate loads flip as a group per thread block, driven by an online threshold that watches L1 hits and pipeline resource conflicts. Notably — like MRPB — the win narrows the gap between naive cache-reliant code and hand-tuned shared-memory code: these mechanisms are programmability subsidies.