5.1–5.1.1Thread Scheduling & Threadblock Assignment

Book pp. 85–88 · ~3 min read

  • threadblock scheduling
  • CTA throttling
  • Equalizer

The book’s final chapter collects research that refuses to stay inside one subsystem. First up: scheduling — because on a machine whose whole strategy is massive parallelism, when things run is architecture. GPU scheduling happens at four distinct levels:

  1. Threads → warps. Consecutive thread IDs are statically fused into warps (alternatives: lane permutation & friends).
  2. Threadblocks → cores. Work arrives in bulk: a threadblock at a timethreadblock schedulingAssigning CTAs to cores in bulk (round-robin, resource-limited); no preemption — a CTA runs to completion before its resources free up.Glossary →, its warp slots, registers and shared memory subscribed together — and since that state is huge, no preemption: CTAs run to completion (§5.1.3 challenges this). Baseline policy: round-robin across cores until some resource runs out; surplus CTAs wait off-core entirely.
  3. Cycle-by-cycle. The fine-grained fetch/issue/operand schedulers of Chapter 3.
  4. Multiple kernels. Once one-kernel-at-a-time, now concurrent via NVIDIA Streams and Hyper-Q — GPU multiprogramming (§5.1.3).

5.1.1 Research on assigning threadblocks to cores

More parallelism is not always better. That counter-intuition drives both proposals here:

  • CTA throttlingCTA throttlingLimiting resident threadblocks per core (watching idle vs memory-delay cycles) to cut memory-system contention from over-subscription.Glossary → (Kayiran et al.): start every core at half its CTA capacity, monitor idle cycles vs memory-delay cycles, and when a core mostly waits on memory, stop assigning — even pause — threadblocks. Fewer resident CTAs → less memory-system interference → net speedup, despite idle warp slots.
  • EqualizerEqualizerRuntime that tunes CTA count, core frequency and memory frequency from four warp-state counters, in energy-saving or performance mode.Glossary → (Sethia & Mahlke) goes further: contention is dynamic, so tune three knobs — CTA count, core frequency, memory frequency — from four per-core counters, each epoch:
Equalizer's decision loop (one epoch)cycle 0 / 4
4 warp-state countersactive · waiting-on-mem · ALU-ready · mem-readyLocal decision (per SIMT core)CTA count + freq preferenceGlobal work distributorissues / pauses CTAsGlobal frequency managermajority vote → chip-wide DVFSThis epoch's operating pointCTAs · core clock · memory clockepoch sample

step 0Each epoch, hardware counters capture: active warps, warps waiting for memory data, warps ready to issue an ALU op, warps ready to issue a memory op. These proxy memory contention, compute intensity, and memory intensity.

One tick = one step of Equalizer's per-epoch decision loop (Sethia & Mahlke). The example core turns out to be memory-intensive.

The same diagnosis feeds two policies:

Compute-intensiveMemory clock ↓Core clock ↑
Memory-intensiveCore clock ↓Memory clock ↑
Cache-sensitiveCTAs ↓CTAs ↓
Unsaturated—CTAs ↑
Equalizer actions by workload class and mode. Click a row for the reasoning.

Both mechanisms echo the chapter-3 scheduling trade-off one level up: equal progress vs. locality, now with energy in the objective function. The per-warp version of this fight — CCWS and its descendants — is next.

Check yourself

  1. 1. Why don't contemporary GPUs preempt running threadblocks?

  2. 2. How does the baseline threadblock scheduler place CTAs?

  3. 3. What signals drive Kayiran et al.'s CTA throttling, and what does it do?

  4. 4. Predict Equalizer (use the animation): the counters show more memory-waiting warps than one CTA holds, and Equalizer runs in ENERGY mode. What happens?

  5. 5. Why are Equalizer's frequency decisions made by a GLOBAL majority function rather than per core?

0 / 5 correct · 5 unanswered