1.1–1.2The Landscape of Computation Accelerators & GPU Hardware Basics

GPUs started life as video-game rendering engines. Today they sit in phones, laptops, datacenters and supercomputers — and an increasing share of what they do isn’t graphics at all (machine learning being the loudest example). This book, and this site, focus on the features that make GPUs fast and energy-efficient for general-purpose computation.

1.1 The Landscape of Computation Accelerators

For decades, each generation of computing hardware got faster per dollar almost automatically: smaller transistors, better architectures, better compilers, better algorithms. By some estimates, about half of those gains came purely from transistor scaling making devices faster. Around 2005 that free ride ended: Dennard ScalingDennard ScalingClassical transistor-scaling rules whose ~2005 breakdown ended easy clock-frequency gains, pushing architecture toward specialization.Glossary → — the classical rule set under which shrinking a transistor also made it proportionally faster and less power-hungry — broke down. Clock frequencies now creep rather than leap. If you want more performance, you must find more efficient hardware architectures.

That is exactly what a computation acceleratorcomputation acceleratorSpecialized hardware that trades generality for energy efficiency (up to ~500× vs. general-purpose cores).Glossary → is: hardware that gives up some generality in exchange for efficiency. And the exchange rate is dramatic — Hameed et al. measured up to 500× better energy efficiency for specialized hardware vs. general-purpose cores. That 500× decomposes into two big pieces:

1. ~10× from vector hardware (the GPU approach): one instruction fetch, decode, and schedule is amortized over many data elements, eliminating most instruction-processing overhead.
2. Most of the rest from minimizing data movement: complex operations that chain several arithmetic steps without round-tripping values through large storage structures such as register files.

The architect’s dilemma is that maximum efficiency and maximum flexibility pull in opposite directions. A fully specialized design — say, Google’s Tensor Processing Unit for deep neural networks — is extremely efficient at one family of computations and useless outside it. At the other end, GPUs are Turing CompleteTuring CompleteAble to run any computation given enough time and memory; what separates GPUs from fixed-function accelerators.Glossary →: given enough time and memory, they can run any computation. Yet for software that uses the hardware well, a GPU is still roughly an order of magnitude more efficient than a CPU. That combination — flexible and efficient — is why many of the fastest and most energy-efficient supercomputers are GPU-based, and why this book bets that “software written in traditional programming languages” will keep needing an efficient home even as fixed-function accelerators proliferate.

1.2 GPU Hardware Basics

A question newcomers often ask: will GPUs replace CPUs? Almost certainly not. GPUs are not stand-alone computing devices — every GPU system pairs the GPU with a CPU, either as separate chips (a discrete GPUdiscrete GPUGPU on an add-in card with its own DRAM, connected to the CPU by a bus (e.g. PCIe).Glossary → on an add-in card) or fused on one die (an integrated GPUintegrated GPUGPU sharing one die and one DRAM space with the CPU (e.g. AMD APUs, mobile SoCs).Glossary →, like AMD’s Bristol Ridge APU or a smartphone SoC). The CPU starts computations, feeds the GPU its data, and — crucially — talks to I/O devices. Operating-system services and device drivers lack the massive parallelism that GPU hardware needs to shine, so even research APIs that offer “I/O from the GPU” quietly assume a nearby CPU.

The memory arrangement is the key difference between the two designs. Discrete GPUs give the CPU and GPU separate DRAM spaces: system memorysystem memoryThe CPU's DRAM (latency-optimized, DDR) in a discrete-GPU system.Glossary → (DDR — optimized for low latency, which CPU code craves) and device memorydevice memoryThe GPU's DRAM (throughput-optimized, GDDR) in a discrete-GPU system.Glossary → (GDDR — optimized for raw bandwidth, which GPU workloads devour). Integrated designs share one DRAM space, typically low-power LPDDR in mobile parts — but sharing brings its own problem: private caches on both sides can disagree about memory contents, a cache-coherence problem hardware designers must solve.

How a computation reaches the GPU

A GPU application begins life on the CPU, which allocates and initializes data. On older discrete GPUs the programmer explicitly copies data into device memory; newer ones (NVIDIA Pascal onward) can migrate pages automatically using virtual-memory support — NVIDIA calls this unified memoryunified memorySoftware/hardware support (NVIDIA Pascal+) that automatically migrates data between CPU and GPU memory via virtual memory.Glossary →. Then the CPU names a kernelkernelThe code a GPU-computing application launches to run across many GPU threads.Glossary → (the code to run on the GPU — Chapter 2 covers these in detail), a thread count, and the data’s location; the driverdriverCPU-side software that conveys the kernel, thread count and data locations to the GPU and signals it to start.Glossary → writes that launch information where the GPU is configured to look and signals it to start. Step through the whole sequence:

Inside the GPU

A modern GPU is a sea of cores. NVIDIA calls them streaming multiprocessors (SMs), AMD calls them compute units (CUs); this book says SIMT coreSIMT coreThis book's name for a GPU core; NVIDIA "streaming multiprocessor (SM)", AMD "compute unit (CU)".Glossary → for both. Each SIMT core runs a SIMTSIMTSingle-instruction, multiple-thread: the execution model GPU cores use to run thousands of scalar threads on SIMD hardware.Glossary → program — single-instruction, multiple-thread — and typically hosts on the order of a thousand threads. Threads on the same core can share data through a fast scratchpad memoryscratchpad memoryFast per-core, software-managed memory through which threads on a core communicate.Glossary → and synchronize with cheap barrier operations. Each core also has first-level instruction and data caches, which act as bandwidth filtersbandwidth filterThe role of per-core L1 caches: reducing traffic sent to lower levels of the memory system.Glossary →: whatever hits in L1 never has to travel down the memory system.

Feeding all those cores requires parallelism in the memory system itself: multiple memory channels, each usually paired with a slice of the last-level cache to form a memory partitionmemory partitionA slice of last-level cache paired with a memory channel, connected to cores via the on-chip interconnect.Glossary →. An on-chip interconnection network (a crossbar, for instance) connects cores to partitions. (Alternatives exist — Intel’s Xeon Phi distributed its last-level cache with the cores instead.)

Why a thousand threads? Latency hiding

GPUs spend their die area differently than CPUs: much more on arithmetic logic units, much less on control. They can afford to because of latency hidinglatency hidingUsing many ready threads so a core keeps issuing while other threads wait on memory.Glossary → — when one group of threads stalls waiting on memory, the scheduler simply issues another. Watch the same toy workload run with one thread group vs. four (memory latency = 6 cycles). Keep an eye on the utilization number:

With one group, the core sits idle for six of every ten cycles. With four, every waiting period is covered by another group’s useful work — the memory latency hasn’t gone away, it has been hidden.

Guz et al. built a simple analytical model that makes this trade-off vivid. Drag the marker across the thread count and watch what happens between the multicore (MC) region and the multithreaded (MT) region — the performance valleyperformance valleyGuz et al. model region where threads overflow the cache but are too few to hide memory latency (between the MC and MT regions).Glossary →:

The price of moving data

With the end of Dennard Scaling, energy efficiency drives architecture. The uncomfortable fact: touching a large memory structure can cost as much as — or vastly more than — the computation itself. Sort the table below by relative cost, and click a row for context (values for a 45 nm process; GPGPU simulators such as GPGPU-Sim now ship energy models precisely because of this):