1.3–1.4A Brief History of GPUs & Book Outline

1.3 A Brief History of GPUs

The GPU’s road from drawing text on a monochrome screen to accelerating machine learning is short — about four decades — and every major turn was a step toward programmability. Walk the milestones:

Computer graphics emerged in the 1960s with projects like Ivan Sutherland’s Sketchpad, splitting early into two traditions: off-line rendering for film animation, and real-time rendering for video games — the demanding, cost-sensitive market that would bankroll GPU innovation. Early video cards, starting with IBM’s Monochrome Display Adapter (1981), only handled text; 2D and then 3D acceleration followed, serving video games and computer-aided design.

Early 3D processors such as the NVIDIA GeForce 256 were essentially fixed-function: fast, but only at the rendering pipeline they hard-wired. The pivotal change came in 2001, when the GeForce 3 introduced vertex and pixel shadersvertex/pixel shadersThe first programmable GPU stages (GeForce 3, 2001), the seed of GPGPU.Glossary → — programmable stages inside the graphics pipeline. Researchers noticed almost immediately that a programmable graphics card is a parallel computer in disguise: by mapping matrix data into textures and expressing math as shading, they implemented linear algebra on stock GPUs. Academic projects then mapped general-purpose programs onto GPUs so the programmer didn’t need to know any graphics at all.

Vendors took the hint. The GeForce 8 series became the first commercial GPU to support general-purpose computing directly, adding two things graphics never needed: the ability to write to arbitrary memory addresses from a shader, and a scratchpad memoryscratchpad memoryFast per-core, software-managed memory through which threads on a core communicate.Glossary → to keep intermediate data on-chip and off the scarce off-chip bandwidth. Fermi then enabled caching of read-write data — general memory-access patterns finally got a cache hierarchy. AMD’s Fusion architecture put CPU and GPU on the same die (the integrated design point from §1.2), and dynamic parallelismdynamic parallelismThe ability to launch new GPU threads from code already running on the GPU.Glossary → let the GPU launch threads itself, without a CPU round-trip. Most recently, NVIDIA’s Volta adds Tensor CoresTensor CoreVolta-generation unit specialized for machine-learning matrix operations.Glossary → aimed squarely at machine learning — specialization growing back inside the general-purpose GPU, the same efficiency-vs-flexibility tension from §1.1 playing out on a smaller stage.

1.4 Book Outline

The rest of the book proceeds hardware-first but software-aware:

• Chapter 2 — Programming Model. Hardware exists to run software, so first: how GPUs are programmed, the code development process, and the compilation flow down to real instruction sets (NVIDIA’s and AMD’s).
• Chapter 3 — The SIMT Core. The architecture of an individual GPU core that executes thousands of threads. Built up incrementally — three successive “approximation loops” of increasing fidelity — ending with a survey of research on core architecture.
• Chapter 4 — Memory System. The first-level caches inside the cores and the internal organization of the memory partitions. GPU computations are frequently limited by off-chip bandwidth, so this chapter matters as much as the cores themselves. Ends with memory-system research.
• Chapter 5 — Crosscutting Research. Work that spans both cores and memory: scheduling, alternative parallelism models, transactional memory, and heterogeneous systems.