Chapter 3 opens the excerpt’s Part II question: how do you judge a DNN accelerator? The popular shortcut is a single efficiency number — operations per second per watt (FLOPS/W, TOPS/W). The chapter’s argument is that real efficiency is a vector of metrics — accuracy, throughput, latency, energy, power, cost, flexibility, scalability — and that reporting only a slice of it paints a misleading picture. The first component of that vector, and the gatekeeper for all the others, is accuracy: does the system produce results good enough to be useful at all?
What accuracy measures — and why its units change with the task
Accuracy accuracy Quality of the result for a given task; units depend on the task (% correct for classification, mAP for detection) and must be read against task/dataset difficulty. defined in ch. 3 — open in glossary indicates the quality of the result for a given task. DNNs are popular precisely because they deliver state-of-the-art accuracy on a wide range of tasks — but “accuracy” is not one number with one unit. It is measured in whatever way the task demands:
- Image classification — the model assigns one label per image, so accuracy is simply the percentage of images classified correctly.
- Object detection — the model must find and localize objects, so accuracy is reported as mean average precision (mAP) mean average precision Accuracy metric for object detection (mAP), capturing the trade-off between true- and false-positive rates. defined in ch. 3 — open in glossary , which reflects the trade-off between the true-positive rate and the false-positive rate.
Two tasks, two meanings of “accurate.” A classifier is right or wrong per image, so a percentage suffices. A detector juggles two failure modes at once — missing real objects vs raising false alarms — so its accuracy metric (mAP) must summarize the whole true-positive/false-positive trade-off.
Beyond accuracy: robustness and fairness
The book notes (its footnote 1) that ideally robustness and fairness should be evaluated in conjunction with accuracy — there is an interplay among the three — but these were active research areas at the time of writing and are outside the book’s scope.
The score means nothing without the exam
The factors that most affect achievable accuracy are the difficulty of the task and of the dataset. Two of the book’s calibration points:
- Classification on ImageNet (~1000 diverse object classes, natural photos) is much more difficult than on MNIST (10 handwritten digits) — a model that performs well on MNIST may do poorly on ImageNet.
- Object detection or semantic segmentation is harder than classification — the model must decide where things are (per-box or even per-pixel), not just what the image shows.
The book’s analogy (its footnote 2): scoring 9/10 on a high-school exam is not the same achievement as 9/10 on a college exam. One must look beyond the score and consider the difficulty of the exam.
The exam analogy: interpret every accuracy number relative to the difficulty of the task and dataset it was measured on.
Why accuracy is a hardware problem
Accuracy would be someone else’s metric if it stopped at the model. It doesn’t. Achieving high accuracy on difficult tasks typically requires more complex DNN models — a larger number of MAC operations mac operation Multiply-accumulate, the dominant operation in DNN processing; the book uses 'operations' and 'MAC operations' interchangeably. defined in ch. 3 — open in glossary , more distinct weights, increased diversity in layer shapes — and that complexity directly impacts how efficiently the hardware can process the model. This causal chain is why accuracy leads the chapter: every later metric (throughput §3.2, energy §3.3, flexibility §3.5) is measured while sustaining some accuracy target, and raising the target makes each of them harder.
Why accuracy is where the metric story starts: the accuracy target shapes the model, and the model shapes every hardware metric that follows.
Reporting accuracy so it can be trusted
Because the number is only meaningful in context, the book’s reporting rule is: evaluate hardware on well-studied, widely used DNN models, tasks, and datasets, so readers can calibrate the significance of the accuracy achieved. Motivated by the impact the SPEC benchmarks had on general-purpose computing, industry and academic organizations assembled MLPerf mlperf Industry/academic benchmark suite of well-studied DNN models for fair comparison of DNN hardware and frameworks, for training and inference. defined in ch. 3 — open in glossary — a broad suite of well-studied DNN models serving as a common yardstick for comparing software frameworks, hardware accelerators, and cloud platforms, covering both training and inference. (Earlier benchmarking efforts, DeepBench and Fathom, have been subsumed by MLPerf.)
The suite spans DNN types (CNNs, RNNs, …) and a deliberately wide task mix — click any cell for what it implies about evaluation:
| Accuracy metric | Typical benchmark | Difficulty context | |
|---|---|---|---|
| Image classification | % correctly classified | MNIST (easy) · ImageNet (hard) | baseline |
| Object detection | mAP | e.g. COCO-style photo sets | harder than classification |
| Semantic segmentation | per-pixel accuracy measures | scene-parsing datasets | harder than classification |
| Translation · speech-to-text · recommendation · sentiment · RL | task-specific | MLPerf suite workloads | varies |
Design takeaway
Never read (or report) an accuracy number bare. Attach the task, the dataset, and ideally a well-studied model from a common suite such as MLPerf — otherwise the number cannot be compared, and (as §3.7 will show) an “efficient” design can hide behind an easy workload.
Check yourself
Check yourself — Accuracy
1.Model A scores 99% on MNIST. Model B scores 80% on ImageNet. What can you conclude about which model is 'better'?
2.Why is object-detection accuracy reported as mAP rather than 'percent correct'?
3.Predict the knock-on effect: a team switches its target task from MNIST digit classification to ImageNet-level classification and wants to keep accuracy high. What happens to the workload their hardware must process?
4.What problem is MLPerf designed to solve?