The last four metrics are quicker stories than throughput and energy — but §3.7’s point is that the set is what matters: leave any one of them out of a report, and the others can be gamed.
3.4 Hardware Cost
Hardware cost hardware cost Monetary cost to build a system; proxied by chip area (with process node) and off-chip bandwidth (packaging/PCB). defined in ch. 3 — open in glossary is the monetary cost to build a system, and it decides financial viability for industry and research alike. Cost constraints follow volume and market: embedded processors face much more stringent limits than cloud processors. (The book’s footnote 7 scopes this carefully: operating cost — electricity, cooling — is already covered by energy/power in §3.3; design cost is noted only as “custom accelerators cost more to design than off-the-shelf CPUs/GPUs”; pricing economics is out of scope.)
Two factors dominate:
- Chip area (mm²) in conjunction with the process technology (e.g., 45 nm CMOS) — this constrains on-chip storage and compute: the number of PEs in a custom accelerator, cores in a CPU/GPU, DSP engines in an FPGA. Without naming a process, report the proxies: on-chip memory capacity (e.g., global-buffer size) and amount of compute (e.g., #PEs).
- Off-chip bandwidth — it dictates packaging and PCB cost and complexity (High Bandwidth Memory to reach DRAM, NVLink between GPUs) and may demand extra die area for transceivers to keep signal integrity at speed. Proxy: off-chip bandwidth off-chip bandwidth Data rate between the chip and external memory or other chips (Gbps); a proxy for packaging and PCB cost. defined in ch. 3 — open in glossary in Gbps, ideally with the number of I/O ports.
The two trade against each other: growing on-chip storage (area ↑, cost ↑) reduces the off-chip bandwidth needed (packaging cost ↓) — so both must be reported for the total cost picture.
What the money buys. Area (with its process node) prices the die’s compute and storage; off-chip bandwidth prices the package, board, and transceivers around it.
Reducing cost is not the objective by itself — the objective is to maximize throughput or energy efficiency per cost: inferences per second per . That connects cost straight back to utilization (§3.2): every PE bought must be kept busy, and every byte of on-chip memory must earn its area by cutting off-chip accesses — i.e., raising operations per off-chip access, exactly the roofline lens of Figure 3.1.
3.5 Flexibility
Flexibility flexibility The range of DNN models a processor supports — functionally (tier 1) and while maintaining efficiency (tier 2). defined in ch. 3 — open in glossary is the range of DNN models a processor can support — together with the ability of the software environment (the mapper mapping The placement and scheduling, in space and time, of every MAC operation and its operand delivery onto the PEs — a compiler for DNN hardware. defined in ch. 3 — open in glossary ) to maximally exploit the hardware for any desired model. With DNN research moving fast, support comes in two tiers:
- Tier 1 — functional: the model runs on the hardware at all.
- Tier 2 — efficient: the hardware maintains high throughput and energy efficiency across different models.
Tier 2 is where designs fail quietly: efficiency must not depend on model properties that cannot be guaranteed. The book’s example: an accelerator that is efficient only when the entire model’s weights fit on-chip performs terribly the day models outgrow it — and model sizes have kept growing (the full book’s §2.4.1). A flexible processor handles even models that exceed on-chip memory gracefully:
Flexibility is a trade with cost: supporting a wider range of workloads (or accelerating them) usually needs added hardware, and specialization is where efficiency comes from — so the objective is to minimize the overhead of flexibility while keeping efficiency across the range. The extra hardware must be a net benefit across multiple workloads. And the workload range keeps widening, because the very techniques that make models efficient also make them diverse: varied layer shapes (efficient network architectures — full book Ch. 9), varied precision per layer and data type (Ch. 7), varied degrees and structures of sparsity sparsity The fraction of zeros in weights/activations; exploiting it means skipping ineffectual MAC operations. defined in ch. 3 — open in glossary (Ch. 8), plus layer types and computations beyond MACs (e.g., activation functions). Getting real benefit from each technique needs hardware support — again only worth it if overhead < gain: a hardware / DNN-model co-design argument.
One more wrinkle: today’s mappers behave like static per-layer compilers. As DNN models become dynamic — changing with the input at runtime — mapping must adapt dynamically while still exploiting the hardware fully.
How to assess flexibility
Evaluate efficiency (inf/s, inf/J) across a wide model range — MLPerf mlperf Industry/academic benchmark suite of well-studied DNN models for fair comparison of DNN hardware and frameworks, for training and inference. defined in ch. 3 — open in glossary as a base, extended with workloads representing efficient techniques (compact architectures, reduced precision, sparsity), matched to the target application. Ideally also publish the range and limits of what the platform supports efficiently: max weights per filter or model, minimum sparsity and its required structure, precision levels (8/4/2/1-bit), layer and activation types.
3.6 Scalability
Scalability scalability How well a design scales throughput and efficiency when resources (PEs, storage, chiplets) are replicated without redesign. defined in ch. 3 — open in glossary asks: if resources are scaled up — more PEs, more on-chip storage, and lately multi-chip chiplets or even wafer-scale chips — do throughput and energy efficiency scale with them without significant redesign (replication only)? Redesign is expensive in time and cost; a scalable design serves low-cost embedded devices and cloud performance parts from one design effort.
The ideals and the realities:
- Throughput would ideally scale linearly and proportionally with #PEs. In practice, utilization of PEs drops at scale and data movement over longer interconnects costs more.
- Energy efficiency would improve with more on-chip storage — likely nonlinearly: an abrupt jump when the whole model suddenly fits on-chip.
- Cost-efficiency at scale: with linear scaling, inferences/s per $ stays constant; it can even improve super-linearly with PE count thanks to increased data sharing across PEs.
What “scalable” really buys
Scalability converts one design effort into a product family: the same architecture, replicated, spans edge to cloud. The evaluation is done under a no-redesign assumption — if scaling up requires re-architecting the NoC or memory system, that cost lands on every future product.
3.7 Interplay Between Different Metrics — report them all
Each metric so far is necessary; none is sufficient. The book’s two strawmen show how omitting one metric games the rest:
§3.7’s two cautionary designs. Any single metric can look excellent if a different one silently absorbs the damage.
The test setup must be reported too: measured silicon or simulation? If simulation — synthesis or post place-and-route, and at what library corner (process corner, supply voltage, temperature)? And how many images were tested?
Finally, the book’s six-step viability evaluation, walked through on a candidate design:
1Accuracy — can it do the task at all?
Gate 1: accuracy on the given task and dataset (say, ImageNet-class detection at the application’s required mAP). Fail here and nothing else matters — this is what Strawman A hides.
| Report | Because | |
|---|---|---|
| Accuracy | accuracy + task + dataset + model | blocks Strawman A |
| Throughput & latency | measured runtime + batch size | batching hides latency |
| Energy & power | chip AND off-chip energy/power, or bytes moved per inference | blocks Strawman B |
| Cost | area+process (or #PEs, buffer KB) + off-chip Gbps | area ⇄ bandwidth trade |
| Flexibility | efficiency across many models + supported limits | brittle designs cliff |
| Scalability | metrics vs #PEs / storage as they scale | one design, many products |
| Test setup | measured vs simulated (synthesis / post-P&R, corner), #images | comparability |
Check yourself
Check yourself — Cost, Flexibility, Scalability & Interplay
1.A paper wants to convey hardware cost without naming a process technology. Per the book, what should it report as proxies?
2.An accelerator is blazingly efficient whenever a DNN's weights all fit on-chip. The book flags this as a flexibility failure. Why?
3.Predict: a design is replicated from 1,000 to 4,000 PEs with no other changes. Per §3.6, what does ideal vs realistic scaling look like?
4.One of the book's 'gaming the metrics' strawmen: a chip of only multipliers reports low cost, high throughput, high accuracy, and low chip power. What single omitted metric hides the trick?
5.Order the book's six-step viability evaluation for a DNN system, starting from the first gate: