Fast is not enough — the inference also has to be affordable in joules and in watts. This section separates those two ideas, decomposes them the same way §3.2 decomposed throughput, and lands on the number that dominates everything: the energy cost of moving data.
Energy efficiency vs power consumption
Energy efficiency energy efficiency Task executions per unit energy; for DNN inference, inferences per joule (or its inverse, joules per inference). defined in ch. 3 — open in glossary is the amount of data processed or task executions completed per unit of energy — generically operations per joule, at task level inferences per joule (or its inverse, joules per inference). It rules at the edge: smartphones, smart sensors, robots, and wearables run on limited batteries, and edge processing is often preferred over the cloud for latency, privacy, or communication-bandwidth reasons.
Power consumption power consumption Energy consumed per unit time (watts); together with energy efficiency it bounds achievable throughput. defined in ch. 3 — open in glossary is energy per unit time (watts). Power turns into heat, so its ceiling is a design criterion: the thermal design power (TDP) thermal design power The power a system's cooling is designed to dissipate (TDP); the ceiling on sustained power consumption. defined in ch. 3 — open in glossary — the power the cooling system is built to dissipate. Power binds in the cloud (data centers have stringent power ceilings driven by cooling costs) and in the hand: users are sensitive to heat, and a phone’s or wearable’s form factor rules out fans.
Where each constraint binds. Power limits sustained speed through heat; energy limits how much total work a battery charge buys.
Power and energy efficiency jointly cap throughput (Eq. 3.5):
\frac{\text{inferences}}{\text{second}} \;\leq\; \text{Max}\!\left(\frac{\text{joules}}{\text{second}}\right) \times \frac{\text{inferences}}{\text{joule}} \tag{3.5}
| term | meaning | units | dictated by |
|---|---|---|---|
| power ceiling (TDP): what the cooling / form factor can dissipate | W = J/s | system design | |
| energy efficiency of the whole stack on this workload | inf/J | hardware + model |
Worked example: a 1 W wearable at 20 inf/J sustains at most 20 inf/s. The same model on a 250 W datacenter card at 5 inf/J manages 1250 inf/s — but pays 0.2 J per inference instead of 0.05 J. Improving inf/J raises the ceiling everywhere; that is why the rest of this section chases joules.
Mirroring Eq. 3.1, inferences per joule decompose (Eq. 3.6) into operations per joule — dictated by both hardware and DNN model — times the inverse of operations per inference, dictated by the model:
\frac{\text{inferences}}{\text{joule}} = \frac{\text{operations}}{\text{joule}} \times \frac{1}{\frac{\text{operations}}{\text{inference}}} \tag{3.6}
Where the joules go: Eq. 3.7 and 3.8
Energy per operation splits into moving the data and doing the math (Eq. 3.7):
\text{Energy}_{\text{total}} = \text{Energy}_{\text{data}} + \text{Energy}_{\text{MAC}} \tag{3.7}
and each component’s joules per operation (where an “operation” can be a MAC or a data movement — the book’s footnote 6) follows the classic switching-energy law (Eq. 3.8):
\frac{\text{joules}}{\text{operation}} = \alpha \times C \times V_{DD}^{2} \tag{3.8}
| term | meaning | units | dictated by |
|---|---|---|---|
| switching activity — how often the capacitance is actually charged | 0–1 | circuit + data | |
| total switching capacitance — grows with wire length and structure size | F | hardware | |
| supply voltage — enters SQUARED | V | hardware |
Worked example: α = 0.5, C = 10 fF, V_DD = 1.0 V → 5 fJ per switch-event; drop V_DD to 0.7 V and energy falls to 0.49× (the quadratic term). Now slide C up 100× — that is what happens when the “wire” becomes an off-chip trace to DRAM: no voltage trick rescues a long wire.
Figure 3.3 — the number every accelerator designer memorizes
Why does data movement dominate? Because capacitance grows with distance, and Eq. 3.8 charges you for every switch of it. The book’s Figure 3.3 (energy per operation in a 45 nm process, relative cost on a log scale vs the 8b add):
Figure 3.3 (recreated; adapted by the book from Horowitz, ISSCC 2014). Table and log-scale bars in one view; memory accesses in red, arithmetic in cyan-blue. A 32b DRAM read costs 640 pJ — over 21,000× the 8b add and ~170× the 32b FP multiply.
What this really says
The multiply is not the expensive part of a multiply-accumulate — the fetch is. One careless off-chip read buys you ~170 32-bit FP multiplies. So the architecture question is never “how do I make MACs cheaper?” first; it is “how many times can I reuse a datum before I pay DRAM prices again?” Precision matters for the same reason: an 8b multiply is ~15× cheaper than a 32b FP one, and narrower operands also shrink every memory access that carries them.
The memory hierarchy as an energy ladder
Capacitance — and therefore energy — increases with the distance data travels to reach the PE: out of the memory it sits in, then across the network between that memory and the PE. Larger memories and longer interconnects (especially off-chip) burn more energy per access than smaller, closer ones. The defense is data reuse data reuse Moving a datum once from large distant memory and using it for many operations out of small local memory, cutting data-movement energy and bandwidth. defined in ch. 3 — open in glossary : move each datum once from the distant large memory, then serve the many operations that need it from a small local one. Optimizing this movement is a major consideration in DNN accelerator design — the dataflow dataflow The order of operations and where data is stored and reused in a DNN accelerator; determines how often each piece of data moves. defined in ch. 3 — open in glossary (the full book’s Chapter 5, outside this excerpt) defines the processing order that determines how much reuse the hierarchy can exploit; and advanced device and memory technologies can shrink the switching capacitance between compute and memory outright — that is Chapter 10 of this excerpt.
Shrinking joules per MAC — and the reporting scope
On the arithmetic side, joules per MAC falls by reducing switching activity switching activity α in energy/operation = α·C·V²DD: how often the switched capacitance is actually charged. defined in ch. 3 — open in glossary and/or capacitance at the circuit or micro-architecture level, and — per Figure 3.3 — by reducing precision (fewer bits per operand and operation; the full book’s Chapter 7), remembering the impact on accuracy must be checked (§3.1). For instruction-based CPUs/GPUs, the same aggregate-instruction trick from §3.2 (SIMD / vector, SIMT / tensor) cuts instruction-bookkeeping energy: one instruction, many MACs.
Reporting rule — the scope question
Over what boundary should energy and power be measured? The whole system (fans, power supplies) is beyond the book’s scope; but ignoring off-chip memory access is misleading, since it varies greatly between designs and often dominates. Report both the chip’s energy/power and the off-chip memory’s — or, if no memory technology is specified, the amount of off-chip traffic: total data read and written per inference. And, as always, on clearly specified, widely used models.
Sparsity, one more time — now in joules
Operations per joule decomposes exactly like operations per cycle did in §3.2 (Eq. 3.9): a hardware term (effectual + unexploited ineffectual operations per joule), the hardware’s skip ability, and the model’s density:
\frac{\text{ops}}{\text{joule}} = \underbrace{\frac{\text{eff.} + \text{unexp. ineff. ops}}{\text{joule}}}_{\text{① hw}} \times \underbrace{\frac{\text{eff. ops}}{\text{eff.} + \text{unexp. ineff. ops}}}_{\text{② skip ability}} \times \underbrace{\frac{1}{\frac{\text{eff. ops}}{\text{ops}}}}_{\text{③ 1 / density}} \tag{3.9}
The trap is the same as in §3.2’s Eq. 3.4 — with a sharper edge: the zero-detection hardware itself burns energy on every operation, lowering term ①. Exploiting sparsity is a net win only if the gain from ③ (density ↓) more than offsets the loss in ①.
| term | meaning | units | dictated by |
|---|---|---|---|
| performed ops per joule of the plain (dense) design | Gops/J | hardware | |
| energy overhead of the added zero-detection/skip hardware | 0–1 | hardware | |
| ② effectual / performed — skip ability | 0–1 | hardware | |
| ③ density (1 − sparsity) | 0–1 | model + input |
Predict before you slide: with 15% overhead and r = 0.9, how dense can the model get before the sparse design loses to the plain one? Answer: the curves cross where (1−h)·r/d = 1, i.e. d ≈ 0.77 — any denser and the “optimization” costs energy. This is the book’s net-benefit condition made visible.
Summary — all MACs are not created equal
The number of MAC operations and weights in a model is not sufficient for evaluating energy: it says nothing about where each operand was accessed and how often it was reused, which set operations per joule. Two models with identical MAC counts can differ vastly in energy — hence, as with throughput, it is often more effective to design DNN models with hardware in the loop (the full book’s Chapter 9).
Reporting rules for energy & power
Report chip and off-chip (DRAM) energy efficiency and power — or off-chip bytes per inference if the memory isn’t specified. Evaluate on clearly specified, widely used models ( MLPerf mlperf Industry/academic benchmark suite of well-studied DNN models for fair comparison of DNN hardware and frameworks, for training and inference. defined in ch. 3 — open in glossary ). A chip-only TOPS/W figure on an unnamed workload conveys almost nothing.
Check yourself
Check yourself — Energy Efficiency and Power
1.Per Figure 3.3 (45 nm), a 32b DRAM read costs ~640 pJ while a 32b FP multiply costs ~3.7 pJ. What design conclusion does the book draw from this gap?
2.In Eq. (3.8), joules/op = α·C·V²DD. Predict: lowering supply voltage from 1.0 V to 0.7 V (all else fixed) changes energy per operation by…
3.A paper reports impressive TOPS/W for its chip but is silent about off-chip memory. Why does the book consider this misleading?
4.Power consumption and energy efficiency jointly bound throughput (Eq. 3.5). A wearable capped at 1 W runs a model at 20 inferences/joule. What is the max sustainable throughput?
5.Why are #MACs and #weights not good proxies for energy consumption?