Throughput and Latency · Efficient DNN Processing

Section 3.1 asked whether the system is good enough; this section asks whether it is fast enough — and immediately splits “fast” into two different quantities that are routinely, and wrongly, treated as one.

Two different meanings of “fast”

Throughput is the amount of data processed, or the number of task executions completed, per unit time. It is what matters when volume is the goal: processing video at 30 frames per second is the bar for real-time performance; in data analytics, higher throughput means more data analyzed in the time available — increasingly urgent as visual data grows exponentially and analysis must trigger action (security screening, medical diagnosis, drug discovery). For DNN inference it is generically reported as operations per second, or at task level as inferences per second (equivalently, runtime in seconds per inference).

Latency is the time between input data arriving and the result being generated, reported in seconds. It is what matters when a response is time-critical: augmented reality, autonomous navigation, robotics.

Throughput ≠ latency: the batching lesson

The two are often assumed to be two views of one number — throughput as 1/latency. They are not, and batching is the canonical counterexample. Processing many inputs together amortizes per-batch overhead such as loading weights once for all inputs, so throughput rises. But inputs arriving at 30 fps must wait for the batch to fill: with a batch of 100, the earliest frames wait 100/30 ≈ 3.3 seconds before processing even begins — unacceptable for high-speed navigation, where it eats the time available for course correction. High throughput and low latency can therefore be at odds, and both must be reported.

Aside — Little’s Law (the book’s footnote 4)

Queuing theory compresses this into Little’s Law :

$\overline{\text{throughput}} = \frac{\overline{\text{tasks-in-flight}}}{\overline{\text{latency}}}$

In DNN terms: throughput in inferences/second, latency in seconds, and inferences-in-flight = the images being processed simultaneously (the batch). Raising tasks-in-flight raises throughput only if latency doesn’t rise proportionally — but batching raises both, which is why it can be counterproductive for latency-bound uses.

Batching lab — watch throughput and worst-frame delay move in opposite directions

\text{worst frame delay} \;=\; \underbrace{\tfrac{B}{\text{fps}}}_{\text{waiting for batch to fill}} \;+\; \underbrace{t_{\text{load}} + B\,t_{\text{proc}}}_{\text{processing the batch}}

term	meaning	units	dictated by
$B$	batch size — inputs processed together	frames	system config
$\text{fps}$	input arrival rate (camera)	frames/s	application
$t_{\text{load}}$	fixed overhead paid once per batch (e.g., loading weights)	ms	hardware
$t_{\text{proc}}$	processing time per frame once weights are resident	ms	hardware + model

batch size B1input frame rate30 fpsper-batch overhead t_load80 msper-frame compute t_proc5 ms

throughput (batch amortized)11.8 inf/s

worst frame delay0.118 s

Worked example: at fps = 30 and B = 100, the fill-wait alone is 100/30 ≈ 3.3 s — the book’s example. Throughput saturates near 1/t_proc as the per-batch overhead amortizes away, while delay keeps climbing linearly with B. The two curves share one log axis so you can watch both trends at once (units differ — read each against its own label).

Decomposing inferences per second

What produces throughput? The book builds it as a chain of factors. First (Eq. 3.1), inferences per second splits into a hardware-and-model term and a model-only term:

$\frac{\text{inferences}}{\text{second}} = \frac{\text{operations}}{\text{second}} \times \frac{1}{\frac{\text{operations}}{\text{inference}}} \tag{3.1}$

Operations per second is dictated by both the hardware and the DNN model; operations per inference is dictated by the DNN model alone (its layers and shapes decide how much work one inference takes). Since the dominant DNN operation is the MAC , the book uses “operations” and “MAC operations” interchangeably.

Next (Eq. 3.2), consider hardware built from many processing elements (PEs) — each a primitive core performing a single MAC:

$\frac{\text{operations}}{\text{second}} = \underbrace{\frac{1}{\frac{\text{cycles}}{\text{operation}}} \times \frac{\text{cycles}}{\text{second}}}_{\text{for a single PE}} \times \;\text{number of PEs} \times \text{utilization of PEs} \tag{3.2}$

The first term is the peak throughput of one PE, the second the parallelism, the last the degradation from failing to keep those PEs busy. Multiplied out with all PEs computing every cycle (utilization = 100%) you get peak performance — the number on the datasheet. Explore how the factors trade:

Eq. 3.1 × 3.2 — the throughput machine, end to end

\frac{\text{inf}}{\text{s}} = \left(\frac{1}{\text{cyc/op}} \times f_{clk} \times N_{PE} \times U\right) \times \frac{1}{\text{ops/inf}}

term	meaning	units	dictated by
$\text{cyc/op}$	cycles one PE needs per MAC (1 if pipelined; more for a multi-cycle MAC)	cycles	hardware
$f_{clk}$	clock frequency — set by the critical path	cycles/s	hardware
$N_{PE}$	number of PEs — parallel MACs available	PEs	hardware
$U$	utilization of PEs — fraction of peak actually achieved	0–1	hardware + model
$\text{ops/inf}$	MAC operations one inference requires	MACs	DNN model

cycles per operation1clock f_clk1 GHznumber of PEs168utilization U0.8MACs per inference724M

peak performance168G MAC/s

achieved ops/sec134G MAC/s

inferences / second186 inf/s

Worked example (the initial setting): a pipelined MAC (1 cycle/op) at 1 GHz across 168 PEs — the PE count of the Eyeriss chip this book’s authors built — peaks at 168 GMAC/s; at 80% utilization it achieves ~134 GMAC/s, and a 724M-MAC inference (AlexNet-class CONV layers) runs at ~186 inf/s. Now drop utilization to 20% and watch the datasheet number stop mattering.

Raising each factor — and what it costs

Peak of one PE. More cycles per second means a higher clock frequency, won by shortening the critical path at the circuit or micro-architectural level; fewer cycles per operation comes from MAC design (a non-pipelined multi-cycle MAC has more cycles per op).
More PEs. The count is set by the PE’s area density times the system’s area cost. With chip area fixed, more PEs means shrinking each PE, or trading on-chip storage area for compute — which can backfire by lowering utilization (less local data → more starvation). Density also improves by sharing one piece of control logic across many MACs, exactly the trick instruction-based CPUs/GPUs use with aggregate instructions (SIMD / vector, SIMT / tensor): one instruction, many operations, less bookkeeping overhead per MAC.
Utilization — the subtle one, decomposed next.

The compute-vs-storage area trade-off: number of PEs and utilization of PEs pull on the same silicon budget. Neither extreme wins by itself — Eq. 3.2 multiplies them.

Where utilization goes

Utilization itself factors (Eq. 3.3):

$\text{utilization of PEs} = \frac{\text{number of active PEs}}{\text{number of PEs}} \times \text{utilization of active PEs} \tag{3.3}$

The first term is distribution: how many PEs receive work at all. It is limited by the flexibility of the architecture — for instance the on-chip network — to support the DNN model’s layer shapes, and within that, by the mapping : the placement and scheduling in space and time of every MAC (and its operand delivery) onto PEs — think of it as a compiler for DNN hardware. (The full book designs on-chip networks and mappings in its Chapters 5–6, outside this excerpt.)

The second term is feeding: active PEs still idle if data doesn’t arrive on time. That depends on the bandwidth and latency of on- and off-chip memory and the network, which in turn depend on the data reuse available in the model and how much of it the memory hierarchy and dataflow (the order of operations and where data lives) can exploit. Larger batch sizes raise reuse — the other reason batching helps throughput. Finally, load imbalance hurts: when exploiting sparsity , PEs that drew less work finish early and sit idle.

Open the box: where a 16-PE array loses its utilization

1The datasheet fantasy: 16/16 PEs computing

Peak performance assumes every PE performs a MAC every cycle — utilization 100%. This number only depends on the hardware: PEs × per-PE peak.

step 1 / 5

The roofline model

The interplay of compute peak and data delivery is captured by the well-known roofline model (Figure 3.1), which relates average bandwidth demand and peak computational ability to achievable performance. On the x-axis is operational intensity — how many operations the workload performs per byte of data moved. At low intensity performance rises along the BW-limited slope (slope = bandwidth); past the inflection point it flattens at the computation-limited peak. The design goal: operate as close to the roof as possible at your workload’s intensity. The impact of all the §3.2 factors together can be captured by Eyexam , the authors’ systematic framework for understanding a DNN processor’s performance limits as a function of the DNN model’s and accelerator’s characteristics — it includes and extends the roofline model (its full treatment is the book’s Chapter 6, outside this excerpt).

Figure 3.1, live — the roofline model

\text{achievable}\; \frac{\text{ops}}{\text{s}} = \min\!\big(\text{peak},\; \text{BW} \times \text{op. intensity}\big)

term	meaning	units	dictated by
$\text{peak}$	peak performance — the flat roof (all PEs busy)	Gops/s	hardware
$\text{BW}$	memory bandwidth — the slope of the left ramp	GB/s	hardware
$\text{op. intensity}$	operations per byte moved — where the workload sits on the x-axis	ops/byte	model + dataflow

peak performance1k Gops/smemory bandwidth25 GB/sworkload op. intensity10 ops/byte

achievable at this intensity250 Gops/s

inflection point (optimal op. int.)40 ops/byte

Both axes are log-scale, as rooflines are usually drawn. Left of the inflection (marker in the sloped region): BW-limited — raising peak does nothing; raise BW or increase reuse to move right. Right of it: computation-limited — now peak (PEs, clock) is what binds. Eyexam, the authors’ analysis framework (full book Ch. 6, not in this excerpt), includes and extends this model to pinpoint which factor caps a specific design.

The model side of Eq. 3.1 — and why the two sides interact

Operations per inference is the DNN model’s lever: efficient layer shapes (“efficient network architectures”, the full book’s Chapter 9) cut MACs per inference. But the same exotic shapes can map poorly onto the PE array — poor utilization — reducing operations per second in Eq. 3.2. Cutting the numerator can shrink the denominator’s factors: the two sides of Eq. 3.1 are coupled through the hardware.

Not all operations are real work: sparsity

A deeper refinement: cycles per operation need not be constant across operations. Anything multiplied by zero is zero, so a MAC with a zero operand is ineffectual — it cannot change the accumulated value. How many MACs are ineffectual depends on the DNN model and the input data. Ideal hardware would skip them all (“exploiting sparsity”); real hardware might only recognize zeros in one operand (say, weights), so ineffectual operations split into exploited (skipped) and unexploited (still performed). What the hardware actually executes = effectual + unexploited ineffectual operations. Eq. 3.4 turns this into a throughput statement:

$\frac{\text{ops}}{\text{cycle}} = \underbrace{\frac{\text{eff.} + \text{unexp. ineff. ops}}{\text{cycle}}}_{\text{① hw constant}} \times \underbrace{\frac{\text{eff. ops}}{\text{eff.} + \text{unexp. ineff. ops}}}_{\text{② hw skip ability}} \times \underbrace{\frac{1}{\frac{\text{eff. ops}}{\text{ops}}}}_{\text{③ 1 / density}} \tag{3.4}$

① is roughly fixed for a given accelerator (how many operations it performs per cycle); ② is the hardware’s ability to avoid wasting slots on ineffectual work (ideally 1 — zero unexploited); ③ blows the count up from performed to total operations — the sparser the model ( density ↓), the more total ops each cycle covers, raising ops/cycle and hence ops/sec in Eq. 3.2.

Eq. 3.4 lab — how sparsity (and the hardware’s skip ability) multiply throughput

\frac{\text{ops}}{\text{cycle}} = A \times r \times \frac{1}{d}

term	meaning	units	dictated by
$A$	① performed ops per cycle (effectual + unexploited ineffectual)	ops/cycle	hardware
$r$	② effectual / performed — 1 means every performed op was real work	0–1	hardware
$d$	③ density: effectual / total ops (= 1 − sparsity)	0–1	model + input data

performed ops/cycle A256skip ability r1density d (1 − sparsity)1

(total) ops per cycle256

speedup vs dense (d = 1, r = 1)1 ×

Worked example: A = 256, perfect skipping (r = 1), 60% zeros (d = 0.4) → 256 × 1 / 0.4 = 640 total ops/cycle — a 2.5× speedup without adding a single PE. But r < 1 (hardware that only detects zero weights, not zero activations) eats directly into the gain.

The catch: skipping needs detection logic. That extra hardware can lengthen the critical path (cycles per second ↓) and fatten each PE (area density ↓ → fewer PEs) — both reducing ops/sec in Eq. 3.2. Whether sparsity exploitation nets out positive is a genuine design trade-off, not a free lunch. (The full book’s Chapter 8 is devoted to it.)

Reduced precision plays the same both-sided game (its Chapter 7): fewer bits per operand cuts the memory bandwidth each operation needs (PEs starve less → utilization ↑) and shrinks each PE (more PEs per mm²) — but supporting multiple precisions adds hardware that can again slow the clock and cost PE density.

Who controls what — Table 3.1

Every factor above belongs to the hardware, the DNN model, or the input data — and improving one often perturbs another. The book’s Table 3.1, cell-clickable:

Table 3.1 — Classification of factors that affect inferences per second
	Hardware	DNN Model	Input Data
operations per inference		✓
operations per cycle	✓
cycles per second	✓
number of PEs	✓
number of active PEs	✓	✓
utilization of active PEs	✓	✓
effectual ops out of (total) ops		✓	✓
effectual + unexploited ineffectual ops per cycle	✓

Dotted-underlined cells have explanations — click one.

Why #MACs is not a proxy for speed — Figure 3.2

The book closes with measured evidence: latency of many DNN models on a Pixel phone, plotted against their MAC counts. If #MACs predicted speed, the points would line up. They don’t:

Figure 3.2 (recreated) — #MACs vs measured latency on a Pixel phone; models from [119]

Click a point to see what it means.

The model affects MACs per inference (shapes, sparsity), but its achieved speed depends on whether the hardware can exploit those properties without giving back utilization, PE count, or clock. Hence the book’s advice: it is often more effective to design DNN models with hardware in the loop (its Chapter 9’s topic).

Reporting rules for throughput & latency

Neither #MACs (model side) nor #PEs × peak (hardware side) suffices. Report measured runtime on clearly specified, well-studied models — ideally the MLPerf suite — and always report batch size alongside throughput, so readers can judge the latency that came with it.

Check yourself

Check yourself — Throughput and Latency

1.A camera feeds frames at 30 fps into an accelerator that processes them in batches of 100. Predict the minimum extra delay some frames experience just from batching.

2.In Eq. (3.2), doubling the number of PEs failed to double a chip's real inferences/sec. Which factor in the equation most likely absorbed the gain?

3.In the roofline model, a workload sits well left of the inflection point (low operational intensity). What limits its performance, and what would help?

4.A model has 60% zeros in its operands (only 40% of MACs are effectual), and the hardware skips ALL ineffectual MACs with no overhead. Per Eq. (3.4), what happens to (total) operations per cycle?

5.Figure 3.2 plots #MACs vs measured latency for many DNN models on a phone. Models with similar #MACs showed a 2× range in latency. Why?

5 questions