§10.2.1Processing in Memory: Fundamentals & NVM

§10.1 moved compute near the memory. Processing in memory processing in memory Performing MACs inside the weight memory itself: inputs drive word lines, storage elements multiply, bit lines accumulate; only results are read out (PIM / in-memory computing). defined in ch. 10 — open in glossary (PIM, also “in-memory computing”) goes further: it moves the compute into the memory that stores the weights. This page covers the general PIM architecture — the crossbar model that every later subsection builds on — and its first technology instantiation, non-volatile memories.

Conventional vs in-memory — what stops moving

DNN processing can be phrased as matrix–vector multiplication (the full book’s Chapter 4 shows FC layers as one big matrix–vector product, and CONV layers reduced to it via the Toeplitz expansion — glossed here where needed). In a conventional design, both operands travel: the input-activation vector and the weight matrix are read from their memories into a MAC array. The number of weights readable at once is capped by the memory interface (read-out logic, ports) — capping operations per cycle, and with it throughput.

PIM’s move (Figure 10.2): put the compute inside the weight memory — weights are never read out. Only the inputs flow in and only computed results (partial sums or output activations) flow out. Two wins:

1. Weight movement disappears — the cost of reading the A×B weight matrix is gone.
2. Weight bandwidth explodes — parallel access is no longer limited by the memory interface; in theory the whole matrix participates at once.

The crossbar: Figure 10.3, live

The typical PIM design is a weight-stationary dataflow dataflow The order of operations and where data is stored and reused in a DNN accelerator; determines how often each piece of data moves. defined in ch. 3 — open in glossary : weights sit still, inputs stream past them. Word lines (WLs) word line The row wire of a memory array; in processing-in-memory it delivers the input activation (voltage or pulse train) to every storage element in the row (WL). defined in ch. 10 — open in glossary deliver the input activations to the storage elements; bit lines (BLs) bit line The column wire of a memory array; in processing-in-memory it accumulates per-cell products (current or charge) into a partial sum read by the ADC (BL). defined in ch. 10 — open in glossary collect the computed partial sums. Each storage element performs a multiplication; the accumulation across all elements on a column happens on the wire. An array of B rows × A columns can, in theory, use all A×B weights simultaneously: A dot products of B elements — A×B MACs — in one cycle. Step it yourself (the numbers are the book’s own toy example from Figure 10.7):

Three more properties of this array, from the book:

• Input reuse for free: each input activation on a word line reaches all A columns — up to A-fold reuse without re-reading the input.
• Tiny “MACs”: a storage element is 10–100× smaller in area (3–10× in edge length) than digital MAC logic, so the wires that deliver inputs are shorter — less routing capacitance, cheaper input delivery.
• Converters at the edges: depending on input/output formats, DACs dac Digital-to-analog converter — in PIM it drives inputs onto word lines; its cost scales with input precision, hence 1-bit DACs plus time coding in practice. defined in ch. 10 — open in glossary drive the word lines (cost scales with input precision) and ADCs adc Analog-to-digital converter — in PIM it resolves the accumulated bit-line value. Its bit count, area (pitch-matching), power, and thermal noise are the central peripheral bottleneck. defined in ch. 10 — open in glossary resolve the bit lines (cost scales with psum precision, which grows with weight/input precision and with B, the number of values accumulated).

The same machine as a loop nest

The full book (its Chapter 5) writes dataflows as loop nests — parallel-for loops run spatially (all iterations at once, in hardware), plain for loops run temporally (one iteration per cycle). An FC layer with M output channels and inputs flattened to length CHW becomes Design 20 — note there is no temporal loop: everything happens in one cycle.

i = Array(CHW)          # input activations
f = Array(M, CHW)       # filter weights
o = Array(M)            # output partial sums

parallel-for m in [0, M):
    parallel-for chw in [0, CHW):
        o[m] += i[chw] * f[m, chw]

Mapping constraints: A ≥ M (a column per output channel) and B ≥ CHW (a row per input element). When they don’t hold with equality, part of the array idles — the under-utilization story of 10.2.4. A 1-D convolution (Design 21) adds one temporal loop over output positions q; each cycle uses all M×S×C weights, reuses each input activation C×S times, and accumulates M psums — with the same input value delivered again for different qs (the Toeplitz-expansion effect: convolution re-reads shifted input windows):

i = Array(C, W)         # input activations
f = Array(M, C, S)      # filter weights
o = Array(M, Q)         # output partial sums

parallel-for m in [0, M):
  parallel-for s in [0, S):
    parallel-for c in [0, C]:
      for q in [0, Q):            # temporal — Q cycles
        w = q + s
        o[m, q] += i[c, w] * f[m, c, s]

Constraints here: A ≥ M and B ≥ C × S.

10.2.1 Non-volatile memories: physics does the MAC

NVM brings the density argument: storage-plus-compute packed at roughly DRAM density, dense enough to hold entire models and replace off-chip memory. The storage elements are programmable resistive devices — memristors memristor A programmable resistive NVM element: conductance G encodes a weight, so Ohm's law I = V·G multiplies and Kirchhoff current summing accumulates. defined in ch. 10 — open in glossary (as cross-point elements they can even sit at wire crossings without an access transistor).

The multiplication is Ohm’s law; the accumulation is Kirchhoff’s current law — the current-based approach current-based pim PIM style that multiplies via device I-V characteristics and accumulates by current summing on the bit line (NVM, 6T SRAM); sensitive to nonlinearity and variations. defined in ch. 10 — open in glossary :

NVM’s three unique challenges

1. Writes are expensive. Programming memristors costs far more than SRAM/DRAM writes — so designs hold all weights resident rather than swapping per layer (the book likens it to FPGA designs like Brainwave and FINN that pin weights at synthesis). Chapter 3’s flexibility flexibility The range of DNN models a processor supports — functionally (tier 1) and while maintaining efficiency (tier 2). defined in ch. 3 — open in glossary warning applies: model size becomes bounded by array capacity.
2. Devices vary and bend. Device-to-device and cycle-to-cycle variations plus nonlinear conductance limit precision to 1–4 bits per device and shape the signaling: instead of encoding inputs as voltage amplitudes, many designs encode them in time — pulse-width modulation pulse-width modulation Encoding an input activation in time at fixed voltage (unary pulses) so a 1-bit DAC suffices; robust to nonlinearity, but costs cycles. defined in ch. 10 — open in glossary at fixed voltage (unary coding), integrating current on a capacitor.
3. No negative resistance. Signed weights need differential signaling — two storage elements (often two whole arrays) per weight — or must be avoided: binary weights as [0,1] with XNOR (or NAND) logic replacing multiplication (reduced-precision tricks from the full book’s Chapter 7).

The book points to in-depth treatments of how these device properties affect DNN processing — and notes work that flips the problem: design the device so it suits DNN processing (including in-place, parallel weight updates for training).

Check yourself