§10.1Processing Near Memory

Chapter 3 ended with a fact that dominated every energy discussion since: data movement, not arithmetic, consumes the energy — both in accessing the memory and in transferring the data. The same physics limits the bandwidth between memory and compute, and with it the whole system’s throughput. Computer architects call this the memory wall memory wall The energy and bandwidth bottleneck of moving data between off-chip memory (e.g., DRAM) and the processor. defined in ch. 10 — open in glossary (specifically: the wall between off-chip memory like DRAM and the processor). Where the full book’s Chapters 5–6 fight it architecturally — spatial architectures that scatter on-chip memory into the PEs — this chapter fights it with technology: advanced memory, process, and fabrication techniques that physically bring compute and memory together.

The chapter’s tour, in order of increasing radicalism:

1. Processing near memory (this section) — keep the compute digital, move the high-density memory next to it: embedded DRAM, 3-D stacks.
2. Processing in memory (§10.2) — move the compute into the memory array, accepting analog computation and its non-idealities.
3. Processing in sensor (§10.3) — same idea aimed at the input side: compute where the data is captured.
4. Processing in the optical domain (§10.4) — photons travel fast and (nearly) for free with distance.

Why off-chip memory is expensive — and what “near” buys

High-density memories typically need a different process technology than logic, so they’re fabricated as separate chips — and every access goes off-chip. Two physical limits follow:

• I/O pads: a chip has only so many pins, capping the width of the data bus (bits per cycle).
• Channel characteristics: the off-chip interconnect’s resistance, inductance, and capacitance limit signaling frequency — and its capacitance is exactly what Eq. 3.8 charges energy for.

Processing near memory processing near memory Bringing digital compute physically close to high-density memory (eDRAM, 3-D stacks) to shorten interconnect: lower access energy, wider and faster interfaces. Also called near-data processing. defined in ch. 10 — open in glossary overcomes both by shortening the wire: access energy falls with interconnect length (less capacitance), and bandwidth rises — primarily because a wider interconnect becomes possible (more bits per cycle, no pad limit), and secondarily because the shorter channel supports a higher clock.

Table 10.1 — the systems, side by side

Tens of megabytes to gigabytes, at tens to thousands of GB/s — and note the Evaluation column: most of these results are simulated, not measured silicon (a §3.7 test-setup reporting point the book makes explicitly).

10.1.1 Embedded high-density memories

The first rung: fabricate the dense memory inside the logic die. eDRAM edram Embedded DRAM fabricated on the compute die — tens of MB on-chip; denser than SRAM but costlier to fabricate and scaling more slowly. defined in ch. 10 — open in glossary is the veteran — DaDianNao uses 16×2 MB eDRAM tiles for weights plus 2×2 MB for input/output activations, all accessible in parallel through 4096-bit rows for extreme bandwidth. Its costs: lower density than off-chip DRAM, higher fabrication cost, and reportedly slower scaling than SRAM — so its density edge erodes over generations. Embedded non-volatile memories (eFlash, MRAM, RRAM, PCRAM) have accordingly gained popularity: denser, non-volatile, and free of eDRAM’s refresh/leakage standby power.

10.1.2 Stacked memory (3-D memory)

The second rung: keep memory and logic as separate dies, but stack them in one package (hence also “in-package memory”).

• DRAM on logic via TSVs through-silicon via A vertical electrical connection through a die that enables 3-D stacking, with much lower capacitance than off-chip traces (TSV). defined in ch. 10 — open in glossary — commercialized as Hybrid Memory Cube (HMC) and High Bandwidth Memory (HBM). Order-of-magnitude bandwidth gain, up to 5× lower access energy than 2-D DRAM, because TSVs carry far less capacitance than off-chip interconnect. Neurocube gives each DRAM vault (vertically stacked banks) its own PE (buffer + MACs) and connects PEs with a 2-D mesh NoC — making where to place weights and activations across vaults the central design decision, since it sets NoC traffic. Tetris pairs HMC with the Eyeriss spatial architecture and its row-stationary dataflow dataflow The order of operations and where data is stored and reused in a DNN accelerator; determines how often each piece of data moves. defined in ch. 3 — open in glossary , and — now that off-chip is cheap — allocates more area to PEs and less to the on-chip buffer, adapting the dataflow to match (the §3.2 area trade, re-decided under new physics).
• SRAM on logic via stacking stacked memory Memory dies stacked on the logic die, connected by TSVs (HMC, HBM) or inductive-coupling TCI — roughly 10× bandwidth and up to 5× lower access energy than 2-D DRAM. Also called 3-D or in-package memory. defined in ch. 10 — open in glossary with TCI — Quest stacks eight SRAM dies (96 MB) holding weights and intermediate activations for layer-by-layer processing; stacked SRAM offers ~10× lower latency than stacked DRAM. The dies communicate by ThruChip Interface (TCI): inductive-coupling die-to-die wireless links, cheaper to integrate than TSVs.
• Monolithic 3-D integration monolithic 3-d integration Fabricating thin layers of logic and memory directly on top of each other, connected by interlayer vias (ILVs) that are orders of magnitude denser than TSVs. defined in ch. 10 — open in glossary — the frontier: fabricate thin layers of memory devices directly on top of logic, connected by interlayer vias (ILVs) with several orders of magnitude denser vertical connectivity than TSVs. N3XT combines on-chip NVM (RRAM, STT-MRAM/MRAM, PCRAM) with carbon-nanotube logic (CNFETs); simulations put ILV energy-delay up to two orders of magnitude below 2-D systems on DNN workloads (vs 8× for TSV) — with the biggest wins where data reuse is low (FC layers at small batch) and traffic across the vias is heaviest.

What near-memory changes for the architect

The book closes the section with the systems-level punchline: the new memories are faster but smaller, so the optimal mappings mapping The placement and scheduling, in space and time, of every MAC operation and its operand delivery onto the PEs — a compiler for DNN hardware. defined in ch. 3 — open in glossary change — and with them a chain of design decisions:

• data allocation — access patterns are constrained (banks in DRAM, vaults in stacks), so where each tensor lives matters;
• NoC design between memory and PEs;
• area split between on-chip memory and compute, now that off-chip communication is cheap (Tetris chose more PEs, smaller buffer);
• memory hierarchy and dataflow, re-optimized for the new movement costs.

Check yourself