§10.3–10.4Processing in Sensor & in the Optical Domain

The chapter closes with the memory-wall playbook applied twice more: first to the other end of the data pipeline — the sensor where the inputs are born — and then to a different carrier of information entirely: light.

10.3 Processing in Sensor

In applications like image processing, moving data out of the sensor can account for a significant share of total energy. The principle that brought compute near the memory (where the weights live) applies verbatim to the sensor (where the inputs are collected): reduce the data read out, and with it the number of expensive ADC conversions. And the price is the same too — computation slides into the analog domain, inheriting its sensitivity to circuit non-idealities. Where near/in-memory processing cut weight movement, near/in-sensor processing cuts input movement.

Two depths of integration:

• Processing near sensor — analog computation in the image sensor’s periphery, before the ADC: switched-capacitor circuits performing 4-bit multiplications (Zhang et al.) and 3-bit × 6-bit MACs (Lee et al.) — both measured on fabricated chips — and RedEye, which performs the entire first convolution layer (convolution, max pooling, quantization) in analog before the ADC (simulation-based).
• Processing in sensor processing in sensor Moving early DNN computation before or inside the image sensor's ADC (switched-capacitor MACs, Angle Sensitive Pixels) to cut input-data movement. defined in ch. 10 — open in glossary — computation inside the pixel array itself: an Angle Sensitive Pixels (ASP) sensor computes the gradient of the input in-pixel, which together with compression cuts data movement out of the sensor by 10×. And because the first layer of a DNN often outputs a gradient-like feature map, the first layer’s computation may be skipped entirely — saving its energy as well.

10.4 Processing in the Optical Domain

The most radical entry: replace electrons with photons. Two physical gifts motivate optical neural networks optical neural network DNN processing with photons: passive multiplication via interferometer meshes or free-space coherent detection; energy goes to electro-optical conversion, and optical non-idealities cost accuracy. defined in ch. 10 — open in glossary (photonic accelerators): photons travel much faster than electrons, and the cost of moving a photon can be independent of distance — the exact opposite of Eq. 3.8’s wire-capacitance tax. Moreover, multiplication can be performed passively — by optical interference, reconfigurable filters, or static phase masks — and photodetection runs at over 100 GHz. Most work targets matrix multiplication, in two contrasting architectures — a programmable interferometer mesh, and free-space coherent detection coherent detection Optical scheme encoding weights and activations on-the-fly into light interfered on a beamsplitter — an output-stationary optical dataflow without on-chip interferometers. defined in ch. 10 — open in glossary :

The energy story mirrors PIM’s periphery problem: computation in the optical domain has negligible power loss — the power is spent converting between electrical and optical domains (generating the light, detecting the photons). So, exactly as with ADC/DAC amortization, the larger the matrix per conversion, the better the economics.

And the accuracy story mirrors PIM too: optical non-idealities — crosstalk between detectors, errors in phase encoding, photodetection noise — degrade DNN accuracy. Bernstein et al.’s response is a hybrid: do the data transfer optically (exploiting distance-independent photon cost) but the MAC computation digitally in the electrical domain, avoiding the analog-optical arithmetic entirely.

Check yourself