A full-stack search technique for domain optimized deep learning accelerators

Modern models leave a TPU-v3 at 15% utilization — depthwise convolutions take 5% of the math but 65% of the time — so FAST searches the hardware datapath, software schedules, and compiler fusion together, discovering accelerators that swap half their DRAM bandwidth for a huge on-chip buffer and deliver 3.7× performance per watt.

Why this problem is worth solving

Inference at planetary scale changed the economics of hardware specialization. Models serving billions of users run on thousands of accelerators, so even modest efficiency gains fund silicon projects — if the design cost is contained and the design is right for the workload. The paper’s return-on-investment framing makes it concrete: a custom inference accelerator breaks even at roughly 2,164–3,534 deployed chips against the baseline, well within a moderate datacenter deployment — and that analysis conservatively ignores latency-driven revenue (a 500 ms delay cost Bing 1.2% of revenue per user) and demand growth (Facebook’s inference demand grew 3.5× in under two years).

The reason new designs are needed is that models drifted away from the ones accelerators were built for.

What people did before, and where it broke

Accelerator DSE tools. Timeloop and MAESTRO search schedules for a given datapath, single layer at a time, convolutions only; Interstellar (Halide-based) covers blocking and spatial choices over scalar-PE grids; dMazeRunner prunes schedule search for conv/FC; ZigZag broadens datapaths but skips fusion. All optimize slices — none sees end-to-end performance, where fusion, weight pinning, and prefetching live. (For the workload-derived-hardware idea in a different domain, see the REVAMP explainer on this site — CGRAs specialized from application statistics.)

Fixed fusion templates. Production compilers fuse pattern-matched sequences (conv+ReLU and friends). The paper measures what that leaves behind: template fusion cannot lift EfficientNet above the memory-bound region, while hypothetical whole-MBConv-block fusion crosses 200 FLOPS/B — and hand-writing block templates per model family does not scale.

The interaction trap. The paper’s own breakdown experiment is the cleanest indictment of slice-wise search: starting from TPU-v3 and adding FAST’s discoveries incrementally, replacing the 16 MB global memory with the discovered 128 MB gives no speedup at all while fusion is disabled — the bandwidth bottleneck simply caps everything. Enable ILP fusion and the same memory unlocks the datapath. Knobs whose value depends on other knobs must be searched jointly.

The idea

Search where the interactions live. FAST’s definition of the design space is its thesis: hardware datapath and schedule and compiler passes, because the profitable designs sit at their intersection. The combined space is estimated at O(10²³⁰⁰) — mapping choices per layer (~10²⁰⁰⁰ for a ResNet-50-sized model) times datapath configurations (~10¹³) times fusion decisions (~10³⁰⁰). No enumeration survives that; the architecture is black-box optimization over a fast, trustworthy evaluator, which is why the validated simulator (within ~8% of real hardware) is as load-bearing as any search heuristic.

Fusion as an optimization problem, not a pattern library. FAST fusion takes XLA’s already-fused HLO regions and decides, via ILP, which intermediate tensors avoid the DRAM round trip and which weights are pinned permanently in SRAM — with the objective being modeled execution time itself. This is what makes large on-chip memories meaningful: the EfficientNet-B7 sweep shows post-fusion operational intensity climbing with global memory size and falling with batch size (bigger working sets fit worse), exactly the compute-vs-memory allocation trade the datapath search needs to explore simultaneously.

Even the algorithm is a knob. BERT’s softmax requires three passes over its input (max, sum of exponentials, normalize). FAST adds a two-pass softmax — maintaining a running maximum and rescaling the running sum on the fly — trading up to 2N extra exponentials for N fewer memory accesses. Whether that trade wins depends on the very vector-throughput and bandwidth parameters being searched, so it enters the space as a hyperparameter. (Readers will recognize the online-softmax germ of FlashAttention here, deployed as a searchable option rather than a fixed kernel.)

What the search actually discovered. The revealing artifact is FAST-Large, the EfficientNet-B7 winner, side by side with a same-process TPU-v3: nearly identical peak compute (131 vs 123 TFLOPS), but half the memory bandwidth (448 vs 900 GB/s), a 4× larger global buffer (128 vs 32 MiB), tiny 8 KiB L1s, and small systolic arrays (32×32 rather than 128×128 — depthwise convolutions, lacking channel-dimension reductions, leave giant arrays idle). The design pays for SRAM with bandwidth money — affordable only because ILP fusion makes the SRAM do bandwidth’s job.

B7 on TPU-v3: 14.8% util → 6.75× headroom
depthwise: 5% FLOPs, 65% runtime

Why it works

• Performance is min(compute, memory) → single-slice optimizations stall at whichever wall they don’t address → only joint search can trade compute, SRAM, bandwidth, and fusion against each other (the buffer-without-fusion ablation is the proof).
• Fusion decided by ILP against modeled time → tensor placement and weight pinning respond to the actual datapath being evaluated → large on-chip memories become bandwidth substitutes, the economic core of the winning designs.
• A validated simulator (8% of real hardware) makes 5,000-trial black-box search meaningful → search quality is bounded by evaluator fidelity, not optimizer cleverness — and comparing against a simulated baseline on the same process keeps the comparison honest.
• The datapath template supersets published designs → the search is free to rediscover TPU-like shapes or abandon them → it abandoned them: small arrays for depthwise geometry, SRAM over bandwidth.
• Perf/TDP tracks Perf/TCO, and ROI scales with deployment volume → the framework outputs business decisions, not just architectures — including when the 2.4× general design beats the 6.4× specialist.

What it costs

• Simulation-first evidence. No chip is taped out; results rest on the validated simulator and analytical area/power models, with the baseline simulated for fairness.
• Search compute. 5,000 trials at 10 minutes–2 hours each, plus a 20-minute ILP budget per fusion solve — datacenter-scale design deserves datacenter-scale search, but it is not free.
• Fixed models. FAST optimizes hardware for given networks; it does not co-evolve the model architecture (acknowledged as a natural extension), and quantization is left orthogonal.
• Inference only. Training, with its different memory and communication profile, is out of scope.
• Specialization risk. A single-workload chip bets on workload longevity; the paper’s own ROI analysis partially hedges this by favoring multi-workload designs at volume.

The evaluation

Workloads span the difficulty spectrum deliberately: the full EfficientNet family (B0–B7, the depthwise stress test), BERT at sequence lengths 128 and 1024 (attention stress test), ResNet-50v2 (TPU-friendly CNN), and two production OCR components (RPN and an LSTM recognizer) that are already TPU-optimized — a designed-in worst case. Baseline: TPU-v3 simulated on the same sub-10-nm process, within the same area/TDP budgets.