New GPUs arrive with 52-week lead times and new models arrive weekly, so everyone needs to know how a model will run on hardware they cannot touch — but cycle-accurate simulators take 18 hours for a small ResNet and direct machine-learning latency predictors collapse on unseen GPUs (up to 724% average error) — so NeuSight stops asking ML to learn the whole latency function and instead decomposes each kernel along the GPU libraries' own tile-and-wave structure, bounds every prediction with the roofline law, and lets a small MLP learn only the one genuinely nonlinear residual (device utilization as a saturating function of wave count) from public spec-sheet features — hitting 8.9% average error, 8.1% on GPUs it never trained on, and 5.4% for 4-GPU distributed training, generalizing even across the NVIDIA/AMD divide.
The argument in one glance
-
Problem
Everyone must plan for hardware they cannot touch
AI models and GPUs co-evolve on incompatible schedules: an H100 order could take 52 weeks to arrive and costs accordingly, while new model architectures appear monthly. Three questions recur across every AI organization — how will new models run on existing GPUs, existing models on new GPUs, and new models on new GPUs? — and answering any of them today requires possessing the hardware and running the workload.
-
What people did
Simulate it, extrapolate it, or learn it
Cycle-accurate simulators (Accel-Sim lineage) model every architectural detail. Analytical frameworks (Paleo, Calculon) decompose latency into compute and communication terms calibrated by measurements. Machine-learning predictors map kernel and GPU features straight to latency: Habitat trains MLPs per operator; Li et al. fit linear regressions on FLOP counts and extrapolate across GPUs by memory-bandwidth ratios.
-
Why it fell short
Each fails exactly where forecasting is needed
Simulators need manual updates for each new architecture (details often proprietary at launch) and take 18 hours for ResNet-50 at batch 256 — hopeless for LLMs. Analytical models calibrate against the target GPU, which is the thing you do not have. And direct ML prediction collapses out of distribution: Habitat errs up to 127% on an A100 it never saw and 724% on average across unseen GPUs; linear regression breaks at small dimensions (under-utilization bends the line) and its bandwidth-ratio extrapolation is simplistic. Scaling the predictor does not help — deeper MLPs and even transformers still err 70–126% out of distribution. The latency function is too entangled with hardware-software interplay for any feature-to-latency mapping to extrapolate.
-
This paper's idea
Give ML only the residual; structure and physics do the rest
NeuSight (Georgia Tech + Microsoft Research) refuses to learn the latency function. Instead it mirrors how GPU libraries actually execute kernels: the output is partitioned into identical TILES dispatched across streaming multiprocessors in WAVES — so kernel latency = per-tile latency × wave count, pure arithmetic. Per-tile throughput is CAPPED by the roofline law (min of compute and bandwidth ceilings). The only learned quantity is utilization — how close a tile gets to the ceiling — modeled as a saturating function alpha − beta/num_waves whose two coefficients a small sigmoid-bounded MLP predicts from five ratio features built entirely from PUBLIC spec-sheet numbers (peak FLOPS, bandwidth, memory, L2, SM count, normalized per-SM). Five specialized MLPs cover the operator taxonomy; a graph pass aggregates operators; a network model adds all-reduce and pipeline transfers for multi-GPU servers. Results: 8.9% average error overall, 8.1% on wholly unseen GPUs (H100, L4, A100-80GB) where Habitat and Li et al. average 724.3%, 5.4% on 4-GPU distributed training — and 8.8–15.7% on AMD GPUs, an architecture family it also never saw.
Why this problem is worth solving
Every serious AI effort now makes hardware decisions in the
dark: which GPUs to procure years out, whether a planned
model fits a latency budget on the current fleet, whether an
architecture tweak will scale on silicon that exists only in a
press release. The prediction target is genuinely hard because
GPU performance is a conflation:
how GPU libraries (cuBLAS, CUTLASS,
cuDNN) actually run a kernel — the output matrix is
partitioned into identical tiles (dimensions typically 32–256),
each tile loading its operands and computing its output slice
independently on one streaming multiprocessor (SM). With
more tiles than SMs, execution proceeds in WAVES: each SM
takes one tile per wave, and kernel latency scales with the
wave count. This software-imposed regularity is what makes
GPU execution predictable at all
The middle path runs through an old friend:
the first-order performance law —
a kernel’s achievable throughput is capped by
min(arithmetic intensity × peak memory bandwidth, peak
FLOPS). It cannot say what a kernel WILL achieve, only what
it cannot exceed — an upper bound derived purely from
spec-sheet numbers
What people did before, and where it broke
- Cycle-accurate simulators. Faithful but doubly disqualified: each new GPU needs hand-modeling of features that are proprietary at launch, and simulation takes 18 hours for ResNet-50 at batch 256 — LLM-scale workloads are out of reach entirely.
- Analytical frameworks (Paleo, Calculon, MAD-Max). Decompose distributed execution well, but calibrate per-device throughput by running representative workloads ON the target GPU — circular for forecasting. NeuSight positions itself as the missing high-fidelity per-device term such frameworks could plug in.
- Habitat (direct MLP). Trains per-operator MLPs from GPU features to latency. In-distribution it works; on unseen dimensions error hits 38%, on an unseen A100 127% — and for kernel-alike operators it needs a physical GPU in hand to scale from.
- Li et al. (linear regression). Latency ∝ FLOPs per GPU, extrapolated across GPUs by bandwidth ratios. Fails at small dimensions (under-utilization bends the line) and the cross-GPU extrapolation is a single scalar ratio.
- Just use bigger ML? The paper tests it: deeper MLPs and transformer predictors still err 70–126% out of distribution. Model capacity cannot substitute for missing inductive structure — the core negative result motivating the design.
How NeuSight works
Arithmetic where possible, physics where certain, ML where necessary
-
Decompose: kernels into tiles and waves new
For each DNN kernel (GEMM, batched matmul, element-wise, softmax, layer-norm), NeuSight looks up the library tile size (a database built from PyTorch Profiler metadata), computes num_tiles = product of ceil(output_dim / tile_dim), and num_waves = ceil(num_tiles / num_SMs). Kernel latency = per-tile latency × num_waves. All arithmetic, no learning — and it mirrors what CUTLASS actually does.
-
Bound: the roofline as a hard cap new
Per-tile achievable throughput is bounded by min(arithmetic_intensity × peak_bandwidth, peak_FLOPS) — spec-sheet physics. The learned component can only DISCOUNT this bound, never exceed it: achievedBW = rooflineBW × utilization, with utilization forced below 1 by a sigmoid. Predictions stay physical even on inputs the MLP has never seen.
-
Learn: utilization as a saturating residual new
The one empirical fact: how close a tile gets to the ceiling. Measurement shows utilization rises with wave count (more concurrent threads hide memory stalls — 53% at batch 32 to 86% at batch 512 on H100) and saturates. NeuSight encodes this shape as utilization = alpha − beta/num_waves and asks an 8-layer, 512-unit MLP only for the two coefficients — a tiny, well-conditioned learning task compared to "predict the latency."
-
Features: public numbers, per-SM normalized new
Five ratio features feed the MLP: tile FLOPs vs per-SM peak, tile memory vs per-SM bandwidth, working set vs per-SM L2 and memory capacity, and arithmetic-intensity vs machine balance. Everything derives from specs published at or before launch (peak FLOPS, bandwidth, memory size, L2, SM count) — no counters, no access to the device. Five specialized MLPs cover the operator taxonomy; unseen exotic operators fall back to memory-bound estimates (their share of runtime: under 0.1% for BERT-Large).
-
Aggregate: graphs, then servers new
A graph extractor walks the PyTorch/TensorFlow model, predicting each kernel and summing along the dataflow (kernels execute sequentially — empirically validated for mainstream frameworks; operator fusion is supported). For multi-GPU servers, a network predictor estimates ring all-reduce (tensor/data parallel) and send/receive (pipeline parallel) latencies and the scheduler composes them with per-device predictions.
-
Train on the past, forecast the future
Training data comes only from older, accessible GPUs (P4, P100, V100, T4, A100-40GB…) and a subset of model dimensions; H100, L4, and A100-80GB are held out entirely, as are full GPT-3/OPT configurations. The evaluation is genuine forecasting: predicting hardware and models the system never measured.
The idea
Learn the residual, not the function. The negative result is the paper’s foundation: direct feature-to-latency ML fails out of distribution, and scaling the model (deeper MLPs, transformers) does not fix it — 70–126% errors persist. The diagnosis is that latency conflates structure (tiling, waves), physics (roofline ceilings), and a genuinely empirical residue (stall-hiding efficiency); asking ML to rediscover the first two from data guarantees extrapolation failure, because arithmetic and physics do not interpolate. NeuSight hard-codes what is known and learns only the residue — a bounded scalar with a known monotone-saturating shape. This is the site’s learned-systems lesson in its cleanest form yet: the learned flash-mapping page bounded its model with error guarantees; here the guardrails are the laws of the machine itself.
Decomposition is what keeps the ML in-distribution. The subtle trick in predicting at tile granularity: model dimensions grow without bound (GPT-3’s GEMMs dwarf the training set), but tile dimensions do not — libraries use 32–256 regardless. A kernel far outside the training distribution decomposes into tiles squarely inside it; the out-of-distribution burden shifts from the learned component to the arithmetic (num_tiles × waves), which extrapolates perfectly by construction. The same normalization move appears in the features: dividing by SM count converts absolute chip scale (which grows every generation) into per-SM ratios (which barely move), so an H100 looks to the MLP like a familiar neighborhood, not a new continent.
Forecast from the spec sheet, by design. The feature set is a deployment decision disguised as feature engineering: peak FLOPS, bandwidth, memory, L2, and SM count are exactly what vendors publish at announcement — the paper notes Blackwell’s key numbers were public before availability. No hardware counters, no profiling runs on the target, no GPU in hand. That is what makes NeuSight a forecasting tool rather than a fitting tool, and the honest held-out protocol (H100/L4/A100-80GB never trained on; 8.1% error anyway; NVIDIA-only training generalizing to AMD at 8.8–15.7%) is the evidence the design earns its name.
Predicting an H100 it has never seen
GPT-3 inference latency, forecast before the hardware arrives
Waves = ceil(tiles / SMs), the unit of sequential SM occupancy. Utilization = fraction of the roofline bound achieved. Numbers from the paper's evaluation.
Step 1 of 6
The question arrives with a spec sheet
How fast will GPT-3 decode on H100? NeuSight has never measured an H100 — but NVIDIA's documentation supplies what the model needs: 66.9 TFLOPS peak, 3.43 TB/s bandwidth, 80 GB memory, 50 MB L2, 132 SMs. The graph extractor walks the PyTorch model into a kernel sequence: GEMMs, batched matmuls, softmaxes, layer-norms, element-wise ops.
no counters, no profiling, no device access — spec-sheet forecasting by design the same pipeline accepted Blackwell's numbers before release
Why it works
- GPU libraries impose tile-and-wave regularity on every kernel → latency decomposes into (learned per-tile time) × (computed wave count) → the unbounded part of the problem is arithmetic that extrapolates exactly, and the learned part stays within the training distribution no matter how large models grow.
- The roofline bound derives from published specs alone → predictions are clamped to physical possibility on any GPU, seen or unseen → out-of-distribution errors are capped where direct-ML predictors run unbounded.
- Utilization has a known qualitative shape (rising with wave count as stalls hide, saturating below one) → encoding the shape and learning two coefficients is a far easier task than learning latency → a small MLP suffices and generalizes.
- Per-SM normalization removes chip scale from the features → generational growth (more SMs, more bandwidth) cancels, leaving ratios that vary little across GPUs → an H100 or an AMD MI210 lands near the training manifold.
- Everything upstream of the MLP is deterministic structure → errors do not compound mysteriously: GEMM-family kernel error (13.8%) flows through the sum to comparable end-to-end error, and the framework prices distributed execution by adding a network term rather than retraining.
What it costs
- A tile database is an implicit library dependency. Tile sizes come from profiling existing library versions; a future CUTLASS that retiles radically (or a fundamentally new execution model) would need re-profiling — the structure that grants generalization is borrowed from software that can change.
- Sequential-kernel and known-schedule assumptions. Kernels are modeled as executing one at a time (validated for mainstream frameworks today), fusion must be enumerated, and distributed prediction requires the user to supply the parallelism strategy and schedule.
- Within-server scope. Multi-GPU support covers one server (NVLink/DGX); cross-server networking, stragglers, and datacenter effects are outside — analytical distributed frameworks are the intended complement.
- Residual error is not negligible. 8–9% average with a 28% tail is transformative next to 60–724% baselines, but procurement decisions hinging on sub-10% differences between GPUs still need real measurement.
- New operator families need new specialists. Five MLPs cover today’s taxonomy; genuinely new kernel classes (not expressible as the five, and too hot to memory-bound-approximate) need training data that, for a new model class, may not exist yet.
The evaluation
Genuine held-out forecasting: H100, L4, and A100-80GB excluded from training entirely; GPT-3/OPT trained only on a subset of dimensions; comparisons against Habitat (MLP) and Li et al. (linear regression) plus larger-predictor ablations; training and inference; single-GPU and 4-GPU distributed; and a cross-vendor test on AMD.
Three ways to predict a GPU you do not have
| Property | Cycle-accurate simulation | Direct ML latency prediction (Habitat / Li et al.) | NeuSight (structure + bounds + residual ML) |
|---|---|---|---|
| What must be known about the target GPU | Feature vectors (specs), but the mapping is learned from GPUs in hand — Habitat's kernel-alike path literally requires a physical GPU | ||
| Time to predict one model | 18 hours for ResNet-50 @ batch 256 — LLMs infeasible | Seconds | |
| Out-of-distribution behavior (unseen GPUs / dimensions) | Accurate if the model is updated — the update is the bottleneck | ||
| Where the intelligence lives | Explicit models of everything | One learned mapping over everything | |
| Generality | Per-architecture effort | Per-operator MLPs, NVIDIA-centric training |
Check yourself
Check your understanding
1. Habitat's MLP errs 724% on unseen GPUs, and upgrading it to deeper MLPs or transformers still leaves 70–126% error. Predict why NeuSight's MLP — smaller than those — succeeds where they fail.
2. Why is the TILE the magic granularity — rather than the kernel, the layer, or the instruction?
3. The only learned quantity is utilization, encoded as alpha − beta/num_waves with a sigmoid clamp. What real hardware behavior does this curve encode, and why does encoding the SHAPE (rather than learning it) matter?
4. NeuSight's features are five ratios built solely from spec-sheet numbers, normalized per-SM. What do these two choices — public inputs and per-SM normalization — each buy?