SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Training

New GPUs can multiply numbers in tiny subbyte formats like FP4, which would slash the enormous cost of training a large language model — but blindly forcing every layer down to 4 bits wrecks quality, and today's frameworks either use one uniform precision everywhere or tune it with brittle heuristics; this paper (University of Michigan, UMass Amherst — Yunjie Pan, Yongyi Yang, Hanmei Yang, Scott Mahlke) presents SNIP, which periodically measures how much each layer's quantization actually hurts training (a forward loss-divergence metric and a backward weight-divergence metric), then solves a small knapsack-style optimization to hand each layer the lowest precision it can tolerate under a compute budget — cutting FLOPs by up to 80 percent on Llama-like models from 1B to 70B while preserving quality.

The argument in one glance

Problem

Subbyte math could make LLM training far cheaper, if it did not break quality

Training a large language model is staggeringly expensive: the paper notes that training Llama 3 with 8 billion parameters took 1.46 million GPU hours on NVIDIA H100s and emitted roughly 420 tons of carbon dioxide. The dominant cost is the matrix multiplications (GEMMs) at the heart of every layer, and the way to make those cheaper is to compute them in lower-precision number formats. Mixed-precision training already does this in part: run the compute-heavy operations in a low-precision format while keeping a few critical operations in higher precision. Modern GPUs keep pushing the low end lower — NVIDIA Hopper added FP8 (8-bit floating point), and the newer Blackwell added FP4 (4-bit), each halving of precision roughly doubling the multiply throughput. An FP8 GEMM runs at twice the throughput of BF16 and gives a 1.3 to 1.4 times end-to-end training speedup; an FP4 GEMM on Blackwell doubles that again. These subbyte formats — fewer than 8 bits — are where the biggest savings live. The catch is that lower precision means coarser numbers, which introduces error into activations, gradients, and weight updates. Push too far, in the wrong place, and training becomes unstable or converges to a worse model. So the prize is real, but only if you can use subbyte precision without paying for it in quality.
What people did

Apply one precision to every layer, or tune it with hand-built heuristics

Existing mixed-precision training frameworks take one of two approaches. The most common is a uniform precision policy: pick a low-precision format and apply it to every GEMM in the network, assuming all layers have identical precision requirements. The alternative is heuristic-based methods that assign precision using hand-crafted rules of thumb — for example, based on a layer type or a fixed sensitivity ranking. Both aim to capture the speed of low precision without a bespoke, per-layer analysis of what each layer actually needs.
Why it fell short

Uniform precision is blunt, and heuristics do not generalize as training moves

Neither approach fits the subbyte regime. A uniform policy assumes every layer tolerates the same precision, but layers are not equal: some carry information that degrades sharply when quantized, while others are robust. Forcing all of them to the same low format either wastes the opportunity (by staying conservatively high everywhere) or destabilizes training (by pushing sensitive layers too low), and it misses the fine-grained gains that come from configuring precision per layer. Heuristic methods try to be selective, but their fixed rules do not generalize: a layer sensitivity ranking that holds early in training may be wrong later, because which layers are fragile shifts across training phases, model sizes, and architectures. So heuristics lead to suboptimal convergence and instability exactly when precision is most aggressive. What was missing was a way to know, quantitatively and continuously, how much quantizing each layer actually harms the model — and to turn that knowledge into a per-layer precision assignment that respects a compute budget.
This paper's idea

Measure each layer's quantization harm, then solve for the best precision mix

SNIP is a fine-grained, adaptive mixed-precision framework for LLM pretraining that supports subbyte precision. Instead of guessing, it measures. Periodically during training, SNIP collects statistics on activations, gradients, weights, and optimizer states, and from them computes two metrics that capture how much a layer's quantization hurts quality. The first, loss divergence, is a forward-pass quantity: the increase in training loss directly caused by quantizing that layer. The second, weight divergence, is a backward-pass quantity: how quantization error propagates through the gradients into the weight updates, pulling the trained weights away from what full precision would produce. Both are grounded in a theoretical result that lets SNIP estimate a layer's quantization error cheaply, mostly from quantities the backward pass already computes (with two small extra noise-injected passes for the harder-to-estimate part). These per-layer harm estimates then feed an Integer Linear Programming (ILP) problem, framed as a knapsack: each layer is a decision variable, each precision format (FP8, FP4) an option, the objective is to minimize total loss-plus-weight divergence, and the constraint is an efficiency budget such as the fraction of FLOPs allowed in FP4. Solving it yields the layer-wise precision scheme that loses the least quality for the compute you are willing to spend, and SNIP applies it asynchronously so training never stalls. Across Llama-like models of 1B, 3B, 7B, and 70B parameters, SNIP reduces FLOPs by up to 80 percent while preserving quality, with minimal overhead.

Why this problem is worth solving

The cost of LLM training is so large that even a fractional saving on the dominant operation is enormous, which is exactly why subbyte precision is worth the difficulty.

Almost all of the compute in training a transformer goes into the matrix multiplications inside its linear layers, repeated across billions of parameters and trillions of tokens. Precision is the most direct lever on that compute: a number format with half the bits can, on the right hardware, run the multiplication at roughly double the throughput. That is why GPU vendors keep adding lower formats — FP8 on Hopper, FP4 on Blackwell — and why each step down translates into a concrete end-to-end speedup (FP8 gives 1.3 to 1.4 times, FP4 roughly doubles the FP8 gain). When a single training run costs over a million GPU hours and hundreds of tons of carbon, shifting even part of the arithmetic to a subbyte format saves real money, time, and emissions. But the same property that makes low precision fast — representing each number in fewer bits — makes it lossy, and training is a delicate iterative process where small errors can compound over many steps. So the reason this problem matters is that the upside is huge and the hardware is already here; the only thing standing in the way is using the aggressive formats without destabilizing the very optimization you are trying to accelerate.

The deeper point is that precision is not one global knob but a per-layer decision that changes as training proceeds, so it demands measurement rather than a fixed rule.

The instinct behind uniform precision is that a network is a homogeneous stack, so one setting should suit all of it. But layers differ in how much they can be quantized before quality suffers: some are numerically fragile, their activations or gradients spanning a wide range that a coarse format mangles, while others are robust. Worse, this fragility is not static — which layers are sensitive shifts across training phases, across model sizes, and across architectures, so a ranking baked in ahead of time drifts out of date. That is why heuristics, however clever, keep failing to generalize. The only reliable way to allocate precision well is to observe, during the actual run, how much quantizing each layer is currently hurting the model, and to re-decide as that picture changes. This reframes precision assignment from a design-time guess into a runtime measurement-and-optimization loop. It is also what makes the problem tractable to optimize: once you can put a number on each layer’s quantization harm and each layer’s compute saving, choosing the best mix under a budget becomes a well-posed optimization rather than an educated guess.

What people did before, and where it broke

Uniform precision for every GEMM. Choose one low-precision format and apply it identically to all layers. Where it broke. Layers have unequal precision needs, so a single setting is either too conservative (leaving speed on the table) or too aggressive on sensitive layers (destabilizing training), and it forfeits the fine-grained gains of per-layer configuration.
Heuristic precision assignment. Use hand-crafted rules — layer type, a fixed sensitivity ranking — to pick precision per layer. Where it broke. The rules do not generalize: layer sensitivity changes across training phases, model sizes, and architectures, so a fixed heuristic leads to suboptimal convergence and instability, especially at subbyte precision.
Precision chosen once, up front. Decide the precision scheme before training and keep it fixed. Where it broke. Because which layers are fragile shifts as training progresses, a static scheme is right early and wrong later, missing chances to relax or tighten precision as the model matures.
Ignoring the backward-pass and optimizer effects. Judge a layer only by its forward-pass error. Where it broke. Quantization error also propagates through gradients into the weight updates, and this weight divergence can compound over iterations and layers — a forward-only view misses a major source of quality loss.
The gap. No framework measured, continuously and cheaply, how much each layer’s quantization actually harms quality in both the forward and backward passes, and turned that into a budget-constrained, per-layer precision assignment that adapts as training moves.

How SNIP works

From uniform low precision to measured, per-layer, budget-aware precision

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Give each layer the lowest precision it can personally tolerate. SNIP rejects the idea that one format suits the whole network; it assigns precision per layer, matching aggressive subbyte formats to robust layers and reserving higher precision for fragile ones. Driving efficiency by tailoring the number of bits to each individual unit rather than applying one width everywhere is the same fine-grained-precision instinct the work on per-variable bitwidth speculation (per-variable bitwidth speculation) covered elsewhere on this site pursues, spending bits only where they are actually needed. SNIP brings that per-unit precision tailoring to the layers of an LLM during training, where the payoff is measured in GPU hours saved.

Put a principled number on how much each layer’s quantization hurts. Rather than rank layers by a rule of thumb, SNIP derives two grounded metrics — forward loss divergence and backward weight divergence — that quantify each layer’s contribution to quality loss, estimated cheaply from statistics the training already produces. Attributing an outcome to individual components with a principled, quantitative influence measure, so decisions rest on measurement rather than intuition, is the same influence-quantification discipline the work on Shapley-value performance attribution (ShapleyIQ) covered elsewhere on this site is built on, turning a vague sense of importance into concrete per-component numbers. SNIP uses its per-layer divergence numbers not to debug but to decide where precision can safely drop.

Turn precision assignment into an optimization under a budget. Once every layer has a harm estimate and a compute saving, SNIP frames the choice as an ILP knapsack — minimize total divergence subject to an efficiency budget — and solves for the exact best mix rather than approximating it. Casting a resource-allocation decision as a constrained optimization solved to meet an explicit budget, instead of hand-tuning it, is the same constraint-aware framing the work on constraint-aware resource scheduling for LLM inference (ExeGPT) covered elsewhere on this site adopts, letting a solver find the configuration that best trades quality against cost. SNIP applies that solver-driven allocation to layer-wise precision, so the precision mix is optimal for whatever FLOP budget the user sets.

Following a precision update through SNIP

One periodic precision-update cycle, from statistics to a new scheme

A trace, drawn from the paper, of SNIP's periodic loop: measure each layer's quantization harm, solve the ILP, and apply the new precision without stalling training.

Step 1 of 5

Run a high-precision reference step and record statistics

At the checkpoint chosen for this cycle, SNIP runs a normal training iteration in BF16 and records the Frobenius norms of inputs, weights, outputs, and gradients, the quantization-error norms, and the optimizer's first and second moments. Because these are simple norms, the cost is negligible.

This high-precision step is the baseline everything else is compared against.

Why it works

Per-layer precision beats one global setting. Layers differ in how much quantization they tolerate → but assigning each its own precision format → so robust layers run in aggressive subbyte formats while fragile ones stay safe.
Measuring harm beats guessing it. Heuristic sensitivity rankings drift out of date as training moves → but computing loss and weight divergence from live statistics → so precision decisions track the model’s actual current fragility.
Counting both passes catches the real damage. A forward-only error view misses gradient-driven harm → but adding backward weight divergence to forward loss divergence → so the compounding error in weight updates is accounted for, not ignored.
Cheap estimation keeps the loop affordable. Exact quantization-impact math is prohibitive → but approximating it from existing gradient norms plus two small noise probes → so the measurement adds negligible-to-minimal overhead to training.
An ILP finds the true best mix. Hand-tuning precision under a budget is guesswork → but solving a knapsack over layers and formats → so the chosen scheme provably minimizes quality loss for the FLOPs allowed.

What it costs (trade-offs and scope)

The measurement is not entirely free. Estimating weight divergence needs two extra noise-injected passes on the GPU each cycle; the paper keeps this to minimal overhead by running the loop only periodically and offloading the analysis and ILP solve to the CPU, but it is a real (if small) cost.
The estimates are approximations. Loss and weight divergence rest on theoretical bounds and single-sample approximations of quantities (like second-order derivatives) that are prohibitive to compute exactly, so they are principled estimates of quantization harm, not exact measurements.
Scope is linear layers and specific formats. The analysis targets the matrix-multiply linear layers of transformer blocks and the FP8 and FP4 formats; the optimizer modeling focuses on AdamW (the paper argues it generalizes to any differentiable optimizer).
Efficiency is modeled, not always measured. Because GPUs natively supporting both FP8 and FP4 were not available, efficiency is approximated as the fraction of FLOPs in FP4 rather than measured end-to-end, and FLOP reduction is the headline efficiency metric.
Results are on Llama-like pretraining. The up-to-80-percent FLOP reduction with preserved quality is demonstrated on 1B, 3B, 7B, and 70B Llama-like models during pretraining; other architectures, tasks, or fine-tuning regimes may behave differently.

The comparison

Choosing precision for low-precision LLM training

Property	Uniform precision	Heuristic assignment	SNIP
Granularity
Adapts during training
Accounts for backward and optimizer effects
How precision is chosen
Reported outcome

A comparison of ways to set precision in low-precision LLM training. Uniform precision applies one format to every layer and is blunt; heuristic assignment picks per-layer precision by fixed rules that do not generalize across training phases. SNIP measures each layer's quantization harm with forward loss-divergence and backward weight-divergence metrics, then solves an ILP knapsack to assign per-layer precision that minimizes quality loss under a FLOP budget — reducing FLOPs by up to 80 percent while preserving quality on Llama-like models from 1B to 70B. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2602.01410, of the ASPLOS 2026 paper). The described system is the paper’s: SNIP is a fine-grained adaptive mixed-precision training framework for LLM pretraining supporting subbyte precision. It periodically collects statistics on activations, gradients, weights, and optimizer states, and defines two metrics — loss divergence in the forward pass (the quantization-induced increase in training loss) and weight divergence in the backward pass (error propagating through gradients into weight updates). These are grounded in two theorems that let a layer’s quantization error be estimated from gradient norms under a small-random-perturbation model, with a couple of extra noise-injected passes to approximate a prohibitive second-order quantity. The metrics parameterize an Integer Linear Programming problem, mapped to a knapsack, that assigns each layer a precision format (FP8, FP4) to minimize total divergence subject to an efficiency budget (approximated as the fraction of FLOPs in FP4); the resulting scheme is applied asynchronously. The headline results — up to 80 percent FLOP reduction with preserved quality — are from the paper’s evaluation on 1B, 3B, 7B, and 70B Llama-like models. Because hardware natively supporting both FP8 and FP4 was not available, efficiency is modeled via FLOPs rather than measured end-to-end. Background on mixed-precision training, FP8/FP4, GEMMs, and the AdamW optimizer is standard context. The step-by-step walkthrough is an illustration of the described mechanism. The exact theorems, estimation procedure, ILP formulation, and full evaluation should be read from the paper.

Background

Why efficient LLMs keep coming back to spending bits where they matter

It helps to place SNIP in the broader push to make large models cheaper by using fewer bits. Training and serving LLMs is dominated by matrix multiplications, and the number of bits each value carries sets both the memory it occupies and the speed of the arithmetic. So a whole line of work asks the same question from different angles: where can precision be lowered without losing what matters? Some efforts design better low-precision number formats so that 4-bit values still capture the range and detail a tensor needs; some quantize after training for cheaper inference; SNIP works during training, deciding per layer and per phase how low each part can go. The recurring lesson is that precision is not uniform across a model or across time — value distributions differ by layer and shift as training proceeds — so the winning strategies measure and adapt rather than apply one global setting. This drive to make low precision usable for large models by engineering how few bits each value really needs is the same terrain the work on a metadata-augmented microscaling format for efficient low-precision LLMs (m2xfp) covered elsewhere on this site occupies, squeezing quality out of subbyte formats. SNIP’s particular contribution is to turn the where-can-precision-drop question into a measured, budget-constrained optimization run continuously through pretraining.

If you remember only three things

1. Training large language models is extraordinarily expensive, and the arithmetic is dominated by matrix multiplications whose cost falls roughly in half each time the number format drops by a power of two. Modern GPUs now support subbyte formats — FP8 on Hopper, FP4 on Blackwell — that promise large training savings, but lower precision introduces error into activations, gradients, and weight updates, so pushing it too far in the wrong place makes training unstable or degrades the model. Existing mixed-precision frameworks either apply one uniform precision to every layer (blunt, and either wasteful or destabilizing) or use hand-built heuristics that fail to generalize because layer sensitivity shifts across training phases, model sizes, and architectures.
2. SNIP is a fine-grained, adaptive mixed-precision framework that measures rather than guesses. Periodically during pretraining it collects cheap statistics on activations, gradients, weights, and optimizer states, then computes two grounded metrics: loss divergence (how much quantizing a layer raises training loss in the forward pass) and weight divergence (how quantization error propagates through gradients into the weight updates in the backward pass). Both are estimated affordably from quantities the backward pass already produces, plus two small noise-injected probe passes for the harder second-order term. These per-layer harm estimates feed an Integer Linear Programming problem shaped like a knapsack, which assigns each layer a precision format (FP8 or FP4) to minimize total quality loss subject to an efficiency budget such as the fraction of FLOPs allowed in FP4.
3. By solving that optimization periodically and applying the result asynchronously — offloading the analysis and solve to the CPU so the GPU never stalls — SNIP continuously re-adapts precision as each layer's sensitivity changes through training. On Llama-like models from 1B to 70B parameters it reduces FLOPs by up to 80 percent while preserving model quality, with minimal overhead. The trade-offs: the divergence metrics are principled approximations rather than exact measurements, the extra probe passes add a small cost, the analysis targets linear transformer layers with FP8/FP4 and the AdamW optimizer, and because hardware supporting both formats natively was unavailable, efficiency is modeled through FLOPs rather than measured end-to-end.

Check yourself

Check your understanding

1. Why is subbyte precision (like FP4) attractive for LLM training, and what makes it risky?
2. What is wrong with uniform precision and heuristic precision assignment for subbyte training?
3. What two metrics does SNIP define, and what does each capture?
4. How does SNIP turn its per-layer measurements into an actual precision assignment?

Sources

The paper is “SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Training” by Yunjie Pan, Yongyi Yang, Hanmei Yang, and Scott Mahlke (University of Michigan; Yongyi Yang also NTT Research; Hanmei Yang at University of Massachusetts Amherst), published at ASPLOS 2026, DOI 10.1145/3779212.3790223. It is available on arXiv as 2602.01410.

This explainer is written from the paper’s full text (the authors’ arXiv copy). SNIP is a fine-grained adaptive mixed-precision training framework for LLM pretraining that supports subbyte precision. It periodically collects statistics on activations, gradients, weights, and optimizer states; defines loss divergence (forward-pass, quantization-induced training-loss increase) and weight divergence (backward-pass, error propagating through gradients into weight updates), estimated cheaply via two theorems and a couple of noise-injected passes; and feeds these into an Integer Linear Programming knapsack that assigns each layer a precision format (FP8, FP4) to minimize total divergence subject to an efficiency budget, applied asynchronously. Evaluated on 1B, 3B, 7B, and 70B Llama-like models, the paper reports up to 80 percent FLOP reduction while preserving quality, with minimal overhead; efficiency is modeled via FLOPs because hardware natively supporting both FP8 and FP4 was unavailable. Background on mixed-precision training, FP8/FP4, GEMMs, and AdamW is standard context. The numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.