M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization

Weiming Hu, Zihan Zhang, Haoyan Zhang, Chen Zhang, Cong Guo, Yu Feng and 5 others

Shrinking a giant language model to 4-bit numbers saves enormous memory, and the standard microscaling formats do it by giving each small block of numbers a shared scale factor — but that scale is a coarse power-of-two that misfits the block's largest value and loses accuracy; this paper (Shanghai Jiao Tong University and Huawei — Weiming Hu, Jingwen Leng, and colleagues) proposes M2XFP, which spends a tiny amount of extra "metadata" — just a quarter of a bit per number — on exactly the elements that need it, using cheap element-level refinement for the runtime-varying activations and offline subgroup refinement for the fixed weights, recovering most of the lost accuracy at an effective 4.5 bits with a lightweight hardware unit that reaches 1.91 times speedup and 1.75 times energy savings.

The argument in one glance

Problem

Squeezing a huge model into 4-bit numbers saves memory but wrecks accuracy

Large language models are enormous — hundreds of billions of parameters, terabytes in normal 16-bit precision — far more than a commodity accelerator can hold or move cheaply. The leading way to fit and run them is low-bit quantization: store each weight and activation in far fewer bits, cutting memory, bandwidth, and energy. The industry has converged on microscaling (MX) formats, which quantize numbers in small blocks and give each block a single shared scaling factor, so a block of tiny 4-bit numbers can still cover a range of magnitudes. These formats are efficient and are built into current accelerators from NVIDIA, AMD, and Microsoft. But at 4 bits per element the accuracy falls off a cliff, and the culprit is the shared scale: it is stored as a coarse power-of-two value, whose steps are too widely spaced to line up with the block's actual largest number, so quantizing to it introduces large rounding error. The model gets small, but it also gets noticeably worse.
What people did

Make the scale factor more precise, or invent a richer number format

The two obvious fixes both target the number system itself. One is to make the shared scale more precise — store it as an 8-bit floating-point value (as in NVFP4) so it can align more finely with the block. The other is to design a custom, more expressive low-bit data type that packs more information into each element. Both keep tinkering with the scale factor and the base data type, the two dimensions everyone has been optimizing.
Why it fell short

Those axes are near their limits, and both have real costs

A more precise 8-bit scale helps with alignment but narrows the dynamic range the block can represent and needs extra rescaling, trading one problem for another. Richer custom data types are expressive but expensive in hardware, especially for activations that change at runtime and must be quantized on the fly. More fundamentally, scaling-factor design has largely converged and data-type innovation is hitting scalability walls, so grinding on those two axes yields diminishing returns. Meanwhile a third design dimension — metadata, a small amount of auxiliary information that could carry extra precision or range without changing the core datapath — is barely explored, and the few attempts are fragmented: outlier schemes break under block-wise formats, and structural metadata balloons to tens of bits per block, defeating the purpose.
This paper's idea

Spend a tiny, principled amount of metadata exactly where it helps most

M2XFP is an algorithm-hardware co-design that systematically exploits the metadata axis. Its insight is that a few extra metadata bits can act as extra mantissa (refining precision) or extra exponent (extending range), and can be applied to a single critical element or to a subgroup's scale — a small design space it explores under an effective-bit-width budget. The key finding is an asymmetry: activations, which vary at runtime, are best served by cheap element-level refinement that can be encoded in real time, while weights, which are fixed, are best served by subgroup-level refinement with an offline scale search. So M2XFP uses a hybrid: element-level extra mantissa for activations, subgroup-level refinement with adaptive scales for weights. At just 0.25 bits of metadata per element — an effective 4.5-bit format — it reaches near-16-bit accuracy, and a lightweight hardware unit (a top-1 decode, an augmented FP4-by-FP4 multiplier, and a streaming quantizer) folds into a systolic array without disrupting the matrix-multiply pipeline, delivering up to 1.91 times speedup and 1.75 times energy savings.

Why this problem is worth solving

Quantization is the enabling technology for running large models at all, and 4 bits is where it gets both most valuable and most fragile.

Quantization stores numbers in fewer bits than full precision to save memory, bandwidth, and energy — critical when a model is too large to fit or move otherwise. A microscaling (MX) format does this in blocks: a small group of elements is quantized to a few bits each (four, in MXFP4), and the whole block shares one scaling factor that stretches the tiny quantized values to cover the block’s actual magnitude range. This block-level sharing is what makes MX efficient — one scale amortized over many elements, with streamlined dequantization — and it is why MX formats are natively supported in commercial accelerators. The weakness surfaces at 4 bits. The shared scale in MXFP4 is stored as a power-of-two value (an 8-bit exponent, no mantissa, called E8M0), so its possible values are spaced by factors of two. To represent a block, the scale is chosen from that coarse ladder to match the block’s largest element, but the ladder rungs rarely land near the true maximum, so the scale is off, and every element quantized against it inherits a large rounding error. The blocks that matter most — those with a big outlier value — are exactly the ones the coarse scale serves worst. Recovering that lost accuracy without giving up the efficiency that made MX attractive is the problem, and at the 4-bit precision the industry wants, it is unsolved.

The trouble is that the usual knobs for fixing it are close to exhausted, which points attention to an overlooked one.

A low-bit quantization format can be improved along three axes. The first is the scaling factor — how precisely the shared scale is stored; making it an 8-bit float instead of a power-of-two (as NVFP4 does) aligns it better but shrinks the representable range and adds rescaling cost. The second is the base data type — the format of each element; custom types can pack more information but are expensive to build in hardware, and prohibitively so for activations that must be re-quantized every inference. Both of these have been heavily worked, and both are near their practical limits: scale design has converged and data-type design is hitting scalability bottlenecks. The third axis is metadata: a small quantity of auxiliary bits attached alongside the data that can encode extra precision or range information, without disturbing the core datapath that the arithmetic units run. This is attractive because it is orthogonal to the arithmetic — the multiplier still sees standard low-bit values, while the metadata quietly corrects the worst errors — so it can add fidelity at low hardware cost. But it has been left fragmentary: outlier-oriented metadata schemes that work at whole-tensor granularity break down in the block-wise MX setting, and structural metadata approaches balloon to tens of bits per block, more overhead than the savings justify. That leaves a wide, under-explored design space precisely on the axis with the most remaining headroom, which is where M2XFP concentrates.

What people did before, and where it broke

MXFP4 with a power-of-two shared scale. Give each block a coarse E8M0 (power-of-two) shared scale and quantize elements to 4 bits. Where it broke. The power-of-two scale’s steps are too widely spaced to match the block’s local maximum, so it misaligns and produces large rounding error, causing severe accuracy loss at 4 bits.
NVFP4 with an 8-bit floating-point scale. Store the shared scale more precisely as an FP8 value to align it better with the block. Where it broke. The finer scale narrows the dynamic range the block can represent and requires additional rescaling, trading the alignment gain for range and overhead costs.
Custom expressive low-bit data types. Design a richer per-element format to carry more information. Where it broke. The expressiveness comes with prohibitive hardware cost, especially for dynamic activations that must be quantized on the fly, so it does not scale to real accelerators.
Outlier or structural metadata (OliVe, MicroScopiQ). Attach metadata to handle outliers or structure. Where it broke. Outlier schemes designed for whole-tensor settings break under block-wise MX formats, and structural metadata often exceeds forty bits per block, an overhead that erases the efficiency benefit.
The gap. No format used lightweight, principled metadata — placed on exactly the elements that matter, and differently for weights than activations — to recover 4-bit accuracy while keeping MX’s efficiency and adding negligible hardware.

How M2XFP works

Recovering 4-bit accuracy by spending a quarter-bit of metadata where it counts

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Attack the overlooked metadata axis, and spend its bits on the elements that actually lose accuracy. The reframing at the heart of M2XFP is that the two dimensions everyone optimizes — the scale factor and the base data type — are near their limits, while a third, metadata, is barely used and orthogonal to the arithmetic. Metadata is attractive precisely because the multiplier keeps seeing plain low-bit numbers while a few side bits carry corrections, so accuracy can be bought cheaply without redesigning the datapath. But the win only materializes if the bits go where the error is, and the paper’s observation is that the damage in a 4-bit block is concentrated on its largest element, mis-served by the coarse shared scale. So M2XFP directs its metadata there — extra mantissa on the single top element of a block, for instance — rather than spreading it thin. Getting an outsized accuracy return from a minimal number of carefully placed bits is the same economy the work covered elsewhere on this site captured in its title, that a single bit can be worth a thousand tensor values when spent on the right information rather than uniformly. M2XFP turns that principle into a concrete metadata format that recovers most of the lost accuracy at a quarter-bit per element.

Give weights and activations different metadata, because they live on different timelines. The subtler idea is that the best metadata strategy is not the same for both operands of a matrix multiply, and the reason is when they are known. Weights are static — fixed after training — so you can afford an expensive offline search to pick the best subgroup-level refinement and adaptive scale for each block, optimizing at leisure. Activations are dynamic — produced fresh during each inference — so any metadata for them must be encoded in real time, cheaply, which favors a simple element-level scheme that a small hardware unit can compute on the fly. M2XFP explores this and codifies the asymmetry into a hybrid format: element-level extra mantissa for activations, subgroup-level refinement with adaptive scales for weights, each matched to its operand’s constraints. Tailoring a quantization scheme differently to weights and activations according to their distinct properties, rather than applying one uniform format, is the same operand-aware design the transformer-inference quantization work covered elsewhere on this site is built on, recognizing that the two sides of a neural network’s computation have different statistics and different budgets and should be quantized accordingly. M2XFP’s version keys the distinction specifically to the static-versus-dynamic timeline that decides whether offline optimization is even possible.

Co-design the format with hardware so the metadata is nearly free to use. A clever format is worthless if handling its metadata stalls the accelerator, so M2XFP is co-designed with the arithmetic that runs it. The metadata is deliberately restricted to operations a systolic array can absorb: a top-1 decode unit locates the block’s critical element that carries extra mantissa, an augmented FP4-by-FP4 processing element performs the slightly-refined multiply, and a streaming quantization engine encodes activations as they flow in — all woven into the matrix-multiply pipeline so the metadata is processed in stride rather than as a separate, blocking step. The result is that the accuracy recovery costs negligible area and does not slow the GEMM, which is what lets the same design deliver both better accuracy and better performance and energy. Building a specialized processing element and datapath so that a richer numeric scheme runs at full speed inside the systolic array, rather than paying for its expressiveness in stalls, is the same hardware-format co-design the multiplier-free GEMM accelerator work covered elsewhere on this site pursues, shaping the arithmetic units around the data representation. M2XFP shows that with the right minimal metadata and matching hardware, 4-bit efficiency and near-full-precision accuracy need not be in conflict.

Following a quantized block through M2XFP

Quantizing and multiplying a 4-bit block with metadata correction

A trace, drawn from the paper's design, of how M2XFP quantizes weights and activations and runs the refined multiply.

Step 1 of 5

Weights are refined offline with a scale search

Before deployment, each weight block is quantized to 4-bit MX with subgroup-level mantissa refinement and an adaptive shared scale chosen by an offline search, since the weights are fixed. The metadata for weights is optimized at leisure to minimize error per block.

Being static, weights can afford this offline optimization that activations cannot.

Why it works

Metadata corrects errors without touching the arithmetic. Redesigning the data type or scale is costly and near its limit → but metadata rides alongside standard low-bit values as side bits the multiplier barely notices → so accuracy is recovered at low hardware cost on the axis with the most headroom.
Placing metadata on the critical element beats spreading it. A 4-bit block’s error concentrates on its largest, worst-scaled value → but element-level extra mantissa refines exactly that element → so a quarter-bit per element recovers most of the loss instead of thinly diluting the benefit.
Matching metadata to operand timelines exploits what each allows. Weights are static and activations dynamic → but weights get an offline subgroup scale search while activations get cheap real-time element encoding → so each operand gets the strongest strategy its timeline permits.
An effective-bit-width budget guides principled choices. Metadata could be spent many ways with unclear payoff → but searching under equivalent bit width measures accuracy recovered per real bit → so the chosen hybrid is the best trade rather than an ad-hoc guess.
Co-designed hardware keeps the metadata nearly free. Handling metadata could stall the accelerator → but a top-1 decode, augmented processing element, and streaming quantizer fold into the systolic array → so the correction adds negligible area and does not slow the GEMM, giving accuracy and speed together.

What it costs (trade-offs and scope)

It is not free precision; it is 4.5 effective bits. M2XFP recovers accuracy by spending 0.25 bits of metadata per element, so the format is genuinely half a bit larger than plain 4-bit MX; the claim is near-full-precision quality at 4.5 effective bits, not at a true 4 bits, so it trades a modest bit-budget increase for the accuracy.
The benefit rests on the error being concentrated. Element-level metadata on the critical value works because a block’s damage clusters on its largest element; distributions where error is spread evenly across a block, or blocks with many comparably-large values, would see less benefit from concentrating the metadata.
It requires custom hardware in the accelerator. The gains depend on the top-1 decode unit, augmented FP4-by-FP4 processing element, and streaming quantization engine being present; running M2XFP on hardware without these extensions would not realize the performance and energy benefits, so it is an accelerator-integrated format, not a drop-in software change.
Weight refinement needs an offline optimization step. The subgroup-level scale search for weights is an offline process, so preparing a model in M2XFP involves that optimization; it is a one-time cost per model but is additional machinery beyond simply casting to a format.
Evidence is a benchmark study against specific baselines. The accuracy-loss reductions and the 1.91-times and 1.75-times gains are averages and maxima over LLM benchmarks against MXFP4, NVFP4, and state-of-the-art MX accelerators; results vary by model and task, and behavior on other formats, models, or hardware is inferred rather than exhaustively shown.

The evaluation

4-bit quantizing a large language model with a microscaling format

Property	MXFP4 (power-of-two scale)	NVFP4 (FP8 scale)	M2XFP
How accuracy is improved
Uses the metadata axis
Weight vs activation treatment
Effective bits per element
Reported outcome

A comparison drawn from the paper. M2XFP is evaluated on LLM benchmarks against MXFP4, NVFP4, and state-of-the-art MX accelerators; the 70.63-percent and 37.30-percent accuracy-loss reductions and the up-to-1.91-times speedup and 1.75-times energy savings are the paper's reported results, at an effective 4.5-bit precision with negligible area cost.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The full paper text was available (the authors’ arXiv version, 2601.19213), so this account of the mechanism — microscaling formats and the power-of-two shared-scale problem, the three design axes with metadata as the under-explored one, the framing of metadata as extra mantissa or exponent applied element- or subgroup-level, the equivalent-bit-width-guided design space exploration, the weight-versus-activation asymmetry, the hybrid format (element-level extra mantissa for activations, subgroup refinement with adaptive scales for weights) at 0.25 bits per element and 4.5 effective bits, and the hardware (top-1 decode, augmented FP4-by-FP4 processing element, streaming quantization engine in a systolic array) — follows the paper’s own description. The headline numbers are the paper’s reported results: on average a 70.63 percent reduction in accuracy loss versus MXFP4 and 37.30 percent versus NVFP4 on LLM benchmarks, near-FP16 accuracy at 4.5 effective bits, and up to 1.91 times speedup and 1.75 times energy savings over state-of-the-art accelerators at negligible area cost. Those are averages and maxima over specific benchmarks, formats, and hardware; per-model results vary. Exact per-benchmark numbers, the design-space-exploration details, and the hardware specifics should be read from the paper itself.

Background

Why the last bits of a quantized model are the hardest

It helps to see why 4-bit quantization is where this fight happens. Dropping a model from 16 bits to 8 bits is relatively forgiving — there is enough resolution left that a straightforward format loses little accuracy. But each further halving of the bit budget squeezes the representable values harder, and around 4 bits the format runs out of room to capture the range and precision that neural network values actually need, so accuracy degrades sharply. The reason low-bit quantization is nonetheless pursued so aggressively is that the payoff scales with the model: a 405-billion-parameter model at 4 bits instead of 16 is a fourfold cut in the memory and bandwidth that dominate the cost of serving it, which can be the difference between fitting on a given accelerator and not. So the industry keeps pushing toward the lowest bit-widths that remain usable, and the whole game becomes recovering the accuracy that the aggressive compression throws away, using every trick — better scales, better data types, and now metadata — to claw it back. This relentless drive to run large models in as few bits as possible while defending their quality is the same pressure the narrow-integer accelerator work covered elsewhere on this site operates under, building hardware to make the smallest viable number formats fast. M2XFP’s contribution is to find that a sliver of well-placed metadata is a cheaper way to defend 4-bit accuracy than making the numbers themselves fancier.

If you remember only three things

1. Large language models must be quantized to low bit-widths to fit and run efficiently, and microscaling (MX) formats like MXFP4 are the industry standard: they quantize elements in blocks with a shared scaling factor. But at 4 bits the shared scale is a coarse power-of-two value that misaligns with each block's largest element, causing large rounding error and severe accuracy loss; making the scale a finer 8-bit float (NVFP4) narrows dynamic range, and richer custom data types cost too much hardware, leaving the "metadata" axis — cheap side information that adds precision without touching the datapath — under-explored and fragmented.
2. M2XFP is an algorithm-hardware co-design that spends a tiny, principled amount of metadata where it helps most. It frames metadata as extra mantissa (precision) or exponent (range) applied to a block's critical element or its subgroup scale, searches this space under an equivalent-bit-width budget, and discovers a weight-versus-activation asymmetry: activations (dynamic) need cheap real-time element-level encoding, weights (static) allow offline subgroup refinement with a scale search. The resulting hybrid — element-level extra mantissa for activations, subgroup refinement with adaptive scales for weights — uses only 0.25 bits of metadata per element (effective 4.5-bit) to reach near-16-bit accuracy.
3. Lightweight hardware (a top-1 decode unit, an augmented FP4-by-FP4 processing element, and a streaming quantization engine) folds into a systolic array without disrupting the matrix-multiply pipeline. On LLM benchmarks M2XFP cuts accuracy loss by 70.63 percent versus MXFP4 and 37.30 percent versus NVFP4, and delivers up to 1.91 times speedup and 1.75 times energy savings over state-of-the-art accelerators at negligible area. It trades a modest half-bit of budget, relies on error being concentrated on a block's top element, and requires the custom hardware to realize its gains.

Check yourself

Check your understanding

1. Why does 4-bit MXFP4 quantization lose so much accuracy?
2. What is the "metadata axis" and why is it attractive?
3. What asymmetry between weights and activations does M2XFP exploit?
4. What are M2XFP's main results and its trade-offs?

Sources

The paper is “M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization” by Weiming Hu, Zihan Zhang, Haoyan Zhang, Chen Zhang, Cong Guo, Yu Feng, and Jingwen Leng (Shanghai Jiao Tong University; several also Shanghai Qi Zhi Institute) and Tianchi Hu, Guanglin Li, Guipeng Hu, and Junsong Wang (Huawei), published at ASPLOS 2026, DOI 10.1145/3779212.3790185. It is licensed CC BY.

This explainer is based on the paper’s full text, obtained from the authors’ arXiv preprint (arXiv:2601.19213). The account of the mechanism — microscaling (MX) formats and the power-of-two (E8M0) shared-scale problem at 4 bits; the taxonomy of three design axes (scaling factor, data type, metadata) and the case that metadata is the under-explored one; the framing of metadata as extra mantissa (precision) or extra exponent (range) applied at element level or subgroup level, with fixed or adaptive shared scales; the equivalent-bit-width (EBW)-guided design space exploration; the identified weight-versus-activation asymmetry (element-level metadata for dynamic activations, subgroup-level refinement with scale search for static weights); the hybrid M2XFP format using element-level extra mantissa for activations and subgroup-level mantissa refinement with adaptive shared scales for weights at 0.25 bits of metadata per element and 4.5 effective bits; and the hardware (a top-1 decode unit, an augmented FP4-by-FP4 processing element, and a streaming quantization engine integrated into systolic arrays without disrupting the GEMM pipeline) — follows the paper’s own description. The reported results cited here are the paper’s own: on average a 70.63 percent reduction in accuracy loss compared to MXFP4 and a 37.30 percent reduction relative to NVFP4 on LLM benchmarks, near-FP16 accuracy at effective 4.5-bit precision, and up to 1.91 times speedup and 1.75 times energy savings over state-of-the-art accelerators at negligible area cost. Exact per-benchmark numbers, the design-space-exploration details, and the hardware specifics should be read from the paper itself.