COMET: Towards Practical W4A4KV4 LLMs Serving

Lian Liu, Long Cheng, Haimeng Ren, Zhaohui Xu, Yudong Pan, Mengdi Wang and 3 others

Modern GPUs have super-fast 4-bit tensor cores, but serving a large language model can't use them, because a few wild outlier values in the activations wreck 4-bit accuracy, and existing quantization either keeps activations at 8 or 16 bits or ignores how the GPU actually computes; this work from the Chinese Academy of Sciences builds COMET, the first practical scheme to run weights, activations, and the KV cache all at 4 bits — quantizing only the outlier channels to 8-bit and co-designing a GPU kernel that hides the conversion cost — for 2.02 times more serving throughput.

The argument in one glance

Problem

GPUs have fast 4-bit tensor cores, but LLM serving cannot use them

Serving a large language model is expensive in two ways: memory, to hold the weights and the ever-growing key-value cache of past tokens, and compute, to run the matrix multiplications through the GPU's tensor cores. Modern GPUs offer low-precision tensor cores that are far faster — an A100's 4-bit cores run at twice the throughput of its 8-bit cores and four times its 16-bit cores. So compressing the whole model to 4 bits would both shrink memory and unlock that speed. The catch is that squeezing everything to 4 bits, especially the activations, has not been made to work in practice without wrecking either accuracy or the very efficiency it aims for.
What people did

Quantize the weights, or both weights and activations to 8 bits

The common compromises each give up something. Weight-only quantization stores 4-bit weights but keeps 16-bit activations, so it saves memory on the weights. Weight-activation quantization like SmoothQuant pushes both down to 8 bits. And a line of finer-grained schemes quantizes activations or the key-value cache per channel or per token to preserve accuracy under aggressive compression.
Why it fell short

Each wastes the hardware, and 4-bit activations break on outliers

Weight-only quantization must dequantize the 4-bit weights back to 16 bits to match the activations before the 16-bit tensor cores can run — wasting compute — and it does nothing for the key-value cache, which dominates memory at long sequences. Eight-bit quantization leaves the twice-as-fast 4-bit tensor cores idle. The fine-grained schemes quantize at a granularity that does not line up with how the GPU computes, so their dequantization overhead swamps the savings. And naive 4-bit activations fail outright: a small cluster of outlier values, hundreds of times larger than the rest, forces everything else to round toward zero.
This paper's idea

Make 4-bit work for the common case, and co-design the GPU kernel

COMET achieves the first practical serving with 4-bit weights, activations, and key-value cache. Its algorithm quantizes most activations to 4 bits and only the rare outlier channels to 8 bits, in blocks sized to match the GPU's compute granularity, clustering the outliers together so few blocks pay the higher precision. Its co-designed kernel then runs this mixed-precision math efficiently — hiding the conversion cost behind computation, laying data out to avoid conflicts, and balancing the fast and slow tiles across the GPU's cores — while the 4-bit cache lifts the memory ceiling.

Why this problem is worth solving

The heart of the opportunity is that an LLM’s two costs are bound by two different hardware limits, and 4 bits helps both — if you can make it work.

LLM inference runs in two phases: a prefill that processes the whole prompt at once, and a decode that generates tokens one at a time, autoregressively, caching each token’s Key and Value activations so they need not be recomputed — the key-value cache, which grows with the sequence and at long lengths dominates memory (over 70 percent of the footprint for a small model at a 128,000-token sequence). Quantization compresses this, but a roofline analysis shows the parts of the model are bound differently. The attention operators that read the cache have a fixed, low compute intensity, so they are memory-bandwidth-bound: shrinking the cache to 4 bits directly speeds them up and, by freeing memory, lets you run larger batches. The weight-times-activation linear layers, by contrast, become compute-bound at large batch, so quantizing their activations to 4 bits lets them use the GPU’s fastest tensor cores. This is why the target is W4A4KV4 — 4-bit weights, 4-bit activations, 4-bit cache — rather than any one of them: 4-bit activations feed the fast cores where compute binds, and a 4-bit cache lifts the memory ceiling where bandwidth binds. Prior work took only half-measures — 4-bit weights but 16-bit activations, or 8 bits across the board — because pushing activations to 4 bits without destroying accuracy or efficiency was the unsolved piece. Solving it is what turns quantization from a modest memory saving into a full exploitation of modern GPU hardware.

The reason 4-bit activations resisted a practical solution is a collision between where the hard values are and how the GPU wants to compute.

Once an LLM passes roughly six billion parameters, a small set of its hidden-state features — under one percent of them — take on magnitudes tens or hundreds of times larger than the rest. These outliers are essential to the model’s behavior and cannot be dropped, but they poison quantization: a min-max scheme that must span an outlier of 30 alongside normal values near 1 will round almost every normal value to zero, gutting accuracy. The natural fix is mixed precision — keep the outliers at higher precision and the normal values at 4 bits — and fortunately the outliers concentrate in specific feature channels, consistently across tokens, so they can be separated out. But here the second problem bites: a GPU tensor core computes at a fixed granularity, a tile of a certain size, and it is only efficient when the quantization blocks are whole multiples of that tile. Prior fine-grained schemes quantized per channel or in irregular clusters, which do not align to the tile, so the GPU must constantly reshuffle and dequantize, and the overhead of that data juggling erases the benefit of the lower precision. So the real problem is not just isolating outliers for accuracy, but doing it in a shape the hardware can execute fast — a joint algorithm-and-systems problem, which is exactly why a co-designed solution is worth building.

What people did before, and where it broke

Weight-only quantization to 4 bits (GPTQ, AWQ, OmniQuant). Store 4-bit weights alongside 16-bit activations. Where it broke. The 4-bit weights must be dequantized back to 16 bits to match the activations before the 16-bit tensor cores run, wasting compute, and it leaves the key-value cache — the real memory bottleneck at long sequences — untouched.
Weight-activation quantization to 8 bits (SmoothQuant). Shift quantization difficulty from activations to weights so both fit in 8 bits. Where it broke. Eight bits is conservative: it leaves the twice-as-fast 4-bit tensor cores idle, capping the speedup well below what the hardware can give.
Fine-grained per-channel or per-token quantization (ZeroQuant, Atom, RPTQ, KVQuant). Quantize activations or the cache at fine granularity to hold accuracy under aggressive compression. Where it broke. The granularity does not align with the GPU’s fixed compute tile, so the constant reshuffling and dequantization overhead cannot be processed efficiently and swamps the savings.
Naive uniform 4-bit activation quantization. Just quantize all activations to 4 bits. Where it broke. Outlier values hundreds of times larger than normal ones force everything else to round toward zero — a full-4-bit scheme raises perplexity by more than 5, far beyond usable.
The gap. Every approach either kept activations too wide to use the fast cores, ignored the cache, or quantized in a shape the GPU could not run efficiently — none achieved 4-bit activations that are both accurate and hardware-efficient. What was missing was a mixed-precision scheme aligned to the GPU’s granularity plus a kernel that executes it fast, which is exactly what practical W4A4KV4 serving needs.

How COMET works

Making all-4-bit serving accurate and hardware-efficient

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Spend full precision only on the rare hard values, and 4 bits on the common case. The whole obstacle to 4-bit activations is a tiny cluster of outliers that would drag the entire tensor’s range wide, but those outliers sit in a few fixed channels, so COMET keeps just them at 8 bits and quantizes the overwhelming majority — over 84 percent of the matrix-multiply work — at 4 bits, with negligible accuracy loss where a uniform 4-bit scheme would be unusable. Exploiting that the difficulty concentrates in a small, identifiable subset, and handling only that subset expensively while the common case runs cheap, is the same strategy the Mixture-of-Experts inference work covered elsewhere on this site uses, singling out the few hot experts for special treatment so the bulk of the serving work stays on the fast path.

Hide the slow conversion work behind the fast computation. Mixed precision forces extra steps — dequantizing and converting 4-bit data to 8-bit — that run on the GPU’s general-purpose cores, which are far slower than the tensor cores, so left in the open they would bottleneck everything. COMET pipelines them, using double-buffered shared memory to convert the next tile while the tensor cores multiply the current one, so the conversion never sits on the critical path and the fast 4-bit cores stay saturated. Overlapping unavoidable auxiliary work behind the main computation so it disappears from the timeline is exactly the discipline the communication-overlapping scheduler covered elsewhere on this site applies, keeping the expensive side-work running quietly underneath the math that was going to happen anyway.

Shape the algorithm to the hardware’s real execution units, then balance the load. A quantization scheme that is accurate but ignores how the GPU actually computes gains nothing, because the tensor core runs at a fixed tile granularity and mismatched blocks force constant reshuffling. COMET sizes its quantization blocks to whole multiples of that tile, lays weights out to avoid memory-bank conflicts, and remaps the fast and slow tiles so no streaming multiprocessor idles at a barrier — making the math fit the silicon. Co-designing the computation to match the GPU’s true execution model and keep every unit busy is the same discipline the recommendation-model compiler work covered elsewhere on this site follows, reshaping the workload to the GPU’s thread-block structure so the hardware is actually utilized rather than merely occupied.

Following a linear layer through COMET

Running a quantized matrix multiply during decode

Tracing one linear layer shows the algorithm and kernel working together.

Step 1 of 5

A linear layer must multiply activations by weights

During token generation, a decoder layer multiplies a batch of activations by a weight matrix. The weights are already 4-bit, but the activations contain a few outlier channels whose values are hundreds of times larger than the rest, which naive 4-bit quantization would ruin.

At large batch this linear layer is compute-bound, so its precision decides how fast the tensor cores run.

Why it works

Isolating outliers into higher precision lets the rest go to 4 bits accurately. Outliers hundreds of times larger than normal values would force everything to round toward zero → but they live in a few fixed channels, so keeping just those at 8 bits and the rest at 4 preserves the range → so over 84 percent of the matrix-multiply work runs at 4 bits with negligible perplexity loss.
Block-wise quantization with channel permutation fits the GPU’s granularity. Per-channel schemes misalign with the fixed tensor-core tile and drown in reshuffling → but blocks sized to the tile, with the scattered outliers permuted into a few blocks, align to the hardware → so fewer than a fifth of blocks need 8 bits and the majority run as efficient 4-bit tiles.
Quantizing the cache to 4 bits relieves the memory-bound bottleneck. The attention operators reading the cache are limited by memory bandwidth, and the cache dominates footprint at long sequences → but its values carry few outliers, tamed further by the model’s normalizations, so a full 4-bit cache is accurate → so memory frees up, larger batches fit, and those memory-bound operators speed up.
Pipelining and fast conversion keep the slow steps off the critical path. Mixed precision needs dequantization and 4-to-8-bit conversion on the slow general cores → but a double-buffered pipeline overlaps that with tensor-core work, and a conflict-free layout with two-instruction conversion makes it cheap → so the fast 4-bit cores stay saturated instead of waiting.
Balancing tiles across cores stops the fast units from idling. The quick 4-bit tiles otherwise wait at barriers for the slower 8-bit tiles, leaving their cores idle → but remapping and splitting tiles gives every streaming multiprocessor a balanced mix → so utilization rises and the kernel runs up to 2.88 times faster than the 16-bit baseline.

What it costs (trade-offs and scope)

It is not truly all-4-bit, and full 4-bit remains impractical. A minority of activations — roughly 16 percent, as low as 8 percent on some models — still need 8 bits for the outliers; a uniform 4-bit scheme raises perplexity by more than 5 and is unusable, so the mixed-precision compromise is essential, not optional.
There is a small but real accuracy cost. Zero-shot accuracy drops about 0.75 percent and perplexity rises around 0.05 versus higher-precision baselines — negligible for most uses, but a genuine loss.
The outlier handling needs calibration and a matching weight permutation. Locating the outlier channels requires calibration data offline, and the channel permutation must be applied to the weights too; the runtime permutation is cheap (under 1 percent) but it is extra machinery.
The kernel fixes its tile partitioning, hurting some shapes. To support mixed-precision tiles, COMET uses a fixed tile-partitioning strategy rather than the shape-adaptive partitioning a plain library uses, so it underperforms on certain matrix shapes.
The gains concentrate at large batch and long sequences. The 4-bit activations help most when the linear layers are compute-bound (large batch) and the 4-bit cache helps most with long outputs; small-batch kernel speedups are more modest (around 1.25 to 1.48 times).
The conversion tricks are hardware-specific. The fast 4-to-8-bit conversion and layout are tuned for the A100’s INT4 and INT8 tensor cores; they adapt to newer GPUs’ FP4 with changes, but the results and low-level optimizations are for this hardware, and this is a serving-efficiency co-design, not a new model or accuracy method.

The evaluation

Three ways to quantize an LLM for serving

Property	Weight-only (W4A16)	Weight-activation (W8A8)
Activation precision		8-bit
Key-value cache		8-bit
Uses the fast 4-bit tensor cores
Dequantization overhead	High, on weights	Moderate
Serving throughput	Baseline	Modest gain

Three ways to quantize an LLM for serving, evaluated on a single A100-80G across the LLaMA family, Mistral, OPT, and Qwen2 models. COMET's FMPQ algorithm runs over 84 percent of matrix-multiply work at 4-bit with only about 0.05 higher perplexity and 0.75 percent lower zero-shot accuracy, and its co-designed W4Ax kernel reaches a 2.88 times kernel speedup over a 16-bit baseline and a 2.02 times end-to-end throughput gain over 4-bit-weight serving — because it makes 4-bit activations and a 4-bit cache both accurate and hardware-efficient, using the fast INT4 tensor cores that prior schemes leave idle.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

COMET is a well-executed algorithm-systems co-design that unlocks something the field had only half-achieved: serving with 4-bit weights, activations, and key-value cache together, using the fast 4-bit tensor cores that prior quantization leaves idle. The algorithmic half (FMPQ) keeps the rare outlier channels at 8 bits and the rest at 4, in blocks matched to the GPU’s tile granularity, so over 84 percent of the compute runs at 4-bit with negligible accuracy loss where uniform 4-bit is unusable; the systems half (the W4Ax kernel) pipelines the mixed-precision math to hide conversion, lays data out to avoid conflicts, and balances tiles across cores. The payoff is real: a 2.88 times kernel speedup and 2.02 times end-to-end throughput on an A100. Read it with its scope. It is not truly all-4-bit — a slice of activations stays 8-bit, and full 4-bit remains impractical; there is a small accuracy cost (about 0.75 percent zero-shot, 0.05 perplexity); the outlier handling needs calibration and a weight permutation; the fixed tile partitioning hurts some matrix shapes; the gains concentrate at large batch and long sequences; and the low-level conversion tricks are tuned to the A100’s tensor cores. The durable lesson is that pushing quantization to the hardware’s fastest precision is not an accuracy problem alone but a co-design problem — the compression only pays if it is shaped to how the silicon actually computes.

Background

Why serving efficiency is now about feeding the fast units, not adding them

It helps to see COMET against where LLM-serving efficiency actually lives now. The tensor cores on a modern GPU are extraordinarily fast, and getting faster with each lower precision — but that speed is only useful if you can keep them fed, and increasingly the binding constraints are elsewhere: the memory to hold the growing key-value cache, the bandwidth to stream it, and the overhead of getting data into the precise format the cores demand. So the leverage has shifted from raw arithmetic to marshaling: compressing what must be stored and moved, and shaping the computation so the fast units never wait. COMET is a study in exactly that — its 4-bit cache attacks the memory wall, its pipeline hides the format-conversion overhead, and its scheduling stops the fast cores idling. The deep-learning-on-GPUs retrospective covered elsewhere on this site makes the general case that the era of simply adding arithmetic is over, and that the bottleneck has migrated to moving and feeding data, so the useful engineering is now in the memory and data path rather than the compute. The broad lesson is that as hardware makes computation ever cheaper per operation, the systems that win are the ones that treat data movement and format as the scarce resource — quantizing, laying out, and scheduling so the abundant compute is actually put to work.

If you remember only three things

1. Serving an LLM is bound by memory (weights and the growing key-value cache) and by feeding the GPU's tensor cores, whose 4-bit units run twice as fast as 8-bit and four times as fast as 16-bit. Compressing to 4 bits everywhere (W4A4KV4) would exploit both, but prior work fell short: weight-only 4-bit quantization dequantizes weights to 16-bit (wasting compute) and ignores the cache; 8-bit weight-activation quantization leaves the fast 4-bit cores idle; fine-grained schemes misalign with the GPU's fixed compute tile and drown in dequantization overhead; and naive 4-bit activations fail because a tiny cluster of outlier values, hundreds of times larger than normal, forces everything to round toward zero (raising perplexity by over 5).
2. COMET achieves the first practical W4A4KV4 serving through algorithm-system co-design. Its FMPQ algorithm exploits that outliers sit in a few fixed channels: it block-quantizes activations at a granularity matched to the tensor-core tile, permutes channels to cluster outliers into a few 8-bit blocks (under 20 percent), and 4-bit-quantizes the rest — over 84 percent of the compute — plus a full 4-bit key-value cache (memory-bound and outlier-light). Its co-designed W4Ax GPU kernel runs the mixed 4-bit/8-bit matrix multiplies with a double-buffered software pipeline that hides the slow CUDA-core dequantization behind tensor-core work, a conflict-free weight interleaving and a two-instruction (versus ten) 4-to-8-bit conversion, and fine-grained SM scheduling (tile remapping and decomposition) that balances the fast 4-bit and slow 8-bit tiles so no core idles at a barrier.
3. On a single A100-80G across LLaMA-1/2/3, Mistral, OPT, and Qwen2 models, FMPQ raises perplexity only about 0.05 and drops zero-shot accuracy about 0.75 percent while running over 84 percent of work at 4-bit, and the W4Ax kernel reaches a 2.88 times kernel speedup over a 16-bit baseline and a 2.02 times end-to-end throughput gain over 4-bit-weight serving (1.17 times over Qserve), largest at big batches and long outputs. Its limits: a slice of activations stays 8-bit and full 4-bit is still impractical, there is a small accuracy cost, outlier handling needs calibration and a weight permutation, the fixed tile partitioning hurts some shapes, and the low-level conversion tricks are tuned to the A100's INT4/INT8 tensor cores.

Check yourself

Check your understanding

1. Why do weight-only (4-bit weight, 16-bit activation) quantization and 8-bit quantization both leave performance on the table?
2. How does COMET make 4-bit activations accurate without wrecking the model?
3. What does COMET's custom W4Ax kernel do to run the mixed-precision math efficiently?
4. Which statement best reflects the scope and limits of COMET?

Sources

The paper is “COMET: Towards Practical W4A4KV4 LLMs Serving” by Lian Liu, Long Cheng, Haimeng Ren, Zhaohui Xu, Yudong Pan, Mengdi Wang, Xiaowei Li, Yinhe Han, and Ying Wang, published at ASPLOS 2025, DOI 10.1145/3676641.3716252. The authors are at the Institute of Computing Technology of the Chinese Academy of Sciences and the University of Chinese Academy of Sciences, North China Electric Power University, ShanghaiTech University, and Zhongguancun Laboratory, and COMET is open-source.

This explainer is written from the full 16-page text. Everything above comes from the paper’s own sections: the introduction and background on LLM inference (prefill and decode phases, the key-value cache growing to dominate memory) and the two quantization families (weight-only and weight-activation), with the roofline analysis showing activation-activation operators are memory-bound while weight-activation operators become compute-bound at large batch; the FMPQ algorithm analyzing outlier distribution (concentrated in fixed channels above roughly six billion parameters), adopting block-wise mixed-precision quantization aligned to the tensor-core granularity (128-channel blocks) with channel permutation to cluster outliers into fewer than 20 percent of blocks, and full 4-bit channel-wise quantization of the outlier-light key-value cache; the COMET-W4Ax kernel design with its SIMT-enhanced double-buffered software pipeline, weight interleaving to avoid shared-memory bank conflicts, fast INT4-to-INT8 conversion via location switch and zero extension (two instructions versus ten), and fine-grained SM scheduling through barrier minimization, tile remapping, and tile decomposition; and the evaluation on an A100-80G-SXM4 reporting FMPQ perplexity increases as small as 0.04 to 0.07 (0.05 with cache quantization) versus a 5.21 rise for uniform W4A4, a 0.75 percent zero-shot accuracy drop, over 84 percent of GEMM at W4A4, kernel speedups of 1.48 to 2.91 times (2.88 times average at large batch over cuBLAS), and 2.02 times end-to-end throughput over TensorRT-LLM W4A16 with 1.17 times over Qserve, plus the ablations isolating the pipeline, weight interleaving, conversion, scheduling, and the separate 1.32-times weight-activation and 1.17-times cache contributions.