MVQ: Towards Efficient DNN Compression and Acceleration with Masked Vector Quantization

Shuaiting Li, Chengxuan Wang, Juncan Deng, Zeyu Wang, Zewen Ye, Zongsheng Wang and 2 others

Vector quantization shrinks a neural network by replacing groups of weights with the nearest entry from a small codebook, but it treats every weight as equally important, so the few weights that matter get dragged toward the many that do not and accuracy collapses — MVQ prunes the unimportant weights first and then runs a masked k-means that ignores the pruned positions, so the codebook fits only what matters, and pairs it with a sparse systolic array that skips the pruned multiplies, reaching 75.2 percent on ResNet-50 at 22 times compression with 70 percent fewer operations and 2.3 times the energy efficiency of the base accelerator.

The argument in one glance

Problem

Weights do not fit on chip, and fetching them from DRAM is where the energy goes

A convolutional network for an edge device has millions of weights. The on-chip memory of a small accelerator cannot hold them, so weights stream in from DRAM, and in the paper's model one DRAM access costs about 200 times as much energy as one multiply-accumulate. Compressing weights is therefore the first lever for low-power inference, but every known method trades compression ratio against accuracy.
What people did

Vector quantization: store an index into a small codebook instead of the weights

Vector quantization (VQ) cuts the weight tensor into short subvectors, clusters them with k-means into k codewords, and stores only a log2(k)-bit index per subvector plus the codebook. With d = 8 weights per subvector and k = 512 codewords, 64 bits of 8-bit weights become a 9-bit index, and the accelerator only needs to load indices and look up codewords on chip.
Why it fell short

k-means treats every weight the same, so the important ones get averaged away

The paper shows the flaw with one experiment on ResNet-18. Mark the 2 largest weights of every 8 as important. Replace only those important weights with their vector-quantized values and accuracy falls from 69.7 to 5.8 percent. Replace only the other 75 percent instead, with a larger total error, and accuracy stays at 37.5 percent. Approximating the important weights well is what matters, and ordinary k-means, which minimizes total error, forces important weights to align with the near-zero ones around them.
This paper's idea

Prune the unimportant weights, then cluster with a mask so only the survivors count

MVQ first applies N:M pruning within each subvector, keeping for example 4 of 16 weights and recording a mask. Then it runs masked k-means: distances between a subvector and a codeword are computed only over the unpruned positions, and each codeword is updated as the average of the unpruned elements assigned to it. The codebook is quantized to 8 bits and fine-tuned with masked gradients. On ResNet-18 at about 22 times compression, this cuts the clustering error on important weights by 85 percent compared to clustering sparse weights with ordinary k-means, raises accuracy by 1.5 points over plain VQ, and removes 70 percent of the multiply operations. A co-designed accelerator with a sparse systolic array turns the removed operations into a 55 percent smaller array and 2.3 times the energy efficiency.

Why this problem is worth solving

Consider a small camera module that must classify images on a battery. Its accelerator has a 2 megabyte level-2 SRAM. ResNet-50 has 25 million parameters, which at 8 bits is 25 megabytes, so most of the weights live in off-chip DRAM and are fetched layer by layer. The paper’s energy model, from Table 8, puts one DRAM access at 200 units, one level-2 SRAM access at 15, one level-1 access at 6, and one multiply-accumulate at 1. Weight fetching, not arithmetic, sets the power budget.

Compression attacks the fetch. The paper reviews the alternatives. One-bit quantization compresses 32 times but loses too much accuracy. Unstructured pruning must store the position of every surviving weight. Structured pruning rarely exceeds 50 percent sparsity.

Vector quantization

Compressing a tensor by cutting it into short subvectors and replacing each with the index of its nearest entry in a small learned codebook.

can reach 20 to 30 times compression and is hardware-friendly, because the accelerator loads short indices and reconstructs weights from an on-chip

codebook

The small table of k learned subvectors (codewords) from which compressed weights are reconstructed by index.

. The question is whether it can do so without wrecking accuracy. Prior VQ methods such as BGD and PQF reach 74.2 percent on ResNet-50 at 22 times compression, against 79.2 percent uncompressed. MVQ reaches 75.2 percent, one point higher, and does it with 73 percent fewer floating-point operations.

What people did before, and where it broke

Ordinary vector quantization: BGD, PQF, DKM. Cluster weight subvectors with k-means or a variant. BGD includes activations in the clustering objective and distills layer by layer; PQF searches for a permutation of weights that clusters better; DKM makes clustering differentiable. Where it broke. All of them minimize error over every weight equally. The paper’s experiment in Table 1 shows that accuracy depends almost entirely on how well the important weights are approximated, and k-means forces them to share a codeword with whatever unimportant weights sit beside them in the subvector.
N:M pruning alone. Keep N of every M weights, a pattern that hardware such as the Nvidia Ampere sparse tensor core can exploit. Where it broke. Compression is limited to the pruning ratio, and pushing past 75 percent sparsity on ResNet-18 drops accuracy below the baseline.
Sparse accelerators: SparTen, CGNet, SPOTS, S2TA. Skip the zero multiplies in pruned models. Where it broke. They exploit sparsity but not codebook compression, so weights still stream from DRAM at close to full width, and their compression ratios are 3 to 10 times.
Earlier VQ accelerators. Two 2019 designs implemented VQ in hardware. Where it broke. Their VQ algorithms suffered large accuracy losses, and the dataflow was not designed around the compressed format.
The gap. Nobody had combined pruning and vector quantization so that the codebook approximates only the weights that matter, nor built an accelerator that profits from both the index-based loading and the sparsity at once.

Where MVQ sits

From a trained model to a running accelerator

Group and prune new

Reshape weights into subvectors of length d along the output-channel axis; apply N:M pruning inside each subvector and fine-tune; keep a mask.
↓ →
Masked k-means new

Cluster the sparse subvectors into k codewords using distances and averages over unpruned positions only.
↓ →
Quantize and fine-tune new

Quantize the codebook to signed 8 bits; fine-tune codewords with masked gradients.
↓ →
Store

Per subvector: a log2(k)-bit index and a compact mask code. Plus one small codebook per layer.
↓ →
Accelerator new

Weight loader looks up codewords by index, applies the mask, and feeds a sparse systolic array that has only N of M multipliers per subvector.

The three algorithm boxes and the accelerator box are the paper's contributions. The base dataflow, EWS, comes from the authors' earlier work.

The idea

Grouping. A convolution weight is a 4-D tensor of output channels, input channels and a kernel plane. MVQ groups along the output-channel axis: each subvector holds d weights that belong to d different output channels at the same input channel and kernel position. The paper chooses this over kernel-wise grouping because d can then be any power of two, which suits both the compression ratio and the hardware array width.

N:M pruning. Inside each subvector of length d, only N of every M weights survive, chosen by magnitude, and d is a multiple of M. This is

N:M pruning

A structured sparsity pattern in which, within every block of M consecutive weights, only N are kept non-zero.

. The sparse model is fine-tuned before clustering. For ResNet-18 the paper sweeps 6:16 to 3:16 and finds 4:16, that is 75 percent sparsity, gives the best clustering accuracy; for parameter-efficient models such as MobileNet-V2, 1:2 (50 percent) is the better balance, because 2:4 gains 1.7 points of pruning accuracy but costs an extra 0.25 bits per weight in mask storage.

Why the mask is cheap. A full bitmask costs one bit per weight, which is too much at 22 times compression. But an N:M block has only a fixed number of possible masks, the binomial coefficient of M choose N. So MVQ stores a code of ceil(log2(M choose N)) bits per block and expands it through a lookup table in the weight loader. For 4:16 that is 11 bits per 16 weights.

Masked k-means. Ordinary

k-means

An iterative clustering method that assigns each vector to its nearest codeword and then moves each codeword to the mean of its assigned vectors.

alternates assignment and update. MVQ modifies both steps with the mask. In masked assignment, a subvector is assigned to the codeword that is nearest when the distance is computed only over its unpruned positions, so a codeword is not penalized for whatever value it holds at positions the subvector has pruned. In masked update, each element of a codeword becomes the average of that element over only the subvectors that keep that position, an element-wise division by the mask count. The pruned zeros never pull the codeword toward zero. On a GPU this is implemented by broadcasting the codebook against every mask pattern and calling a batched distance function.

Codebook quantization and fine-tuning. The codebook is quantized to signed 8-bit values with one learned scale per codebook, using learned step-size quantization. Then the network is fine-tuned end to end. In the forward pass, weights are reconstructed by looking up each codeword and applying the mask. In the backward pass, gradients flow to the codewords as masked averages, again so pruned positions contribute nothing.

The accelerator. The base is the authors’ Enhanced Weight Stationary (EWS)

systolic array

A grid of processing elements in which data flows between neighbours each cycle; here the weights stay fixed in each element while activations stream through.

, which keeps weights stationary in the array and adds register files so activations and partial sums are reused across output channels, input channels and kernel positions rather than refetched from SRAM. MVQ adds two things. First, an assignment-aware weight loader: the codebook is loaded once into a codebook register file with L/d read ports for an array L elements wide, and thereafter only indices and mask codes are read from level-2 SRAM; the mask is expanded and applied with AND gates to produce the sparse weight vector. Second, a sparsity-aware tile: since only N of M weights in a stationary subvector are non-zero, a group needs only Q = N/M times d multipliers instead of d. Cascaded leading-zero counters turn the d-bit mask into Q position codes, and demultiplexers route each product to the right position in the adder tree, which keeps its full depth. A zero-gated processing element prefetches the next weight and activation and holds the multiplier inputs still when the next product would be zero, cutting switching power.

Tracing masked k-means on two subvectors

How the mask changes clustering

This is the paper's Figure 4 example. Two subvectors of length 4, 2:4 pruned, are both assigned to one codeword. A dot in a position means it is pruned to zero. Distances are squared Euclidean.

Step 1 of 6

The inputs

Subvector 1 keeps positions 1 and 2 with values 0.7 and 0.7; positions 3 and 4 are pruned. Subvector 2 keeps positions 2, 3 and 4 with 0.5, 0.5, 0.5, and position 1 is pruned (in this illustration the paper draws it with three survivors). The current codeword is 0.6, 0.6, 0.6, 0.6.

s1 = [0.7, 0.7, ., .]  mask1 = 1100 | s2 = [., 0.5, 0.5, 0.5]  mask2 = 0111 | c = [0.6, 0.6, 0.6, 0.6]

Why it works

Accuracy lives in the important weights. The Table 1 experiment shows that quantizing only the 25 percent largest weights costs 64 points of accuracy while quantizing the other 75 percent costs 32, despite the latter having larger total error. Because MVQ removes the unimportant weights before clustering, the codebook’s capacity is spent entirely on the weights that decide accuracy.
Pruning shrinks the clustering problem. Fewer non-zero values per subvector means fewer distinct patterns to represent, so the same k codewords fit the survivors more tightly. Higher pruning rates reduce clustering error, which is why the paper pushes sparsity as far as pruning accuracy allows.
Masking removes the pull of zeros. Without the mask, sparse subvectors still drag codewords toward zero at every pruned position (case B in the ablation: total error falls but important-weight error barely moves). With the mask, the error on important weights drops 85 percent relative to clustering sparse weights with ordinary k-means.
The same sparsity pays twice in hardware. Because the weights are stationary in the array, a subvector with N of M non-zeros needs only N/M of the multipliers, permanently. And because weights are stored as indices, the weight-loading datawidth from SRAM shrinks by the compression ratio, which the roofline analysis identifies as the bottleneck for arrays of 32 by 32 and larger.

What it costs

Three stages of training. Prune and fine-tune, cluster, then fine-tune again. For classification the sparse model is trained with SR-STE; for detection and segmentation that was unstable and the simpler ASP method was used instead.
Mask storage. The compact mask code is small but not free. On MobileNet-V2, moving from 1:2 to 2:4 costs 0.25 bits per weight. Cases in the ablation that do not store a mask cannot reduce operations at all, so the mask is the price of the FLOP savings.
Codebook register file area. The codebook needs L/d read ports for an L-wide array, and its area grows with array size. At 64 by 64 the sparse tile saves enough that the total is still 55 percent smaller than base EWS, but the register file is a visible cost in Table 7.
Depthwise convolutions gain little. They map to the diagonal of the array and have few parameters, so weight loading is not their bottleneck; the paper reports pointwise-only results for MobileNet.
Parameter-efficient models tolerate less pruning. MobileNets and EfficientNet lose noticeable accuracy already at 50 percent sparsity, so their compression ratio and FLOP savings are lower than for ResNets.
The energy figures exclude DRAM in one place. The TOPS-per-watt comparison in Figure 19 omits main-memory energy; the DRAM savings are reported separately as a 2 to 4.1 times reduction in data-access cost.

The evaluation

The algorithm is evaluated on ImageNet classification (ResNet-18, ResNet-50, MobileNet-V1 and V2, EfficientNet, AlexNet, VGG-16), COCO detection and segmentation (Mask R-CNN with a ResNet-50 backbone) and Pascal VOC segmentation (DeepLab-V3 with a MobileNet-V2 backbone). The accelerator is synthesized in a 40 nanometre library at 0.99 volts with Synopsys Design Compiler and measured on gate-level simulation with PrimeTime PX, inside the open-source PULP system, at array sizes 16, 32 and 64.

Algorithm: accuracy at matched compression ratio

Property	Prior VQ (PQF, BGD, or PvQ)	MVQ
ResNet-50, about 22x, top-1 (uncompressed 79.2)
ResNet-18, about 22x, top-1 (uncompressed 69.7)	PQF 68.2
MobileNet-V2, about 16x (uncompressed 71.7)	PvQ 2-bit uniform 59.1
Mask R-CNN on COCO, 26x, box AP (uncompressed 37.9)	BGD 33.9, PQF 36.3
DeepLab-V3 on VOC, mIoU (uncompressed 72.9)	PvQ at 16x: 17.6

From Tables 4, 5 and 6 and Figure 13 of the paper. Compression ratio counts assignments, masks and codebook against 32-bit weights. Layer-wise codebooks (one per layer) beat a single cross-layer codebook.Underlined values explain themselves when clicked.

Hardware: 64 by 64 array, 40 nm, 75 percent sparsity, ResNet-18

Property	EWS base (8-bit weights)	EWS-C (ordinary VQ)
Accelerator area, square millimetres	4.24
Speedup over EWS base	1.0	—
Energy efficiency, TOPS per watt (excluding DRAM)	2.9	3.0
Data-access energy including DRAM	1.0	—
Versus prior sparse accelerators, normalized to 40 nm	—	—

From Tables 7 and 9 and Figures 15, 17 and 19. Level-1 SRAM is 128 KB for the 16x16 array and 256 KB for larger ones; level-2 is 2 MB throughout.Underlined values explain themselves when clicked.

Where this stops working

MVQ depends on there being unimportant weights to prune. Parameter-efficient networks such as MobileNet and EfficientNet already lose accuracy at 50 percent sparsity, so their compression and FLOP savings are roughly half those of ResNets, and depthwise layers see almost no hardware benefit. The method is evaluated on convolutional networks; nothing in the paper covers transformers or attention layers. The accelerator’s headline efficiency figures exclude DRAM energy, and the DRAM savings depend on activations fitting on chip, which fails for the large early layers of VGG-16. Finally, the pipeline requires three training stages and a choice of pruning pattern per model family, and for detection and segmentation the pruning method had to be swapped because the classification recipe was unstable.

Background

Why important weights drown in ordinary k-means

A subvector of 8 weights taken across output channels typically contains one or two large values and several near zero. k-means assigns it to the codeword nearest in all 8 dimensions and then moves that codeword to the mean of everything assigned to it. The many near-zero entries dominate both decisions, so codewords cluster around small values and the occasional large weight is approximated by whatever small value the codeword holds in that position. Pruning removes the near-zero entries; masking stops the zeros left behind from having the same effect. Case B in the ablation isolates the difference: pruning alone drops total error from 1153 to 518 but leaves important-weight error at 498, while pruning plus masking brings it to 251.

If you remember only three things

1. Vector quantization stores an index per subvector into a small codebook, so an accelerator loads a few bits instead of full weights, but ordinary k-means minimizes error over all weights equally and drags the few important ones toward the many near-zero ones. The paper shows accuracy depends almost entirely on the important weights: quantizing only the largest 25 percent costs 64 points on ResNet-18, quantizing the rest costs 32.
2. MVQ prunes each subvector with an N:M pattern (4:16 for ResNets, 1:2 for MobileNets), stores the mask as a compact code, and clusters with masked k-means: distances count only unpruned positions and each codeword element is the average over the subvectors that keep it. After 8-bit codebook quantization and masked-gradient fine-tuning, important-weight clustering error falls 85 percent and ResNet-50 reaches 75.2 percent at 22x compression, 1 point above PQF, with 70 percent fewer operations.
3. A co-designed accelerator on the EWS dataflow loads indices and mask codes, expands them through a lookup table, and runs a sparse systolic tile with only N/M of the multipliers per group, routing products with leading-zero counters and demultiplexers. At 64x64 in 40 nm it is 55 percent smaller, 1.2 to 1.8x faster and 2.3x more energy-efficient than base EWS, and 1.73x more efficient than the best prior sparse accelerator, with 2 to 4.1x less data-access energy.

Check yourself

Check your understanding

1. In the paper's motivating experiment, replacing only the 25 percent largest weights with their VQ approximation produced higher total error or lower total error than replacing the other 75 percent, and which hurt accuracy more?
2. What does masked k-means change relative to ordinary k-means?
3. Predict what happens to the sparse systolic tile if a layer is 2:4 pruned instead of 4:16.
4. Why does MVQ store a short code per block rather than a bitmask with one bit per weight?
5. Why does the speedup from MVQ grow with array size in the roofline analysis?

Sources

The paper is “MVQ: Towards Efficient DNN Compression and Acceleration with Masked Vector Quantization” by Shuaiting Li, Chengxuan Wang, Juncan Deng, Zeyu Wang, Zewen Ye, Zongsheng Wang, Haibin Shen and Kejie Huang, Zhejiang University, published at ASPLOS 2025, Volume 1, DOI 10.1145/3669940.3707268. This explainer was written from the full text of the arXiv preprint (2412.10261, version 2, December 2024). Numbers are taken from Tables 1 to 9 and Figures 10 to 20 of that version.