Mugi: Value Level Parallelism For Efficient LLMs

When you multiply matrices in low precision, the numbers can only take a few distinct values, so the same product gets computed over and over — "value-level parallelism" (VLP) computes each once and reuses it, but it was built only for the plain matrix multiplies of older networks; this paper (University of Central Florida and Carnegie Mellon — Daniel Price, John Paul Shen, Di Wu, and colleagues) extends VLP to the trickier operations LLMs need (nonlinear functions like softmax, mixed-precision small-batch matrix multiplies) and packs it into one chip, Mugi, that runs everything faster, greener, and on the same hardware array.

The argument in one glance

Problem

Low-precision matrix multiply recomputes the same products over and over

The dominant cost in running a neural network is matrix multiplication (GEMM), which accounts for over ninety percent of the runtime, so an enormous amount of research has gone into making it efficient. One of the biggest wins has been low numerical precision: representing the activations and weights in narrow formats (like 8-bit or 4-bit numbers) instead of full precision, which is much cheaper and, done carefully, barely hurts accuracy. But low precision creates a curious kind of waste. When a number can only take a few distinct low-precision values, a large matrix multiply ends up computing the same product — the same value times the same weight — many times over, because that value recurs all across the input. A technique called value-level parallelism (VLP) exploits this: instead of redundantly recomputing a repeated product, compute each distinct product once and reuse it wherever that value appears. On large batches of low-precision data, where values repeat constantly, VLP can save a great deal of work. The catch is that VLP was designed for exactly one thing — the plain activation-weight matrix multiplies of older networks, with the same low precision on both sides. Modern large language models are not that simple, and it was an open question whether VLP could help them at all.
What people did

Apply value-level parallelism to LLMs as-is

The natural idea is to take VLP, which already speeds up low-precision matrix multiply, and use it to accelerate transformer-based large language models. LLMs spend most of their time in matrix multiplies too, so on the surface VLP's value-reuse trick should carry over: find the repeated low-precision products and compute each only once. Prior VLP hardware, such as a design called Carat, demonstrated this for symmetric low-precision (both sides in 8-bit) matrix multiply.
Why it fell short

LLMs break VLP's assumptions in four ways

Applying VLP to LLMs directly runs into four problems, because LLMs do more, and do it differently, than the workloads VLP was built for. First, LLMs are full of nonlinear operations — softmax in attention, and activation functions like SiLU and GELU — which are far more complex than the simple ReLU of older networks and take significant runtime; prior VLP hardware cannot do these at all. Second, LLMs use asymmetric, mixed-precision quantization: to fit in memory, they keep weights and the key-value (KV) cache in very low precision (like 4-bit integers) while activations stay in a wider format (like 16-bit), so a matrix multiply has one narrow side and one wide side — but prior VLP assumed both sides share the same low precision. Third, real-time LLM inference uses small batch sizes (as small as 8) to meet latency targets of a couple hundred milliseconds, whereas VLP's savings came from large batches where values repeat often; on small batches there is far less repetition to exploit. Fourth, and separately, existing AI chips use separate hardware units for matrix multiply and for the nonlinear operations, which adds chip area and, crucially, embodied carbon — the emissions from manufacturing the silicon — that can outweigh the carbon of actually running the model. So VLP as it stood could neither handle the operations LLMs need, nor match their precision and batch patterns, nor help with the sustainability cost of extra hardware.
This paper's idea

Extend value-level parallelism to LLMs and fuse it into one array: Mugi

Mugi is a new VLP architecture that overcomes all four problems and runs full LLM workloads on a single unified array. First, it brings value-level parallelism to nonlinear operations for the first time. The trick is a value-centric input approximation: rather than the usual approach of feeding a precise input to hardware that approximates the output, Mugi approximates the input (mapping it to a nearby representative value) and then produces a precise output for that approximate input — and it assigns greater accuracy to the more important input values, so the approximation is concentrated where it matters. This lets softmax, SiLU, and GELU run on the same value-reuse machinery as matrix multiply, and it beats existing nonlinear approximations on end-to-end LLM accuracy, performance, and efficiency. Second, Mugi optimizes VLP for the asymmetric, small-batch matrix multiplies LLMs actually use — the wide-times-narrow (for example, 16-bit activations against 4-bit weights) products produced by weight-only quantization, KV-cache quantization, and group-query attention — which prior VLP could not handle. Third, because Mugi can now do both the nonlinear operations and the matrix multiplies with value-level parallelism, it reuses the same hardware array for both, eliminating the separate nonlinear units that added area and manufacturing carbon. The result, on multiple LLMs, is large gains: throughput and energy efficiency up to 45 times and 668 times for nonlinear softmax operations, and 2.07 times and 3.11 times for full LLMs, while cutting operational carbon by 1.45 times and embodied carbon by 1.48 times.

Why this problem is worth solving

Matrix multiply and the operations around it dominate LLM cost, so squeezing redundant work out of low-precision computation pays off broadly, and doing it sustainably matters more as hardware advances.

Running large language models is enormously expensive in compute and energy, and the overwhelming majority of that cost is matrix multiplication, which is why so much effort targets it. Low precision is the biggest lever for efficiency, but it introduces exactly the redundancy value-level parallelism can remove — the same low-precision products recomputed again and again. Eliminating that redundant work directly reduces the energy and time of the most expensive part of inference, across every LLM. And there is a second dimension that is easy to overlook: sustainability. As chip technology advances, the carbon emitted just to manufacture the silicon (embodied carbon) becomes a larger share of a chip’s total footprint, sometimes exceeding the carbon of running it. So dedicating separate hardware to nonlinear operations, which sits idle much of the time, is not just an area cost but a carbon cost. A design that both removes redundant computation and reuses one array for everything therefore improves efficiency and sustainability at once, which is a compounding win as LLMs and the hardware for them keep scaling.

The key is a shift in where the approximation happens, which is what lets one value-reuse engine handle the messy operations LLMs need.

Value-level parallelism works by recognizing that low-precision values repeat, so a distinct product need only be computed once. That is straightforward for a plain matrix multiply, but nonlinear functions like softmax do not look like reusable products, which is why prior VLP could not touch them. Mugi’s insight is to reframe how a nonlinear function is approximated. The conventional way is output approximation: take a precise input and compute an approximate output, usually with special-purpose circuits. Mugi does the opposite — input approximation: snap the input to a nearby representative value, and produce a precise output for that value. This matters because a small set of representative input values is exactly the kind of repetition VLP thrives on: once the inputs are quantized to representatives, the nonlinear function becomes a reuse-friendly mapping the same array can evaluate. And by making the approximation value-centric — spending more accuracy on the input values that matter most to the model’s output and less on the rest — Mugi keeps end-to-end accuracy high while maximizing reuse. This single reframing is what unifies the nonlinear operations and the matrix multiplies under one mechanism, which in turn is what lets Mugi fuse them onto one array and cut the area and embodied carbon that separate units would cost.

What people did before, and where it broke

Use low precision to speed up matrix multiply. Represent activations and weights in narrow formats. Where it broke. Low precision creates repeated products across a large multiply, so the same value-times-weight is recomputed many times, wasting work.
Apply value-level parallelism to symmetric low-precision GEMM (Carat). Reuse repeated products for matrix multiplies where both sides share the same low precision. Where it broke. LLMs also need nonlinear operations, use asymmetric mixed precision, and run small batches, none of which prior VLP supports.
Handle nonlinear operations with dedicated hardware units. Add separate matrix and vector units, approximating outputs from precise inputs. Where it broke. The extra units add chip area and embodied (manufacturing) carbon, which can outweigh the carbon of running the model.
Run LLM inference at large batch to amortize. Use big batches so low-precision values repeat and VLP saves work. Where it broke. Real-time LLM inference must use small batches to meet latency targets, so there is far less repetition to exploit.
The gap. No VLP design supported nonlinear operations, asymmetric small-batch matrix multiply, and a single shared array for both, so none could accelerate full LLM workloads efficiently and sustainably.

How Mugi works

Bringing value reuse to every LLM operation on one array

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Compute a repeated value once and reuse it, instead of redundantly recomputing. Mugi’s foundation is value-level parallelism: because low precision makes values repeat, each distinct product is computed a single time and reused everywhere it recurs. Accelerating computation by detecting that the same work appears many times and doing it only once, rather than repeating it, is the same principle the work on coarse-grained circuit deduplication (Don’t Repeat Yourself) covered elsewhere on this site is built on, eliminating redundant repeated computation. Mugi applies this value-reuse idea across an LLM’s matrix multiplies and, newly, its nonlinear operations.

Implement the arithmetic with a temporal, unary style tuned for efficiency. VLP encodes a value as activity in time — a value becomes a spike at a particular cycle, and a multiplication becomes an accumulation over cycles — a temporal, unary way of computing rather than conventional parallel arithmetic. Redesigning the low-level arithmetic in a temporal or unary form to gain energy efficiency, rather than using standard binary datapaths, is the same approach the work on energy-efficient convolutions with temporal arithmetic covered elsewhere on this site takes, computing in the time domain for efficiency. Mugi builds on this temporal-coding substrate to make value reuse cheap in hardware.

Spend precision on the values that matter and skimp on the rest. Mugi’s value-centric approximation gives more accuracy to important input values and less to unimportant ones, keeping end-to-end quality high while shrinking the work. Allocating numerical precision unevenly, concentrating bits where they most affect the model and saving them where they do not, is the same idea the work on a metadata-augmented microscaling data format for efficient low-bit LLM inference (M2XFP) covered elsewhere on this site develops, matching precision to importance. Mugi applies this principle both to its nonlinear approximation and to its asymmetric mixed-precision matrix multiply.

Following an LLM layer through Mugi

Running attention and a feed-forward layer with one value-reuse array

A trace, drawn from the paper, of how Mugi evaluates both matrix multiplies and nonlinear operations on the same array.

Step 1 of 5

A mixed-precision matrix multiply arrives

A layer needs an activation-weight multiply where the activations are wide (say 16-bit) but the weights are very narrow (say 4-bit) from weight-only quantization — an asymmetric multiply prior VLP could not handle.

LLM quantization makes one side narrow and one side wide.

Why it works

Low precision makes values repeat, so reuse saves real work. A large low-precision multiply recomputes the same products → but computing each distinct product once and reusing it → so the redundant computation is eliminated where values recur.
Input approximation makes nonlinear ops reuse-friendly. Softmax and activations do not look like reusable products → but snapping inputs to a small set of representatives with precise outputs → so nonlinear functions become mappings the same value-reuse array can evaluate.
Value-centric accuracy keeps quality while shrinking work. Approximating every value equally would hurt accuracy → but giving important values more precision and others less → so end-to-end LLM accuracy stays high with a small representative set.
Asymmetric support matches how LLMs actually quantize. Prior VLP needed both sides at the same low precision → but handling wide-times-narrow multiplies from weight-only and KV-cache quantization → so Mugi accelerates the real LLM operations, not just symmetric ones.
One array for both cuts area and manufacturing carbon. Separate matrix and nonlinear units add silicon and embodied carbon → but running both on the same value-reuse array → so the extra hardware, and the carbon of making it, is removed.

What it costs (trade-offs and scope)

Approximation trades some accuracy for efficiency. Value-centric input approximation is still approximation; Mugi keeps end-to-end accuracy high by concentrating precision on important values, but a workload sensitive to the approximated values could see quality slip, so the value-importance judgment matters.
The gains depend on value repetition. Value-level parallelism saves work only where low-precision values recur; Mugi extends this to small batches and asymmetric precision, but a setting with little repetition offers less to reuse, so the benefit is workload- and precision-dependent.
It is a specialized architecture, not a drop-in. Mugi is a new VLP hardware design with temporal-coding datapaths and a fused array; realizing its benefits means building and using that architecture, not running on a conventional accelerator.
The nonlinear and GEMM fusion constrains the design. Making one array serve both matrix multiply and nonlinear approximation requires the value-centric reframing and imposes structure on how operations map onto the hardware, which is a co-design rather than a free addition.
Reported gains span a very wide range and are metric-specific. The headline numbers (up to 45 times and 668 times for softmax, 2.07 times and 3.11 times for LLMs, carbon reductions) vary enormously by operation and metric; the softmax figures are for isolated nonlinear operations, and the end-to-end LLM and carbon numbers are more modest and depend on the models and hardware assumptions.

The evaluation

Accelerating LLM computation with value-level parallelism

Property	Conventional accelerator	Prior VLP (Carat)	Mugi
Reuses repeated low-precision products
Nonlinear operations (softmax, SiLU, GELU)
Asymmetric mixed precision
Small-batch efficiency
Area and embodied carbon

A comparison of ways to accelerate LLM computation. By bringing value-level parallelism to nonlinear operations via value-centric input approximation, supporting asymmetric small-batch matrix multiply, and reusing one array for both, Mugi reports up to 45 times throughput and 668 times energy efficiency for nonlinear softmax, 2.07 times and 3.11 times for full LLMs, and cuts operational and embodied carbon by 1.45 times and 1.48 times. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2601.10823, of the ASPLOS 2026 paper). The described system is the paper’s: Mugi is a value-level parallelism (VLP) architecture for efficient LLMs. VLP, from prior work such as Carat, improves large-batch, low-precision matrix multiply (GEMM) by reusing repeated products rather than recomputing them, implemented with a temporal (unary) coding scheme in which a value is a spike in time and multiplication becomes accumulation over cycles. Mugi extends VLP to LLMs in three ways: it generalizes VLP to nonlinear approximation using a value-centric input-approximation approach (approximating the input to a representative value and producing a precise output, assigning greater accuracy to more important values), applied to softmax, SiLU, and GELU; it optimizes VLP for asymmetric, small-batch GEMMs using LLM-oriented quantization (weight-only quantization, KV-cache quantization) and group-query attention, introducing BF16-INT4 GEMM support; and it reuses one array for both nonlinear operations and GEMM, reducing on-chip area and embodied carbon. The paper reports throughput and energy-efficiency improvements up to 45 times and 668 times for nonlinear softmax operations, and 2.07 times and 3.11 times for full LLMs, and decreases in operational carbon by 1.45 times and embodied carbon by 1.48 times. The background above on GEMM, quantization, temporal coding, and nonlinear activation functions is standard context provided to make the approach intuitive. The step-by-step walkthrough is an illustration of the described mechanism. Exact circuit designs, the approximation method, and the full evaluation and baselines should be read from the paper.

Background

Why efficient AI hardware keeps hunting for redundancy to remove

It helps to see Mugi within a broad pattern: much of the progress in AI hardware efficiency comes not from doing arithmetic faster but from noticing that a great deal of the arithmetic is redundant, and finding ways to avoid it. Low precision, sparsity, and quantization all, in different ways, reduce the amount of distinct work a model actually requires — and once the distinct work shrinks, the opportunity is to compute each distinct thing once and reuse it, rather than blindly repeating it across a huge tensor. Value-level parallelism is a sharp instance of this: it turns the fact that low-precision values repeat into a concrete reuse mechanism. What Mugi adds is the recognition that an LLM is not just matrix multiplies — it is matrix multiplies interleaved with nonlinear functions, at mixed precisions and small batches — and that the same reuse philosophy, with the right reframing (approximate the input, not the output), can cover all of it on one piece of hardware. Fusing the operations onto a single array then attacks a cost that is becoming central as chips advance: the carbon embodied in manufacturing silicon, which extra dedicated units make worse. This attention to the full cost of an accelerator, operational and embodied together, rather than raw performance alone, is the same lens the work on a first-order carbon model to assess processor sustainability (FOCAL) covered elsewhere on this site applies, weighing what a chip costs to build against what it costs to run. Mugi’s contribution is to show that extending value-level parallelism to the whole of an LLM, and running it on one shared array, improves throughput, energy, and sustainability at once.

If you remember only three things

1. Matrix multiply (GEMM) dominates neural-network runtime (over ninety percent), and low numerical precision is the main efficiency lever — but low precision means values repeat, so a large multiply recomputes the same value-times-weight products many times. Value-level parallelism (VLP) exploits this by computing each distinct product once and reusing it wherever the value recurs, implemented with a temporal (unary) coding scheme. Prior VLP (Carat) worked only for the plain, symmetric-precision, large-batch matrix multiplies of older networks, and did not fit LLMs, which also need nonlinear operations (softmax, SiLU, GELU), use asymmetric mixed precision (wide activations against narrow 4-bit weights or KV cache), run small batches for real-time latency, and pay area and embodied manufacturing carbon for separate nonlinear hardware units.
2. Mugi is a new VLP architecture that overcomes all four issues and runs full LLM workloads on one shared array. It brings value-level parallelism to nonlinear operations for the first time via value-centric input approximation: rather than approximating the output of a precise input, it snaps the input to a nearby representative value (giving more accuracy to important values) and produces a precise output, turning softmax and activations into reuse-friendly mappings the value-reuse array can evaluate. It optimizes VLP for the asymmetric, small-batch matrix multiplies LLMs actually use (weight-only and KV-cache quantization, group-query attention, BF16-INT4 GEMM). And because both nonlinear operations and matrix multiplies now use value-level parallelism, it reuses one array for both, eliminating separate nonlinear units and the area and embodied carbon they cost.
3. On multiple LLMs, the paper reports Mugi improves throughput and energy efficiency by up to 45 times and 668 times for nonlinear softmax operations, and 2.07 times and 3.11 times for full LLMs, while cutting operational carbon by 1.45 times and embodied carbon by 1.48 times. The trade-offs: value-centric approximation trades some accuracy for efficiency and depends on judging value importance well, VLP's savings depend on values actually repeating, Mugi is a specialized architecture (with temporal-coding datapaths and a fused array) rather than a drop-in, and the reported gains span a very wide, metric-specific range — the largest numbers are for isolated softmax, while the end-to-end LLM and carbon figures are more modest.

Check yourself

Check your understanding

1. What is "value-level parallelism" (VLP), and what waste does it exploit?
2. Why did prior value-level parallelism not fit large language models?
3. How does Mugi bring value-level parallelism to nonlinear operations like softmax?
4. What does running both matrix multiply and nonlinear operations on one array achieve, and what does Mugi report overall?

Sources

The paper is “Mugi: Value Level Parallelism For Efficient LLMs” by Daniel Price, Prabhu Vellaisamy, John P. Shen, and Di Wu (University of Central Florida and Carnegie Mellon University), published at ASPLOS 2026, DOI 10.1145/3779212.3790189. It is available on arXiv as 2601.10823.

This explainer is written from the paper’s full text (the authors’ arXiv copy). Mugi is a value-level parallelism (VLP) architecture for efficient LLMs. VLP, from prior work such as Carat, accelerates large-batch, low-precision matrix multiply by reusing repeated products rather than recomputing them, using a temporal (unary) coding scheme. Mugi generalizes VLP to nonlinear approximation via a value-centric input-approximation approach (approximating the input to a representative value and producing a precise output, assigning greater accuracy to more important values) for softmax, SiLU, and GELU; optimizes VLP for asymmetric, small-batch matrix multiplies using LLM quantization (weight-only quantization, KV-cache quantization) and group-query attention, including BF16-INT4 GEMM; and reuses one array for both nonlinear operations and matrix multiply, reducing on-chip area and embodied carbon. The paper reports throughput and energy-efficiency improvements up to 45 times and 668 times for nonlinear softmax operations, and 2.07 times and 3.11 times for full LLMs, and decreases in operational carbon by 1.45 times and embodied carbon by 1.48 times. Numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.