Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation

Yaoyao Ding, Bohan Hou, Xiao Zhang, Allan Lin, Tianqi Chen, Cody Hao Yu and 2 others

Squeezing a language model's weights down to a few bits per number makes it far cheaper to serve, and odd bit widths like 5, 6, or 7 hit the sweet spot between accuracy and speed — but GPUs and their tools only make power-of-two widths (4 and 8) fast, so nobody could generate efficient kernels for the in-between ones; this paper (University of Toronto, CMU, AWS — Yaoyao Ding, Gennady Pekhimenko, Tianqi Chen, and colleagues) presents Tilus, a tile-level GPU programming language whose algebraic layout system exposes registers and the memory hierarchy, letting it compile fast kernels for any bit width from 1 to 8 and beat the best existing kernels by up to 2.6 times.

The argument in one glance

Problem

The most useful low-precision bit widths are the ones GPUs make slow

Serving a large language model is dominated by matrix multiplications that read enormous weight matrices from memory. Quantization — storing each weight in fewer bits — is the leading way to make this cheaper: dropping from 16-bit to 4-bit weights cuts memory, bandwidth, and time-to-generate-a-token by roughly four times. But 4-bit is often too aggressive, losing noticeable accuracy, while 8-bit leaves efficiency on the table. The accuracy-efficiency sweet spot for many models sits at the in-between widths — 5, 6, or 7 bits, and sometimes 3 — which preserve quality while still shrinking the model. The catch is that GPUs and their software stacks are built around power-of-two bit widths: 4-bit and 8-bit values pack neatly into bytes and words and map onto the hardware's instructions, but 3, 5, 6, and 7 bits do not. A 5-bit value straddles byte boundaries, needs awkward bit-shuffling to unpack, and has no native hardware support, so a naive implementation is slow enough to erase the benefit. As a result, the very bit widths that would give the best accuracy-efficiency trade-off have no efficient GPU kernels, and generating them is an open problem. The demand for flexible quantization is real and growing, but the tooling to exploit it does not exist.
What people did

Write kernels by hand, or generate them with a high-level tile compiler

Two approaches exist for producing low-precision GPU kernels. One is to write them by hand: experts like Marlin and QuantLLM craft highly optimized kernels for a specific format on specific hardware. The other is to generate them with a compiler: tools like Triton let a programmer write kernels at the level of tensor tiles processed by a thread block, and Ladder extends a scheduling compiler to emit low-precision code. These automate away much of the tedium of GPU programming and are widely used for standard formats.
Why it fell short

Hand-written kernels do not generalize, and tile compilers hide the control you need

Both approaches fall short exactly where flexible low precision needs them. Hand-written kernels are narrow and brittle: Marlin supports only 4-bit signed integers and not the newest GPUs, QuantLLM covers only 5- and 6-bit floats and lacks finer quantization granularity, and each is time-consuming to build and hard to extend to a new bit width or architecture. Compiler-generated kernels are more general but pay for it with abstraction. Triton's tile model simplifies GPU programming, but it has no built-in low-precision types, so a developer must hand-code the fiddly bit-level packing and unpacking, and it does not expose the full GPU memory hierarchy, so it cannot perform the fine-grained register management and memory-access optimizations that low-precision kernels live or die by. Ladder supports low precision but only at power-of-two bit widths, and its scheduling primitives cannot express crucial optimizations like software pipelining, so it underperforms — especially at batch sizes above one, which continuous batching makes the common case during LLM decoding. The root problem is that high-level GPU abstractions, meant to ease programming, hide the low-level control — over registers, over data layout across threads, over the memory hierarchy — that efficient arbitrary-precision computation requires. What was missing was a programming model that keeps the productivity of tiles but gives back that control.
This paper's idea

A tile-level language whose algebraic layout system exposes registers and memory

Tilus is a tile-level GPGPU programming language, backed by a virtual machine, designed specifically for low-precision computation. It keeps the productivity of the tile model — you write the computation of a thread block over tensor tiles — but, unlike Triton, it exposes the GPU's hierarchical memory (registers, on-chip shared memory, global memory) and gives explicit, fine-grained control over where data lives and how it moves. Its central innovation is an algebraic layout system: a compact, composable algebra that specifies exactly how the elements of a tile are distributed across the GPU's threads and their registers. This unified representation exposes the registers to the kernel developer and lets the compiler reason about and transform layouts systematically, which is what makes fine-grained optimization possible. On top of this, Tilus provides first-class support for arbitrary low-precision data types — signed integers, unsigned integers, and floating-point numbers at any bit width from 1 to 8 — so the awkward 3, 5, 6, and 7-bit formats become ordinary types rather than hand-coded bit tricks. Tilus programs are then compiled into highly efficient GPU code through automatic vectorization and instruction selection. The result extends the reach of efficient low-precision kernels across the full spectrum of bit widths and type kinds, and it outperforms the state of the art on the formats those systems do support: the paper reports speedups of 1.75 times over Triton, 2.61 times over Ladder, 1.29 times over the hand-written QuantLLM, and 1.03 times over the hand-written Marlin.

Why this problem is worth solving

Low precision is one of the biggest levers on the cost of serving LLMs, and unlocking the full range of bit widths lets each model be served at its own best point on the accuracy-efficiency curve.

Serving an LLM is largely a memory-bandwidth problem: generating each token requires reading the model’s weights from memory, and the fewer bits each weight takes, the less data must be moved and the faster the token comes out. Quantization is therefore central to making LLM serving affordable, and the choice of bit width is a direct dial between quality and cost. But the useful settings of that dial are not limited to 4 and 8. Four bits maximizes savings but can degrade a model’s accuracy unacceptably; eight bits is safe but wastes half the potential savings; and for many models the right answer is somewhere in between — 5, 6, or 7 bits — where accuracy is preserved and cost is still substantially cut. Different models, and even different layers, have different sensitivities, so the ability to pick any bit width is what lets a deployment land exactly on its best trade-off rather than being forced to the nearest power of two. Yet this flexibility is worthless without efficient kernels: a 6-bit model that runs slower than an 8-bit one because the hardware cannot handle 6-bit values gains nothing. So the value of solving this is to convert a known algorithmic opportunity — flexible-precision quantization — into a real performance win, by making the odd bit widths as fast to run as the standard ones. That is what turns a research curiosity into something a serving system can actually deploy.

The deeper lesson is about the tension between programming abstraction and hardware control, and how a well-designed layout algebra can resolve it.

GPU programming has a persistent trade-off. Write in raw CUDA and you get total control over registers, shared memory, and how data is arranged across the thousands of threads — control that low-precision kernels desperately need, because packing 6-bit values efficiently means carefully choosing which thread holds which bits and how they are unpacked with the fewest operations. But raw CUDA is painful and error-prone. Reach for a high-level tile compiler and programming gets easy, but the abstraction hides exactly those low-level details, so the optimizations that matter become impossible to express. Tilus’s resolution is that the missing ingredient is a first-class representation of layout — how tile elements map onto threads and registers. By making layout an explicit, algebraic object that both the programmer and the compiler can manipulate, Tilus exposes the registers without forcing the programmer down to raw CUDA: you still write tile-level code, but you can now specify and transform the data-to-thread mapping that determines performance, and the compiler can generate efficient vectorized instructions from it. This is the crux, because for arbitrary-precision computation the layout is the optimization — where the bits of each low-precision value sit across threads dictates how cheaply they can be loaded, unpacked, and fed to the matrix-multiply units. Turning that from an implicit, hidden detail into an explicit, composable algebra is what lets one system be both programmable and fast across every bit width.

What people did before, and where it broke

Hand-written low-precision kernels (Marlin, QuantLLM). Expertly optimize a kernel for one format on one architecture. Where it broke. They are narrow — Marlin only 4-bit signed integers without the newest GPUs, QuantLLM only 5- and 6-bit floats without fine granularity — and time-consuming to extend to new bit widths or hardware.
Triton’s tile programming model. Write thread-block kernels over tensor tiles, automating much GPU programming. Where it broke. It has no built-in low-precision types, forcing manual bit-level packing, and it does not expose the full memory hierarchy, so it cannot do the fine-grained register and memory-access optimizations low precision needs.
Ladder’s scheduling-based low-precision compiler. Extend a schedule-oriented compiler to emit low-precision kernels. Where it broke. It is restricted to power-of-two bit widths, and its primitives cannot express key optimizations like software pipelining, so it underperforms, especially at the batch sizes above one that continuous batching makes common.
Power-of-two-only tooling. Optimize the whole stack for 4-bit and 8-bit values that pack neatly into bytes. Where it broke. The accuracy-efficiency sweet spot often lies at 3, 5, 6, or 7 bits, which this tooling makes inefficient, so the best trade-offs are unreachable in practice.
The gap. No system offered the productivity of the tile model together with the fine-grained control over registers, layout, and memory needed to generate efficient kernels for arbitrary low-precision types.

How Tilus works

From a high-level tile kernel to fast arbitrary-precision GPU code

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Make flexible low precision the point, not an afterthought. Tilus is built specifically to compute in low precision, treating arbitrary bit widths and type kinds as first-class so LLMs can be served at their best accuracy-efficiency point. Making non-standard, sub-byte numeric formats efficient so that models can run at reduced precision without losing quality is the same low-precision-formats goal the work on a metadata-augmented microscaling data format for efficient low-precision LLMs (m2xfp) covered elsewhere on this site pursues, engineering the numeric representation for the accuracy-cost trade-off. Tilus attacks the same objective from the kernel-generation side, making any bit width from 1 to 8 fast to execute on the GPU.

Keep the tile abstraction but hand back the hardware. Tilus stays a tile-level programming language — you write a thread block’s work over tiles — yet exposes the registers and memory hierarchy that high-level tile compilers hide. Providing a tile-based programming model for GPU kernels that combines productivity with control over how work maps to the hardware is the same tile-oriented compiler approach the work on a task-mapping programming paradigm for deep-learning tensor programs (Hidet) covered elsewhere on this site is built on, letting the programmer direct the mapping rather than leaving it to a fixed schedule. Tilus, from the same lineage, extends that philosophy with fine-grained memory control and native low-precision support.

Give data layout across threads a first-class algebra. Tilus’s core mechanism is an algebraic layout system describing how a tile’s elements are distributed over threads and registers, which the compiler can reason about and transform. Making the layout of tensor data across a GPU’s threads an explicit, manipulable object in the compiler, rather than an implicit consequence of scheduling, is the same layout-centric approach the work on an intermediate representation for optimized tensor computations on GPUs (Graphene) covered elsewhere on this site takes, treating data placement as a first-class part of the program. Tilus turns that layout representation into an algebra tailored to packing and unpacking arbitrary-precision values efficiently.

Following a 6-bit kernel through Tilus

From a 6-bit matrix-multiply to fast GPU code

A trace, drawn from the paper, of how Tilus generates an efficient kernel for a non-power-of-two bit width.

Step 1 of 5

Declare 6-bit weights as a first-class type

The developer writes a matrix-multiplication kernel in Tilus using a 6-bit integer type for the weights directly, instead of manually packing six-bit values into bytes and writing bitwise unpacking code as they would in Triton.

Arbitrary bit widths from 1 to 8 are ordinary types in Tilus.

Why it works

First-class odd bit widths remove the packing burden. Manually coding 6-bit packing in a high-level tool is error-prone and slow → but making any 1-to-8-bit type first-class → so awkward formats become ordinary types the compiler handles.
An explicit layout algebra unlocks the key optimization. For low precision the data-to-thread mapping determines speed → but representing that layout as a manipulable algebra → so the compiler can generate the cheapest loads and unpacking for each bit width.
Exposing the memory hierarchy enables what tile compilers hide. High-level tiles conceal registers and shared memory → but giving explicit control over placement and movement → so optimizations like software pipelining become expressible.
Keeping the tile model preserves productivity. Raw CUDA gives control but is painful to write → but staying at the thread-block tile level → so developers get the control without abandoning a productive abstraction.
Automatic vectorization turns layouts into fast code. Even a good layout is useless without efficient instructions → but auto-vectorization and instruction selection from the layout → so the generated kernel actually runs at hardware speed.

What it costs (trade-offs and scope)

More control means more to specify. Exposing registers, layout, and the memory hierarchy gives power but also asks the developer to make decisions a fully-automatic scheduler would hide, so writing a top-performing Tilus kernel still requires understanding the hardware.
The gains concentrate in low-precision, memory-bound kernels. Tilus targets the quantized matrix multiplications that dominate LLM serving; its advantages are largest there, and it is a GPGPU language rather than a general application framework.
It is a new language and toolchain. Adopting Tilus means writing kernels in its tile-level language and compiling through its virtual machine, rather than reusing an existing kernel library, though it aims to keep programming simpler than raw CUDA.
Bit widths are covered from 1 to 8. The first-class type support spans 1 to 8 bits; formats outside that range, or fundamentally different numeric schemes, are beyond the demonstrated scope.
Results are on specific GPUs, models, and baselines. The reported speedups (1.75, 2.61, 1.29, and 1.03 times over Triton, Ladder, QuantLLM, and Marlin) are measured on particular hardware and workloads against those systems on their supported formats; other settings may differ.

The comparison

Generating low-precision GPU kernels

Property	Hand-written kernels	High-level tile compilers	Tilus
Bit-width coverage
Register and layout control
Memory hierarchy access
Effort to add a new format
Reported performance

A comparison of ways to generate low-precision GPU kernels. Hand-written kernels (Marlin, QuantLLM) are fast but narrow and hard to extend; high-level tile compilers (Triton, Ladder) are general but hide the register and memory control low precision needs, and Ladder is limited to power-of-two bit widths. Tilus keeps the tile programming model but exposes registers and the memory hierarchy through an algebraic layout system, supports any bit width from 1 to 8 as first-class types, and auto-vectorizes to fast code — beating Triton, Ladder, QuantLLM, and Marlin by 1.75, 2.61, 1.29, and 1.03 times on their supported formats. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2504.12984, of the ASPLOS 2026 paper). The described system is the paper’s: Tilus is a tile-level GPGPU programming language, backed by a virtual machine, with dedicated support for low-precision computation. It keeps a thread-block-level tile programming model but exposes the GPU’s hierarchical memory (registers, shared, global) for fine-grained control, and its central contribution is an algebraic layout system specifying how a tile’s elements are distributed across threads and registers, which exposes registers to the developer and lets the compiler reason about layouts. It provides first-class arbitrary low-precision data types — signed integers, unsigned integers, and floating-point numbers with bit widths from 1 to 8 — and compiles programs to efficient GPU code via automatic vectorization and instruction selection. The paper motivates the work by the gap that GPUs and tooling optimize power-of-two bit widths while non-power-of-two widths (3, 5, 6, 7) often give better accuracy-efficiency trade-offs. The reported results — extending efficient kernels across bit widths 1 to 8 and both integer and floating-point kinds, and outperforming Triton, Ladder, QuantLLM, and Marlin by 1.75, 2.61, 1.29, and 1.03 times respectively (up to about 2.6 times) on their supported kernels — are from the paper’s evaluation on specific GPUs and workloads. Background on LLM serving, quantization, GPU memory hierarchy, and tile programming is standard context. The step-by-step walkthrough illustrates the described mechanism. Exact layout-system semantics, the virtual machine, and full results should be read from the paper.

Background

Why fast deep-learning kernels keep coming down to how data maps onto the hardware

It helps to see Tilus within the broad effort to generate fast deep-learning kernels without hand-writing assembly for every case. The central difficulty in this field is the enormous gap between a high-level tensor operation and the machine code that runs it well: a matrix multiply must be tiled, its data laid out across threads and memory levels, its loads vectorized, and its computation overlapped with data movement, and small choices in how the data maps onto the hardware make large differences in speed. A long line of compilers and languages has attacked this by inventing abstractions that let programmers or search procedures express these mappings — where each piece of a tensor lives, which thread touches it, how it moves through the memory hierarchy — while still generating efficient code. Low-precision computation makes this harder and more important at once, because sub-byte values that do not align to hardware boundaries need especially careful placement to be unpacked cheaply. This drive to bridge high-level tensor programs to efficient hardware execution by giving the mapping and layout a real, optimizable representation is the same terrain the work on an abstraction for automatic tensorized program optimization (TensorIR) covered elsewhere on this site occupies, making the tensor-to-hardware mapping something the compiler can manipulate. Tilus’s distinctive contribution is to center that representation on an algebra of layouts and pair it with first-class arbitrary-precision types, so that the awkward, high-value bit widths finally get kernels as fast as the standard ones.

If you remember only three things

1. Quantizing an LLM to fewer bits per weight is a leading way to make serving cheaper, since generating each token is bottlenecked on reading weights from memory. But 4-bit is often too aggressive (accuracy loss) and 8-bit wastes savings, so the best accuracy-efficiency trade-off frequently lies at non-power-of-two widths like 3, 5, 6, or 7 bits. The problem is that GPUs and their software stacks are built for power-of-two widths that pack neatly into bytes; a 5-bit value straddles byte boundaries with no native hardware support, so naive kernels are too slow to be worth it. Existing options do not close the gap: hand-written kernels (Marlin, QuantLLM) are fast but narrow and hard to extend, and high-level tile compilers (Triton, Ladder) hide the fine-grained register and memory-access control that efficient low precision requires, with Ladder limited to power-of-two widths.
2. Tilus is a tile-level GPGPU programming language, backed by a virtual machine, built for low-precision computation. It keeps the productivity of the tile model — writing a thread block's work over tensor tiles — but, unlike Triton, exposes the GPU's hierarchical memory (registers, shared, global) with explicit control over placement and movement. Its central innovation is an algebraic layout system: a composable algebra specifying exactly how a tile's elements are distributed across threads and registers, which exposes the registers to the developer and lets the compiler reason about and transform layouts. It adds first-class arbitrary low-precision types — signed and unsigned integers and floats at any bit width from 1 to 8 — so odd formats are ordinary types rather than hand-coded bit tricks, and compiles everything to efficient GPU code via automatic vectorization and instruction selection.
3. The key insight is that for arbitrary-precision computation the data layout across threads is the optimization — where each low-precision value's bits sit determines how cheaply they load and unpack — so making layout an explicit, manipulable algebra resolves the usual tension between a productive abstraction and the hardware control performance demands. Tilus extends efficient low-precision kernels across the full spectrum of bit widths (1 to 8) and both integer and floating-point kinds, and on the formats existing systems support it outperforms Triton, Ladder, QuantLLM, and Marlin by 1.75, 2.61, 1.29, and 1.03 times respectively (up to about 2.6 times). The trade-offs: the extra control asks more of the developer, gains concentrate in low-precision memory-bound kernels, and it is a new language and toolchain, with results measured on specific GPUs, models, and baselines.

Check yourself

Check your understanding

1. Why are non-power-of-two bit widths (like 5, 6, or 7 bits) both valuable and hard to run fast on GPUs?
2. Why do existing approaches fail to generate efficient arbitrary-precision kernels?
3. What is Tilus's central innovation?
4. What does Tilus support and achieve?

Sources

The paper is “Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation” by Yaoyao Ding, Bohan Hou, Xiao Zhang, Allan Lin, Tianqi Chen, Cody Hao Yu, Yida Wang, and Gennady Pekhimenko (University of Toronto, Carnegie Mellon University, and Amazon Web Services, among the affiliations), published at ASPLOS 2026, DOI 10.1145/3760250.3762219. It is available on arXiv as 2504.12984.

This explainer is written from the paper’s full text (the authors’ arXiv copy). Tilus is a tile-level GPGPU programming language, backed by a virtual machine, for low-precision computation: it keeps a thread-block-level tile programming model but exposes the GPU’s hierarchical memory for fine-grained control, contributes an algebraic layout system that specifies how a tile’s elements distribute across threads and registers (exposing registers and enabling compiler optimization), and provides first-class arbitrary low-precision types (signed and unsigned integers and floats, bit widths 1 to 8), compiling to efficient GPU code via automatic vectorization and instruction selection. It targets the gap that GPUs optimize power-of-two bit widths while non-power-of-two widths often give better accuracy-efficiency trade-offs. Evaluated against Triton, Ladder, QuantLLM, and Marlin, the paper reports speedups of 1.75, 2.61, 1.29, and 1.03 times respectively (up to about 2.6 times) on their supported kernels, and coverage across bit widths 1 to 8 for both integer and floating-point types. Background on LLM serving, quantization, GPU memory hierarchy, and tile programming is standard context. The numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.