Syno: Structured Synthesis for Neural Operators

Neural networks are built from a fixed menu of operators like convolution and matrix multiply, and the tools we have either rearrange those existing operators (architecture search) or speed up their implementation without changing what they compute (tensor compilers) — nobody invents genuinely new operators; this paper (Tsinghua — Yongqi Zhuo, Zhengyuan Su, Chenggang Zhao, Mingyu Gao) presents Syno, which synthesizes brand-new operators from fine-grained primitives defined on tensor dimensions, using canonicalization to skip redundant duplicates and a shape-guided tree search to find valid, trainable operators — replacing the originals in a model for 1.37 to 2.06 times average speedups at under 1 percent accuracy loss.

The argument in one glance

Problem

We optimize how neural operators are arranged and run, but never invent new ones

A neural network is assembled from a small menu of building blocks called operators — convolutions, matrix multiplications, pooling — each of which takes tensors in and produces tensors out. Two big families of tools try to make networks more accurate and faster, but both take that menu as fixed. Neural architecture search (NAS) automatically designs better models by trying different ways to compose the existing operators into cells and layers; it can find clever arrangements, but every arrangement is still built from convolutions and matmuls. Tensor compilers take a given operator and generate a faster low-level implementation of it — better loop nests, tiling, fusion — but they are forbidden from changing what the operator computes: they only produce semantically equivalent versions. So one family rearranges the blocks and the other polishes each block, but neither can create a genuinely new block. That leaves a whole axis of optimization untouched: what if the best operator for a task is not a convolution at all, but some new computation nobody has written down? Discovering such operators automatically is the unexplored direction this paper takes.
What people did

Treat it as program synthesis and compose operators from basic atoms

The natural framing is program synthesis — the classic idea of automatically generating a program that meets a specification. If you could synthesize operators from very basic programming primitives rather than from whole convolutions, you would have the fine granularity needed to explore a huge space of possible computations and invent new ones. Early attempts to apply synthesis to neural networks existed, and generic program synthesis is a mature field, so the obvious move is to point these tools at the problem of building neural operators atom by atom.
Why it fell short

Naive synthesis produces mostly-invalid, mostly-redundant operators with no clear target

Fine-grained synthesis of neural operators turns out to be much harder than generic program synthesis, for three reasons. First, validity: a neural operator must satisfy special properties to be usable and trainable — it should be differentiable, and it should not arbitrarily replicate or discard tensor elements — but an operator composed from arbitrary low-level atoms will almost never satisfy these, so the vast majority of random candidates are useless. Second, redundancy: countless different compositions compute the same or nearly the same thing, and those semantically-equivalent variants are exactly what tensor compilers already explore, so a synthesizer that keeps generating them wastes its effort rediscovering what compilers handle, instead of finding genuinely new operators. Third, the target is fuzzy: traditional program synthesis searches for a program that is correct against a crisp specification, but neural networks tolerate small errors and there is no exact right answer — the goal is better accuracy and speed, which is a soft, dual objective unlike anything classic synthesis was built for. And the design space is enormous. Existing program synthesis approaches do not scale to this, and early NN synthesis work stayed at the coarse level of composing whole operators. A new approach was needed.
This paper's idea

Synthesize from dimension-level primitives, canonicalize duplicates, and search with a shape guide

Syno is an end-to-end framework that automatically synthesizes novel operators to replace the ones in a given network, improving accuracy and/or speed. It solves the three problems with three ideas. First, instead of arbitrary atoms, Syno composes operators from a carefully designed set of fine-grained primitives whose semantics are defined on tensor dimensions (coordinates) — primitives like splitting, merging, unfolding, sharing, and reducing dimensions. These are built so that any operator composed from them automatically maintains the properties neural operators need (differentiability, no replication or discard), so validity comes for free rather than being checked after the fact. Second, to avoid drowning in redundant candidates, Syno applies expression simplification and canonicalization: it rewrites each synthesized operator into a canonical form (using a term-rewriting system) so that equivalent variants collapse to the same thing and can be pruned, letting the search skip the semantic duplicates that tensor compilers already cover. Third, Syno makes the search structured: it builds operators bottom-up, iteratively sampling and adding primitives, and to ensure the result actually fits the network it introduces a shape-distance metric that guides synthesis toward matching the required input and output tensor dimensions. The whole search is framed as a Markov decision process and solved with Monte Carlo Tree Search. Syno generates PyTorch and TVM code for its discovered operators and finds ones that beat standard convolutions and matmuls even on NAS-optimized models, with average speedups of 1.37 to 2.06 times at under 1 percent accuracy loss.

Why this problem is worth solving

The reason inventing operators is worth the difficulty is that the operator menu is a hidden ceiling on both accuracy and speed that the existing tools cannot lift.

Almost all effort to improve neural networks works within a fixed vocabulary of operators. Architecture search asks which arrangement of convolutions and matmuls is best; tensor compilers ask how to run a given convolution fastest. Both are valuable, but both inherit whatever limitations the operators themselves have. A convolution has a particular computational structure and a particular cost, and if that structure is not the most efficient way to capture the patterns a task needs, no rearrangement and no faster implementation can fix it — you are optimizing around a fixed shape. This is why synthesizing new operators opens a genuinely different axis: it lets the computation itself change, not just its arrangement or its code. The paper shows this pays off concretely, finding operators that are faster than standard convolutions and matrix multiplications while barely losing accuracy, even when the surrounding network was already optimized by NAS. The three approaches are complementary — you can take a NAS-designed topology, swap in Syno-synthesized operators, and then let a tensor compiler optimize their implementation — so operator synthesis is not a replacement for the existing tools but a new layer of freedom beneath them, and unlocking it is worth real effort precisely because it reaches what the others cannot.

The deeper design lesson is that in a search this hard, the right move is to make the primitives carry the guarantees, so the search never wastes time on invalid or redundant candidates.

A search-based synthesizer lives or dies by how much of its effort lands on useful candidates. If most compositions are invalid, or most are duplicates of things already seen, the search drowns before it finds anything good. Syno’s central bet is to fix this at the source — in the primitives and their algebra — rather than by filtering afterward. By defining primitives on tensor dimensions with semantics that inherently preserve differentiability and never replicate or discard elements, every operator the search can build is valid by construction, so the search space contains only trainable operators. By canonicalizing expressions so equivalent forms collapse, the search treats the many spellings of one computation as a single point, and can deliberately skip the semantic-equivalence variations that tensor compilers already own. Together these turn an intractable space of mostly-garbage candidates into a much smaller space of distinct, valid operators. This is a general principle in program synthesis and search: encode your constraints into the representation so the generator can only produce good candidates, instead of generating freely and validating later. Syno’s contribution is to find the right dimension-level representation for neural operators, where validity and canonicality both fall out naturally, and then to guide the remaining search toward operators that actually fit the target network.

What people did before, and where it broke

Neural architecture search (NAS). Automatically compose existing operators (convolutions, matmuls, pooling) into better model structures, optimizing accuracy or accuracy-plus-speed. Where it broke. It only recombines the fixed menu of operators in a coarse, black-box way, so it is bounded by those operators and cannot invent new ones.
Tensor compilers. Generate faster low-level implementations of each operator through tiling, fusion, and scheduling. Where it broke. They must preserve the operator’s semantics, so they only explore equivalent variants — they make an operator faster but never make a new one.
Generic program synthesis. Automatically generate programs meeting a specification. Where it broke. It targets crisp correctness, does not easily handle the soft accuracy-and-speed goals of neural operators, and does not scale to the enormous fine-grained design space.
Coarse-grained NN synthesis. Early attempts synthesized networks by composing whole basic operators. Where it broke. Staying at the level of existing operators leaves the large potential of truly novel, fine-grained operators unexploited.
The gap. No method synthesized genuinely new neural operators from fine-grained atoms while guaranteeing they are valid and trainable, pruning the redundant equivalents compilers already handle, and steering the search toward operators that fit the target network.

How Syno works

From a fixed operator menu to synthesized, validated, network-fitting operators

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Automatically discover better neural designs instead of hand-crafting them. Syno searches for network improvements automatically, framing operator discovery as a large search over compositions rather than a task for human intuition. Letting an automated search explore the space of neural network designs to find high-quality, hardware-efficient ones, rather than relying on manual design, is the same automated-design premise the work on hardware-optimized neural architecture search (HyperScale) covered elsewhere on this site is built on, searching for good architectures under performance constraints. Syno pushes that automation a level finer — instead of searching over arrangements of existing operators as NAS does, it searches over the operators themselves, synthesizing new computations.

Represent the computation with a structured tensor abstraction. Syno’s primitives and canonical forms are a small, principled intermediate representation for tensor computations, defined on dimensions, that makes valid transformations easy to express and reason about. Designing a structured intermediate abstraction for tensor programs so that transformations are systematic and analyzable, rather than ad hoc, is the same representation-first approach the work on an abstraction for automatic tensorized program optimization (TensorIR) covered elsewhere on this site takes, giving tensor computations a clean algebra to manipulate. Syno’s dimension-level primitives play that role for operator synthesis, where the abstraction must also guarantee validity and support canonicalization.

Search a huge design space efficiently with a principled algorithm. Because the space of synthesizable operators is vast, Syno relies on a structured stochastic search — a Markov decision process explored by Monte Carlo Tree Search, guided by a shape-distance heuristic — rather than brute force. Using a smart, budgeted search to navigate an enormous space of program or configuration candidates toward good ones is the same search-based optimization idea the work on a fast and portable Bayesian compiler optimization framework (BaCO) covered elsewhere on this site is built on, spending its search budget where it is most likely to pay off. Syno applies tree search to operator synthesis, using shape distance to keep candidates heading toward operators that actually fit the target network.

Following an operator synthesis through Syno

From an output tensor shape to a novel, trained operator

A trace, drawn from the paper, of how Syno synthesizes and evaluates one candidate operator to replace a layer in the backbone network.

Step 1 of 5

Fix the operator's required input and output shapes

To replace a layer, the new operator must consume and produce tensors of specific dimensions dictated by the backbone. Syno takes these required input and output shapes as the specification the synthesized operator has to satisfy.

Matching these shapes is what makes the operator droppable into the model.

Why it works

Dimension-level primitives make every candidate valid. Arbitrary code atoms almost never form a trainable operator → but primitives defined on tensor dimensions that preserve differentiability and elements → so the search space contains only valid neural operators by construction.
Canonicalization removes the redundant flood. Countless compositions compute the same thing → but rewriting each into a canonical form and pruning matches → so the search treats equivalent operators as one and skips what compilers already handle.
Shape distance keeps the search on target. A synthesized operator is useless if its dimensions do not fit the model → but guiding synthesis by distance to the required input and output shapes → so candidates converge on operators that actually drop into the network.
Tree search navigates the vast space efficiently. Brute force cannot cover the enormous operator space → but framing it as a Markov decision process explored by Monte Carlo Tree Search → so the budget concentrates on the most promising branches.
Closing the loop with real training and benchmarks aims the search well. Accuracy and speed are soft, hardware-dependent goals → but generating real PyTorch and TVM code and measuring outcomes → so the search optimizes for true results rather than a proxy.

What it costs (trade-offs and scope)

Synthesis plus training is a heavy search. Exploring the operator space and training candidate networks to measure accuracy is computationally expensive; Syno makes the search efficient with canonicalization and tree search, but it is still an offline, resource-intensive discovery process.
It targets linear operators to replace existing layers. Syno searches for novel linear operators to swap into a given backbone; it is designed around replacing operators in an existing topology rather than inventing arbitrary nonlinear structures from nothing.
Small accuracy loss is part of the deal. The speedups come with a bounded accuracy drop (under 1 percent on CIFAR-100, and 1 to 2 percent on ImageNet in the paper’s results), so the discovered operators trade a little quality for speed rather than being strictly better on both.
Gains depend on hardware and compiler. The speedups vary by backend — mobile CPU, mobile GPU, server GPU, and whether TVM or torch.compile is used — so the benefit of a synthesized operator is tied to the target stack, not universal.
The discovered operators are unconventional. Because they are novel computations rather than familiar convolutions, they may be harder to interpret, reason about, or trust than standard operators, even though the paper finds interesting algorithmic insights in them.

The comparison

Improving neural networks: three levers

Property	Neural architecture search	Tensor compilers	Syno
What it changes
Can invent new computations
Building blocks
Handles redundant equivalents
Reported outcome

A comparison of three complementary ways to improve neural networks. Neural architecture search rearranges existing operators; tensor compilers speed up an operator's implementation without changing what it computes; neither can invent a new operator. Syno synthesizes genuinely new operators from fine-grained, dimension-level primitives that guarantee validity, canonicalizes away redundant equivalents, and uses shape-guided Monte Carlo Tree Search to find operators that fit the network — reaching 1.37 to 2.06 times average speedups at under 1 percent accuracy loss, even on NAS-optimized models. The three can be layered together. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2410.23745, of the ASPLOS 2025 paper). The described system is the paper’s: Syno is an end-to-end framework for neural operator synthesis that, given a backbone network, searches for novel linear operators to replace its layers and improve accuracy and/or speed. Its key techniques are a set of fine-grained primitives defined on tensor coordinates (dimensions) that preserve neural-operator properties such as differentiability and no replication or discard of elements; expression simplification and canonicalization (via a term-rewriting system) to eliminate redundant, semantically-equivalent candidates that tensor compilers already handle; and a structured bottom-up search guided by a novel shape-distance metric and formulated as a Markov decision process solved with Monte Carlo Tree Search, with PyTorch and TVM code generators for evaluation. The reported results — average speedups of 1.37 to 2.06 times across various hardware and compilers at under 1 percent accuracy loss even on NAS-optimized models, with specific figures such as up to 4.73 times on ImageNet with TVM, and GPT-2 training accelerated 1.1 times with perplexity improving from 111 to 99 — are from the paper’s evaluation on five vision models and GPT-2. Background on NAS, tensor compilers, and program synthesis is standard context. The step-by-step walkthrough illustrates the described mechanism. Exact primitive definitions, the canonicalization and shape-distance algorithms, and full results should be read from the paper.

Background

Why optimizing neural networks keeps climbing to new levels of automation

It helps to place Syno in the arc of automating neural network optimization. The field has steadily moved decisions that were once made by hand into automated search and compilation. First, model architectures: neural architecture search replaced hand-designed topologies with automatically discovered ones. Then, implementations: tensor compilers replaced hand-written kernels with automatically scheduled and tuned ones. Each step took a layer of the stack that experts used to craft manually and turned it into a search or optimization problem, expanding what could be explored far beyond human patience. Syno extends this trajectory to a layer that had stayed manual — the operators themselves. The recurring pattern is the same: define a structured space of possibilities, find a representation that makes valid candidates easy to generate and reason about, and search it efficiently. This drive to automate the design and optimization of deep-learning computations by giving them a clean abstraction and a search procedure is the same terrain the work on a task-mapping programming paradigm for deep-learning tensor programs (Hidet) covered elsewhere on this site occupies, systematizing what used to be expert craft. Syno’s distinctive step is to automate not the arrangement or the implementation of operators but their invention, showing that new operators — not just new networks or new kernels — can be discovered by machine.

If you remember only three things

1. Neural networks are built from a fixed menu of operators like convolution and matrix multiplication, and the two main optimization tools both take that menu as given: neural architecture search only recombines existing operators into better arrangements, and tensor compilers only produce faster implementations of an operator without changing what it computes. Neither can invent a genuinely new operator, leaving an entire axis of optimization — the computation itself — untouched. Doing operator synthesis from fine-grained atoms is hard because arbitrary compositions are almost never valid trainable operators, they generate floods of redundant semantically-equivalent candidates (which compilers already handle), and neural networks have a soft accuracy-and-speed goal rather than the crisp correctness target classic program synthesis assumes.
2. Syno is an end-to-end framework that synthesizes novel operators to replace those in a given network. It composes operators from fine-grained primitives defined on tensor dimensions (splitting, merging, unfolding, sharing, reducing) that are designed so any operator built from them is automatically valid — differentiable, with no replicated or discarded elements — making validity free rather than checked afterward. It applies expression canonicalization (via term rewriting) so equivalent candidates collapse to one form and can be pruned, letting the search skip the duplicates compilers already optimize. And it searches the space with a structured, guided procedure: bottom-up synthesis steered by a shape-distance metric toward the required input and output dimensions, formulated as a Markov decision process and solved with Monte Carlo Tree Search, with PyTorch and TVM code generation to measure real accuracy and speed.
3. The result is that Syno discovers operators faster than standard convolutions and matmuls even inside NAS-optimized models, achieving average speedups of 1.37 to 2.06 times across mobile CPUs, mobile GPUs, and server GPUs and across TVM and torch.compile, at under 1 percent accuracy loss on CIFAR-100 (with up to 4.73 times on ImageNet at 1 to 2 percent loss), and it accelerates GPT-2 training 1.1 times while improving perplexity from 111 to 99. The three approaches are complementary — a NAS topology, Syno operators, and a tensor compiler can be layered — and the trade-offs are that synthesis-plus-training is an expensive offline search, the gains come with a small accuracy cost and depend on the hardware and compiler, and the discovered operators are unconventional and less familiar than standard ones.

Check yourself

Check your understanding

1. What gap in neural network optimization does Syno target?
2. Why is fine-grained synthesis of neural operators so difficult?
3. How does Syno ensure the operators it synthesizes are valid and non-redundant?
4. How does Syno search the design space, and what did it achieve?

Sources

The paper is “Syno: Structured Synthesis for Neural Operators” by Yongqi Zhuo, Zhengyuan Su, Chenggang Zhao, and Mingyu Gao (Tsinghua University), published at ASPLOS 2025, DOI 10.1145/3676642.3736118. It is available on arXiv as 2410.23745.

This explainer is written from the paper’s full text (the authors’ arXiv copy). Syno is an end-to-end neural operator synthesis framework that, given a backbone network, searches for novel linear operators to replace its layers. It uses fine-grained primitives defined on tensor dimensions that preserve neural-operator properties (differentiability, no replication or discard), so synthesized operators are valid by construction; expression canonicalization via term rewriting to prune redundant semantically-equivalent candidates that tensor compilers already handle; and a bottom-up search guided by a shape-distance metric, framed as a Markov decision process solved with Monte Carlo Tree Search, with PyTorch and TVM code generators. Evaluated on five vision models and GPT-2, the paper reports average speedups of 1.37 to 2.06 times across various hardware and compilers at under 1 percent accuracy loss even on NAS-optimized models (up to 4.73 times on ImageNet at 1 to 2 percent loss), and GPT-2 training accelerated 1.1 times with perplexity improving from 111 to 99. Background on NAS, tensor compilers, and program synthesis is standard context. The numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.