CIM-MLC: A Multi-level Compilation Stack for Computing-In-Memory Accelerators

Chips that compute inside the memory array itself promise to sidestep the cost of moving data, but no two of them look alike — different memory technologies, different depths of hierarchy, and, most awkwardly, different levels at which they let software issue commands — so every research group writes its own compiler by hand; CIM-MLC (Chinese Academy of Sciences) proposes an abstraction with three tiers, a scheduling level per tier, and the honest finding that a level helps only when the hardware has room for it.

The argument in one glance

Problem

Moving data costs more than computing on it

A neural network is mostly multiply-and-accumulate, and on a conventional chip most of the time and energy goes into shuttling weights and activations between memory and arithmetic units rather than into the arithmetic. Computing inside the memory array itself removes that traffic entirely — the array holds the weights and performs the multiplication where they sit.
What people did

Build accelerators that compute in memory, and many people have

Dozens of designs exist, using several memory technologies. Some organise chips into cores into arrays; some are a flat sheet of arrays. Some let you invoke a whole convolution; some let you invoke one matrix-vector multiply; some let you switch on individual rows of the array. Each reports impressive numbers on the networks its authors ran.
Why it fell short

Every one of them needs its own compiler, written by hand

Mainstream deep learning compilers optimise for chips where memory and arithmetic are separate, so their central techniques are about improving data locality — which is the problem this hardware does not have. The compilers written for this hardware are each bound to one architecture, and typically schedule at one granularity, so they degrade or fail when the architecture or the exposed granularity changes.
This paper's idea

Abstract the hierarchy, and match a scheduling level to each level of it

Describe any such accelerator as three tiers — chip, core, array — with a parameter set per tier, and observe that the granularity at which a design lets software issue commands corresponds one-to-one with those tiers. Then run a scheduling pass per granularity, each inheriting the decisions of the coarser one above it, and stop wherever the hardware stops exposing control.

Why this problem is worth solving

The premise is a genuine inversion of how chips normally work.

Performing arithmetic inside the memory array rather than reading values out to a separate arithmetic unit. For neural networks the trick is that a grid of memory cells holding a weight matrix can multiply an input vector by that entire matrix in one step, using physics — drive the rows with voltages proportional to the inputs and the current summing down each column is the dot product. This removes almost all the data movement that dominates conventional accelerators, but it also means the computation is inseparable from the physical layout of the memory, so everything about the array becomes a compiler concern.

is a genuinely different computing paradigm, and the tools built for the old one do not transfer.

The unit of hardware is where the diversity concentrates.

The grid of memory cells at the heart of one of these accelerators, which stores a chunk of a weight matrix and performs a matrix-vector multiplication against it in a single operation. Three of its properties dominate compilation: how large it is, since a matrix bigger than the grid must be split across several; how many bits each cell holds, since low-precision cells force a value to be spread over several columns or several grids; and how many rows may be driven simultaneously, since some designs restrict this to limit power draw or analog error, which turns one operation into several.

is the reason a compiler for one of these chips does not work on another.

What people did before, and where it broke

Use a mainstream deep learning compiler. The obvious starting point, and these frameworks are mature. Where it broke. Their core optimisations — loop unrolling, tiling, reordering — exist to improve data locality, because they target chips where memory and arithmetic are separate. This hardware has no such problem, and what it needs instead is help using the array efficiently, which those frameworks have no concept of.
Write a compiler for your own architecture. What most groups do, and it produces good results on that architecture. Where it broke. The mapping and scheduling policies do not generalise — two designs with apparently similar hierarchies still differ in array size and array count, which changes the mapping entirely. It is a large amount of duplicated effort with nothing shared.
Use polyhedral methods to find and map the matrix multiplications automatically. A more principled route that two published tools take. Where it broke. Both target one memory technology, and both assume there is ample capacity for loading and duplicating parameters — which is exactly the assumption that fails on the resource-constrained designs where capacity is the binding limit.
Expose a specific programming interface to the compiler. Several designs do this, giving the compiler a defined vocabulary to target. Where it broke. The interface is bound to that design, and because different designs expose control at different granularities, a compiler written against one interface has the wrong optimisation space for another.
Schedule at matrix-vector granularity. What most published work does, since that is the natural unit of the hardware. Where it broke. It misses the coarser question of how to duplicate and pipeline whole operators across the chip, and the finer question of when to activate individual rows. The paper notes that existing work supports pipelining across layers but not at matrix-vector granularity, which is where its own power result comes from.
The gap. Everyone treated the level at which the hardware is programmable as a fixed backdrop. It is actually the variable — it determines what the compiler can even express — so it belongs in the abstraction rather than being baked into a particular tool.

How CIM-MLC compiles a network

From a network description to a stream of hardware operations

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

Around 3.2 times faster than the prior general compiler for this hardware, and 75 percent lower peak power on one architecture.

The idea

The level at which a design lets software issue commands is an architectural property, and it belongs in the abstraction. This is the observation the whole design rests on, and it is easy to underrate. Everyone knows these accelerators differ in memory technology and hierarchy depth. What the paper adds is that they also differ in something that changes the compiler’s job qualitatively rather than quantitatively — one design accepts a whole convolution and places it on cores, another accepts a single matrix-vector multiply and places it on arrays, another exposes individual rows so software can decide which to switch on. Those are not three levels of detail about the same problem; they are three different problems, with different decision variables and different sizes of search space. So the abstraction describes any such accelerator as three tiers with a parameter set apiece, and records which of three corresponding granularities the design exposes — and the compiler simply runs whichever levels apply. Building one abstraction that several incompatible hardware projects can share, rather than each writing its own compiler, is the same argument as the shared stencil compilation work covered elsewhere on this site.

Run a pass per granularity, coarse to fine, each inheriting the last. The alternative structures are both worse in known ways. Optimising only at graph level, as conventional compilers do, cannot see the array resources at all. Optimising only at matrix-vector level, as most prior work does, misses both the question of how many copies of an operator to spread across the chip and the question of when to fire individual rows. And optimising everything at the finest granularity at once produces a search space the paper describes as intractable. Layering solves this by making each level a smaller problem constrained by the one above: the coarsest pass searches duplication counts by dynamic programming under the core budget, then balances the pipeline, then trims for network and buffer bandwidth; the middle pass inherits those counts and recomputes duplication per array; the finest pass inherits both and rearranges weight placement so row activations line up. Decomposing a joint hardware-and-schedule search into levels that constrain each other rather than solving it flat is the same tactic as the speculative inference co-exploration work covered elsewhere on this site.

The most interesting result is where a level buys nothing, and the paper reports it. Applying the stack to three real published accelerators produces three different stories, and they are more informative than the headline. On a design exposing whole operators, only the coarsest pass applies, and duplication combined with pipelining gives around 3.7 times. On a design exposing arrays, the staggered activation cuts peak power by 75 percent — worth noting because 83 percent of the power is array activation itself. On a design exposing individual rows, the three levels together give around 2.3 times, but the middle level contributes nothing at all, because that macro has too few arrays per core for array-level duplication to have anywhere to go. The paper says this plainly. It also reports the other side of the trade: duplication at the coarsest level raises normalised peak power by 5 to 16 times on one network family, which is the same quantity the middle level exists to reduce. The optimisations are not simply additive, and what any of them is worth depends on what the hardware has spare. Finding that a technique’s value is bounded by the concrete limits of the memory hardware rather than by the idea is the same lesson as the commodity memory processing work covered elsewhere on this site.

Scheduling one convolution three ways

The same operator, on three accelerators that expose three different granularities

A convolution followed by a rectifier, compiled for hardware that accepts operators, arrays, or rows.

Step 1 of 5

Describe the hardware once

Fill in the three parameter sets — cores per chip, network type and its cost matrix, buffer size and bandwidth; arrays per core and the same for the level below; then array dimensions, converter precisions, cell technology, bits per cell, and how many rows may be driven at once.

A separate mechanism handles a subtlety this glosses over: designs differ in how they spread a value bit-width across the hardware. A binding scheme states whether the bits of a number go across adjacent columns of one array or across separate arrays, so one abstraction covers both conventions.

Why it works

Programming granularity determines what the compiler can express. A design accepting whole operators gives the compiler placement and duplication decisions and nothing finer → while one exposing rows gives it activation scheduling too → so recording the granularity in the abstraction is what lets one compiler serve both instead of two compilers serving one each.
Layering keeps each search small. The coarsest pass decides duplication under a core budget, which is a manageable problem → and each finer pass inherits those decisions as fixed constraints rather than reopening them → which is why the combined space is broader than single-level scheduling without being the intractable flat search.
Duplication and pipelining fail in opposite regimes. Duplicating an operator across cores helps most when the network is small enough that copies fit, falling from 25.4 times to 3.1 as models grow → while pipelining adjacent operators helps more as networks get deeper, rising from 2.3 times to 4.7 → so the search has to trade them against each other rather than maximising either.
Firing arrays as their inputs arrive cuts the peak, not the total. A conventional schedule waits for every array in a multiplication before starting, so all are live together → whereas firing each on arrival means fewer are live at any instant → and since array activation is 83 percent of the power, that reduces the peak by 75 percent on one design.
Restricted row activation is a scheduling problem, not just a slowdown. Some designs cap how many rows may be driven at once to limit analog error → so a result spanning more rows than the cap takes several cycles and stalls whatever consumes it → and rearranging where weights sit lets the consumer start on partial results instead of waiting.

What it costs (trade-offs and scope)

The largest number in the paper is not a comparison against prior work. The 123 times figure for one network family is duplication and pipelining measured against no optimisation at all. The comparison against the existing general compiler for this hardware is around 3.2 times, and the comparisons against the three real accelerators’ own schedules are 3.7, 2.3 times and a power reduction.
Everything is simulated. A functional simulator written for this work, checked against a mainstream framework, plus performance and power simulators extended from three existing open-source ones. No hardware was involved.
Unspecified parameters are treated as ideal, and the paper says so. The example given is that the on-chip buffer is assumed to have enough bandwidth that data movement hides inside computation. For hardware whose entire premise is that data movement is the problem, idealising the buffer is not a small assumption.
One of the three scheduling levels contributes nothing on one of the three accelerators. Array-level optimisation gives no improvement on the macro with few arrays per core, which the paper states directly. The multi-level structure is worth having, and how much of it pays depends on the hardware having spare capacity at that level.
Two of the optimisations pull against each other. Duplication at the coarsest level raises normalised peak power by 5 to 16 times on one network family, while the array-level pipeline exists to reduce peak power. The speedup and the power results come from different experiments on different hardware.
Each real accelerator exercises only the levels it exposes. That is correct by construction, but it means no single design is evaluated across all three levels against its own baseline except the row-exposing macro, where one level is inert.
Duplication degrades sharply with model size. From 25.4 times on the smallest network in a family to 3.1 on the largest, so the technique that dominates the coarse level is the one that stops working as models grow.
Scope is inference on quantised convolutional and transformer networks. All models are quantised to eight bits and evaluated on one image dataset; training is not addressed.
The generality claim rests on the abstraction rather than on breadth of evaluation. The comparison table shows the abstraction covering more device types, interfaces and granularities than any prior tool, which is a design argument; the empirical support is three accelerators and one prior compiler.

The evaluation

Three ways to compile a network for in-memory hardware

Property	Mainstream deep learning compilers	Compilers written for one such architecture
What they optimise for		Efficient use of the arrays, for one specific design
How much hardware they can describe	None of it — arrays and row activation have no representation
Scheduling granularity	Computation graph only
Measured result	Not applicable — no model of the hardware
Peak power	Not modelled

Three published accelerators using two memory technologies and three programming granularities, plus one prior general compiler, on quantised convolutional and transformer networks in simulation.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The headline figure of 123 times is not a comparison with prior work — it is duplication plus pipelining measured against no optimisation at all. The comparison with the existing general compiler for this hardware is around 3.2 times, and against the three real accelerators’ own schedules the results are 3.7 times, 2.3 times, and a 75 percent peak power reduction. Everything is simulated: a functional simulator written for this work and checked against a mainstream framework, plus performance and power simulators extended from three existing ones. Parameters not explicitly set are treated as ideal, which the paper states, with the example that buffer bandwidth is assumed sufficient for data movement to hide inside computation — a meaningful assumption for hardware whose premise is that data movement is the problem. The most useful negative result is reported honestly: on one of the three accelerators the array-level pass contributes nothing at all, because that design has too few arrays per core for it to work with. And two of the optimisations pull in opposite directions, since coarse duplication raises normalised peak power by 5 to 16 times on one network family while the array-level pipeline exists to reduce it. The generality argument rests mainly on the abstraction covering more of the design space than prior tools; the empirical support is three accelerators.

Background

Why this hardware resists having a compiler at all

Conventional accelerators, however varied, share an abstraction: some arrangement of arithmetic units fed by a memory hierarchy, where the compiler’s job is to arrange the data movement well. That shared shape is what lets one framework target many chips. Computing in memory breaks it, because the computation is performed by the storage, so the physical layout of the array is not an implementation detail the compiler can abstract over — the array’s dimensions decide how a matrix is split, the bits per cell decide how a number is spread, and the number of rows that may be driven at once decides how many operations a single multiplication becomes. Two designs can be architecturally similar and still need completely different mappings. The field therefore ends up with the situation this paper describes, where every group writes its own compiler, and the way out is to find an abstraction that admits the diversity rather than hiding it — the same problem, from the other end, that intermediate representations for hardware accelerators such as the work covered elsewhere on this site are trying to solve.

If you remember only three things

1. Accelerators that compute inside the memory array eliminate the data movement that dominates conventional chips, but the computation becomes inseparable from the physical array, so array size, bits per cell and how many rows may be driven at once all become compiler concerns — which is why mainstream deep learning compilers, whose core techniques exist to improve data locality, do not transfer.
2. CIM-MLC observes that the granularity at which a design lets software issue commands is itself an architectural property that changes the compiler job qualitatively — whole operators, single matrix-vector multiplies, or individual row activations — and builds an abstraction of three hardware tiers matched one-to-one with three scheduling levels, each inheriting the decisions of the coarser one above it.
3. The most instructive results are internal rather than headline: on one of three real accelerators the array-level pass contributes nothing because that design has too few arrays per core, coarse duplication raises peak power by 5 to 16 times while the array-level pipeline exists to lower it, and the 123 times figure is measured against no optimisation rather than against prior work, where the margin is around 3.2 times.

Check yourself

Check your understanding

1. Why do mainstream deep learning compilers not work for this hardware?
2. What does the paper add to the observation that these accelerators are diverse?
3. Why schedule at three granularities in sequence rather than all at once?
4. What is the most informative result in the evaluation?

Sources

The paper is “CIM-MLC: A Multi-level Compilation Stack for Computing-In-Memory Accelerators” by Songyun Qu, Shixin Zhao, Bing Li, Yintao He, Xuyi Cai, Lei Zhang and Ying Wang, at the Chinese Academy of Sciences and collaborating institutions, published at ASPLOS 2024 Volume 2, pages 185-200, DOI 10.1145/3620665.3640359. The first two authors contributed equally. It is open access under a Creative Commons Attribution licence and the project is public.

This explainer is written from the full text, read from the authors’ copy on arXiv (2401.12428). Everything above comes from the paper’s own sections 2 through 4: the survey classifying published accelerators along memory device, architecture hierarchy and programming interface, including the observation that write cost differences make some designs freeze weights in place while others update them freely; the argument that mainstream compilers optimise for memory efficiency where this hardware needs in-situ computation efficiency, and the assessment of prior tools as bound to specific devices, assuming ample capacity for duplication, or omitting the coarser operator interface; the section 3.2 hardware abstraction with its three tiers, the full parameter lists at each, the three computing modes in one-to-one correspondence with the tiers, the virtual crossbar and dimension-binding scheme for value bit-width, and the row activation limit; the section 3.3 scheduling levels including the dynamic programming duplication search with its pipeline balancing and bandwidth trimming, the sub-graph construction for networks that do not fit, the duplication recomputation formula, the staggered array activation with its worked example and roughly 30 percent peak power figure, and the weight remapping for limited row activation; the meta-operator vocabularies given in formal syntax for each granularity; and the section 4 evaluation covering the simulator construction and the statement that unspecified parameters are treated as ideal, the three accelerator case studies with their individual results including the array-level pass contributing nothing on one of them, the comparison against the prior general compiler, the network study showing duplication and pipelining trading off with model depth and the peak power increase from duplication, and the sensitivity studies over core count, array count, array size and row activation limit.