CINM (Cinnamon): A Compilation Infrastructure for Heterogeneous Compute In-Memory and Compute Near-Memory Paradigms

Asif Ali Khan, Hamid Farzaneh, Karl Friedrich Alexander Friebel, Clément Fournier, Lorenzo Chelini and Jerónimo Castrillón

Compute-in-memory and compute-near-memory chips do work right where the data lives, escaping the bottleneck of shuttling data to a far-off processor, and they are orders of magnitude faster and more efficient — but every one of them is programmed by hand in its own low-level dialect, which Meta called the biggest obstacle to using them; this work from TU Dresden builds CINM, or Cinnamon, the first compiler that generalizes over these wildly different devices through a stack of MLIR abstractions, so one high-level program lowers to a memristor crossbar or a UPMEM near-memory processor, matching or beating hand-tuned code while being about fifteen times more concise.

The argument in one glance

Problem

Memory-centric chips are fast and efficient, but nearly impossible to program

Conventional computers spend most of their time and energy shuttling data between memory and a distant processor — on a phone, that movement alone burns most of the system power. A new class of chips fixes this by computing where the data already lives: compute-in-memory uses the physical properties of memory cells to calculate in place, and compute-near-memory puts small processors right next to the memory. They are orders of magnitude faster and more efficient. The catch is that each one is a radically different device, programmed only through its own low-level, hardware-specific library.
What people did

Hand-write low-level code, or build a compiler for one chip and one kernel

In practice, a programmer writes directly against each device's library, managing the mapping of work, the load balancing across compute units, the movement of data between memory levels, and the synchronization all by hand. Where compilers exist, each one targets a single kind of hardware and often a single application — a matrix multiply on a memristor crossbar, say. Meta publicly called programmability perhaps the biggest challenge to using these chips at all.
Why it fell short

One-off tools cannot span devices, and future systems will mix many of them

Because every existing compiler is tied to one device and one domain, moving an application to a different memory-centric chip means rewriting it against a different low-level interface, and there is no path at all to a heterogeneous machine that combines several of them. The programming models are too low-level and too specific, so these powerful chips stay locked away from ordinary programmers — and the heterogeneous systems everyone expects to build have no way to be programmed as a whole.
This paper's idea

One compiler that generalizes over all of them through layered abstractions

CINM, pronounced Cinnamon, is the first end-to-end compiler that generalizes across these diverse compute-in-memory and compute-near-memory devices. Built on MLIR, it lets a high-level program descend through a hierarchy of abstraction levels — a device-agnostic top that decides which device runs each kernel, paradigm-level layers for in-memory and near-memory computing, and device-specific layers underneath — applying the right optimization at each. The same high-level kernel lowers to a memristor crossbar or a UPMEM near-memory processor, matching or beating hand-tuned code while being far more concise.

Why this problem is worth solving

The whole point of these chips is a different answer to where computation happens, so it is worth being precise about the two families.

In a conventional (von Neumann) computer, data lives in memory and computation happens in a separate processor, so every operation means moving data across a narrow channel between the two — and that movement, not the arithmetic, is where most time and energy go. Two paradigms attack this. Compute-near-memory (CNM) places compute units physically next to the memory — for example UPMEM integrates a small general-purpose processor, a data processing unit, onto each DRAM bank, giving thousands of them running in parallel. This slashes the trip to a distant CPU, though data still moves between the memory and the nearby unit. Compute-in-memory (CIM) goes further and computes in place, exploiting the physics of the memory devices themselves — memristor cells arranged in a crossbar perform a matrix-vector multiply in constant time, and other technologies do logic or population counts directly in the array — so data never moves to a compute unit at all. The trade-offs differ too: in-memory computing is often analog and fixed-function and wears out its cells on writes, while near-memory computing is more general but still moves data locally.

The reason a compiler, not just faster hardware, is the missing piece is a gap between how people write programs and how these chips are driven.

Each of these chips exposes only a low-level, device-specific library, and leaves everything hard to the programmer: deciding which parts of the computation to offload, balancing load across thousands of compute units, orchestrating data movement between memory levels, and managing synchronization and consistency by hand. Worse, every chip is different — UPMEM, a memristor crossbar, and a Samsung near-memory DRAM share almost nothing in their programming models — so code written for one does not transfer, and a change to the hardware can force a near-total rewrite of the application. The compilers that exist each solve this for a single device and often a single kernel. As systems become heterogeneous, mixing several such accelerators with a CPU, this fragmentation becomes untenable: there is no single, retargetable way to express a program and let a compiler map it to whichever device suits it best.

What people did before, and where it broke

Program each chip by hand in its low-level library. Write directly against the device API, managing offload, load balancing, data movement, and synchronization manually. Where it broke. It demands deep per-device expertise, produces code that does not transfer to any other chip, and can require a full rewrite when the hardware changes.
Build a compiler for one CIM device and one kernel. Tools like OCC automatically map a specific pattern — a matrix multiply — onto a specific memristor crossbar. Where it broke. They target one homogeneous architecture and one application domain, so they neither generalize across the many device types nor handle a heterogeneous mix.
Build a compiler for one CNM system. Frameworks such as PIMFlow or XLA-NDP compile for a particular near-memory architecture, often only for one workload like convolutional networks. Where it broke. Again single-target and single-domain, with no shared abstraction that other devices could reuse.
Offer a high-level library for one system. SimplePIM gives a friendlier C++ interface for UPMEM specifically. Where it broke. A library uses kernels as-is on one device and cannot fuse operations or retarget; for most of these chips such libraries do not even exist.
Extract a data-flow graph and optimize at the LLVM level. Infinity Stream works from a single low-level IR to schedule hybrid in- and near-memory execution. Where it broke. A single flat IR makes it hard to express the domain-level and device-level optimizations that different abstraction heights make natural.
The gap. Every path was tied to one device, one domain, or one level of abstraction, so none could offer a single high-level program a route to many different memory-centric chips, let alone to a heterogeneous system built from several — which is exactly what programming this emerging hardware needs.

How CINM works

From a high-level kernel to diverse memory-centric hardware

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Generalize over utterly different hardware with a hierarchy of abstractions. The devices CINM targets share almost nothing — an analog memristor crossbar and a RISC core welded to a DRAM bank are worlds apart — and the earlier one-off compilers each collapsed into a single device because they had no way to separate what is common from what is specific. CINM’s foundation, built on the MLIR compiler framework, is to make a program descend through a stack of intermediate representations: a device-agnostic top where the program says what to compute, paradigm-level layers for in-memory and near-memory computing, and device-specific layers at the bottom where each chip’s quirks live. Optimizations that apply to any device are done high up; optimizations that depend on a chip’s physics are done low down; and the application never mentions a device. That same progressive-lowering discipline is what the CAM compiler covered elsewhere on this site applies to one specialized corner of this space — content-addressable search hardware — and it is in fact a direct extension of CINM, sharing its layered structure to reach a device family CINM’s own authors folded back in.

Choose the right device for each kernel, rewriting the work to fit. In a heterogeneous machine with several accelerators, the compiler must decide where each piece of a program should run — a question the old single-device tools never had to ask. CINM’s top-level dialect makes this decision over a fixed, well-understood set of primitives, and crucially it will rewrite operations to expose acceleration that is not visible in the original form: a convolution, which no target runs natively, becomes an image-to-column reshaping plus a matrix multiply, which both an in-memory crossbar and a near-memory processor can accelerate. It also carries a placeholder for cost models, so that once devices can be compared quantitatively the mapping can be automated. Deciding, per unit of work and per the workload’s shape, which hardware resource should handle it is the same placement discipline the workload-aware page-table work covered elsewhere on this site follows, matching each decision to the demand rather than applying one fixed rule everywhere.

Tile to the hardware, and optimize for its real costs. A memristor crossbar is a fixed, tiny array — sixty-four by sixty-four cells — while real tensors are enormous, so nothing fits without being cut up. CINM applies a generic tiling pass, told each device’s array size, that partitions a kernel into hardware-sized tiles and merges the partial results, and then it tunes for what actually costs on that device. On non-volatile memory a write is slow and physically wears the cell, so CINM reorders the tiled loops specifically to reduce the number of writes; on a near-memory processor it tiles for the small fast scratchpad instead. Carving a computation too large for fixed hardware units into pieces that fit, and stitching the partial answers back together, is the same partitioning problem the parallelization work covered elsewhere on this site solves for large models, dividing a computation across limited devices so that the whole of it can run.

Following a convolution through CINM

Compiling a 2D convolution to memory-centric hardware

Tracing one kernel from PyTorch to a memristor crossbar shows the stack at work.

Step 1 of 5

Import the kernel

The input is a 2D convolution from a PyTorch model, brought in through the linalg tensor dialect. Convolution dominates the runtime of most machine-learning models, but none of the target chips are built to run a convolution directly.

The front end is shared, so the same entry point serves TensorFlow, ONNX, and other frameworks.

Why it works

One layered IR lets you write once and optimize per device. These chips are radically different, from analog crossbars to RISC cores inside DRAM → but a stack of MLIR dialects keeps the program device-agnostic at the top and applies device-aware transformations only at the bottom → so a single high-level kernel targets any of them without the application ever naming a device.
Target selection sends each kernel to the hardware that suits it. A heterogeneous machine has several accelerators, each good at different things → but a top-level dialect that reasons over a fixed set of primitives and rewrites operations to expose acceleratable shapes can decide where each kernel runs → so a convolution becomes a matmul bound for a crossbar while other work goes to a near-memory processor.
Tiling to the hardware makes fixed-size accelerators handle any problem. A memristor crossbar is a fixed 64-by-64 array, far smaller than real tensors → but a generic tiling pass, told each device’s array size, splits kernels into array-sized pieces and merges the partial results → so arbitrarily large computations run without the programmer managing the split.
Optimizing for the hardware’s real costs beats a generic transform. On non-volatile memory a write is slow and wears the cell, unlike on a CPU → but a device-aware pass that reorders the tiled loops specifically to cut the number of writes respects that cost directly → so the compiler earns a sevenfold write reduction and a large speedup a hardware-blind pass would miss.
A compiler can match hand-tuned code while being far more concise. Experts hand-write hundreds of lines of low-level device code per kernel → but CINM generates code that meets or beats those hand-tuned versions from a high-level program about fifteen times shorter → so the productivity gain does not cost performance.

What it costs (trade-offs and scope)

Automated cost models and target selection are not actually implemented. The framework provides an interface where cost models can plug in, but the current mapping uses simple heuristics — a large matrix multiply goes to the crossbar, most else to UPMEM — or a user hint. The automated heterogeneous-mapping policy is explicitly left to future work.
Heterogeneous systems are described, not evaluated. Every experiment pairs the host CPU with a single accelerator (CPU-plus-UPMEM or CPU-plus-memristor), because no real machine combining in-memory and near-memory devices was available to test on. The heterogeneity story is architectural.
Only two backends are demonstrated. UPMEM near-memory and memristor crossbars were chosen because they had open tools to compare against; support for other chips like Samsung’s near-memory DRAM or content-addressable memories is described or handled in sister work, not benchmarked here.
The in-memory results are simulated. Only UPMEM ran on real hardware; the memristor-crossbar numbers come from the gem5 simulator using the prior OCC setup.
Benchmark preparation was partly manual. The non-idiomatic near-memory benchmarks had no MLIR front end, so they were hand-translated, and only seven of nine benchmark categories were evaluated to limit that effort.
The wins over hand-tuned code are real but moderate. Against the hand-optimized PrIM suite on UPMEM the speedup averages about 1.6 to 2 times, mostly from better partial-result and scratchpad management; on one kernel the hand-tuned version still wins, and a couple of in-memory kernels use more energy for lack of data reuse.
The conciseness metric is a rough proxy. The roughly fifteenfold lines-of-code reduction compares very different programming models and, as the authors themselves note, can mislead.

The evaluation

Programming memory-centric accelerators

Property	Device libraries	OCC (one CIM target)	Hand-tuned PrIM
Who writes it		Compiler, one target
Devices it spans	One at a time		UPMEM only
Device-agnostic input	No	Partly	No
Performance	Baseline		The CNM baseline
Code size	Verbose	Not reported	Hundreds of lines of C

Programming compute-in-memory and compute-near-memory accelerators, evaluated on a real UPMEM machine (16 DIMMs) for near-memory and the gem5 simulator with a PCM memristor crossbar for in-memory, against the hand-optimized PrIM suite and the OCC compiler. CINM matches OCC and reaches up to 30 times the CPU on in-memory matmul kernels (5 times less energy), beats hand-tuned PrIM code on UPMEM by about 1.6 to 2 times, and expresses applications roughly 15 times more concisely than low-level device code.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

CINM is a genuine first: a single, retargetable compiler that generalizes over deeply different compute-in-memory and compute-near-memory chips through MLIR’s progressive lowering, so one high-level program reaches a memristor crossbar or a UPMEM near-memory processor, matching or beating hand-tuned code while being about fifteen times more concise. Read its claims with their scope. The headline of a general, heterogeneous compiler is partly a promise: cost models and automated device selection are an interface, not an implementation, and no heterogeneous machine was actually evaluated — every test pairs the CPU with one accelerator. Only two backends are demonstrated, the in-memory numbers are from simulation while only UPMEM is real hardware, some benchmarks were hand-translated, and the wins over expert code, though real, are a moderate 1.6 to 2 times on UPMEM with one kernel still favoring the hand-tuned version. The durable contribution is the architecture itself: a layered set of abstractions — device-agnostic at the top, device-aware at the bottom — that gives this fragmented, fast-moving hardware space its first common compiler, extensible enough that adding a new chip need not disturb the layers above it.

Background

Why computing is moving into the memory

It helps to step back and ask why anyone builds a chip that computes inside its memory, because the answer explains this entire field. For decades, making programs faster meant making the processor’s arithmetic faster. But on today’s data-heavy workloads the arithmetic is rarely the bottleneck; the cost is in moving data between memory and a distant processor, over and over, across a channel that has not kept pace — on a mobile device, that movement alone can consume most of the system’s energy. Compute-near-memory and compute-in-memory attack this at the root by doing the work where the data already sits, which is why they can be orders of magnitude more efficient: not because they compute faster, but because they barely move anything. The same realization drives the deep-learning-on-GPUs retrospective covered elsewhere on this site, which measures whole real applications and concludes that data movement, not raw compute, is now the frontier. CINM matters because hardware this powerful is useless if only a handful of experts can program it — a common compiler is what turns a fragmented pile of clever memories into something a programmer can actually target.

If you remember only three things

1. Compute-in-memory (which calculates using the physical properties of memory cells, like a memristor crossbar doing a matrix-vector multiply in place) and compute-near-memory (which puts small processors right next to the memory, like UPMEM putting one on each DRAM bank) escape the von Neumann bottleneck of shuttling data to a distant CPU and deliver order-of-magnitude gains — but each chip is a radically different device programmed only through its own low-level library, with mapping, load balancing, data movement, and synchronization all left to the programmer, and existing compilers each target just one device and one application.
2. CINM (Cinnamon) is the first end-to-end compiler that generalizes across these diverse devices, built on MLIR so a high-level program (entered through linalg, TOSA, or Torch) lowers through a hierarchy of dialects: a device-agnostic cinm layer that performs target selection and rewrites operations (a convolution into a matmul) to expose acceleration, paradigm-level cim and cnm layers, and device-specific layers (memristor, UPMEM) that apply hardware-aware optimizations such as tiling to the crossbar size and reordering loops to minimize the slow, wearing writes on non-volatile memory — so device-agnostic optimizations happen high up and device-aware ones low down, and adding a new chip need not change the higher layers.
3. On a real UPMEM machine and a simulated memristor crossbar, CINM matches the OCC compiler and reaches up to 30 times the CPU on in-memory matmul kernels (5 times less energy), beats the hand-optimized PrIM suite on UPMEM by about 1.6 to 2 times, and is roughly 15 times more concise than low-level device code — but automated cost models and target selection are only an interface (current mapping is heuristic), no heterogeneous system was actually evaluated, only two backends are shown, the in-memory results are simulated, and some benchmarks were hand-translated.

Check yourself

Check your understanding

1. What is the difference between compute-in-memory and compute-near-memory?
2. How does CINM generalize over such different devices?
3. Why does CINM reorder loops to minimize writes on the memristor target?
4. Which statement best reflects the limits of CINM?

Sources

The paper is “CINM (Cinnamon): A Compilation Infrastructure for Heterogeneous Compute In-Memory and Compute Near-Memory Paradigms” by Asif Ali Khan, Hamid Farzaneh, Karl F. A. Friebel, Clement Fournier, Lorenzo Chelini, and Jeronimo Castrillon. The authors are at TU Dresden (with ScaDS.AI and the Barkhausen Institut) and Intel Switzerland; this explainer uses the extended version on arXiv (2301.07486, updated May 2024), which is part of the ASPLOS collection this site covers.

This explainer is written from the full 16-page text. Everything above comes from the paper’s own sections: the introduction and background motivating the von Neumann bottleneck and the compute-in-memory and compute-near-memory paradigms (including UPMEM’s data processing units and memristor crossbars, and the quoted industry note that programmability is the biggest challenge of processing-in-memory); the taxonomy and programmability landscape of these systems and the argument for shared abstractions; the CINM framework section detailing the MLIR-based progressive lowering through the cinm, cim, cnm, and device-specific (memristor, UPMEM) dialects, the target-selection and operation-rewriting mechanism, the workgroup model for near-memory computing, the compulsory tiling and write-minimizing loop interchange for in-memory computing, the cost-model interface, and the recipe for adding new devices; and the evaluation section reporting the in-memory results (matching OCC, about 10 times the CPU baseline on matmul, 7 times fewer writes and 12.4 times speedup from write minimization, 30 times speedup and 5 times energy reduction fully optimized, with mv and conv using 30 to 40 percent more energy), the UPMEM results (device-aware optimizations 40 to 47 percent faster, and CINM-generated code 1.6 to 2 times faster than the hand-optimized PrIM suite), and the roughly 15-fold reduction in lines of code versus low-level UPMEM programming.