Be CIM or Be Memory: A Dual-mode-aware DNN Compiler for CIM Accelerators

Compute-in-memory chips run neural networks fast by doing the math inside the memory arrays, and modern ones can flip each array between two roles — a compute unit or plain scratchpad memory — but the compilers that map networks onto these chips ignore that flexibility and freeze the split; this work from the Chinese Academy of Sciences builds CMSwitch, a compiler that decides, for each part of a network, how many arrays should compute versus store, because a convolutional network is starved for compute while a large language model is starved for memory, and getting the split right gives an average 1.31 times speedup over the best prior compilers.

The argument in one glance

Problem

CIM chips can flip arrays between compute and memory, but compilers ignore it

Compute-in-memory (CIM) accelerators do neural-network math inside the memory arrays. Modern chips let each array switch dynamically between compute mode (doing matrix math) and memory mode (acting as scratchpad). The right split depends on the workload, but existing compilers treat it as fixed.
What people did

Compile assuming compute and memory resources are static

Existing CIM compilers map a network onto the arrays and schedule it treating the amount of compute and the amount of memory as fixed, optimizing weight placement and scheduling within that frozen split.
Why it fell short

The wrong split starves the workload

A convolutional network is compute-hungry (high arithmetic intensity), while a large language model is memory-hungry (very low arithmetic intensity) and cannot fit its huge weights in compute mode. A fixed split serves one and starves the other, and even within one model the ideal balance varies by layer — so a static schedule wastes the dual-mode capability.
This paper's idea

A dual-mode-aware compiler that allocates modes per segment (CMSwitch)

CMSwitch puts the compute-memory mode switch into the compiler's optimization space via a new hardware abstraction, then a pass using dynamic programming and mixed-integer programming finds the best network segments and their per-segment mode allocations, expressed through a tailored meta-operator. This adaptive splitting gives an average 1.31 times speedup over state-of-the-art CIM compilers.

Why this problem is worth solving

Compute-in-memory is one of the most promising answers to the memory wall for neural networks, and getting the most from it depends on the compiler.

A compute-in-memory (CIM) accelerator merges computation and storage: instead of fetching weights from memory to a separate processing unit, the weights are stored in a memory array — a crossbar of resistive or SRAM cells — and the array itself performs the multiply-accumulate operations at the heart of a neural network, in place, in massively parallel fashion. This dramatically cuts the data movement that dominates conventional accelerators and directly attacks the memory wall, where performance is limited not by arithmetic but by moving data, which makes CIM especially competitive for data-intensive deep-neural-network inference. But a raw CIM chip is hard to use, so a compiler is needed to map a network onto it: deciding which arrays hold which layer’s weights, how the computation is scheduled across the arrays, and how data flows between them. A good CIM compiler lowers the barrier to adopting the architecture and can make a large difference to performance, which is why compilation for CIM has been an active research area, with tools tuned for various array technologies.

The newest CIM chips add a flexibility that changes the compiler’s job, and existing compilers do not exploit it.

Modern CIM designs make each memory array dual-mode: by resetting the control signals to its input driver, an array can switch dynamically between compute mode, where it performs matrix math on the weights stored in it, and memory mode, where it serves as ordinary on-chip scratchpad memory, for example caching the activations flowing through the network. This is powerful because different workloads need very different balances of compute and memory, and the split is zero-sum — an array devoted to compute cannot also be memory. The determining factor is arithmetic intensity, the ratio of computation to data moved. A convolutional network like ResNet-50 has high arithmetic intensity (on the order of dozens of operations per byte), so it is compute-hungry and wants many arrays in compute mode. A generative large language model like LLaMA doing single-batch inference has very low arithmetic intensity (around two), and enormous parameters that cannot all be held in compute mode at once, so it is memory-hungry and wants more arrays as scratchpad. The ideal balance even varies across the layers and stages of a single model. Yet existing CIM compilers treat the compute-versus-memory resources as static, scheduling as if the split were fixed — so on a dual-mode chip they leave much of the benefit on the table.

What people did before, and where it broke

Map DNNs onto CIM assuming a fixed compute-memory split. Optimize weight placement and scheduling within a static allocation of compute and memory arrays. Where it broke. Modern chips can switch each array’s mode, and a fixed split cannot adapt to a workload’s actual compute-versus-memory needs, leaving performance unrealized.
Optimize only the scheduling of the computation. Focus the compiler on ordering operations and mapping weights, given fixed resources. Where it broke. It ignores the array-mode decision entirely, so the most consequential knob on a dual-mode chip is never turned.
Configure the chip for one workload class. Set the compute-memory balance for, say, convolutional networks. Where it broke. Large language models have the opposite need — memory over compute — so a single static configuration serves one class and starves the other, and no single split fits all layers even within one model.
Ignore the interdependence of mode and mapping. Treat which arrays compute and where weights go as separable decisions. Where it broke. The number of arrays in each mode affects both the current operator’s efficiency and the scheduling of later operators, so the decisions are entangled and cannot be made independently.
The gap. No compiler jointly decided array modes and weight mapping across a network to match each part’s compute-memory needs. Providing that, via a dual-mode-aware optimization, is what CMSwitch does.

How CMSwitch works

From a static compute-memory split to per-segment mode allocation

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Make the compiler aware of a hardware capability it had been ignoring. CMSwitch’s foundation is to bring the array’s compute-or-memory mode switch, previously a hidden hardware feature, into the compiler’s optimization space by adding a new hardware-abstraction attribute that represents each array’s dual-mode ability. Only once the compiler can see and reason about this knob can it be optimized; prior compilers, blind to it, could never turn it. Lifting a target accelerator’s distinctive low-level capability into the compiler’s model so the compiler can actually optimize for it, rather than compiling against a generic abstraction that hides what the hardware can do, is the same move the in-memory-accelerator compiler work covered elsewhere on this site makes, teaching the compiler the specifics of the compute-in-memory substrate so it can generate code that exploits the hardware instead of merely running on it.

Match the compute-memory split to what each part of the network actually needs. The heart of the approach is recognizing that the right balance is workload-dependent and even segment-dependent: a compute-hungry convolution and a memory-hungry language-model layer want opposite allocations, and forcing one static split onto both wastes the chip. CMSwitch therefore allocates modes adaptively across the network, giving compute-bound segments more computing arrays and memory-bound segments more scratchpad, so the reconfigurable hardware is continually retuned to the demand in front of it. Reconfiguring a flexible accelerator on the fly to match the shifting compute-versus-memory character of the workload, rather than committing to one fixed configuration, is the same discipline the mode-switching parallelism work covered elsewhere on this site follows, changing the hardware’s operating mode as the workload’s needs change so that each phase is served by the configuration that suits it instead of a single compromise.

Formulate the entangled decision as an optimization and solve it. The mode-allocation and weight-mapping decisions are interdependent and explode combinatorially — with m arrays there are two-to-the-m mode allocations, multiplying an already hard CIM scheduling problem into an exponential space — so CMSwitch does not search naively but formulates the joint problem and solves it with dynamic programming to segment the network and mixed-integer programming to allocate resources within each segment. This turns an intractable brute-force search into a structured optimization with a principled answer. Casting a combinatorially explosive compilation decision as a mathematical optimization problem handed to a solver, rather than relying on heuristics or exhaustive enumeration, is the same design the solver-aided compiler work covered elsewhere on this site is built on, expressing the compiler’s hard placement-and-scheduling choices as constraints an optimizer can resolve so the generated plan is optimal within the model rather than merely feasible.

Following a network through CMSwitch

Compiling a DNN for a dual-mode CIM accelerator

A trace of CMSwitch compiling a network, following the paper's dual-mode-aware optimization.

Step 1 of 5

Model the chip's dual-mode arrays

CMSwitch represents the CIM accelerator with a hardware abstraction in which each array carries a mode attribute — it can be a compute unit doing matrix math or a memory scratchpad — so mode is a variable the compiler can set.

Prior compilers omitted this attribute entirely.

Why it works

The optimal split genuinely differs by workload. Because arithmetic intensity ranges from high in convolutions to very low in single-batch language models → the ideal compute-memory balance is opposite for the two → so a compiler that adapts the split beats any single fixed configuration.
Exposing mode to the compiler is prerequisite. Because prior compilers had no representation of the mode switch → they could not optimize a knob they could not see → so adding a hardware-abstraction attribute is what makes the whole optimization possible.
Per-segment allocation fits intra-model variation. Because even one model’s compute-memory needs vary across layers and stages → segmenting the network and allocating modes per segment tracks that variation → so no part is forced into a split meant for another.
Joint optimization respects the interdependence. Because array mode and weight mapping affect each other and later operators → deciding them together rather than separately captures their coupling → so the schedule is coherent instead of locally greedy.
A solver tames the exponential space. Because two-to-the-m mode allocations make brute force hopeless → dynamic programming plus mixed-integer programming structures the search → so the compiler reaches a principled allocation in tractable time.

What it costs (trade-offs and scope)

It requires dual-mode CIM hardware. CMSwitch’s benefit exists only on chips whose arrays can actually switch between compute and memory modes. On fixed-function CIM, or conventional accelerators, the core idea does not apply, so the gains depend on that emerging hardware capability being present.
Compile-time optimization has a cost. Dynamic programming and mixed-integer programming to jointly segment, allocate, and map are more expensive at compile time than a static schedule; this is an offline cost paid once per model, but the solver’s scalability bounds how large or complex a problem can be handled.
The model must capture mode-switch costs. Switching an array’s mode and reconfiguring is not free, and the quality of the allocation depends on the compiler’s model of those costs and of the hardware; an inaccurate abstraction would yield suboptimal or invalid schedules.
Gains vary widely by model. The average 1.31 times speedup spans a range — modest for some networks and much larger for memory-bound large models (for one large language model the paper reports well over a 1.7 times average against the main baseline) — so the benefit is concentrated where the compute-memory mismatch under a static split was worst.
It optimizes allocation, not the underlying arrays. CMSwitch schedules a given dual-mode chip well; it does not change the array technology, precision, or peripheral design, so it is complementary to hardware advances rather than a substitute, and its ceiling is set by the chip.
Results are against specific baselines and configurations. The speedups are relative to particular state-of-the-art CIM compilers on particular models and on a specific assumed hardware configuration (array counts, on-chip memory); different baselines or chips would shift the numbers.

The evaluation

Compiling DNNs for dual-mode CIM accelerators

Property	Static-split CIM compiler	Single-workload configuration
Uses the arrays' mode switch
Adapts split to the workload
Compute-heavy nets (CNNs)	Adequate if configured so	Good if tuned for CNNs
Memory-heavy nets (LLMs)		Poor if tuned for CNNs
Speedup over prior CIM compilers	Baseline	Baseline

Compiling deep neural networks for dual-mode compute-in-memory accelerators, as compared in the paper. CIM accelerators do neural-network matrix math inside memory arrays, and modern designs let each array switch dynamically between compute mode (doing the math) and memory mode (on-chip scratchpad for activations) — a zero-sum split, since an array cannot be both. Existing CIM compilers treat compute and memory resources as static, optimizing weight mapping and scheduling within a fixed split, so they never use the mode switch. But the ideal split is workload-dependent: convolutional networks have high arithmetic intensity and want many computing arrays, while large language models in single-batch inference have very low arithmetic intensity and huge parameters and want more scratchpad memory, and the balance even varies across a model's layers and stages. CMSwitch is a dual-mode-aware compiler that brings the compute-memory mode switch into the optimization space via a new hardware-abstraction attribute, then uses a compilation pass combining dynamic programming (to segment the network) and mixed-integer programming (to allocate compute-versus-memory arrays and map weights per segment) to find the best joint allocation, expressed through a tailored meta-operator. This adaptive allocation delivers an average 1.31 times speedup over state-of-the-art CIM compilation works across a wide range of real DNNs, with the largest gains on memory-bound large models where a static split hurt most.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

CMSwitch is a clean compiler contribution that catches up to a hardware capability the software had ignored. Compute-in-memory accelerators fight the memory wall by doing neural-network math inside the memory arrays, and modern CIM chips add a genuinely useful trick: each array can switch between being a compute unit and being plain scratchpad memory. This matters because different workloads want opposite balances — a convolutional network is compute-hungry with high arithmetic intensity, while a large language model doing single-batch inference is memory-hungry with very low arithmetic intensity and parameters far too large to all sit in compute mode — and the balance even shifts across a single model’s layers. The paper’s observation is that existing CIM compilers ignore all this, scheduling as if the compute-memory split were fixed, and thereby leaving the dual-mode benefit unused. CMSwitch fixes it by exposing the mode switch to the compiler through a new hardware-abstraction attribute, then jointly deciding, per network segment, how many arrays compute versus store and where weights map, using dynamic programming to segment and mixed-integer programming to allocate — taming a search space that is exponential in the number of arrays. The result is an average 1.31 times speedup over state-of-the-art CIM compilers, larger on the memory-bound large models where a static split hurt most. Read the scope carefully. The approach requires dual-mode CIM hardware and is only as good as its model of mode-switching costs and the chip; the joint optimization adds compile-time cost bounded by solver scalability; the gains vary widely by model and are measured against specific baselines and a specific hardware configuration; and it optimizes how a given chip is used rather than improving the chip itself. The durable idea is that when hardware gains a new degree of freedom — here, arrays that can be compute or memory on demand — the compiler must be extended to see and optimize that freedom, because a capability the software cannot represent is a capability the system cannot use, and matching a reconfigurable accelerator to each workload’s real compute-memory character can be worth a substantial, and workload-dependent, speedup.

Background

Why reconfigurable accelerators push work onto the compiler

It helps to see CMSwitch within a broad pattern: as accelerators become more flexible, more of the burden of using them well shifts onto the compiler, which must decide how to configure the hardware for each workload. A dual-mode CIM array that can be compute or memory is one instance of a general trend toward reconfigurable substrates whose potential is unlocked only by software smart enough to exploit their flexibility. The processing-in-memory work for large-model inference covered elsewhere on this site reflects the same frontier from the workload side, confronting how memory-bound large language models stress the balance between compute and memory bandwidth in memory-centric accelerators — precisely the tension CMSwitch resolves at compile time by reallocating arrays between roles. The common thread is that the memory wall and the rise of both compute-heavy and memory-heavy neural workloads have made the compute-versus-memory balance a first-class design variable, and hardware that can adjust that balance is only as good as the compiler that sets it. As CIM and other reconfigurable accelerators mature and as workloads span the gulf from convolutions to trillion-parameter language models, compilers that treat hardware configuration as part of the optimization — matching each model, and each part of each model, to the resources it needs — are likely to be essential to realizing what the hardware promises.

If you remember only three things

1. Compute-in-memory (CIM) accelerators run neural networks by performing the multiply-accumulate math directly inside the memory arrays, cutting the data movement that causes the memory wall, and a compiler maps a network onto the chip by placing weights and scheduling computation. Modern CIM chips add dual-mode arrays: each array can switch dynamically between compute mode (doing matrix math on stored weights) and memory mode (acting as on-chip scratchpad for activations), a zero-sum split since an array cannot be both. The ideal balance is workload-dependent, set by arithmetic intensity (computation per byte): a convolutional network like ResNet-50 has high arithmetic intensity and wants many computing arrays, while a large language model like LLaMA in single-batch inference has very low arithmetic intensity (around two) and parameters far too large to all hold in compute mode, so it wants more scratchpad memory — and the balance even varies across a single model's layers. But existing CIM compilers treat the compute-memory split as static, so on a dual-mode chip they leave the capability and its performance untapped.
2. CMSwitch is a dual-mode-aware DNN compiler that closes this gap. First, it brings the compute-memory mode switch into the compiler's optimization space by adding a new hardware-abstraction attribute representing each array's dual-mode ability, since a knob the compiler cannot see cannot be optimized. Second, it matches the split to the workload adaptively, giving compute-bound segments more computing arrays and memory-bound segments more scratchpad. Third, because the mode-allocation and weight-mapping decisions are interdependent and the search space is exponential (m arrays give two-to-the-m mode allocations, multiplying an already hard CIM scheduling problem), it formulates the joint problem and solves it with dynamic programming to segment the network and mixed-integer programming to allocate compute-versus-memory arrays and map weights within each segment. The resulting plan is expressed through a tailored meta-operator, a generalized representation of the compiled schedule that the accelerator executes with its arrays switching modes per segment.
3. Across a wide range of real-world DNNs, CMSwitch achieves an average 1.31 times speedup over state-of-the-art CIM compilation works, with the largest gains on memory-bound large models (for one large language model, well over 1.7 times average against the main baseline) where a static compute-memory split hurt most. Scope: it requires dual-mode CIM hardware and is only as good as its model of mode-switching costs and the chip; the joint optimization adds compile-time cost bounded by solver scalability; the gains vary widely by model and are relative to specific baselines and a specific hardware configuration; and it optimizes how a given chip is used rather than improving the chip. The durable lesson is that when hardware gains a new degree of freedom — arrays that can be compute or memory on demand — the compiler must be extended to represent and optimize that freedom, because a capability the software cannot express is one the system cannot use, and matching a reconfigurable accelerator to each workload's compute-memory character can be worth a substantial, workload-dependent speedup.

Check yourself

Check your understanding

1. What is the dual-mode capability of modern CIM arrays, and why does the right balance depend on the workload?
2. Why do existing CIM compilers fail to exploit dual-mode arrays?
3. How does CMSwitch decide the mode allocation, and how does it handle the huge search space?
4. What did CMSwitch achieve, and what are its limits?

Sources

The paper is “Be CIM or Be Memory: A Dual-mode-aware DNN Compiler for CIM Accelerators” by Shixin Zhao, Yuming Li, Bing Li, Yintao He, Mengdi Wang, Yinhe Han, and Ying Wang, published at ASPLOS 2025, DOI 10.1145/3676641.3716248. The authors are at the Institute of Computing Technology and the Institute of Microelectronics of the Chinese Academy of Sciences and the University of Chinese Academy of Sciences; the compiler is named CMSwitch.

This explainer is based on the full text of the paper, obtained from the authors’ arXiv version (arXiv:2502.17006). The problem framing (compute-in-memory accelerators performing multiply-accumulate inside memory arrays to mitigate the memory wall for DNN inference; the role of a CIM compiler in mapping weights and scheduling; the dual-mode capability of modern CIM arrays that can switch between compute and memory modes by resetting input-driver control signals, with the zero-sum constraint that a compute-mode array cannot serve as memory; and why the ideal split is workload-dependent, illustrated by ResNet-50’s high arithmetic intensity around 66 versus LLaMA 2’s average around 2 for single-batch inference, and why existing compilers treating resources as static miss this), the CMSwitch design (integrating the compute-memory mode switch into the optimization space via a new hardware-abstraction attribute; the two challenges of an exponential two-to-the-m mode-allocation search space entangled with weight mapping and the interdependence between array-mode scheduling and mapping; the compilation optimization pass using dynamic programming to identify network segments and mixed-integer programming to allocate modes and map weights; and the tailored meta-operator to express the result), and the results (an average 1.31 times speedup over state-of-the-art CIM compilation works, with per-model ranges against the CIM-MLC baseline including up to roughly 1.30 times on some models and an average around 1.73 times for OPT-13B, and comparisons to baselines including PUMA and CIM-MLC) are all drawn directly from the paper. Background explanations of compute-in-memory, multiply-accumulate, arithmetic intensity, the memory wall, dynamic programming, and mixed-integer programming are standard and provided to make the paper’s contribution understandable.