Moving data between memory and the CPU costs far more than the math itself, so a promising idea is to compute INSIDE the memory — modern CPUs have huge L3 caches, and turning their SRAM cells into bit-serial processors yields millions of tiny compute units (about 128x a vector unit's throughput); the catch is that in-memory computing is painful to use — it needs data laid out in a special transposed format with operands lined up in the same column, it often works best when SPLIT with near-memory computing (compute next to the banks), and the fastest code hardcodes chip details that break portability; this paper (UCLA and UT Austin) builds Infinity Stream, whose key idea is a new program representation, the tensor dataflow graph, that places tensors on an abstract "lattice" grid (hiding the real bitlines and banks) and makes data alignment and reuse first-class, unifying in- and near-memory so a runtime can fuse them and JIT-compile the graph to whatever hardware it lands on — turning plain C into code that runs 2.6x faster with 75% less data traffic than state-of-the-art near-memory computing, and 5.1x over a high-end multicore.
The argument in one glance
-
Problem
The real cost is moving data, so compute where the data lives
In modern systems, moving data between memory and the CPU cores costs more time and energy than the arithmetic it feeds — and it only gets worse as core counts grow. Two "memory-centric" ideas attack this. NEAR-memory computing adds specialized compute hardware right next to the memory banks (say, next to each L3 cache bank), so results are produced without hauling data all the way to a core. IN-memory computing goes further and turns the memory ARRAYS THEMSELVES into processors: using bit-serial logic inside the SRAM bitlines, every column of the cache becomes a tiny arithmetic unit. Because modern CPUs carry enormous L3 caches (over 100MB), this yields MILLIONS of bit-processors working in parallel — on the order of 128 times the throughput of a wide vector unit. The potential is enormous.
-
What people did
Build in-memory and near-memory accelerators
A large body of work builds exactly these devices: bit-serial compute SRAMs that perform bitwise or arithmetic operations in place, and near-data engines that offload computation to sit beside the cache. The hardware works and the raw parallelism is real. The question is whether ordinary programs can actually USE it.
-
Why it fell short
In-memory computing is powerful but almost unusable, on three fronts
Three unsolved problems block adoption. First, TRANSPARENT ORCHESTRATION: bit-serial logic needs the data laid out in a special TRANSPOSED format — each value stored vertically down a bitline column — with the operands of every operation ALIGNED in the same column, plus careful tiling and reuse to limit data traffic. Doing this by hand is a nightmare. Second, FUSION: the best plan often SPLITS a computation between in-memory (which is great for regular, massively-parallel work) and near-memory (better for irregular access like A[B[i]] and for final reductions) — which needs a single execution model spanning both. Third, PORTABILITY: the fastest implementations hardcode low-level details (bitlines per array, SRAM array sizes, instruction formats) and runtime values (tensor dimensions), so they cannot run unchanged on a different chip or a future generation. Existing in-memory systems are either domain-specific or dump all of this on the programmer, and none solve all three.
-
This paper's idea
A tensor dataflow graph that unifies in-/near-memory and JIT-compiles anywhere
This paper (UCLA and UT Austin — Wang, Liu, Arora, John, Nowatzki) builds Infinity Stream, which makes in-memory computing programmer-friendly and portable. Its key observation: the parts of a program that benefit from in-memory computing have very simple, AFFINE (strided) access to multidimensional tensors — regular enough that a compiler can analyze them perfectly and work out the layout, tiling, and data movement automatically. So it makes parallel tensor access and relative memory alignment FIRST-CLASS in a new intermediate representation, the tensor dataflow graph (tDFG). Tensors live on an abstract global LATTICE SPACE (an N-dimensional grid that hides the real bitlines, arrays, banks, and network), and explicit Move and Broadcast nodes reify the alignment and reuse the hardware needs, so the compiler can optimize data movement — searched with equality-graph rewrites. The same tDFG expresses BOTH in-memory and near-memory computation, so a runtime can FUSE them, choosing the best mode per region. And it is a portable "virtual ISA," like a GPU's PTX: the compiler emits a hardware-independent tDFG, and a just-in-time compiler lowers it to the actual SRAM sizes and bit-serial commands at runtime. From plain C with minimal annotations, Infinity Stream runs 2.6x faster with 75% less data traffic than state-of-the-art near-memory computing, and 5.1x over a high-end multicore.
Why this problem is worth solving
For decades the answer to “make computing faster” was faster cores, but
the wall now is data movement: a modern chip spends most of its time and
energy shuttling operands to and from the cores, not computing. The two
ways to attack that are worth pinning down. A
both move computation
toward the data instead of the data toward the computation, but at
different depths. NEAR-memory computing places compute units NEXT TO the
memory (for example, beside each L3 cache bank), so a result is produced
close to where its inputs live and does not travel to a core — but the
data is still READ OUT of the SRAM arrays into that nearby unit, so
throughput is limited by how fast the arrays can be read. IN-memory
computing goes inside the array: it augments the SRAM so that each
bitline (each column of the memory) computes on the bits stored in it,
using BIT-SERIAL logic — processing one bit position per cycle, so an
n-bit add takes on the order of n cycles, but with every column doing it
at once. Because a large L3 cache has millions of columns, in-memory
computing offers enormous parallelism (roughly 128 times a vector unit)
at very high energy efficiency, at the cost of a peculiar execution
model
That peculiar execution model is exactly what has kept in-memory
computing on the shelf, and it is what Infinity Stream’s representation
is built to tame. A
Infinity Stream’s
program representation. Bit-serial in-memory computing has two hard
requirements: data must be TRANSPOSED (each value laid out vertically
down one bitline column, one bit per wordline) so the column can process
it bit by bit, and the operands of an operation must be ALIGNED in the
SAME column so the bitline can combine them. The tensor dataflow graph
(tDFG) makes these first-class. It places every tensor on a shared,
abstract GLOBAL LATTICE SPACE — an N-dimensional grid whose cells will be
mapped to physical bitlines at runtime, but which in the IR just
expresses WHERE each tensor sits relative to the others. Two tensors in
the same lattice cell are aligned and can be computed together; a MOVE
node shifts a tensor to bring operands into alignment; a BROADCAST node
spreads a small reused tensor across a dimension. Compute nodes act
element-wise on the overlap of their inputs with no ordering, exposing
the data parallelism, while the lattice positions capture the data
ALIGNMENT the hardware demands — so the compiler can reason about, and
minimize, the physical data movement
What people did before, and where it broke
- Domain-specific in-memory accelerators. Build bit-serial compute SRAMs tuned for one domain (machine learning, graph processing, databases). Fast for that domain, but not general and not programmable from ordinary code.
- Bit-parallel in-cache computing (Compute Cache). Support in-place bitwise operations in CPU caches, but only the less general bit-parallel layout and single-dimension operations, without a high-level compiler.
- Near-data programming with intrinsics or dataflow (PEI, Livia, near-stream computing). Make near-memory offload programmable via instructions or stream abstractions. Useful, but they do not manage the transposition or guarantee bitline alignment that IN-memory computing needs, so they cannot drive it.
- Hardcoded high-performance in-memory kernels. Exploit low-level microarchitecture details for speed. Fast on one chip, but fixing those details prevents portability across microarchitectures and array sizes.
- The gap. No system offered transparent orchestration, fused in-/near-memory execution, AND portability, all from a general-purpose language. That three-way gap is what Infinity Stream closes.
How Infinity Stream makes in-memory computing usable
Lift the program into a tensor dataflow graph, fuse the two modes, and JIT to the hardware
-
The substrate: compute in and near memory
Bit-serial in-SRAM computing turns each cache column into an arithmetic unit — millions of them, about 128x a vector unit — but needs data transposed and operands column-aligned. Near-memory computing sits beside the banks and suits irregular access and reductions. Using either from ordinary code is the challenge.
-
Lift the program into a tensor dataflow graph new
The parts that benefit from in-memory computing have simple, affine tensor access, so the compiler extracts a tensor dataflow graph from plain C. Tensors are positioned on an abstract global lattice grid that hides the physical bitlines, arrays, and banks, letting the compiler reason about layout and movement portably.
-
Reify alignment and reuse as Move and Broadcast nodes new
A Move node shifts a tensor along a lattice dimension to bring operands into the same bitline column (satisfying the hardware alignment requirement); a Broadcast node spreads a small reused tensor across a dimension. Making these explicit lets the compiler see and minimize the physical data movement rather than leaving it implicit.
-
Unify and fuse in- and near-memory execution new
The same tDFG expresses both paradigms, with dependencies crossing between them — an in-memory reduction can feed a near-memory final reduction, and a near-memory indirect stream can feed in-memory tensor ops. The runtime chooses per region: regular parallel work in-memory, irregular or reduction work near-memory.
-
Optimize by equality-graph rewriting new
Infinity Stream applies algebraic rewrites (associativity, commutativity, moving a Move or Broadcast past a compute, expanding or shrinking a tensor) to generate many equivalent tDFGs in an equality graph, then picks the one with the least data traffic and computation — for example merging two identical shifts, or reusing a shifted result instead of recomputing it.
-
JIT-lower to the actual hardware new
The optimized tDFG is a portable virtual ISA, like a GPU's PTX. The compiler emits a fat binary with tDFG configurations for common array sizes; at runtime, once tensor sizes and the SRAM array sizes are known, a JIT compiler chooses the transposed data layout and tiling and lowers the graph into concrete bit-serial commands — so one binary runs on many microarchitectures.
The idea
Fuse in-memory and near-memory instead of choosing — pick the mode per region. In-memory computing is unbeatable at regular, massively-parallel work but weak at irregular access and at the final stages of a reduction (where its parallelism halves each round); near-memory computing is the reverse. Infinity Stream refuses to pick a side: the single tensor dataflow graph expresses both, with dependencies crossing freely, so the runtime runs each region wherever it is fastest — regular tensor math in the bitlines, indirect access and final reductions beside the banks. This reject-the-false-either/or, hybridize-and-assign-per-subproblem discipline is one the site rewards again and again — the memory-allocation page fusing a heuristic with a solver, the accelerator-mapping page combining annealing with an ILP, the accelerator-companion page pairing a general ALU with specialized memory logic — and here it is worth a further 24% beyond in-memory alone, because the two modes cover each other’s weak spots. The PointNet++ study makes it concrete: some layers run best in the core, some near-L3, some in-L3, and only fusing all three wins.
Reify the hardware’s alignment requirement into the representation, so a compiler can optimize it. The reason in-memory computing was unprogrammable is that its defining constraint — operands must sit in the same bitline column — lived below the abstraction, as a chore the programmer had to satisfy by hand. Infinity Stream lifts it into the IR: the global lattice space makes relative position first-class, and Move and Broadcast nodes make the alignment and reuse EXPLICIT operations the compiler can see, count, and rewrite. Once data movement is a first-class term, minimizing it becomes an optimization problem rather than a manual one. This invent-the-right-representation, make-the-implicit-explicit move is the same one the site’s memory-timing-defense page makes when it turns a request pattern into a graph it can shape, and the branch-predictor page makes when it turns hidden state into an addressable object — the payoff in every case is that a thing you could not previously reason about becomes something you can optimize.
Compile to a portable virtual ISA and defer the hardware specifics to a JIT. Hardcoding bitline counts and array sizes buys speed and destroys portability; Infinity Stream keeps both by splitting compilation in two, exactly as GPUs do with PTX. The static compiler produces a hardware-independent tDFG — the difficult program analysis done once — and a runtime JIT, knowing the actual tensor and SRAM sizes, chooses the transposed layout and tiling and emits the bit-serial commands. This is the site’s recurring plan-what-you-can-offline, decide-the-rest-at-runtime split on its static-versus-dynamic axis: the same discipline the runtime graph-alignment and elastic-scheduling pages use when they defer the decisions that depend on data only known at runtime. Portability is not a tax here; it is what lets one binary exploit whatever memory arrays it lands on.
From plain C to millions of bitline processors
Making the bitlines usable
Compiling ordinary tensor code to fused in-/near-memory execution. Numbers from the paper's 64-core evaluation.
Step 1 of 5
Why in-memory is powerful but unusable
Bit-serial in-SRAM computing turns each cache column into an arithmetic unit — about 128 times a vector unit's throughput on a large L3 — but it demands data transposed into a vertical layout, operands aligned in the same column, and careful tiling and reuse. Doing this by hand, and hardcoding chip details for speed, is what made it unusable and unportable.
a 64-core system with a 144MB L3 exposes roughly 4 million bitlines, each a tiny processor an n-bit operation takes about n cycles (bit-serial), but every column runs in parallel
Why it works
- Data movement dominates cost, and a large L3 has millions of bitlines → computing in the SRAM arrays offers about 128x a vector unit’s throughput at high efficiency → the ceiling is high enough to be worth a hard programming model.
- The workloads that suit in-memory computing have affine tensor access → their layout, tiling, and movement are fully analyzable at compile time → the orchestration can be automated instead of left to the programmer.
- Bit-serial computing requires operands aligned in the same column → making position and alignment first-class in the tDFG (lattice space, Move, Broadcast) → data movement becomes an explicit term the compiler can minimize with rewrites.
- In-memory excels at regular parallel work and near-memory at irregular access and final reductions → a unified graph lets the runtime run each region in its best mode → fusion adds 24% over in-memory alone and covers each mode’s weaknesses.
- Hardcoding array sizes and bitline counts breaks portability → emitting a hardware-independent tDFG and JIT-lowering it at runtime → one binary exploits whatever memory arrays it runs on, with about 11% JIT overhead.
What it costs (trade-offs and scope)
- One in-memory job at a time. Only a single thread can reserve the L3 for in-memory computing (it locks the L3), so concurrent in-memory sharing across programs is not supported in this implementation.
- Data must already fit L3, and no register spilling. The input is assumed pre-tiled to fit the L3; otherwise in-memory computing is disabled. All kernels must fit in the available SRAM “registers” — register spilling is left to future work.
- The JIT is on the critical path. Lowering costs about 11% of runtime and can exceed 50% when lowered commands cannot be reused or for the boundary tiles of high-dimensional tensors; the no-JIT configuration shows a further 19% is left on the table.
- It targets dense, affine tensor code. The analyzable structure is exactly what makes automatic orchestration possible, so irregular, control-heavy code (beyond the near-memory indirect streams) is out of scope — control flow and predication are future work.
- Real silicon cost, and a dataflow the programmer still shapes. In-memory parallelism halves each reduction round, so programmers should prefer outer-product formulations and still tile for L3; and the design modifies the SRAM arrays (sense amps, decoders, compute logic) for a 6.52% whole-chip area overhead.
The evaluation
An LLVM-based compiler and a gem5-20 cycle-level simulator with partial AVX-512, on a 2 GHz 64-core system with a 144MB shared L3 (about 4 million bitlines) and DDR4, on 13 dense fp32 OpenMP workloads (stencils, DWT, Gaussian elimination, convolutions, matrix multiply, k-means, an MLP gather) plus a PointNet++ case study, versus a high-performance multicore (Base) and a near-stream near-memory baseline (Near-L3).
Ways to compute in the memory hierarchy
| Property | Multicore (core-centric) | Near-memory (Near-L3) | Infinity Stream (fused) |
|---|---|---|---|
| Where compute happens | |||
| Effective parallelism | |||
| Data orchestration | |||
| In-/near-memory fusion | |||
| Speed and traffic |
Check yourself
Check your understanding
1. In-memory computing offers millions of bitline processors, yet it had been almost unusable from ordinary code. Predict the core reason, and how Infinity Stream addresses it.
2. Infinity Stream unifies in-memory and near-memory computing in one representation rather than committing to either. Predict why fusing them beats choosing one.
3. Fast in-memory kernels normally hardcode microarchitecture details like SRAM array size, which locks them to one chip. Predict how Infinity Stream keeps performance without sacrificing portability.
4. Infinity Stream claims the workloads that benefit from in-memory computing can be orchestrated automatically. Predict what property of those workloads makes automatic orchestration possible.
Sources
- Infinity Stream: Portable and Programmer-Friendly In-/Near-Memory Fusion (author PDF, UCLA PolyArch) — ASPLOS 2023 (full text read).
- ACM DL: Infinity Stream (DOI 10.1145/3582016.3582032) — published version.