PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference

Generating LLM tokens one at a time leaves a GPU's compute 79% idle while its memory bandwidth gasps — CENT deletes the GPU entirely, spreading Llama-scale models across 32 CXL-networked memory devices whose DRAM banks multiply-accumulate in place (99% of transformer arithmetic) with RISC-V-plus-accelerator sidecars for the softmax-shaped remainder, delivering 2.3× the throughput of four A100s at similar power, 2.9× less energy, and 5.2× more tokens per dollar.

Why this problem is worth solving

The economics of LLM serving hinge on a mismatch between what inference needs and what accelerators sell.

The classic remedy, batching, is blocked by a uniquely LLM-shaped obstacle: the

What people did before, and where it broke

• The density tax fights the capacity hunger. Near-bank units cost 25–75% of memory density depending on technology. LLMs are the workload that can least afford it: parameters plus KV caches demand hundreds of gigabytes. A PIM-only box built from a few devices simply cannot hold the model.
• Hybrids keep the cost problem in the rack. AttAcc and NeuPIMs (both covered elsewhere on this site) pair 8 A100-class GPUs with 8 PIM stacks — GPUs for the compute-heavy prefill, PIM for decode. But prefill is ~2% of end-to-end GPU inference time, so the hybrid retains the single most expensive component to accelerate the smallest slice; their 3-year TCO comes out 3.5× and 2.6× above CENT’s.
• Near-memory compute gives back the bandwidth. Processing-near-memory (PNM) — CMOS compute in the controller or buffer device, as in Samsung’s CXL-PNM for GPT — preserves density and area-efficiency but reads memory at ordinary interface speeds, forfeiting exactly the internal-bandwidth advantage that made in-memory computing attractive.

The missing piece was capacity scaling without GPUs — and that is what

How CENT works

The idea

If decoding is a memory workload, build the server out of memory. CENT’s central bet is architectural subtraction. Prior systems treated PIM as an attachment to a GPU; CENT observes that the piece of inference the GPU is uniquely good at — high-intensity prefill GEMMs — is ~2% of end-to-end time, and its own PNM+PIM compute is still ~49% of the GPU baseline’s TFLOPS. So it removes the GPU, its 300 W board power, and its $10,000 price from the design entirely, and lets the density-taxed PIM chips multiply until capacity fits the model — which is exactly what CXL memory expansion is for. The cost structure inverts: memory (priced at 10× commodity DRAM for PIM) becomes the system, and the “accelerator” budget drops to a 19 mm² controller die per device.

Split the transformer by arithmetic frequency, not by layer. The enabling measurement: multiply-accumulates are over 99% of a transformer block’s arithmetic. So CENT provisions lavishly for MACs where bandwidth is free (beside every bank) and handles everything else — exponentials for softmax, roots for RMSNorm, SiLU — in a thin CMOS layer shared per device. This is the anti-UPMEM position: a general-purpose near-bank core would pay density and throughput for generality that 1% of operations need. The RISC-V cores are the escape hatch for model evolution: new activation functions become firmware, not silicon.

Cheap links are enough when the tensors that travel are small. Distributing a model across 32 devices sounds like it demands NVLink, but decoding’s inter-device traffic is embedding vectors — 16 KB per block-to-block hop for Llama2-70B — not weights or caches, which never move. That is why CXL’s modest bandwidth (with ~8× lower latency than RDMA and switching to thousands of nodes) suffices, and why pipeline parallelism doubles as the batching mechanism: with 80 blocks spread over 27 devices, 80 pipeline stages hold 80 in-flight queries, recovering throughput that GPU-style batching gets from GEMM fusion — without the KV-capacity squeeze, because capacity scales with device count.

One token through CENT

inter-device traffic per hop: one embedding vector (16 KB)
weights + KV cache: resident, never moved

Why it works

• Decoding’s arithmetic is one pass over resident data → moving compute to the banks converts the bottleneck resource from 8 TB/s of external bandwidth to 512 TB/s of internal bandwidth → the memory-bound stage speeds up 2.5× while the compute-bound stage (prefill, 2% of time) barely matters to the total.
• MACs are >99% of transformer arithmetic → specializing near-bank silicon to MACs and exiling rare ops to shared CMOS sidecars minimizes the density tax per unit of useful throughput → more model fits per device than any general-purpose near-bank design, and RISC-V firmware absorbs model evolution.
• Weights and KV caches never leave their banks; only 16 KB embedding vectors travel → interconnect bandwidth demand is tiny → cheap, switchable CXL suffices where NVLink would be overprovisioned, and capacity scales by adding $371 devices, not$10,000 GPUs.
• Pipeline stages hold independent queries → batch size equals pipeline depth, decoupled from any single device’s memory → CENT gets GPU-batching’s throughput benefit without the KV-cache capacity squeeze, which is precisely what collapses GPU batches at long context (128 → 8 from 4K to 32K).
• No GPU in the bill of materials → hardware cost drops 2.8× ($14,873 vs$42,128) and power ~8× per device-vs-GPU → the throughput and efficiency gains compound into 5.2× tokens per dollar.

What it costs

• Prefill is genuinely slower. GPUs beat CENT 2.5× on the compute-intensive prefill stage (CENT has ~49% of their TFLOPS) — CENT’s latency for the prefill portion is 1.4× worse. For workloads dominated by long-prompt understanding rather than generation, the paper itself suggests disaggregation: GPUs for prefill, CENT for decode.
• Simulated, not fabricated. Results come from a Ramulator2-based simulator with synthesized controller RTL, GDDR6-PIM timings from Samsung/SK Hynix parts, and analytical CXL models — no rack exists. Cost claims lean on estimates (PIM at 10× DRAM price, A100 at a conservatively halved $10K, NRE amortized over a projected 3M-unit volume).
• The density tax is paid, just distributed. 32 devices carry 512 GB where the same GDDR6 without PIM would hold more; CXL expansion hides but does not repeal the capacity loss.
• Block granularity quantizes scaling. Throughput plateaus when transformer blocks don’t divide evenly across devices (Llama2-70B’s 80 blocks map cleanly to 40 devices but not 44) — splitting one block across devices costs more communication than it gains.
• BF16 only, inference only. Near-bank PUs do BF16 multiply-accumulate; training, and the GEMM-heavy workloads GPUs excel at, are out of scope. Grouped-query attention — now standard — narrows CENT’s edge (1.2× on 70B vs 2.3× geomean).
• A new software stack. Custom ISA, instruction traces per model, host-driven orchestration — none of the CUDA ecosystem applies.

The evaluation

Llama2 7B/13B/70B, each query 512 prefill + 3584 decode tokens, against 4×A100 80 GB running vLLM at its saturating batch of 128 — system sizes chosen for similar average power. Baselines beyond GPUs: Samsung’s CXL-PNM (near-memory, no in-bank compute) and the GPU-PIM hybrids AttAcc and NeuPIMs on GPT3-175B. Performance comes from a cycle-level Ramulator2 model plus functional verification; power from an activity-based model (measured GDDR6 currents, synthesized RTL); costs from a 3-year TCO model covering hardware, power, and NRE.

Check yourself

Sources

• arXiv preprint 2502.07578 (v3) — full text used for this explainer
• ACM DL: PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference — ASPLOS 2025
• Open-source CENT simulator