Accelerating Retrieval-Augmented Generation

Derrick Quinn, Mohammad Nouri, Neel Patel, John Salihu, Alireza Salemi, Sukhan Lee and 2 others

RAG grounds LLMs in retrieved documents, and this paper's profiling surprise is that EXACT nearest-neighbor search — up to 97% of inference time — beats approximate search end-to-end, because approximation's accuracy collapse forces so many extra documents into the prompt that generation eats all the retrieval savings; IKS therefore accelerates exact search where it belongs — near memory — as a CXL.cache-coherent expander whose eight near-memory accelerators stream a 512 GB vector database at full LPDDR5X bandwidth — 13.4–27.9× faster search, 1.7–26.3× faster inference, with DRAM that doubles as pooled server memory when idle.

The argument in one glance

Problem

Grounding LLMs means searching, and searching is the bottleneck

LLMs hallucinate, forget rare facts, and go stale — their knowledge is frozen into parameters at training time. Retrieval-Augmented Generation fixes this by fetching relevant documents from an external corpus (encoded as embedding vectors in a vector database) and handing them to the model alongside the query. But finding the truly nearest vectors exactly means scanning the whole database: on a Xeon with 8 DDR5 channels feeding an H100, exact search over a 50 GB corpus consumes up to 97% of end-to-end inference time.
What people did

Approximate the search, or brute-force it on GPUs

The standard answer is Approximate Nearest Neighbor Search — graph indexes like HNSW that visit a fraction of the vectors — tuned via parameters trading recall for speed. Alternatively, throw the brute-force scan at GPU memory bandwidth, or build one of the emerging bespoke ANNS accelerator ASICs.
Why it fell short

Measured end-to-end, the approximations backfire

The paper's profiling delivers the counterintuitive verdict. ANNS recall losses translate into GENERATION accuracy collapse — up to 53% worse answers at equal document count, worst for the biggest un-fine-tuned models. Compensating by retrieving more documents inflates the prompt (each document ≈127 tokens, 131 kB of KV cache for Llama-8B) until generation swallows the retrieval savings: a higher-quality-but-slower ANNS with 16 documents beats a faster ANNS with 128 documents on BOTH accuracy and throughput. GPUs are the wrong tool too: ENNS runs at ~0.5 flops/byte while an H100 provisions ~296 — a 600× mismatch — and HBM capacity is far too expensive to park cold embeddings in. And ANNS accelerators are complex and task-specific, chasing a moving algorithmic target.
This paper's idea

Accelerate the simple thing, next to the memory it streams

Exact search is embarrassingly simple — dot products with zero reuse, regular access, trivial parallelism, a top-K at the end — which makes it the PERFECT near-memory workload. IKS (Intelligent Knowledge Store) is a CXL type-2 memory expander: eight LPDDR5X packages, each fronted by a Near-Memory Accelerator with 64 processing engines that evaluate 64 queries in parallel against the streaming shard, coordinated through a cache-coherent CXL.cache doorbell interface (no interrupts, no kernel, 1.6× better than MMIO-style offload). Result: 13.4–27.9× faster exact search over 512 GB, 1.7–26.3× faster end-to-end RAG inference — and when idle, IKS is just disaggregated DRAM, rescuing the server's most expensive component from stranding.

Why this problem is worth solving

retrieval-augmented generation — augmenting an LLM’s frozen “parametric” knowledge with documents fetched at query time from an external corpus. A retrieval model encodes the query, searches a vector database for the K most similar document embeddings, and the winners are fed to the generative model with the prompt. The backbone of grounded assistants, search copilots, and enterprise QA

has become the default answer to LLM hallucination and staleness — which quietly makes vector search core datacenter infrastructure. The retrieval side uses

bi-encoder neural networks that map queries and documents into one high-dimensional embedding space where semantic similarity is inner-product similarity; the corpus becomes a vector database (tens to hundreds of gigabytes of embeddings), and retrieval becomes K-nearest-neighbor search

, and the systems question is where the nearest-neighbor cycles come from.

What elevates this paper above “accelerate the bottleneck” is its first half: a careful profiling study that overturns the field’s default assumption. The received wisdom says exact search cannot scale, so production systems reach for

approximate nearest-neighbor search — indexes (graph-based HNSW, quantization-based IVF-PQ) that examine only a fraction of vectors, trading recall for speed via tuning knobs like HNSW’s efSearch. The workhorse of every production vector database

. The paper measures what recall loss does to end-to-end answer quality — not retrieval metrics — and finds the trade is far worse than assumed: the generative model is only as good as its evidence, and feeding it more lower-quality evidence costs generation time faster than it restores accuracy. The conclusion — exact search is not a luxury but the accuracy-per-second optimum, and its very simplicity makes it the right thing to accelerate — reframes the whole RAG-acceleration agenda around a workload that is almost comically hardware-friendly: streaming dot products at memory bandwidth.

What people did before, and where it broke

ANNS everywhere. HNSW and friends are the industry standard. The paper tunes them honestly (M=32, efConstruction=128, efSearch swept; IVF-PQ tried too) and finds: at K=1, generation accuracy drops 23–53% versus exact retrieval across FiDT5, Llama-8B, and Llama-70B; at K=16, still 14–45%. Big K compensation fails economically — every document is ~127 tokens whose KV cache alone is 131 kB/token ×127 for Llama-8B — so ANNS-2 with 16 documents beats ANNS-1 with 128 on accuracy AND throughput (by 3% and 128%). Notably, the accuracy damage is worst for the largest, non-fine-tuned models.
GPU brute force. ENNS at batch 1–16 runs at ~0.5 flops/byte; an H100 provisions 296 (592 for int8). The GPU’s die idles while its HBM streams — and parking hundreds of gigabytes of cold embeddings in 80 GB HBM parts is a capacity-economics absurdity (8 GPUs to hold 512 GB).
CPU brute force. The roofline shows even this memory-bound kernel failing to saturate the CPU’s 256 GB/s — cache hierarchy overheads and limited cores get in the way — while consuming the host entirely.
Bespoke ANNS accelerators. Emerging ASICs chase graph-traversal ANNS — complex, task-specific designs coupled to fast-moving algorithms. ENNS needs none of that; its simplicity is an asset the accelerator landscape was ignoring.

How IKS works

The Intelligent Knowledge Store

A CXL type-2 memory expander new

IKS hangs off the CPU as a PCIe/CXL card holding 512 GB of LPDDR5X across eight packages. Type-2 means it speaks BOTH CXL.mem (host maps its DRAM as ordinary far memory) and CXL.cache (the device can coherently cache host memory) — one shared address space, no partitioned-accelerator-memory rewrites of the vector database, and the DRAM disaggregates for other applications when search is idle.
↓ →
Scale-out near-memory accelerators new

Each LPDDR5X package is fronted by its own Near-Memory Accelerator chip implementing the memory controllers plus accelerator logic — eight small chips rather than one big one, keeping each die at the minimum perimeter its 8 memory channels demand (PHY shoreline sets area), improving yield, and scaling bandwidth with capacity. Each NMA uplinks to the CXL controller via ×2 PCIe 5.0 — deliberately oversubscribed, harmless in both modes.
↓ →
64 processing engines per NMA, broadcast-fed new

Each NMA holds 64 processing engines (query scratchpad + dot-product unit + top-K unit + output scratchpad) so up to 64 queries evaluate in parallel. A fixed broadcast network streams each embedding element from DRAM to ALL active engines at once — batched queries get data reuse for free. Each dot-product unit packs 68 FP16 MAC units at 1 GHz, exactly saturating the package's bandwidth; embeddings are laid out dimension-major across 68 vectors so address generation is trivial.
↓ →
Offload via cache coherence, not drivers new

The host writes query vectors and parameters into cacheable shared context buffers, rings a doorbell cache line, and parks the core with umwait(). NMAs poll the doorbell (coherence traffic moves it), stream their shards, write partial top-32 lists plus a completion doorbell, and the host wakes. No interrupts, no kernel crossings, no MMIO: measured 1.6× higher offload throughput than a non-temporal-write (CXL.io-style) datapath.
↓ →
Software finishes the job new

Coherence between host and NMAs is SOFTWARE-managed — safe because ENNS has no data reuse — and address translation is coarse-grained thanks to the regular access pattern. The host CPU merges eight partial top-32 lists (negligible cost), hands the top-K documents to the generative model, and the vector database (Faiss) keeps handling everything else: updates, index management, allocation into IKS DRAM via a modified mapping scheme.
↓ →
Scale by adding units

Multiple IKS cards shard the corpus: retrieval over a 2 TB corpus on four units takes just 100 µs longer than 512 GB on one — near-perfect weak scaling, because search is embarrassingly parallel and the interface overhead is tiny.

From Cornell, UMass Amherst, and Samsung (ASPLOS 2025). Evaluated by emulation: a performance model (1.1 TB/s aggregate, 69.9 TFLOPs) composed with real-system measurements on Sapphire Rapids hosts + H100 generation, against CPU (AVX-512 and AMX) and GPU baselines.

The idea

Measure the pipeline, not the stage. The paper’s most exportable result needed no hardware at all: retrieval quality and generation cost are coupled through K, so optimizing retrieval speed in isolation optimizes the wrong thing. Lower recall demands more documents for the same answer quality; more documents linearly inflate transformer input — attention compute, KV cache, time-to-first-token — so the “fast” retriever loses end-to-end. The clean statement is the paper’s ANNS-2@16 vs ANNS-1@128 comparison (higher accuracy and 2.3× the throughput for the slower-but-better retriever), and the corollary: since high-quality retrieval dominates runtime whether exact or approximate, you might as well accelerate exact — which buys the best accuracy and the simplest hardware target simultaneously.

Simplicity is the acceleration strategy. ENNS has the four properties near-memory architects dream about: zero data reuse (caches useless — software-managed coherence is safe), pure dot-products plus top-K (tiny logic), perfectly regular access (coarse address translation, trivial address generation), and embarrassing parallelism (shard the corpus, merge tiny top-K lists). IKS’s design reads as the workload’s properties transcribed into silicon: MAC counts sized to exactly saturate LPDDR5X bandwidth (compute-bandwidth balance, no waste), broadcast networks converting query batches into free reuse, eight small dies instead of one big one because memory PHYs — not logic — set the floorplan. Contrast the bespoke-ANNS-ASIC path: complex, algorithm-coupled, and obsoleted by every index innovation. IKS bets on the operation that cannot change: scanning all the vectors.

CXL.cache is the interface that makes offload worth it. Fine-grained offload usually dies on its overheads — drivers, interrupts, copies, partitioned address spaces. IKS’s interface is a producer-consumer doorbell in coherent shared memory: context buffers are ordinary cacheable lines, notification is a cache-line bounce, the waiting core sleeps in userspace via umwait(). That’s 1.6× better than an MMIO-style path, requires no vector-database rewrite (one address space — the CPU still owns updates and management), and the same coherence machinery makes IKS a dual-use device: a memory expander that happens to search. The stranded-DRAM argument closes the economics: memory is the priciest component in modern servers, and an accelerator whose DRAM pools into the general capacity when idle never sits as dead capital — the anti-GPU argument in one design choice.

One query through IKS-accelerated RAG

A Natural Questions query, from text to grounded answer

Embedding = 768-d BERT vector (FP16). NMA = near-memory accelerator (one per LPDDR5X package, 8 total). PE = processing engine (64 per NMA; one query each). K = documents passed to the generative model. Numbers from the paper's 512 GB-corpus evaluation with Sapphire Rapids host + H100 generation.

Step 1 of 7

Encode the query

A BERT-based query encoder turns the question into an embedding vector — the same space where the entire corpus (Wikipedia-scale, chunked into 100-word documents) was embedded offline and laid out dimension-major across IKS's LPDDR5X packages.

dense retrieval: similarity = inner product in embedding space
corpus sharded 8 ways, one shard per NMA

Why it works

Generation cost scales with retrieved-document count, and recall losses must be repaid in K → retrieval quality and generation time are one coupled budget → exact search, though slower in isolation, minimizes end-to-end time at target accuracy — the profiling result that redirects the whole acceleration target.
ENNS has zero reuse, regular access, and trivial parallelism → caches, TLB precision, and hardware coherence add nothing — software-managed coherence and coarse translation suffice → the accelerator reduces to MACs + top-K sized to memory bandwidth, cheap enough to co-locate with every DRAM package.
Memory PHYs, not logic, set die floorplans → eight small NMAs (one per package) instead of one monolith → bandwidth scales with capacity, yield improves, and the broadcast network turns query batching into free data reuse.
CXL.cache makes host-device communication a cache-line producer-consumer protocol → userspace offload with no interrupts, kernel crossings, or address-space partition (1.6× over MMIO-style) → offload overhead stays microseconds, preserving near-perfect weak scaling to 2 TB.
Type-2 CXL means the accelerator IS a memory expander → idle capacity pools into the server’s general DRAM via CXL.mem → the most expensive server component is never stranded, fixing the economics that damn GPU-resident vector search.

What it costs

Emulated, not fabricated. IKS numbers come from a performance model composed with real-system measurements — standard for architecture papers, but no silicon exists, and the CXL.cache interface rides on an emerging ecosystem (type-2 devices remain rare).
Brute force scales linearly. Exact search reads the whole corpus every query; IKS makes the constant excellent, but a 10× bigger corpus is 10× more work — sublinear ANNS asymptotics win eventually at some corpus size and query rate; the paper’s claim is about today’s RAG scales (tens of GB to 2 TB).
The accuracy verdict is workload-scoped. QA on Natural Questions with specific models and honest-but-particular HNSW tunings; ANNS research continues to close recall gaps, and workloads more tolerant of retrieval noise shift the crossover.
Offload granularity fits batch retrieval. 64 queries per NMA and top-32 caps are design points; final aggregation costs grow with unit count, and multi-node IKS deployments are unevaluated.
LPDDR5X trades bandwidth-per-package for cost and power — the right choice for streaming scans, but capacity-optimized rather than latency-optimized; IKS-resident data serves other applications at far-memory, not local-DRAM, speeds.

The evaluation

Two halves. The profiling study: FiDT5, Llama-8B, and Llama-70B on Natural Questions, ENNS vs tuned HNSW variants, K from 1 to 128, batch 1 (latency-critical) and 16 (throughput), corpora 50–512 GB — establishing the accuracy-throughput frontier and the 97%-of-inference-time retrieval share. The IKS evaluation: emulated IKS against Sapphire Rapids (AVX-512 and AMX), with generation on an H100, plus scaling to four units and 2 TB.

Where RAG retrieval can run

Property	ANNS (tuned HNSW) on CPU	ENNS on CPU / GPU
Answer quality (generation accuracy)		The gold standard — perfect recall by definition
End-to-end economics of the trade
Hardware fit		CPU cannot saturate its own DRAM bandwidth on this kernel; GPU die idles at 600× compute overprovision
Capacity & scaling	Index fits in host DRAM; graphs add memory overhead	Host DRAM (capacity-capped) or HBM (cost-absurd)
Measured performance	Faster search, slower answers: negated end-to-end	Baseline (CPU AVX-512/AMX)

Profiling: Natural Questions (KILT), BERT dense retrieval, Faiss with Intel MKL BLAS backend, HNSW honestly tuned (M=32, efConstruction=128, efSearch 2048/10000); FiDT5 (Fusion-in-Decoder), Llama-8B, Llama-70B generation on H100. Host: dual Xeon 4416+ (8×DDR5-4000, 256 GB/s). IKS: emulated — 8 NMAs × 64 PEs × 68 FP16 MACs @1 GHz, 512 GB LPDDR5X, 1.1 TB/s aggregate, CXL.cache doorbell interface measured at 1.6× over MMIO-style offload.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

IKS is an emulated design riding an emerging CXL type-2 ecosystem, and its headline argument — exact beats approximate end-to-end — is an empirical claim about particular workloads (QA datasets, specific models, honest but specific HNSW tunings), not a theorem: ANNS quality keeps improving, and brute force’s linear scaling must eventually lose to sublinear indexes as corpora grow beyond the evaluated 2 TB. The accuracy collapse numbers are most dramatic for models prompted with plaintext documents rather than fine-tuned retrieval readers. None of this blunts the paper’s durable lessons: profile pipelines end-to-end before accelerating a stage, and when a stage IS the target, prefer the simplest formulation of it that hardware can saturate.

Background

The memory-compute continuum gains a rung

IKS slots neatly into this site’s memory-and-LLM threads. On the where-compute-meets-DRAM continuum, it sits between CENT’s in-bank PIM (compute inside the DRAM die) and conventional hosts: near-package accelerators behind a CXL.cache interface — while Melody’s characterization work maps the very CXL fabric IKS builds on. The intellectual rhyme, though, is with the site’s proxy-objective lessons: TLP showed schedule cost beats program cost as a prediction target, Orion showed bootstrap count is the wrong minimization objective, GAIA showed carbon savings must be priced per unit of everything else — and this paper shows retrieval SPEED is the wrong objective for retrieval, because K couples it to generation. The recurring discipline: find the true end-to-end cost surface before optimizing a stage of it. IKS is what falls out when that discipline points at RAG — an accelerator for the operation everyone assumed was the naive baseline.

If you remember only three things

1. The profiling half overturns RAG's default assumption: approximate retrieval's recall losses collapse generation accuracy (up to 53% at equal K, worst for large un-fine-tuned models), and repaying them with more documents inflates generation cost (~127 tokens and ~16.6 MB of KV cache per document) until the fast retriever loses on BOTH axes — a higher-quality ANNS with 16 documents beats a faster one with 128 — so exact search, at up to 97% of inference time, is simultaneously the bottleneck and the accuracy-per-second optimum.
2. IKS accelerates that bottleneck by transcribing ENNS's simplicity into near-memory silicon: a CXL type-2 expander with eight LPDDR5X packages each fronted by a small NMA whose 64 broadcast-fed processing engines (68 FP16 MACs each, sized to exactly saturate package bandwidth) stream the shard past up to 64 queries at once, coordinated by a cache-coherent doorbell interface (userspace, no interrupts, 1.6× over MMIO-style) with software-managed coherence — safe precisely because the workload has zero reuse.
3. The measured result: 13.4–27.9× faster exact search over 512 GB and 1.7–26.3× faster end-to-end RAG inference with near-perfect weak scaling to 2 TB across four cards — while type-2 duality lets idle IKS DRAM pool into the server's general memory, rescuing the costliest server component from stranding and fixing the capacity economics that make GPU-resident vector search absurd.

Check yourself

Check your understanding

1. Approximate search (ANNS) is faster than exact search per query — that's its entire purpose. Predict why the paper finds ANNS-based RAG LOSES end-to-end, on throughput as well as accuracy.
2. ENNS does strictly more work than ANNS — it scans every vector. Why does the paper argue it is nonetheless the RIGHT target for hardware acceleration?
3. IKS is a CXL TYPE-2 device — it implements both CXL.mem and CXL.cache — rather than a DIMM-based PIM module or a DMA-style PCIe accelerator. What do the two protocols each buy?
4. Why does IKS use EIGHT small NMA chips — one per LPDDR5X package — instead of one larger accelerator die, and what makes its weak scaling (2 TB on 4 cards ≈ 512 GB on 1 card + 100 µs) so clean?

Sources

arXiv preprint 2412.15246 — full text used for this explainer
ACM DL: Accelerating Retrieval-Augmented Generation — ASPLOS 2025