A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs

Hongsun Jang, Jaeyong Song, Changmin Shin, Si Ung Noh, Jaewon Jung, Jisung Park and 1 other

Running a large language model on very long inputs in big batches is limited not by computation but by moving data — the key-value cache (the per-token attention memory) grows with batch size and context length until it must spill from GPU memory to host memory and SSD, and shuttling it back over the interconnect dominates the run; this paper (Seoul National University — Hongsun Jang and colleagues) presents HILOS, a system that puts the memory-heavy attention computation into off-the-shelf near-storage accelerators (an FPGA on the SSD, next to the flash where the cache lives) so only small results cross the interconnect, and adds cooperative caching, delayed writeback, and a memory-efficient attention accelerator — reaching up to 7.86 times the throughput of conventional SSD offloading while using up to 85 percent less energy.

The argument in one glance

Problem

Running a long-context model in big batches is bottlenecked by moving data, not computing

A large language model, or LLM, generates text one token at a time, and to do so efficiently it keeps a key-value cache: for every token processed, it stores that token's key and value vectors so later tokens can attend to it without recomputation. In offline inference — batch jobs where latency does not matter, such as processing a huge dataset — you want to run many requests together in a large batch to make good use of the hardware. But the decoding step is memory-bound, meaning the expensive graphics processing unit, or GPU, spends most of its time waiting for data rather than computing. Worse, the key-value cache grows with both the batch size and the context length (how long each input is), so with long inputs and large batches it becomes far too big to fit in GPU memory. It has to be pushed out to host memory and then to solid-state drive (SSD) storage. Now, on every decoding step, this enormous cache must be read back across the system interconnect to the GPU to compute attention. That transfer, not the arithmetic, dominates the run: key-value cache movement can account for more than 60 percent of inference time.
What people did

Extend the GPU memory hierarchy down to host memory and SSD

The established answer is offloading-based batched inference, used by systems like FlexGen and DeepSpeed. It extends the GPU's memory hierarchy with host memory, spilling over to SSD storage when host memory runs out. Batching many requests lets the model weights be reused across them, raising throughput without buying enough GPUs to hold everything in fast memory, which would be prohibitively expensive.
Why it fell short

Offloading only moves the data further away, so the transfer cost grows

Offloading makes long-context, large-batch inference possible on modest hardware, but it does not remove the fundamental cost — it relocates the huge key-value cache to storage that is even further from the GPU, so the cache must travel a longer path back on every step. As batch size and context length grow, the cache grows, and the input and output transfer overhead grows with it, until moving the cache across the interconnect dominates everything else. For very long contexts, even keeping the cache in host memory can fail, forcing it onto SSD where the transfer cost is highest. The core problem is that the data the GPU needs for attention lives far away and must be dragged to the compute, over and over. What is missing is a way to avoid moving the key-value cache at all — to do the memory-intensive attention work where the cache already is.
This paper's idea

Compute attention inside the storage device, next to the cache

HILOS is a framework for high-throughput, cost-effective offline LLM inference built on near-storage processing (NSP): computing directly inside the storage device rather than shipping data to the GPU. Its core idea is attention near storage. Instead of reading the giant key-value cache off the SSD and across the interconnect to the GPU, HILOS runs the memory-intensive attention operations on a small accelerator that sits on each storage device — an FPGA (a chip you configure into custom hardware) built into an off-the-shelf smart SSD, right next to the flash memory holding the cache. Only the small computed results travel back to the host, so interconnect traffic collapses; because attention traffic scales with context length, this benefit grows linearly the longer the context. It also gives aggregate memory bandwidth that scales with the number of storage devices. On top of this, HILOS adds three optimizations: cooperative caching, which splits the attention work between the host and the storage devices to use the host resources freed up by offloading; delayed key-value cache writeback, which hides storage write latency and reduces wasteful repeated writes when new tokens are added; and a memory-efficient attention accelerator that transposes data on the fly so it sustains high throughput without redundant storage writes. HILOS is the first NSP-offloading system on off-the-shelf products to support contexts beyond 100,000 tokens, and it reaches up to 7.86 times the throughput of conventional SSD offloading while using up to 85 percent less energy.

Why this problem is worth solving

Serving large language models is increasingly a memory problem, not a compute problem, and that mismatch wastes the most expensive hardware.

A large language model produces text token by token. During this decoding phase, generating each new token requires attending to every previous token, and to avoid recomputing their representations the model stores a key vector and a value vector for each one; together these form the key-value cache. This cache is what makes generation fast, but it is large, and critically it grows in two directions at once: with the context length (a longer input means more tokens to remember) and with the batch size (more simultaneous requests each keep their own cache). For offline inference — batch workloads such as benchmarking, information extraction, or bulk data processing, where you care about total throughput rather than fast responses — you want very large batches and can tolerate long contexts. But decoding is fundamentally memory-bound: for each token the model mostly reads a lot of cached data and does relatively little arithmetic, so a costly GPU sits largely idle waiting on memory. Buying many GPUs just to hold the cache in fast memory is cost-prohibitive, and it wastes their compute. So the economical path is to store the cache in cheaper, larger, slower memory and stream it in — which turns the whole problem into one of moving the key-value cache efficiently.

The catch is that the standard fix, offloading, treats the symptom while making the underlying transfer worse.

Offloading-based batched inference extends the GPU’s memory hierarchy downward: the key-value cache lives in host memory, and when that overflows it spills to SSD storage, which is plentiful and cheap. Batching lets the model weights be reused across many requests, so throughput improves. But offloading does not eliminate the cost of the cache; it relocates the cache further from the GPU, which means that on every decoding step the data has to travel a longer path back to the compute. The transfer happens over the system interconnect (the PCIe bus connecting storage, host, and GPU), whose bandwidth is limited. As the batch and the context grow, the cache grows, so more and more data must cross that interconnect each step, until the transfer time swamps the actual computation — measured at more than 60 percent of inference time in batched inference, and worse for long contexts where the cache no longer even fits in host memory and must come from SSD. In other words, the more you scale up the workload to use the hardware well, the more the interconnect becomes the wall. The data the GPU needs is far away, and the design keeps dragging it to the compute. That is exactly the pattern near-storage processing is built to break.

What people did before, and where it broke

Fit everything in GPU memory with more GPUs. Buy enough graphics processing units to hold the model and key-value cache in fast memory. Where it broke. It is cost-prohibitive, and because decoding is memory-bound, the added GPU compute sits underutilized, so it is an expensive answer to a memory problem.
Offload the key-value cache to host memory (FlexGen, DeepSpeed). Extend the memory hierarchy into host DRAM and reuse weights across a batch. Where it broke. For long contexts and large batches the cache exceeds host memory, and even before that, streaming it back to the GPU each step saturates the interconnect.
Spill the cache further, to SSD storage. When host memory overflows, keep the cache on cheap, large SSDs. Where it broke. The data is now even further from the GPU, so the transfer path is longest exactly when the cache is biggest; input and output overhead comes to dominate, with cache movement exceeding 60 percent of inference time.
Compress or approximate attention. Reduce how much cache must move by pruning or approximating. Where it broke. These help but do not change the fundamental structure of dragging cached data to distant compute, and can trade away model quality; the movement problem remains.
The gap. No system for offline long-context inference avoided moving the key-value cache in the first place by doing the memory-intensive attention where the cache already lives, on real, off-the-shelf storage hardware.

How HILOS works

Moving attention to the data instead of the data to attention

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Do the work where the data already is. The foundational move in HILOS is near-storage processing: rather than dragging the key-value cache across the interconnect to the GPU, it computes the memory-intensive attention right inside the storage device, so only tiny results travel back. This inverts the usual data flow and removes the transfer that was the bottleneck. Placing computation inside the storage device so massive data never has to be shipped to a distant processor is the same principle the work on a high-performance in-storage processing system for genome sequence analysis covered elsewhere on this site is built on, filtering and computing where the data lives to avoid moving it. HILOS applies that principle to LLM attention, turning the storage devices holding the cache into the place attention happens.

Attack the key-value cache, the specific thing that scales badly. The reason long-context batched inference is so hard is that the key-value cache grows with both batch size and context length, and it is the movement of that cache that dominates. HILOS targets exactly this: attention is the memory-intensive operation on the cache, so moving attention to storage cuts precisely the traffic that grows. Recognizing that the attention memory of batched generative inference is the scaling bottleneck and moving that specific computation close to the memory holding it is the same insight the work on unleashing processing-in-memory for batched transformer-based generative inference covered elsewhere on this site is built on, relocating attention to where its data sits. HILOS realizes the same idea on off-the-shelf storage devices rather than custom memory, so it works on real systems today.

Use the storage device’s own bandwidth instead of the interconnect’s. A single interconnect has fixed bandwidth, but a rack of storage devices has enormous aggregate internal bandwidth to their own flash. By computing inside each device, HILOS taps that internal bandwidth in parallel, and the total scales with the number of devices rather than being capped by the shared link. Extracting the full internal bandwidth of storage devices to feed a memory-bound serving workload is the same lever the work on maximizing SSD bandwidth utilization for huge embedding models covered elsewhere on this site pulls, turning storage from a bottleneck into a bandwidth source. HILOS uses that bandwidth for attention over the key-value cache, so throughput rises as more devices are added.

Following a decoding step through HILOS

Generating one token over a terabyte-scale key-value cache

A trace, drawn from the paper, of how HILOS computes one decoding step without moving the cache.

Step 1 of 5

The model needs to attend over a huge cached history

A large batch of long-context requests is decoding the next token. The key-value cache for all of them is terabytes in size and lives on the smart SSDs, far too big for GPU or even host memory. In the old design, all of it would be read to the GPU for this one step.

The attention data the GPU needs is far away and enormous.

Why it works

Computing at the cache removes the dominant transfer. The key-value cache is huge and the interconnect is the bottleneck → but attention runs inside the storage device so only small results move → so the traffic that consumed more than 60 percent of inference time nearly disappears.
The benefit grows with context length. Attention traffic scales with how many tokens are cached → but near-storage attention sends back only fixed-size results regardless of cache size → so the longer the context, the larger the saving, growing linearly with context length.
Aggregate bandwidth scales with devices. One shared interconnect has fixed bandwidth → but each storage device reads its own flash at full internal speed in parallel → so total effective bandwidth rises with the number of devices instead of being capped by the link.
Cooperative caching uses the resources offloading frees. Moving attention to storage leaves host memory and compute idle → but splitting the attention work between host and devices puts them to use → so cache input and output shrinks further and no resource is wasted.
Delayed writeback and online transpose keep storage efficient. New tokens add small entries and attention needs transposed keys → but HILOS batches writebacks and transposes blocks on the fly → so it avoids write amplification and redundant transposed copies, sustaining high throughput on real SSDs.

What it costs (trade-offs and scope)

It targets offline inference, not low-latency serving. HILOS is built for batch workloads that tolerate higher latency in exchange for throughput; it is not aimed at online serving with tight time-to-first-token and per-token latency targets, where the trade-offs differ.
It depends on near-storage-capable hardware. The approach requires smart SSDs with on-device accelerators (an FPGA next to the flash); on ordinary storage without near-storage processing, the core benefit is unavailable, so it presumes that class of device.
The accelerator must fit the attention variant. The near-storage accelerator implements attention (the paper covers standard multi-head attention and others) within the limited logic and memory of an on-SSD FPGA; supporting a very different attention mechanism, or larger per-device state, requires reworking the accelerator, though the authors provide tooling to customize it.
Gains scale with how offload-bound the workload is. The largest speedups (up to 7.86 times, and 5.3 to 7.8 times over SSD-based offloading for very long contexts) appear when the key-value cache is huge and interconnect-bound; workloads whose cache fits comfortably in faster memory have less transfer to eliminate and gain less.
On-device compute and bandwidth are modest per device. Each SSD’s accelerator is small compared with a GPU, so the design relies on many devices working in parallel and on the fact that attention here is memory-bound; it would not accelerate compute-bound work, and its advantage is specifically in avoiding data movement.

The evaluation

Serving long-context batched LLM inference cheaply

Property	Offload to host memory (FLEX DRAM)	Offload to SSD (FLEX SSD)	HILOS (near-storage)
Handles very long contexts (over 100K tokens)
Key-value cache moved over the interconnect
Effective bandwidth
Throughput
Energy

A comparison of ways to run offline long-context batched LLM inference. HILOS computes attention inside off-the-shelf smart SSDs, avoiding key-value cache transfers, reaching up to 7.86 times the throughput of conventional SSD offloading (5.3 to 7.8 times for very long contexts where host-memory offloading fails) while using up to 85 percent less energy.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (arXiv:2502.09921). The mechanism described is the paper’s: HILOS is a near-storage-processing framework for high-throughput, cost-effective offline inference of long-context LLMs, implemented on off-the-shelf near-storage products (smart SSDs with on-device FPGAs). Its core is attention near storage, which offloads the memory-intensive attention operations to near-storage accelerators so only computed results cross the system interconnect, with the traffic benefit growing linearly with context length and aggregate bandwidth scaling with the number of devices. On top of this it adds three optimizations: cooperative caching, which partitions the attention workload between the host and the near-storage devices to exploit freed host resources and minimize key-value cache input and output; delayed key-value cache writeback, which hides storage write latency and mitigates write amplification for newly generated entries; and a memory-efficient attention accelerator using matrix-vector units with an in-place online transpose to avoid redundant storage writes. HILOS is the first NSP-offloading system on off-the-shelf products supporting contexts beyond 100,000 tokens, is integrated with a PyTorch-based inference framework, and reports up to 7.86 times throughput over conventional SSD-based solutions (5.3 to 7.8 times over storage-based offloading for very long contexts) with up to 85 percent lower energy. The source code is publicly available. Exact accelerator parameters, the cooperative-caching policy, and the full evaluation should be read from the paper.

Background

Why compute keeps moving toward the data

It helps to see HILOS as part of a broad response to a hard physical fact: for data-heavy workloads, moving the data costs far more time and energy than computing on it. For decades the default was to bring data to the processor, but as datasets grew into the terabytes and interconnect bandwidth failed to keep pace, that default became the bottleneck. The response, across many domains, has been to push computation toward the data instead — into memory (processing-in-memory), into storage (near-storage and in-storage processing), and into the network. Large language model serving has made this especially urgent, because the key-value cache turns generation into a data-movement problem, and the industry is now attacking it from every layer: smarter batching and scheduling, compression, custom memory, and offloading. The broader effort to relieve the memory pressure of LLM serving by offloading it cleverly across the system is the same landscape the work on network-accelerated memory offloading for large language models covered elsewhere on this site maps, tackling the same memory wall from the networking side. HILOS contributes the storage side of that answer, showing that ordinary smart SSDs can compute attention where the cache lives and turn a movement-bound workload into a scalable, energy-efficient one.

If you remember only three things

1. Offline large-language-model inference in big batches is memory-bound, not compute-bound: the key-value cache (the per-token attention memory) grows with both batch size and context length, so for long contexts it becomes terabytes and must be offloaded to host memory and SSD storage. Then, on every decoding step, moving that cache back across the interconnect to the GPU — not the arithmetic — dominates, accounting for more than 60 percent of inference time. Offloading (FlexGen, DeepSpeed) enables the workload but only pushes the cache further away, making the transfer the wall.
2. HILOS avoids moving the cache by computing attention near storage: it runs the memory-intensive attention on a small FPGA built into each off-the-shelf smart SSD, right next to the flash holding the cache, so only tiny results cross the interconnect and effective bandwidth scales with the number of devices. It adds cooperative caching (split attention between host and devices to use freed host resources), delayed key-value cache writeback (hide write latency and cut write amplification for new tokens), and a memory-efficient attention accelerator that transposes key blocks on the fly instead of storing a redundant copy.
3. On a real system with smart SSDs, HILOS is the first near-storage offloading solution on off-the-shelf products to support contexts beyond 100,000 tokens, reaching up to 7.86 times the throughput of conventional SSD offloading (5.3 to 7.8 times for very long contexts where host-memory offloading fails) while using up to 85 percent less energy. It targets offline, memory-bound batch inference and needs near-storage-capable hardware; its gains are largest exactly when the cache is huge and interconnect-bound.

Check yourself

Check your understanding

1. Why is offline long-context batched LLM inference bottlenecked by data movement rather than computation?
2. What is HILOS's core idea?
3. Besides attention near storage, what three optimizations does HILOS add, and what does each fix?
4. Predict: as the context length gets longer and longer, how does HILOS's advantage over conventional SSD offloading change, and why?

Sources

The paper is “A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs” by Hongsun Jang, Jaeyong Song, Changmin Shin, Si Ung Noh, Jaewon Jung, Jisung Park, and Jinho Lee (Seoul National University and collaborators), published at ASPLOS 2026, DOI 10.1145/3779212.3790119. It is licensed CC BY.

This explainer is written from the paper’s full text (arXiv:2502.09921). The paper introduces HILOS, a near-storage-processing framework for high-throughput, cost-effective offline inference of long-context large language models on off-the-shelf near-storage products (smart SSDs with on-device FPGAs). Offloading-based batched inference extends the GPU’s memory hierarchy with host memory and storage but suffers substantial input and output overhead, primarily due to the large key-value cache sizes that scale with batch size and context window length, with key-value-cache-related work accounting for more than 60 percent of inference time in batched inference. HILOS’s core is attention near storage, which offloads the memory-intensive attention operations to near-storage accelerators so only computed results cross the interconnect, with the benefit growing linearly with context length and aggregate bandwidth scaling with the number of devices. On top of this it adds cooperative caching (partitioning the attention workload between host and near-storage devices to exploit freed host resources and minimize cache input and output), delayed key-value cache writeback (hiding storage write latency and mitigating write amplification for new entries), and a memory-efficient attention accelerator (matrix-vector units with an in-place online transpose to avoid redundant storage writes). HILOS is the first NSP-offloading system on off-the-shelf products supporting contexts beyond 100,000 tokens, is integrated with a PyTorch-based inference framework, and achieves up to 7.86 times throughput over conventional SSD-based solutions (5.3 to 7.8 times over storage-based offloading for very long contexts) while reducing energy consumption by up to 85 percent. No numbers beyond the paper’s own have been introduced; exact accelerator parameters, the cooperative-caching policy, and the full evaluation should be confirmed in the paper.