GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecture

GPUs wait on CPUs to fetch their data from SSDs, and the CPU software stack — not the flash — is the bottleneck; BaM maps NVMe queues and doorbells straight into GPU memory so a hundred thousand GPU threads can each demand their own 512-byte reads, hitting 45.8 million IOPs and matching a host-DRAM system that costs 21.7× more.

Why this problem is worth solving

The memory-capacity wall for accelerators is not a niche inconvenience — it decides which workloads GPUs can serve at all. Graphs with hundreds of millions of nodes, embedding tables for recommenders, GNN feature stores: all are arrays of tens of GB to tens of TB, accessed in patterns computed at runtime (the traversal decides which neighbor list is next). Against that, an A100 offers 80 GB.

The conventional escape routes each fail a different way. Host DRAM costs more per GB than flash by an order of magnitude and must be pre-loaded — for data-dependent workloads, much of what is loaded is never touched. CPU-orchestrated tiling turns the CPU into a data butler: it cannot know which bytes the GPU will need, so it ships whole partitions (

The quiet enabler is that storage hardware outgrew its software. NVMe SSDs are simple queue machines —

What people did before, and where it broke

Proactive tiling. CPU code partitions the dataset, pipelines chunks to the GPU, launches kernels per chunk. Works for regular access; for data-dependent workloads it amplifies I/O (the CPU cannot predict need), multiplies kernel launches, and burns programmer effort on partition/overlap logic.

Memory-mapped files with GPU page faults. Elegant abstraction, serial reality: faults traverse CPU driver and OS machinery whose thread-level parallelism is orders of magnitude below the GPU’s request rate.

GPUfs / ActivePointers. Pioneering GPU-side file abstractions — but data transfers still route through CPU software, capping miss-handling throughput at a level BaM exceeds by 20.7× even while ActivePointers serves misses from fast CPU DRAM and BaM from actual SSDs.

NVIDIA GPUDirect Storage (GDS). Moves the data path peer-to-peer (SSD → GPU memory, no host bounce) but keeps the control path on the CPU — fine for large sequential transfers, collapsing at fine granularity: BaM saturates ~25 GB/s at 4 KB blocks where the GDS/CPU stack cannot.

The idea

Latency tolerance is a parallelism budget — and the GPU is rich. The foundational arithmetic is Little’s Law: throughput × latency = required outstanding requests. Fifty-one million 512-byte accesses per second against Optane’s 11 µs needs 561 requests in flight; against a consumer SSD’s 324 µs, 16,524. For a CPU-driven control path those are absurd numbers; for a GPU running hundreds of thousands of threads, they are a rounding error — if the software lets threads issue requests without serializing. BaM’s whole design flows from this: don’t reduce storage latency, hide it under thread parallelism, which the GPU has in three orders of magnitude more than the CPU.

Queues that thousands of threads can share. The NVMe protocol was designed for a few CPU cores; its doorbell registers are write-only and must be updated with monotonic tails — a natural critical section that would strangle GPU-scale concurrency. BaM’s queue protocol dissolves it: an atomic ticket counter assigns each thread a slot in a virtual queue (slot = ticket mod queue size, turn = ticket div queue size); a turn_counter array lets each physical slot act as an ordered sub-queue; threads copy commands into their slots in parallel and set bits in a mark bit-vector; one winning thread takes the lock, advances the tail past all consecutively marked entries, and rings the doorbell once for the whole batch. Completion handling mirrors this, lock-free polling included. The result: enqueue/dequeue at GPU thread counts with critical sections measured in a handful of instructions — the difference between 823 K and 45.8 M IOPs.

A cache built for a million concurrent probes. OS memory management supports every legacy need and pays in critical sections. BaM’s cache pre-allocates everything at startup, reduces locking to line insertion/eviction, and puts the GPU’s strengths — massive memory bandwidth, huge atomic throughput — to work on metadata. Warp-level coalescing collapses duplicate probes; miss coalescing collapses duplicate storage requests. Hot data serves at 430 GB/s — GPU-memory speeds, 11.2× ActivePointers’ cache.

Fine granularity is the point, not a detail. Configurable cache lines down to 512 bytes mean a graph traversal fetches a neighbor list, not the megabyte around it. This is where the economics flip: data-dependent workloads on tiled or big-block systems drown in amplification; BaM’s on-demand 512-byte reads cut traffic so much that four SSDs at PCIe bandwidth keep up with host DRAM on end-to-end graph analytics — because the DRAM system must first spend minutes loading files it may barely use.

bam::array: mmap-like, storage-backed
warp coalescer: 1 probe per unique line

Why it works

• Storage latency × target throughput = required in-flight requests (Little’s Law) → the GPU’s thread count covers even consumer-SSD latencies effortlessly → latency tolerance comes from parallelism the accelerator already has, not from faster (pricier) media.
• The CPU contributed nothing essential to the control path → moving queues, buffers, and doorbells into GPU space deletes the page-fault handler, kernel crossings (36.4% of latency under io_uring), and CPU-GPU synchronization in one stroke.
• NVMe’s serialization points (tails, doorbells) are batchable → the ticket/turn/mark protocol lets command-copying parallelize and reduces the critical section to a tail bump plus one doorbell per batch → queue throughput scales to GPU thread counts.
• On-demand 512-byte access eliminates amplification → four SSDs of PCIe bandwidth outperform pre-loading into host DRAM end-to-end → capacity moves to flash at 4.3–21.8× lower cost per GB with performance parity.
• Everything is off-the-shelf (A100 features, NVMe spec, a small driver) → the design is a software architecture, adoptable without new silicon — and storage-medium-agnostic, so cost/performance is a dial (consumer 980pro: 2.7–3.2× slower than Optane, far cheaper).

What it costs

• GPU memory spent on infrastructure. Queues, I/O buffers, and the software cache all live in the GPU’s scarce 80 GB.
• Cache logic in every access. Software cache probes and queue protocol instructions tax each miss; the design amortizes but cannot eliminate them.
• Trust model. User-level access to raw NVMe queues bypasses kernel file-system enforcement; the paper points to emerging frameworks for securing accelerator-centric storage rather than solving isolation itself.
• Ordering subtleties. GPUDirect RDMA write ordering over PCIe requires care (a read must follow writes to guarantee visibility) — handled, but the kind of detail that makes such systems delicate.
• Read-centric evaluation. Graph/data analytics are read-heavy; writes reach 10.6 M IOPs but endurance and write-path behavior under sustained load get less attention.

The evaluation

The prototype is deliberately mundane hardware: an A100-80GB, dual EPYC host, and a PCIe expansion chassis with up to 10 SSDs of three classes (Intel Optane P5800X, Samsung enterprise, Samsung 980pro consumer). Workloads: microbenchmarks, BFS and connected-components graph analytics on five large graphs (GAP-kron, GAP-urand, Friendster, MOLIERE_2016, uk-2007-05), and I/O-bound data analytics.