OS2G: A High-Performance DPU Offloading Architecture for GPU-based Deep Learning with Object Storage

Zhen Jin, Yiquan Chen, Mingxu Liang, Yijing Wang, Guoju Fang, Ao Zhou and 9 others

Training deep-learning models means constantly hauling huge datasets out of cheap cloud object storage and into GPUs, but the software that fetches and shuffles that data hogs the host CPU and copies the data through it, leaving the expensive GPUs waiting; this work from Zhejiang University builds OS2G, which moves the storage client onto a DPU — a smart network card with its own processor — and opens a direct path from the DPU straight into GPU memory, bypassing the host entirely, so a ResNet image model trains in up to about half the time while using up to about 60 percent less CPU.

The argument in one glance

Problem

Feeding GPUs from object storage burdens the host CPU

Deep-learning training reads huge datasets, and cheap, scalable object storage (like Amazon S3) is attractive for holding them. But the storage client that fetches and processes objects runs on the host CPU and copies the data through the host on its way to the GPU, so the host becomes a bottleneck that starves the expensive GPUs.
What people did

Offload the storage client to a DPU

A DPU — a network card with its own processor, built to offload infrastructure work — can run the storage client instead of the host CPU, freeing the host. This is a promising direction.
Why it fell short

Naive offloading is slow and still copies through the host

Two problems appear. Running the existing storage client unchanged on the DPU degrades performance badly, because the DPU's cores are weaker than the host's. And even with the client offloaded, the data still travels from the DPU to the host and only then to the GPU — a redundant hop that keeps consuming host resources.
This paper's idea

Fast DPU client plus a direct DPU-to-GPU path (OS2G)

OS2G offloads the object storage client to the DPU and enables direct data transfer from the DPU to the GPU. A high-performance OS2G client on the DPU uses asynchrony, prefetching, and concurrency to overcome its weaker cores, and a GPUDirect DPU technique moves data straight from the DPU-accelerated storage to the GPU, fully bypassing the host. It cuts ResNet18 training time by up to about half and host CPU use by up to about 60 percent.

Why this problem is worth solving

The expensive part of a training cluster is the GPUs, and starving them on data wastes money.

Training a deep-learning model means feeding a GPU a continuous stream of data — for an image model, millions of images, epoch after epoch. Those datasets are large, so where to store them matters. Object storage, the kind of cheap, massively-scalable cloud storage behind services like Amazon S3, is increasingly attractive because it is inexpensive and scales to any size. The catch is how the data gets from object storage to the GPU. A program called a storage client runs on the host CPU, fetches the objects over the network, and processes them, and then the data is copied through host memory on its way into the GPU. Every one of these steps — the network fetching, the client processing, the copies — consumes host CPU cycles and memory bandwidth. In a training cluster packed with GPUs, this data-feeding work becomes a bottleneck: the host CPU cannot prepare and deliver data fast enough, so the expensive GPUs sit idle waiting for their next batch. Since the GPUs are the costly resource, any time they spend waiting is wasted money. Reducing the host CPU burden of feeding data, and delivering it to the GPU faster, is therefore directly valuable.

The natural fix hits two snags that this work has to solve.

A DPU, or Data Processing Unit, is a smart network card: besides moving packets, it has its own general-purpose processor cores and accelerators. Its purpose is to take over infrastructure work — networking, storage, security — that would otherwise run on the host CPU, freeing the host for the application. So the obvious idea is to move the object storage client off the host and onto the DPU: then the host CPU is no longer burdened with fetching and processing data. But two problems get in the way. First, a DPU’s cores are weaker than a server’s host cores, so simply running the existing storage client on the DPU unchanged makes it much slower — the client was written assuming powerful host processors, and it degrades badly on the DPU. Second, offloading the client is not enough by itself. Even with the client on the DPU, the data it fetches still follows the old path: it is transferred from the DPU into host memory, and only then from the host into the GPU. That middle hop through the host is redundant — the host is now just a relay — yet it still copies the data through host memory and consumes host resources. To make DPU offloading actually pay off, both problems must be solved: the client must run fast on the weak DPU cores, and the data must reach the GPU without a pointless detour through the host.

What people did before, and where it broke

Host-CPU object storage clients (S3FS, S3Connector). Run the storage client on the host, fetch objects over the network, process them, and copy them to the GPU. Where it broke. All of this runs on the host CPU, so feeding data becomes a bottleneck that starves the GPUs and consumes host resources.
Naively offload the existing client to the DPU. Move the same storage client onto the DPU unchanged to free the host. Where it broke. The DPU’s cores are much weaker than the host’s, so the unmodified client runs far slower there, severely degrading performance.
Offload the client but keep the old data path. Run the client on the DPU, but still route fetched data from the DPU to the host and then to the GPU. Where it broke. The hop through the host is redundant now that the host is only relaying, yet it still copies data through host memory and consumes host resources.
The gap. No design both ran the storage client fast on the DPU and delivered data directly from the DPU to the GPU without the host detour. Doing both is what OS2G provides.

How OS2G works

From a host-bound data pipeline to a direct DPU-to-GPU path

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Move the storage work off the host and onto a dedicated processor. The first insight is that the host CPU should not be spending its cycles fetching and processing storage objects at all, because that starves the GPUs it is supposed to serve. OS2G relocates the object storage client onto the DPU, a processor built for exactly this kind of infrastructure work, so the host is freed. Taking a data-access burden off the general-purpose host and handing it to dedicated hardware built to do it is the same move the userspace-storage work covered elsewhere on this site makes, routing storage access through hardware that handles it directly so the host software stack drops out of the path.

Send the data straight to where it is needed, skipping the middle hop. Freeing the host is undone if the data still detours through it, so the second insight is to build a direct path from the DPU to the GPU. Rather than copying data from the DPU into host memory and then into the GPU, OS2G’s GPUDirect DPU technique moves it straight from the DPU-accelerated storage into GPU memory, bypassing the host entirely. Removing a needless intermediate hop and going straight to the destination is the same shortcut the direct page-table indexing work covered elsewhere on this site takes, reaching the final target directly instead of stepping through an intermediary that adds nothing but cost.

Hide the weak processor’s latency by overlapping and anticipating. A DPU’s cores are slower than a host’s, so a client written for powerful hosts stalls on them. OS2G’s client compensates by not waiting: it issues requests asynchronously, prefetches the objects the training job will need next before they are asked for, and serves many requests concurrently, so the slower cores stay busy and the GPU is not left waiting. Covering the latency of a slower component by overlapping work and fetching ahead of demand is the same technique the secure-computation pipelining work covered elsewhere on this site uses, overlapping communication with computation so that the time spent moving data is hidden behind useful work rather than paid in full.

Following a training batch through OS2G

Delivering a batch of images from object storage to the GPU

A high-level trace of the approach as described in the abstract; the exact client and data-path internals are beyond the abstract this page is based on.

Step 1 of 5

The GPU needs the next batch

A training step needs its next batch of images, which are stored as objects in an object store like an S3-compatible service.

Training reads the whole dataset many times, so this happens constantly.

Why it works

The storage client burdens the host CPU. Because reading from object storage runs a client on the host that fetches, processes, and copies data → these steps consume host CPU cycles and memory every batch → so the host becomes the bottleneck that starves the GPUs.
A DPU can take that work off the host. Because a DPU is a networked processor built for infrastructure work → moving the storage client onto it removes that load from the host CPU → so host resources are reclaimed.
Weak DPU cores need latency hiding. Because a DPU’s cores are slower than the host’s, so a naive client stalls on them → asynchrony, prefetching, and concurrency overlap and anticipate the work → so the DPU keeps the GPU fed despite its slower cores.
The host hop is redundant. Because data offloaded only to the client still travels from the DPU to the host to the GPU → that middle hop copies data through host memory for no reason → so a direct DPU-to-GPU path removes the copy and the host involvement.
Together they free the CPU and speed training. Because the client runs on the DPU and data flows straight to the GPU → the host is barely touched and copies are cut → so ResNet18 trains in up to about half the time using up to about 60 percent less host CPU.

What it costs (trade-offs and scope)

It requires a DPU. OS2G assumes the cluster has DPUs — smart network cards with their own processors — and the ability to run a storage client and a direct data path on them. This is hardware not every deployment has.
The direct path needs supporting technology. Moving data straight from the DPU into GPU memory relies on direct-transfer capabilities between the storage-side hardware and the GPU. It depends on the platform providing such a path and on the software to drive it.
The client must be re-engineered, not just relocated. The gains depend on the OS2G client’s asynchrony, prefetching, and concurrency; simply moving an existing client to the DPU does not work, so this is real engineering specific to the DPU.
Prefetching helps predictable access, less so random. Reading ahead works when the training job’s upcoming data is predictable, as it often is in dataset iteration; workloads with highly irregular access patterns would benefit less from the prefetching.
The benefit tracks how host-bound the pipeline was. The large gains come from removing host CPU work and copies. A pipeline that was not host-bound, or already used direct paths, has less to reclaim.
This page is written from the abstract. The full text was not available when this explainer was prepared, so the client’s internal design, the GPUDirect DPU mechanism, and the full evaluation are summarized only at the level the abstract states, without invented specifics.

The evaluation

Three ways to feed object-storage data to training GPUs

Property	Host-CPU client	Naive DPU offload
Where the storage client runs
Data path to the GPU	Host to GPU
Frees the host CPU
Handles the DPU's weaker cores	Not applicable
Training result (ResNet18)	Baseline	Degraded

Three ways to deliver object-storage data to training GPUs, as described in the paper's abstract. A host-CPU client (like S3FS or S3Connector) runs the fetching, processing, and copying on the host, making it a bottleneck that starves the GPUs. Naively offloading that client to a DPU runs it on much weaker cores and still routes data from the DPU through the host to the GPU, a redundant hop. OS2G runs a re-engineered client on the DPU — using asynchrony, prefetching, and concurrency to overcome the weak cores — and adds a GPUDirect DPU technique that moves data straight from the DPU-accelerated storage into GPU memory, bypassing the host entirely. Compared with S3FS and S3Connector, OS2G reduces ResNet18 training time by 34.3 percent and 50.4 percent and cuts host CPU consumption by 61.9 percent and 57.7 percent respectively. These figures are from the abstract; the full paper was not available for this explainer.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is written from the paper’s abstract, because the full text was not available when it was prepared, so it stays at the level the abstract supports and adds only standard, well-known background about object storage, DPUs, and direct GPU data transfer. The idea is clean and practical. Feeding deep-learning GPUs from cheap object storage burdens the host CPU, which fetches, processes, and copies the data, starving the expensive accelerators. Offloading the storage client to a DPU is the obvious move but fails two ways: the DPU’s weak cores make a naive client slow, and the data still detours through the host on its way to the GPU. OS2G addresses both, running a re-engineered client on the DPU that uses asynchrony, prefetching, and concurrency to overcome the slow cores, and adding a direct DPU-to-GPU data path that bypasses the host. The reported results are strong: versus two common clients, ResNet18 training time falls by about 34 and 50 percent and host CPU use by about 62 and 58 percent. Read the specifics with care. The internal design of the client, exactly how the direct DPU-to-GPU path is implemented, and the full evaluation are in the paper and not reproduced here. As trade-offs, the approach needs DPUs and a supporting direct-transfer path, the client must be re-engineered rather than merely relocated, the prefetching favors predictable access, and the benefit is largest when the pipeline was host-bound to begin with. The durable idea is that as CPUs become the bottleneck in data-heavy GPU pipelines, moving the data work to a dedicated processor and building a direct path to the GPU can reclaim both the CPU and the accelerators.

Background

Why offloading and direct paths keep reshaping data pipelines

It helps to see OS2G within a broad shift in how data-center systems are built: as specialized accelerators like GPUs get faster, the general-purpose host CPU that shuttles data to them becomes the bottleneck, and the response is to take work off the host and to forge direct paths between the components that actually need to talk. Here the storage client moves to a DPU and the data flows straight to the GPU, so the host stops being a relay. The warehouse-scale memory-traffic work covered elsewhere on this site pursues the same underlying goal from a different angle, cutting the needless data movement that burdens the host so its scarce resources go to real work rather than shuffling bytes. The common thread is that moving data is often the real cost, not computing on it, and the biggest wins come from either eliminating unnecessary movement or routing it around the busy general-purpose core. As accelerators proliferate and datasets grow, this pattern — offload the plumbing to purpose-built hardware, and connect producers to consumers directly — recurs across storage, networking, and machine learning. The general lesson is that a fast accelerator is only as useful as the pipeline feeding it, and keeping that pipeline off the host’s critical path is increasingly where system performance is won.

If you remember only three things

1. Deep-learning training reads huge datasets, and cheap, scalable object storage (like Amazon S3) is attractive for holding them, but feeding that data to GPUs burdens the host CPU: a storage client on the host fetches and processes objects and copies them through host memory to the GPU, making the host a bottleneck that starves the expensive accelerators. Offloading the client to a DPU (a network card with its own processor) is promising, but naively doing so is slow because the DPU's cores are weaker than the host's, and even then the data still detours from the DPU through the host to the GPU, a redundant hop that keeps consuming host resources.
2. OS2G offloads the object storage client to the DPU and enables direct data transfer from the DPU to the GPU. Its high-performance OS2G client runs on the DPU and uses asynchrony, prefetching, and concurrency to overcome the weaker cores and keep the GPU fed. Its GPUDirect DPU technique optimizes the data path so data moves straight from the DPU-accelerated storage into GPU memory, fully bypassing the host and its redundant copy. Together these free the host CPU and deliver data to the GPU faster.
3. Compared with the S3FS and S3Connector clients, OS2G reduces ResNet18 training time by 34.3 percent and 50.4 percent, and cuts host CPU consumption by 61.9 percent and 57.7 percent, respectively. Its scope: it requires DPUs and a supporting direct-transfer path to the GPU, the client must be re-engineered for the DPU rather than merely relocated, prefetching favors predictable access, and the benefit is largest when the pipeline was host-bound. Note: this explainer is written from the abstract, as the full text was not available, so the client and data-path internals and full evaluation are summarized only at the level the abstract states.

Check yourself

Check your understanding

1. Why does feeding deep-learning GPUs from object storage burden the host CPU?
2. Offloading the storage client to a DPU seems obvious. Why does naive offloading fail?
3. What are the two key techniques in OS2G?
4. A cluster's data pipeline is already not host-bound — the host CPU has plenty of spare capacity and the GPUs are never waiting on data. What does OS2G's reasoning imply?

Sources

The paper is “OS2G: A High-Performance DPU Offloading Architecture for GPU-based Deep Learning with Object Storage” by Zhen Jin, Yiquan Chen, Mingxu Liang, Yijing Wang, Guoju Fang, Ao Zhou, Keyao Zhang, Jiexiong Xu, Wenhai Lin, Yiquan Lin, Shushu Zhao, Wenkai Shi, Zhenhua He, Shishun Cai, and Wenzhi Chen, published at ASPLOS 2025, DOI 10.1145/3676641.3716265. The authors are at Zhejiang University.

This explainer is written from the paper’s abstract, because the full text was not available at the time of writing: the publisher’s copy was behind automated-access protection, the paper’s open-access status was closed, and no preprint or author copy could be retrieved. The problem statement (object storage being cost-effective and scalable for deep learning but burdening the host CPU through intensive processing and multiple data movements, and the two shortcomings of DPU offloading — the severe degradation of naively offloading the existing client, and the redundant data movement from the DPU to the host and then to the GPU) and the description of OS2G (offloading the object storage client to a DPU and enabling direct DPU-to-GPU data transfer, with a high-performance OS2G client using asynchrony, pre-reading, and concurrency, and a GPUDirect DPU technique optimizing the data path to bypass the host) come directly from the abstract, as do the results (compared with S3FS and S3Connector, reducing ResNet18 execution time by 34.3 percent and 50.4 percent and decreasing CPU consumption by 61.9 percent and 57.7 percent, respectively). The surrounding explanation of object storage, storage clients, DPUs and SmartNICs, direct GPU data transfer, and deep-learning data pipelines is standard, widely-known background provided to make the abstract’s claims understandable, and is not specific to this paper. The internal design of the OS2G client, the GPUDirect DPU mechanism, and the detailed evaluation are not reproduced here because they were not available.