NDPipe: Exploiting Near-data Processing for Scalable Inference and Continuous Training in Photo Storage

A data center full of photos runs machine learning on those images constantly — classifying, tagging, and retraining as new pictures arrive — but the usual way ships the huge image files across the network from the storage servers to a separate cluster of expensive GPUs, wasting bandwidth and energy; this paper (DGIST and collaborators — Jungwoo Kim, Sungjin Lee, and colleagues) presents NDPipe, which instead puts inexpensive commodity GPUs into the photo storage servers themselves and processes images where they already live, pooling the collective power of many weak servers by partitioning the neural network and exploiting data parallelism with low synchronization, delivering 1.39 times more inference throughput and 2.64 times faster training at the same energy budget.

The argument in one glance

Problem

A data center of photos runs ML on them constantly, but the images live far from the GPUs

A large photo storage system holds a staggering number of images, and modern services do not just store them — they run machine learning on them all the time. Every uploaded photo may be classified, tagged, checked for content, or turned into features for search and recommendation, which is inference. And because the world keeps changing and new photos keep arriving, the models themselves must be retrained continuously to stay accurate, which is training. Both are heavy, ongoing workloads over the entire photo collection. The awkward part is where the compute happens: the photos sit on storage servers, but the machine learning traditionally runs somewhere else, on a dedicated cluster of powerful GPUs. So to process the images you must first move them, and there are a lot of images.
What people did

Ship the images to a central cluster of powerful GPUs

The conventional approach separates storage from compute: photo storage servers hold the images, and a separate cluster of high-end GPUs does the inference and training. When the models need to run, the images are read out of storage and sent over the data center network to the GPU cluster, which processes them and, for training, updates the models. Storage and compute are scaled and managed independently.
Why it fell short

Moving the images wastes bandwidth and energy, and centralizes the compute

Image files are large and numerous, so continuously reading them out of storage and pushing them across the network to a distant GPU cluster consumes a great deal of network bandwidth and energy just to relocate data that never needed to move. The powerful central GPUs are expensive, and the whole pipeline is bottlenecked by the cost of transporting images rather than the cost of computing on them. Meanwhile the storage servers, which already hold the data, sit doing little more than serving files. Paying to haul enormous image datasets to the compute, instead of bringing the compute to the data, is the inefficiency at the heart of the conventional design.
This paper's idea

Put cheap GPUs in the storage servers and process images where they live

NDPipe is a photo storage system that flips the arrangement: it performs the inference and training near the data, inside the photo storage servers themselves. It equips each storage server with an inexpensive commodity GPU and uses the collective intelligence of many such servers to do the machine learning right where the images already are, so the images never have to be shipped to a separate cluster. For inference, each server processes its own photos locally. For training, where one weak server cannot handle a whole model efficiently and naively splitting work across many servers would incur heavy synchronization, NDPipe efficiently partitions the neural network and exploits the data parallelism of the many storage servers to get high training throughput with low synchronization cost. It optimizes the near-data-processing engine to make full use of each server's components. At the same energy budget, this delivers 1.39 times higher inference throughput and 2.64 times faster training than typical photo storage systems.

Why this problem is worth solving

The scale of photo storage makes moving the data the dominant cost, which is exactly the setting near-data processing exists for.

A photo storage service operates at enormous scale — billions of images, growing constantly — and the machine learning done over those images is relentless: inference to classify, tag, moderate, or extract features from every photo, and continuous training to keep the models accurate as the distribution of uploaded content shifts. The defining characteristic of this workload is that the data is huge and the compute must touch all of it, repeatedly. In a conventional design, storage and compute are separated: the images live on storage servers, and a dedicated GPU cluster does the learning, so every inference pass and every training epoch requires reading the images and transmitting them across the network. Because image files are large, this data movement dominates — the network bandwidth and the energy spent shuttling pictures can exceed the cost of the computation itself. Near-data processing is the principle of placing computation next to where data resides so that this movement is avoided, and photo storage is close to an ideal case for it: the data is voluminous, it is already spread across many storage servers, and the operations over it are regular and parallelizable across images. Bringing the modest compute needed for image inference and training to the servers that already hold the photos, rather than hauling the photos to the compute, is therefore a natural and high-leverage optimization, which is why it is worth building a whole storage system around.

The catch is that the compute available near the data is weak, so the challenge shifts from moving data to orchestrating many small engines.

If you put compute into storage servers, you cannot put a top-tier GPU in each one — that would defeat the cost and energy savings that motivated near-data processing. So the compute near the data is deliberately inexpensive and individually weak, and the design must extract performance from the collective rather than any single node. For inference this is relatively easy: each photo can be processed independently, so a storage server simply runs the model on its own images, and the aggregate throughput of many servers is large with essentially no coordination. Training is where it gets hard. A neural network being trained is a single shared model whose weights all the work must agree on, so training across many machines normally requires them to synchronize their updates — an all-reduce of gradients every step — which is costly and scales poorly, and it can be worse when each machine is weak and the model does not fit comfortably on one. Naively data-parallel training over many storage servers would drown in synchronization overhead. So the real problem near-data photo processing must solve is how to harness a swarm of cheap, weak, distributed compute engines to train a shared model efficiently, keeping the synchronization cost low while still using every server’s capacity — which is precisely where NDPipe concentrates its design, in how it partitions the model and arranges the parallelism.

What people did before, and where it broke

Separate storage and a central GPU cluster. Keep images on storage servers and do all inference and training on a dedicated cluster of powerful GPUs, shipping images across the network. Where it broke. The images are large and numerous, so continuously moving them consumes heavy network bandwidth and energy, and the expensive central GPUs are underused relative to the transport cost — the pipeline is bottlenecked by data movement, not compute.
Scaling up the central GPU cluster. Add more high-end GPUs to raise throughput. Where it broke. It does nothing about the data-movement bottleneck and raises cost and energy, since the images still have to be read and transmitted from storage regardless of how much compute waits at the other end.
Naive near-data inference only. Do inference near the data but keep training centralized. Where it broke. Continuous training is a first-class need for a photo service, so leaving it centralized still requires shipping images for every training pass; a design that only accelerates inference leaves the harder, movement-heavy half of the workload unaddressed.
Naive data-parallel training across storage servers. Split training batches across the many storage-server GPUs directly. Where it broke. With weak per-server GPUs and a shared model, straightforward data parallelism incurs heavy per-step synchronization (gradient all-reduce), so the synchronization cost dominates and the collective throughput does not scale.
The gap. No photo storage system placed inexpensive commodity GPUs in the storage servers and orchestrated their collective compute — partitioning the model and combining it with data parallelism to keep synchronization low — so that both inference and continuous training could run near the data at high throughput and low energy.

How NDPipe works

Doing image inference and training inside the storage servers that hold the photos

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Bring the compute to the photos instead of the photos to the compute. The organizing move in NDPipe is to refuse the conventional separation of storage and compute for this workload, because for a photo service the data is so large and so numerous that moving it dominates everything. Rather than read images out of storage and ship them across the network to a distant GPU cluster, NDPipe puts the compute inside the storage servers and processes each image where it already lives, so the images never travel. This is the essence of near-data processing, and photo storage is an especially good fit because the images are already spread across many servers and the per-image work is independent and regular. The payoff is measured at the same energy budget: by not spending energy on transporting data, the same power delivers substantially more useful inference and training. Relocating computation into the storage layer so that a data-heavy workload avoids the cost of hauling its data to a separate compute tier is the same principle the in-storage acceleration work covered elsewhere on this site is built on, executing the work where the bytes reside rather than paying to move them out. NDPipe applies that principle to the specific and voluminous case of images and the machine learning done over them.

Make many cheap, weak GPUs act as one, by partitioning the model. Putting compute in every storage server only pays off if the compute is cheap, so NDPipe uses inexpensive commodity GPUs, which are individually weak — and that weakness is the design constraint. Inference tolerates it easily, since each server just runs the model on its own photos, but training a single shared model is harder because no one weak server can carry the whole model well and the servers must still agree on the weights. NDPipe’s answer is to efficiently partition the neural network across the servers, dividing the model so each commodity GPU handles a piece sized to its capability, and then to layer data parallelism over that partitioning. This lets the collective of weak engines behave like a capable trainer, with the work split so that no single node is overwhelmed. Splitting a model across multiple devices so that a large training job fits and flows through limited-capacity hardware, rather than demanding one machine hold and run it all, is the same partitioning discipline the pipeline-parallel training work covered elsewhere on this site is built on, dividing a network into stages spread across devices to overcome any one device’s limits. NDPipe’s twist is to do this over the swarm of storage servers so their pooled commodity GPUs become the training substrate.

Keep the synchronization cheap, so the collective actually scales. The reason naive distributed training over many weak nodes fails is synchronization: every step, the machines must reconcile their model updates, and that coordination cost grows with the number of nodes and can swamp the useful work, especially when each node is slow. NDPipe treats keeping this cost low as a first-class goal, combining the model partitioning with data parallelism specifically so that high training throughput is achieved with low synchronization overhead, and it tunes its near-data engine so each server’s CPU, GPU, and storage bandwidth are all kept busy rather than bottlenecking one another. The aim is that adding more storage servers adds more training capacity rather than more coordination drag, which is what lets a large number of inexpensive servers collectively outrun a conventional setup at the same energy. Designing the parallel training so that scarce hardware is fully utilized and the coordination between many modest machines does not become the bottleneck is the same efficiency-from-commodity-hardware ambition the work on training large models on commodity GPU servers covered elsewhere on this site pursues, wringing strong training performance out of affordable, limited machines by managing their coordination carefully. NDPipe pursues it in the storage layer, so the machines doing the training are the ones already holding the data.

Following a training round through NDPipe

Continuously retraining a photo model across storage servers

An illustrative trace, based on the paper's abstract, of how NDPipe trains and serves models near the photo data. Internal details beyond the abstract were not available for this draft.

Step 1 of 5

New photos accumulate on the storage servers

As users upload images, they land on the photo storage servers spread across the data center. The models that classify or tag these photos must be kept current, so continuous training runs over the growing collection — and the images are already where the training will need them.

Because the data lives on the storage servers, near-data processing avoids ever shipping it to a GPU cluster.

Why it works

Not moving the images removes the dominant cost. Continuously shipping large image files to a GPU cluster burns network bandwidth and energy → but processing each photo on the server that holds it eliminates that transport → so the same energy delivers more useful inference and training.
A swarm of cheap GPUs beats a few costly ones for this work. One powerful central GPU is expensive and bottlenecked by data delivery → but many inexpensive commodity GPUs in the storage servers process data in place → so collective throughput rises without premium hardware or transport.
Model partitioning fits training to weak nodes. A single weak GPU cannot train a whole model well → but partitioning the network across servers gives each a manageable piece → so the collective can train a model no individual node could handle efficiently.
Low-sync parallelism lets the collective scale. Naive data-parallel training over many nodes drowns in per-step synchronization → but combining partitioning with data parallelism keeps synchronization low → so adding servers adds capacity instead of coordination drag.
Full component utilization avoids internal bottlenecks. A server’s CPU, GPU, or storage bandwidth could each stall the others → but tuning the near-data engine keeps all of them busy → so each server contributes its full capability to the aggregate throughput.

What it costs (trade-offs and scope)

This is a draft built from the paper’s abstract. The full paper was behind a publisher access wall for this draft (no arXiv version, no open-access or author copy located, and no web-archive snapshot), so this account is assembled from the paper’s complete abstract plus standard near-data-processing and distributed-training background. The exact model-partitioning scheme, the synchronization strategy, the near-data-engine design and how it balances CPU, GPU, and storage, the hardware configuration, and the evaluation setup and baselines are not reproduced here and should be read from the paper.
The reported figures are the abstract’s own, and at a fixed energy budget. The 1.39-times inference and 2.64-times training gains are the abstract’s results measured at the same energy as typical photo storage systems; the specific workloads, models, hardware, and baselines behind those numbers are not detailed in this draft.
It requires hardware in the storage servers. Putting a commodity GPU in each storage server is a real hardware and integration cost, and it turns storage nodes into compute nodes with the added power, cooling, and management that implies; the approach assumes this augmentation is worthwhile relative to a separate GPU cluster.
The gains depend on the workload fitting near-data processing. The benefit rests on the compute being modest enough for commodity GPUs and the per-image work being parallelizable in place; workloads needing very large models or heavy per-image compute that a weak GPU cannot handle, even partitioned, would fit less well.
It targets photo storage specifically. NDPipe is designed for image inference and continuous training in photo storage; other data types, access patterns, or ML workloads are outside its stated scope, and how the design generalizes is not addressed by the abstract.

The evaluation

Running image inference and training over a photo data center

Property	Central GPU cluster	Inference-only near-data	NDPipe
Where compute runs
Moves large image files
Trains near the data
Handles weak distributed GPUs
Reported outcome (per abstract)

A qualitative comparison based on the paper's abstract; the paper's main body, including its model-partitioning scheme, synchronization strategy, near-data-engine design, and exact baselines, was not available for this draft. The 1.39-times and 2.64-times figures are the abstract's reported results at the same energy budget.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This is an honest draft assembled from the paper’s complete abstract plus standard background on near-data processing and distributed training; the full text was not accessible (no arXiv preprint, no open-access or author copy found, and the ACM Digital Library version returned a bot-verification challenge), so the internal design is not reproduced. What is solid and comes directly from the abstract: photo storage systems must run inference and continuous training over huge image collections; NDPipe leverages near-data processing by distributing storage servers with inexpensive commodity GPUs and using their collective intelligence to do inference and training near the image data; it efficiently partitions deep neural network models and exploits the data parallelism of many storage servers to achieve high training throughput with low synchronization costs; it optimizes the near-data-processing engine to maximally utilize each server’s components; and, at the same energy budget, it exhibits 1.39 times higher inference throughput and 2.64 times faster training than typical photo storage systems. The explanation of why moving images is costly, why weak distributed GPUs make training hard, and how partitioning and data parallelism help is standard background and the natural reading of the abstract, not a claim about NDPipe’s specific internals. Read the exact partitioning and synchronization mechanisms, the engine design, the hardware, and the evaluation in the paper itself (an artifact is available at github.com/dgist-datalab/NDPipe); treat this page as a faithful account at the level of the abstract, not of the implementation.

Background

Why near-data processing keeps returning

It helps to see NDPipe as one instance of an idea that has recurred throughout computing history: bring the computation to the data rather than the data to the computation. The reason it keeps coming back is a persistent, widening gap — moving data is slow and energy-hungry, while computing on it, in relative terms, keeps getting cheaper, so any workload that touches a lot of data spends more and more of its budget on transport. When the data is small this does not matter, but at the scale of a data center full of photos, or a genome database, or a warehouse of logs, the cost of shuttling bytes across networks and memory buses comes to dominate, and the natural response is to push a little compute out to where the bytes already sit. Storage servers, which have historically been passive shelves for data, become an attractive place to put that compute precisely because the data is already there and often idle-adjacent hardware can be added cheaply. The engineering challenge is always the same: the compute you can afford to place near the data is limited, so you must design the workload to fit modest, distributed engines rather than assume a powerful central one. That drive to execute data-intensive computation in or beside the storage that holds the data, sidestepping the movement tax, is the same premise the in-storage genome-processing work covered elsewhere on this site is built on, turning the storage device itself into a site of computation. NDPipe carries the idea to photo storage and to the specific demands of continuously training and serving image models.

If you remember only three things

1. Photo storage systems in data centers hold enormous image collections and must run machine learning on them continuously — inference over every photo and continuous training to keep models current — but the conventional design separates storage from compute, so images are read from the storage servers and shipped across the network to a separate cluster of powerful GPUs, wasting network bandwidth and energy hauling large files that never needed to move.
2. NDPipe flips this by doing the inference and training near the data: it equips each photo storage server with an inexpensive commodity GPU and uses the collective intelligence of many such servers to process images where they already live, so nothing is shipped to a central cluster. Inference is easy (each server runs the model on its own photos independently), but training a shared model across weak distributed GPUs is hard, so NDPipe efficiently partitions the neural network across servers and combines that with data parallelism to reach high training throughput at low synchronization cost, and it tunes its near-data engine to fully use each server's CPU, GPU, and storage.
3. At the same energy budget, NDPipe delivers 1.39 times higher inference throughput and 2.64 times faster training than typical photo storage systems. This explainer is a draft written from the abstract without the paper body, so the exact model-partitioning scheme, synchronization strategy, near-data-engine design, hardware, and evaluation baselines should be confirmed against the paper (an artifact is available). The approach requires adding commodity GPUs to storage servers, depends on the compute fitting weak distributed hardware, and targets image inference and continuous training in photo storage specifically.

Check yourself

Check your understanding

1. Why is the conventional way of running machine learning over a photo data center inefficient?
2. What is NDPipe's core idea?
3. Why is training harder than inference in NDPipe, and how does it address that?
4. What should a reader keep in mind about this explainer, and what did the paper report?

Sources

The paper is “NDPipe: Exploiting Near-data Processing for Scalable Inference and Continuous Training in Photo Storage” by Jungwoo Kim, Seonggyun Oh, Jaeha Kung, Yeseong Kim, and Sungjin Lee (Daegu Gyeongbuk Institute of Science and Technology and collaborators), published at ASPLOS 2024 (Volume 3), DOI 10.1145/3620666.3651345. It is licensed CC BY.

This explainer is a draft. The paper’s main text was not accessible at the time of writing (there is no arXiv preprint, no open-access or author copy was located, and the ACM Digital Library PDF returned a bot-verification challenge), so it is written from the paper’s complete abstract together with standard background on near-data processing and distributed deep-learning training. The abstract supplies the problem and approach: NDPipe is a photo storage system that accelerates training and inference for image data by leveraging near-data processing in photo storage servers; it distributes storage servers with inexpensive commodity GPUs in a data center and uses their collective intelligence to perform inference and training near image data; by efficiently partitioning deep neural network models and exploiting the data parallelism of many storage servers, it achieves high training throughput with low synchronization costs; it optimizes the near-data-processing engine to maximally utilize the system components in each storage server; and, given the same energy budget, it exhibits 1.39 times higher inference throughput and 2.64 times faster training than typical photo storage systems. The explanation of why moving images is costly, why weak distributed GPUs make training the hard part, and how model partitioning and data parallelism help is standard background and the natural reading of the abstract, not a claim about NDPipe’s specific internal design. The paper’s exact model-partitioning scheme, synchronization strategy, near-data-engine design, hardware configuration, evaluation setup, and baselines are not reproduced here and should be read from the paper; an artifact is available at github.com/dgist-datalab/NDPipe. No numbers beyond the abstract’s have been introduced.