SoCFlow: Efficient and Scalable DNN Training on SoC-Clustered Edge Servers

Some edge data centers are built from hundreds of ordinary mobile phone chips packed into a server — cheap and power-thrifty — but they run user apps that are busy only at peak hours and sit idle the rest of the time; this work from Peking University proposes to fill those idle cycles with AI model training, and builds SoCFlow, which sidesteps the weak network between the chips by aggregating results in small groups and only occasionally across them, and squeezes every processor on each mobile chip by training in mixed precision, matching a data-center GPU's training speed at a fraction of the energy.

The argument in one glance

Problem

Edge servers made of phone chips sit idle much of the time

A SoC-Cluster edge server packs hundreds of mobile system-on-chips — the processors found in phones — into one machine, because they are energy-efficient and run mobile apps directly. But the user-triggered apps they host follow a tidal pattern, busy at peak and idle otherwise, so the servers waste many free cycles.
What people did

Fill the free cycles with deep learning training

To harvest those idle cycles, co-locate deep learning training on the SoC-Cluster, running training in the troughs when the user workloads are quiet.
Why it fell short

A SoC-Cluster is a poor fit for training as it stands

Distributed training normally synchronizes gradients across all workers every step, but the network between the mobile chips is weak and quickly becomes the bottleneck. And each mobile chip has several small, heterogeneous processors — a CPU, a GPU, and a neural accelerator — that a naive trainer leaves mostly idle, so it fails to use the chips' real capability.
This paper's idea

Group-wise, delayed aggregation and mixed precision (SoCFlow)

SoCFlow is the first framework to train deep learning models efficiently on a SoC-Cluster. It uses group-wise parallelism with delayed aggregation to train fast and scalably without being throttled by the weak network, and a data-parallel mixed-precision algorithm that unleashes the heterogeneous processors of each mobile chip. It reaches convergence up to hundreds of times faster on 32 chips, and matches a data-center GPU's training speed at several times less energy.

Why this problem is worth solving

There is a large pool of underused, energy-efficient hardware at the edge waiting to be put to work.

A system-on-chip, or SoC, is a whole small computer on a single chip — the CPU, graphics processor, a neural accelerator, and memory controllers all together — of the kind that powers a smartphone. A SoC-Cluster is a server architecture that packs a large number of these mobile chips, hundreds of them, into one machine. This design is gaining ground in industrial edge computing for two reasons: mobile chips are extremely energy-efficient for the work they do, and because they are the same processors phones use, they can run existing mobile applications directly, which is valuable for services like hosting mobile apps or cloud gaming close to users. The problem the paper observes is utilization. The workloads these servers run are user-triggered, so their load rises and falls with human activity — a strong tidal pattern, busy during peak hours and nearly idle at night or off-peak. As a result, the deployed servers are far from fully utilized, and an enormous number of processor cycles simply go to waste. Those free cycles are valuable: if they could be filled with useful work, the operator would get more computation out of hardware and energy it is already paying for. The natural candidate to soak up that idle capacity is deep learning training, which is compute-hungry and can run in the background whenever the user workloads are quiet.

The obstacle is that these clusters were not built for training, and training punishes their weaknesses.

Training a deep neural network across many machines is usually done with data parallelism: each worker holds a copy of the model, processes a different slice of the data, and computes gradients — the adjustments to the model. To keep the copies consistent, the workers must combine their gradients every step, averaging them across all workers, which is a heavy, all-to-all communication over the network. On a normal training cluster the network is fast, so this is manageable. A SoC-Cluster has two weaknesses that make it manageable no longer. First, the network connecting the mobile chips is weak — these were mobile chips, not data-center accelerators, and the interconnect between them is a bottleneck, so synchronizing gradients every step stalls training on communication. Second, each mobile chip is not one processor but several small, heterogeneous ones — a CPU, a GPU, and a neural accelerator, each with different strengths and different numeric-precision support. A trainer written for a normal accelerator uses only part of this, leaving most of each chip idle. So a naive attempt to train on a SoC-Cluster is throttled by the network and wastes the chips’ compute, and the challenge is to work with these constraints rather than against them.

What people did before, and where it broke

Leave the SoC-Cluster to its user workloads. Run only the mobile-app or cloud-gaming workloads the servers were bought for. Where it broke. Those workloads are tidal, so the servers sit largely idle off-peak, wasting the energy-efficient hardware’s free cycles.
Train with standard data parallelism. Co-locate training but synchronize gradients across all chips every step, as on a normal cluster. Where it broke. The weak inter-chip network cannot carry that all-to-all traffic, so training is throttled by communication and does not scale.
Train using only each chip’s main accelerator. Run the training compute on, say, the mobile GPU alone. Where it broke. Each mobile chip has several heterogeneous processors, so using only one leaves most of the chip’s capability idle, wasting compute.
The gap. No framework trained deep learning models efficiently on a SoC-Cluster by both dodging the weak network and using the heterogeneous mobile processors fully, while preserving accuracy. Providing that is what SoCFlow does.

How SoCFlow works

From network-throttled, single-processor training to group-wise mixed precision

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Harvest the idle cycles of hardware you already run. SoCFlow’s premise is that the SoC-Cluster’s tidal underutilization is an opportunity: energy-efficient chips sitting idle off-peak are free capacity, and filling them with deep learning training extracts more useful work from hardware and power the operator already pays for. Turning the idle capacity a system leaves behind into productive work for another job is the same harvesting the idle-flash-bandwidth storage work covered elsewhere on this site performs, reclaiming a resource one tenant is not using so another can put it to work rather than letting it go to waste.

Synchronize less often to dodge the weak network. The reason data-parallel training struggles on a SoC-Cluster is that averaging gradients across all chips every step floods the weak interconnect. SoCFlow’s group-wise parallelism with delayed aggregation cuts that traffic by combining gradients frequently within small groups but only occasionally across groups, so the expensive global synchronization happens far less often. Performing an expensive synchronization only occasionally, rather than paying its full cost at every step, is the same discipline the lazy-storage-barrier work covered elsewhere on this site follows, issuing its costly ordering operation only when it is truly forced instead of once for every unit of work.

Run in mixed precision to use every processor on the chip. A mobile chip is not one accelerator but a bundle of heterogeneous processors that excel at different numeric precisions, and a trainer that uses only one of them leaves most of the chip idle. SoCFlow trains data-parallel in mixed precision so the CPU, GPU, and neural accelerator each contribute at the precision they handle best. Running the computation at reduced and varied precision to unlock the efficiency of the underlying hardware, where the training tolerates it, is the same bet the low-bit language-model serving work covered elsewhere on this site makes, dropping precision to exploit what the model can absorb and what the hardware does fastest.

Following a training job through SoCFlow

Training a model on an idle SoC-Cluster

A high-level trace of the approach as described in the abstract; the exact algorithms are beyond the abstract this page is based on.

Step 1 of 5

Place training on idle chips

A deep learning training job is placed on a SoC-Cluster whose mobile-app workloads are in an off-peak trough, so many of its mobile chips are idle and available.

The training runs in the tidal troughs when users are quiet.

Why it works

Idle infrastructure is worth harvesting. Because SoC-Cluster servers are busy only at peak and sit idle off-peak → their free cycles are wasted capacity on energy-efficient hardware → so co-locating training on them turns idle chips into useful work.
Step-by-step synchronization overwhelms a weak network. Because data-parallel training normally averages gradients across all workers every step → the weak network between mobile chips becomes the bottleneck → so a naive trainer stalls on communication and does not scale.
Grouping and delaying the aggregation cuts the traffic. Because SoCFlow averages gradients within small groups often and across groups only after a delay → far less data crosses the slow interconnect → so training scales without being throttled by communication.
Mixed precision unlocks the whole chip. Because a mobile chip has heterogeneous processors that excel at different precisions → training in mixed precision lets the CPU, GPU, and neural accelerator all contribute → so the chip’s full capability is used instead of a fraction.
Efficiency, not just speed, is the payoff. Because mobile chips are built for energy efficiency and SoCFlow keeps them all busy → the cluster matches a data-center GPU’s training speed at a fraction of the energy → so it trains models at several times less energy for the same accuracy.

What it costs (trade-offs and scope)

Delayed aggregation is an approximation. Synchronizing gradients across groups only occasionally departs from the strict every-step averaging of standard data parallelism. The paper reports it preserves convergence accuracy, but delaying synchronization is a change to the training dynamics that has to be managed to stay accurate.
Mixed precision must not hurt convergence. Using lower and varied numeric precision speeds the heterogeneous processors, but precision that is too low can harm a model’s final accuracy, so the mixed-precision scheme has to be chosen carefully to preserve it.
It targets the SoC-Cluster setting. SoCFlow’s techniques are shaped by a weak inter-chip network and heterogeneous mobile processors. Their benefit is specific to that architecture, not to a normal high-network training cluster.
The gains depend on there being idle cycles. Harvesting free cycles pays off because the servers are tidally underutilized. A SoC-Cluster kept busy by its user workloads has little idle capacity for training to fill.
It shares hardware with production workloads. Co-locating training on servers that also serve users means the two must coexist, and the training must yield when user demand returns, which is a scheduling concern beyond the training speed itself.
This page is written from the abstract. The full text was not available when this explainer was prepared, so the exact group-wise algorithm, the mixed-precision scheme, and the full evaluation are summarized only at the level the abstract states, without invented specifics.

The evaluation

Three ways to run deep learning training relative to a SoC-Cluster

Property	Naive training on SoC-Cluster	Commodity server GPU
Harvests idle edge capacity		No, separate hardware
Handles the weak inter-chip network		Not applicable
Uses the heterogeneous SoC processors		Not applicable
Convergence accuracy	Preserved	Preserved
Training speed and energy	Slow	Fast, high energy

Three ways to run deep learning training in relation to a SoC-Cluster edge server, as described in the paper's abstract. Naive data-parallel training on the SoC-Cluster is throttled by the weak inter-chip network and uses only part of each mobile chip. A commodity data-center GPU (an NVIDIA V100) trains fast but is separate hardware and draws much more power. SoCFlow trains on the SoC-Cluster itself, using group-wise parallelism with delayed aggregation to avoid the network bottleneck and a data-parallel mixed-precision algorithm to unleash each mobile chip's CPU, GPU, and neural accelerator. The abstract reports that SoCFlow outperforms all baselines on training speed while preserving convergence accuracy — a 1.6 to 740 times convergence speedup with 32 chips — and that, under the same power budget as an NVIDIA V100 GPU, it reaches comparable training speed while reducing energy consumption by 2.31 to 10.23 times at the same accuracy. These figures are from the abstract; the full paper was not available for this explainer.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

SoCFlow is a resourceful systems contribution that notices free capacity where others see a fixed-purpose server: SoC-Cluster edge machines, built from hundreds of energy-efficient mobile chips, sit idle much of the time because their user workloads are tidal, and that idle time can be spent training deep learning models. The two techniques are well matched to the hardware’s weaknesses. Group-wise parallelism with delayed aggregation avoids drowning the weak inter-chip network by synchronizing gradients mostly within small groups and only occasionally across them, and a data-parallel mixed-precision algorithm uses each mobile chip fully by letting its CPU, GPU, and neural accelerator all contribute at the precision they do best. The reported results are impressive: up to hundreds of times faster convergence on 32 chips than baselines, at the same accuracy, and — most strikingly — matching an NVIDIA V100 data-center GPU’s training speed under the same power budget while using two to ten times less energy. Read the specifics with care. The exact group-wise algorithm, how the mixed precision is assigned, and the full evaluation are in the paper and not reproduced here. As trade-offs, delaying synchronization and lowering precision both depart from standard training and must be managed to preserve accuracy, the benefit is specific to the SoC-Cluster setting and to there being idle cycles to harvest, and co-located training must coexist with the production workloads. The durable idea is that energy-efficient hardware sitting idle is a resource, and that adapting a demanding workload to a weak network and heterogeneous processors — rather than the reverse — can make that hardware competitive with a power-hungry GPU.

Background

Why matching the workload to the hardware's quirks unlocks efficiency

It helps to see SoCFlow within a broad theme in machine-learning systems: large efficiency gains come not from faster hardware but from reshaping a workload to fit the hardware’s real constraints and to reclaim capacity that would otherwise be wasted. Here the constraints are a weak network and small heterogeneous processors, and the wasted capacity is tidal idle time; SoCFlow adapts training to all three. The cooperative-scheduling work covered elsewhere on this site pursues the same instinct on a different machine, reshaping how work is scheduled to respect the processor’s heterogeneous structure and load so the hardware is used well rather than fought against. The common thread is that a system runs best when the software is designed around the specific shape of the machine — its network, its mix of processors, its load over time — instead of assuming a uniform, always-available, well-connected pool of compute. As computing spreads to the edge and onto diverse, energy-constrained hardware, this matters more: the machines are weaker and more idiosyncratic, but there are many of them and much of their capacity is idle. The general lesson is that harvesting that idle, heterogeneous, weakly-connected capacity — by co-designing the workload to its quirks — can turn hardware bought for one purpose into a surprisingly capable and energy-efficient platform for another.

If you remember only three things

1. A SoC-Cluster edge server packs hundreds of energy-efficient mobile system-on-chips — phone processors — into one machine, valued for running mobile apps at the edge. But its user-triggered workloads are tidal, busy at peak and idle otherwise, so the servers waste many free cycles. Filling them with deep learning training is attractive, but a SoC-Cluster is a poor fit: distributed training synchronizes gradients across all workers every step, which floods the weak inter-chip network, and each mobile chip has several small heterogeneous processors (CPU, GPU, neural accelerator) that a naive trainer leaves mostly idle.
2. SoCFlow, the first framework to train deep learning models efficiently on a SoC-Cluster, uses two techniques. Group-wise parallelism with delayed aggregation partitions the chips into groups, combines gradients frequently within a group but only occasionally across groups, so far less traffic crosses the weak network and training scales without being throttled by communication. A data-parallel mixed-precision training algorithm runs the computation at reduced and varied precision so each mobile chip's CPU, GPU, and neural accelerator all contribute at the precision they handle best, unleashing the whole chip. Together these preserve convergence accuracy while training fast.
3. The abstract reports that SoCFlow outperforms all baselines on training speed while preserving convergence accuracy, with a 1.6 to 740 times convergence speedup using 32 chips, and that under the same power budget as an NVIDIA V100 data-center GPU it reaches comparable training speed while using 2.31 to 10.23 times less energy at the same accuracy. Its scope: the techniques target the SoC-Cluster setting (weak network, heterogeneous chips), the gains depend on there being idle cycles to harvest, delaying synchronization and lowering precision must be managed to keep accuracy, and training must coexist with production workloads. Note: this explainer is written from the abstract, as the full text was not available, so the exact algorithms and full evaluation are summarized only at the level the abstract states.

Check yourself

Check your understanding

1. Why does a SoC-Cluster edge server have free cycles worth harvesting?
2. Why does standard data-parallel training struggle on a SoC-Cluster?
3. What are SoCFlow's two techniques, and what does each address?
4. The most striking result is against an NVIDIA V100 GPU. What does it show, and why does it matter?

Sources

The paper is “SoCFlow: Efficient and Scalable DNN Training on SoC-Clustered Edge Servers” by Daliang Xu, Mengwei Xu, Chiheng Lou, Li Zhang, Gang Huang, Xin Jin, and Xuanzhe Liu, published at ASPLOS 2024, DOI 10.1145/3617232.3624847. The authors are at Peking University and the Beijing University of Posts and Telecommunications.

This explainer is written from the paper’s abstract, because the full text was not available at the time of writing: the publisher’s copy was behind automated-access protection, the paper’s open-access status was closed, and no preprint or separately hosted author copy could be retrieved. The problem statement (the SoC-Cluster architecture of massive mobile system-on-chips, its energy efficiency and mobile-application compatibility, and its underutilization due to user-triggered workloads with significant tidal phenomena, motivating the co-location of deep learning tasks to harvest free cycles) and the description of SoCFlow (the first framework to efficiently train deep learning models on a SoC-Cluster, using group-wise parallelism with delayed aggregation to train fast and scalably without being influenced by the network bottleneck, and a data-parallel mixed-precision training algorithm to fully unleash the heterogeneous processors of the mobile SoCs) come directly from the abstract, as do the results (outperforming all baselines on training speed while preserving convergence accuracy, a 1.6 to 740 times convergence speedup with 32 SoCs, and comparable training speed to an NVIDIA V100 GPU under the same power budget while reducing energy consumption by 2.31 to 10.23 times at the same convergence accuracy). The surrounding explanation of mobile systems-on-chips, distributed data-parallel training and gradient synchronization, network bottlenecks, heterogeneous processors, and mixed-precision training is standard, widely-known background provided to make the abstract’s claims understandable, and is not specific to this paper. The exact group-wise algorithm, the mixed-precision scheme, and the detailed evaluation are not reproduced here because they were not available.