Frugal: Efficient and Economic Embedding Model Training with Commodity GPUs

Embedding models — the huge lookup tables that turn user, item, or node IDs into vectors for recommendation and graph learning — need lots of memory but little compute, so cheap commodity GPUs ought to be perfect, except those cards have weak interconnects and, lacking PCIe peer-to-peer, must bounce every GPU-to-GPU transfer through host memory; this paper (Tsinghua — Minhui Xie, Youyou Lu, and colleagues) builds Frugal, which has each GPU proactively flush the parameters others will need into host memory ahead of time, moving half the communication off the critical path, and prioritizes those flushes with a two-level priority queue, matching datacenter-GPU throughput at 4.0 to 4.3 times better cost-effectiveness.

The argument in one glance

Problem

Cheap GPUs look perfect for embedding models, until they have to talk to each other

Embedding models are the workhorses of sparse learning: they turn massive collections of ID-type features — a user ID, an item ID, a graph node or edge ID — into learned vectors, and they power recommendation systems and graph learning. The defining trait of these models is that they are memory-hungry but compute-light: the embedding tables are enormous, but the arithmetic per training step is modest. That profile makes commodity GPUs — the inexpensive consumer-grade cards — look ideal, because you are paying for cheap compute you do not need much of, rather than for the premium compute of a datacenter GPU. But there is a catch that shows up the moment training is spread across several of these cards: the model's embedding parameters are split among the GPUs, so every step they must constantly exchange parameters with one another, and commodity GPUs are bad at exactly that.
What people did

Just run the standard multi-GPU embedding training on commodity cards

The obvious thing is to take the training approach that works on datacenter GPUs — shard the embedding tables across the GPUs and, each step, have every GPU gather the embedding rows it needs from whichever GPU owns them — and run it unchanged on a box of commodity GPUs, trusting the interconnect to carry the traffic.
Why it fell short

Commodity interconnects are weak and cannot move data GPU-to-GPU directly

Commodity GPUs have deficient communication resources, in two ways. Their interconnect bandwidth is low to begin with, and, crucially, they lack PCIe peer-to-peer support — the ability for one GPU to read or write another GPU's memory directly over the bus. Without it, a transfer from GPU A to GPU B cannot go straight across; it must be bounced through host memory, copied first from GPU A into system RAM and then from system RAM into GPU B. That is two hops instead of one for every cross-GPU exchange, and since embedding training does a great deal of this exchange, the doubled communication cost dominates and training crawls.
This paper's idea

Flush the data others will need ahead of time, and prioritize the flushes

Frugal is an embedding-training system built for commodity GPUs, and its key move follows from the bounce. Since every cross-GPU transfer is really two legs — GPU-to-host, then host-to-GPU — Frugal makes each GPU proactively flush its own parameters that other GPUs will later access into host memory ahead of time, off the critical path. When another GPU needs those parameters, the slow first leg is already done, so only the host-to-GPU leg remains on the critical path, decoupling half the communication overhead. Because this proactive flushing competes with the foreground training for the same scarce bandwidth, Frugal assigns each flush a priority — flushing first the parameters that GPUs are about to access and deferring the rest — and it uses a tailored two-level priority queue so this prioritization scales. The result is that a box of cheap GPUs matches datacenter-GPU throughput at 4.0 to 4.3 times better cost-effectiveness.

Why this problem is worth solving

The appeal is an unusually good match between a workload and cheap hardware, wasted by one weakness.

An embedding model handles ID-type features — discrete identifiers like a user, an item, or a graph node — by keeping a large table that maps each ID to a learned vector, its embedding. In recommendation and graph-learning settings there are enormous numbers of such IDs, so the tables are huge, often far larger than the rest of the model. But looking up a few rows and combining them is cheap arithmetic, so an embedding model spends most of its resource budget on memory capacity and bandwidth, not on raw floating-point throughput. This is the opposite of a dense compute-bound model. Commodity GPUs — consumer-grade cards — are prized for delivering a lot of compute per dollar, and while their peak compute may trail datacenter GPUs, that hardly matters for a workload that barely uses the compute in the first place. So on paper, running memory-intensive, compute-light embedding models on cheap GPUs is a near-ideal pairing: you get the memory you need without paying for premium compute you would leave idle. Realizing that pairing would make training these models substantially cheaper, which is exactly why the one obstacle standing in the way — communication — is worth attacking rather than conceding.

That obstacle is specific and structural, not a matter of tuning.

In a multi-GPU training job the embedding table is partitioned, so each GPU owns a shard and, every step, each GPU must fetch the embedding rows it needs for its current batch from whatever GPUs own them — a constant, all-to-all exchange of parameters. On datacenter GPUs this is fast because the hardware supports direct GPU-to-GPU transfer: with PCIe peer-to-peer (or a dedicated link like NVLink), one GPU can read or write another GPU’s memory straight across the interconnect in a single hop. Commodity GPUs lack this. Without peer-to-peer support, the GPUs cannot address each other’s memory over the bus, so a transfer from GPU A to GPU B has to be staged through the host: first a copy from GPU A’s memory into pinned host memory, then a second copy from host memory into GPU B. Every cross-GPU exchange is thus two transfers over an already-low-bandwidth bus instead of one direct transfer, and because embedding training is communication-heavy — the exchange happens for every batch — this doubled, serialized cost becomes the bottleneck that makes naive commodity-GPU training slow. The problem is baked into the hardware’s capabilities, so the fix has to be a smarter way to schedule the unavoidable two-hop traffic, not a faster link.

What people did before, and where it broke

Train embedding models on datacenter GPUs. Use high-end cards with fast direct GPU-to-GPU links (peer-to-peer or NVLink) so the constant embedding exchange is cheap. Where it broke. It works well but is expensive, paying a large premium for compute the memory-bound embedding model barely uses, which is the cost inefficiency Frugal aims to remove.
Run standard sharded embedding training unchanged on commodity GPUs. Take the datacenter approach and drop it onto cheap cards. Where it broke. Commodity GPUs lack PCIe peer-to-peer and have low bandwidth, so every GPU-to-GPU transfer must bounce through host memory as two hops; the doubled communication dominates and throughput collapses.
Rely on the interconnect and overlap what you can reactively. Issue the cross-GPU fetches when needed and hope communication hides behind computation. Where it broke. Embedding models have little computation to hide communication behind, and the two-hop bounce is fully on the critical path when a parameter is needed, so there is not enough compute to mask it.
Naively flush everything to host memory in advance. Push parameters to host memory ahead of time without regard to order. Where it broke. The proactive flushing itself consumes the same scarce bandwidth as foreground training, so flushing indiscriminately contends with and slows the very training it means to help.
The gap. No system restructured the unavoidable two-hop traffic so that the slow first leg is done ahead of time and off the critical path, while scheduling that background work by priority so it does not starve the foreground training on a bandwidth-starved commodity interconnect.

How Frugal works

Turning a forced two-hop bounce into one critical-path hop through proactive, prioritized flushing

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Move the unavoidable slow half of the transfer out of the moment you need it. The whole design turns on reframing the forced two-hop bounce. A commodity GPU with no peer-to-peer support cannot avoid staging cross-GPU data through host memory, so each transfer is a GPU-to-host copy followed by a host-to-GPU copy. The naive approach pays for both at the instant the data is requested, putting two slow copies squarely on the critical path. Frugal’s insight is that the first leg does not have to wait for the request: a GPU knows which of its own parameters other GPUs will eventually want, so it can flush those into host memory proactively, in the background, before anyone asks. Then, when the request comes, the parameter is already in host memory and only the second leg has to happen synchronously. This does not make the interconnect faster or eliminate a copy; it relocates half the cost to a non-critical path where it overlaps with other work, which on a communication-bound workload is the difference that matters. Taking a cost that sits on the critical path and shifting it earlier so it overlaps with computation rather than blocking it is the same principle the work on overlapping communication with dependent computation covered elsewhere on this site is built on, decoupling a transfer from the point of use so it stops being a serial bottleneck.

Embrace the cheap hardware and engineer around its one real weakness. Frugal is, at heart, a bet that commodity GPUs are the right hardware for embedding models despite their reputation, provided you neutralize their specific deficiency. The pairing is genuinely apt — memory-intensive, compute-light models on cards that are cheap precisely because they skimp on the compute you would not use — and the only thing standing in the way is the weak, peer-to-peer-less interconnect. So rather than concede to expensive datacenter GPUs, Frugal accepts the commodity cards as they are and builds the software to make their one liability tolerable, converting a hardware limitation into a scheduling problem. The payoff is measured in cost-effectiveness: matching datacenter-GPU throughput at a several-fold lower cost, which is the whole point of choosing frugal hardware. Insisting that inexpensive, ostensibly ill-suited commodity GPUs can serve a demanding training workload if the system compensates for their interconnect is the same conviction the commodity-GPU-server training work covered elsewhere on this site is built on, extracting serious training performance from affordable hardware by working around, rather than around-buying, its limits.

Schedule the background work by priority, or it eats the foreground. Proactive flushing is only free if it does not steal bandwidth the training needs right now, and on a bandwidth-starved commodity interconnect that is a real danger, because the flushes and the foreground fetches draw from the same narrow pipe. Flushing everything indiscriminately would have the background optimization contending with and slowing the very training it exists to speed up. Frugal’s answer is to make flushing discriminating: it assigns each flush a priority reflecting how soon the parameter will be needed, pushes the imminently-needed parameters first, and defers the rest, so the limited bandwidth is spent where it pays off soonest. And because managing priorities over many operations can itself become a scalability bottleneck, Frugal tailors a two-level priority queue to keep those priority operations efficient as the system grows. Carefully arbitrating a scarce, shared interconnect so the most valuable transfers go first rather than treating all traffic alike is the same discipline the work on discovering better communication paths for PCIe GPU clusters covered elsewhere on this site pursues, wringing performance out of a constrained PCIe fabric by being deliberate about what moves and when.

Following a training step through Frugal

Fetching a needed embedding parameter without paying the full two-hop cost

An illustrative trace, based on the paper's abstract, of how Frugal serves a cross-GPU parameter request. Internal details beyond the abstract were not available for this draft.

Step 1 of 5

Embedding parameters are sharded across commodity GPUs

Each GPU owns a shard of the embedding table. During a training step, a GPU processing its batch will need embedding rows that live on other GPUs, and those other GPUs will likewise need rows this GPU owns. With no peer-to-peer support, any such transfer must route through host memory.

This all-to-all-style parameter exchange happens every step, so its cost dominates on a weak interconnect.

Why it works

Prefetching the slow leg removes it from the critical path. Each cross-GPU transfer is two serial copies through host memory → but proactively flushing the GPU-to-host copy in advance leaves only the host-to-GPU copy at the moment of need → so each fetch is effectively one hop, halving the communication on the critical path.
Proactive work overlaps with, rather than blocks, training. A reactive two-hop bounce fully stalls a request → but doing the first leg in the background overlaps it with other work → so communication that a compute-light model cannot hide behind computation is instead hidden in time.
Priority spends scarce bandwidth where it pays soonest. Flushing everything indiscriminately contends with foreground fetches → but prioritizing imminently-needed parameters and deferring the rest → so the limited interconnect serves the most valuable transfers first and the background work does not starve training.
A two-level priority queue keeps prioritization scalable. Managing priorities over many flush operations can itself become a bottleneck → but a tailored two-level priority queue handles the priority operations efficiently → so the benefit holds as the system and operation count grow.
The approach matches the hardware to the workload economically. Datacenter GPUs overpay for compute an embedding model barely uses → but making commodity GPUs viable removes that premium → so Frugal reaches datacenter-level throughput at 4.0 to 4.3 times better cost-effectiveness.

What it costs (trade-offs and scope)

This is a draft built from the paper’s abstract. The full paper was behind a publisher access wall for this draft (no arXiv version, no open-access or author copy located, and no web-archive snapshot), so this account is assembled from the paper’s detailed abstract. The exact flushing mechanism, the priority policy, the two-level priority queue design, the system architecture, the datasets and baselines, and the precise per-experiment numbers are not reproduced here and should be read from the paper.
The reported figures are the abstract’s own. The claim of matching datacenter-GPU throughput with a 4.0 to 4.3 times cost-effectiveness improvement is the abstract’s reported result; the specific models, hardware, and conditions behind that range are not detailed in this draft.
Proactive flushing needs to predict what others will access. The benefit depends on flushing the right parameters before they are requested, which requires knowing or estimating each GPU’s upcoming accesses; how accurately Frugal does this, and what happens on a miss (a parameter needed but not yet flushed, forcing the full two-hop bounce), is exactly the internal detail the abstract does not spell out.
It consumes host memory and host-bus bandwidth. Staging parameters through host memory uses system RAM and the host-side bus for the flushes; the design assumes this capacity and bandwidth are available and that the flushing, even prioritized, stays within budget, which may bound how far the approach scales.
Scope is memory-intensive, compute-light embedding models. The whole premise is that commodity GPUs suit workloads that need memory more than compute; models that are compute-heavy would not benefit from cheap GPUs the same way, so the approach targets sparse-learning embedding training specifically, not deep-network training in general.

The evaluation

Training embedding models across multiple GPUs

Property	Datacenter GPUs	Naive commodity GPUs	Frugal on commodity GPUs
GPU-to-GPU transfer
Communication on critical path
Background flush vs training
Hardware cost
Reported outcome (per abstract)

A qualitative comparison based on the paper's abstract; the paper's main body, including its exact flushing mechanism, priority policy, two-level priority queue, datasets, and baselines, was not available for this draft. The 4.0-to-4.3-times cost-effectiveness figure is the abstract's reported result.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This is an honest draft assembled from the paper’s detailed abstract; the full text was not accessible (no arXiv preprint, no open-access or author copy found, and the ACM Digital Library version returned a bot-verification challenge), so the internal design is not reproduced. What is solid and comes directly from the abstract: embedding models are memory-intensive and compute-light, making commodity GPUs attractive; those GPUs have low interconnect bandwidth and no PCIe peer-to-peer support, so cross-GPU transfers must bounce through host memory as two hops; Frugal’s key idea is proactive flushing — each GPU flushes parameters others will access into host memory ahead of time, decoupling half the communication to non-critical paths; Frugal prioritizes flushes to avoid contending with foreground training and uses a two-level priority queue for scalability; and it reports matching datacenter-GPU throughput with 4.0 to 4.3 times better cost-effectiveness. The explanation of sharded embedding training and the two-hop host-memory bounce is standard background and the natural reading of the abstract, not a claim about Frugal’s specific internals. Read the exact flushing and prediction mechanism, the priority policy, the queue design, and the evaluation in the paper itself; treat this page as a faithful account at the level of the abstract, not of the implementation.

Background

Why memory-bound models and cheap GPUs are a natural match

It helps to see why this pairing is worth so much effort. GPUs are usually chosen for their floating-point throughput, and the premium ones cost so much because they push that throughput to the limit. But not every important workload is compute-bound. Embedding models — the backbone of recommendation and graph learning — spend their budget on holding and moving enormous lookup tables, doing only light arithmetic on the rows they fetch. For such a workload, paying for a datacenter GPU’s peak compute is paying for a capability that sits mostly idle, while what you actually need is memory capacity and the ability to move data around, which cheaper cards can supply at a fraction of the price. The economic logic of matching a memory-bound workload to memory-oriented, inexpensive hardware, rather than defaulting to the most powerful compute, is compelling — and it is the same logic behind the storage-and-caching work for scaling embedding tables covered elsewhere on this site, which treats the embedding table’s size and access as the first-class problem to design the system around rather than the model’s compute. Frugal’s contribution is to remove the one thing that spoils the match on commodity GPUs, their inability to move data efficiently between each other, so the natural fit can actually be realized.

If you remember only three things

1. Embedding models turn massive ID-type features (user/item IDs in recommendation, node/edge IDs in graph learning) into learned vectors; they are memory-intensive but compute-light, so cheap commodity GPUs are a natural fit because you avoid paying for premium compute the model barely uses. But training them across multiple commodity GPUs performs poorly, because those cards have low interconnect bandwidth and no PCIe peer-to-peer support — so every cross-GPU transfer of the sharded embedding parameters must be bounced through host memory as two hops (GPU-to-host, then host-to-GPU) instead of one direct hop, and that doubled, critical-path communication dominates.
2. Frugal's key idea is proactive flushing: since the two-hop bounce is unavoidable, each GPU flushes its own parameters that other GPUs will access into host memory ahead of time, off the critical path, so when a parameter is requested only the second (host-to-GPU) leg remains — decoupling half the communication overhead. Because this background flushing shares bandwidth with foreground training, Frugal assigns each flush a priority (flushing imminently-needed parameters first, deferring the rest) and uses a tailored two-level priority queue so the prioritization scales.
3. Applied to recommendation and graph-embedding training, Frugal greatly increases commodity-GPU throughput and reaches datacenter-GPU-level throughput at 4.0 to 4.3 times better cost-effectiveness. This explainer is a draft written from the abstract without the paper body, so the exact flushing and access-prediction mechanism, the priority policy, the two-level priority queue design, the datasets, and the baselines should be confirmed against the paper.

Check yourself

Check your understanding

1. Why are commodity GPUs an attractive but troublesome choice for training embedding models?
2. What is Frugal's key idea, "proactive flushing"?
3. Why does Frugal need to prioritize its flush operations, and how does it keep that scalable?
4. What should a reader keep in mind about this explainer, and what did the paper report?

Sources

The paper is “Frugal: Efficient and Economic Embedding Model Training with Commodity GPUs” by Minhui Xie, Shaoxun Zeng, Hao Guo, Shiwei Gao, and Youyou Lu (Tsinghua University), published at ASPLOS 2025, DOI 10.1145/3669940.3707245.

This explainer is a draft. The paper’s main text was not accessible at the time of writing (there is no arXiv preprint, no open-access or author copy was located, and the ACM Digital Library PDF returned a bot-verification challenge), so it is written from the paper’s detailed abstract together with standard background on multi-GPU embedding-model training. The abstract supplies the problem framing (embedding models learn representations of massive ID-type features in sparse learning such as recommendation and graph learning; they are memory-intensive and compute-light, suiting cost-efficient commodity GPUs; but commodity GPUs have low communication bandwidth and no PCIe peer-to-peer support, so cross-GPU communication must be bounced through host memory), the key idea (proactive flushing, in which each GPU proactively flushes its own parameters that other GPUs will access into host memory, decoupling half of the communication overhead to non-critical paths), the priority mechanism (assigning priorities to flush operations, prioritizing parameters GPUs will access while deferring others, to alleviate contention with foreground training), the scalability mechanism (a tailored two-level priority queue for operations involving priorities), the application domains (recommendation models and graph embedding), and the reported result (significantly increasing training throughput on commodity GPUs and achieving similar throughput to existing systems on datacenter GPUs, with a 4.0 to 4.3 times improvement in cost-effectiveness). The explanation of sharded embedding training and the two-hop host-memory bounce is standard background and the natural reading of the abstract, not a claim about Frugal’s specific internal design. The paper’s exact flushing and access-prediction mechanism, the priority policy, the two-level priority queue design, the system architecture, the datasets, and the baselines are not reproduced here and should be read from the paper; no numbers beyond the abstract’s have been introduced.