RAP: Resource-aware Automated GPU Sharing for Multi-GPU Recommendation Model Training and Input Preprocessing

Recommendation models are retrained continuously on data that arrives every second, and turning that raw data into training tensors is done by fleets of dedicated processors whose power draw has been reported as exceeding that of the machines doing the actual training; RAP (UC San Diego, Amazon and Pacific Northwest) observes that the training accelerators are periodically idle — because these models alternate between compute-heavy and memory-heavy layers — and fills those gaps with the preprocessing work, planning the whole schedule offline so the two never collide.

Why this problem is worth solving

The shape of the model is what creates both the opportunity and the difficulty.

The thing being harvested needs a definition before it can be scheduled.

What people did before, and where it broke

• Run preparation on dedicated processor nodes. The production answer, and the only framework supporting this style of continuous preparation is built for it. Where it broke. Cost and power. It also does not scale with the training: once the preparation processors are saturated, adding accelerators to the training side buys nothing, because the pipeline stalls waiting for data.
• Optimise the training and assume the data is ready. Most of the recent work on these models does this — caching hot embeddings, sharding tables cleverly, improving distributed training. Where it broke. The assumption. It holds for offline training on a prepared dataset and not for the continuous retraining that production actually runs.
• Offload preparation to specialised hardware. Programmable logic devices have been used for this. Where it broke. It needs someone with the expertise to hand-optimise each preparation operation for that hardware, which is a high bar for a set of operations that changes as features change.
• Use the accelerators, but run preparation and training in sequence. A vendor library does exactly this for image and video pipelines. Where it broke. Sequential execution means the preparation time is simply added to the training time. The resource is shared but the latency is not hidden.
• Share the accelerator with the standard mechanisms. Separate streams with different priorities, or the vendor’s multi-process service, both of which let two workloads’ kernels overlap. Where it broke. Neither knows anything about how much room is actually available at each moment. They issue preparation kernels whenever they are ready, which produces contention with the training and inflates its latency — measured here as leaving 30 to 50 percent of the achievable throughput on the table.
• The gap. Everyone treating this as a generic accelerator-sharing problem missed two things specific to it: the leftover capacity here is not random noise but a repeating pattern created by the model’s own alternating structure, and the preparation work is not an unrelated tenant but the supplier of the very data the training is about to consume, which means where it runs determines whether anything has to be sent over the interconnect.

How leftover capacity becomes a schedule

The idea

The idle time is structured, and the structure comes from the model itself. A recommendation model is two very different computations bolted together — dense layers that saturate the arithmetic units, and embedding lookups that saturate memory bandwidth while leaving the arithmetic units largely idle. Because training marches through these phases in the same order every iteration, utilisation rises and falls in a repeating pattern that can be measured once and relied on. That is what makes offline planning possible at all: the gaps are not stochastic, they are a property of the model’s architecture, so a schedule computed ahead of time remains valid. It is worth noticing that the same imbalance shows up as a problem in other work and is attacked from the opposite direction — the hyperscale architecture search covered elsewhere on this site found the dense layers taking much longer than the embedding layers in a production model and redesigned the model to rebalance them, where this paper leaves the imbalance alone and fills the resulting hole with something useful.

Two unlike quantities turn out to measure the same thing, which makes the whole problem tractable. The central modelling difficulty is that leftover capacity and preparation cost are expressed in incompatible units — one is an amount of unused parallelism spread over an interval, the other is a kernel with a shape and a size. The observation that resolves it is geometric: leftover capacity is the area above the utilisation curve during a training operation, and a preparation kernel run on its own occupies the device fully, so its runtime is also an area on that same graph. Standalone runtime therefore becomes the common currency, and the paper validates this rather than asserting it, showing that different operation types follow a consistent relationship between standalone and overlapped latency while a more obvious candidate metric, thread-group count, does not. What remains is predicting that runtime cheaply, which a small model trained offline does to within ten percent for the large majority of kernels. Replacing measurement in an inner search loop with a learned surrogate cheap enough to call thousands of times is the same manoeuvre the Bayesian autotuner covered elsewhere on this site makes to avoid paying a compile-and-run for every candidate.

Kernel fusion here goes the unusual way, and then has to be undone in part. Fusion normally means chaining operations vertically so an intermediate result stays in registers instead of going back to memory. That is not the problem with these preparation operations. They are defined per input feature and are simply too small — dozens or hundreds of tiny kernels, each spending more time being launched than computing. So independent operations of the same type are merged horizontally, side by side, into one kernel with more threads. Choosing which to merge is constrained by type and by dependencies and the choices conflict with one another, so it is posed as an integer program. The awkward part, which the paper is open about, is that this program maximises merging with no notion of a resource budget, so its answer is often a kernel too large for the gap it needs to fit into — and a separate pass then splits it back down until the predicted runtime fits. Reshaping the same computation to match whatever capacity is currently available, rather than committing to one shape at compile time, is the same response to co-location that the multi-tenant compiler covered elsewhere on this site takes by keeping a family of code versions and choosing among them at runtime.

Planning one training iteration

The measurement is what separates this from generic sharing mechanisms, which have no per-phase notion of how much room exists and simply issue work whenever it is ready.

Why it works

• The gaps repeat, so they can be planned rather than reacted to. The alternation between dense and embedding phases happens identically every iteration → so a profile taken once stays accurate for as long as the model and input distribution hold → which means the entire schedule can be computed offline and compiled into code, avoiding the runtime overhead that preemption-based sharing pays.
• Treating both quantities as areas makes them comparable. Leftover capacity and kernel cost have no obvious common unit → but both are areas on a utilisation-against-time graph, since a kernel running alone occupies the device fully → so standalone runtime can stand in for capacity consumed, which is the substitution that makes a cost model possible at all.
• A learned predictor is what makes the search affordable. The joint placement and scheduling problem has an enormous space of candidate plans → measuring each on hardware would take far longer than the training it is meant to accelerate → a model trained offline on roughly 11,000 measurements answers in negligible time and lands within ten percent for 92.9 to 98.5 percent of kernels depending on operation type.
• Merging sideways addresses the actual defect in these kernels. The usual reason to fuse is to keep intermediates out of memory → but these operations are element-wise and lightweight, so their problem is launch overhead and idle hardware rather than memory traffic → widening one kernel to do many features’ work at once fixes the right thing, and the measured utilisation at the point where latency starts to rise goes from 77.6 percent to 92.8 percent.
• Neither extreme placement is right, so the search starts at one and walks toward the other. Splitting preparation by batch matches how the dense layers are parallelised but forces sparse features across the interconnect → splitting by which device owns the data eliminates that traffic entirely but leaves the load lopsided → starting from the zero-traffic placement and trading traffic for balance only when the cost model approves gets 4.3 and 4.0 times lower exposed latency than either heuristic alone.

What it costs (trade-offs and scope)

• The motivating numbers are cited, not measured. The claims that a training job needs many dedicated preparation nodes, and that storage and preparation exceed half of a datacentre’s power, come from a published industrial analysis by another group. This paper’s own evaluation is a single eight-accelerator node on public advertising datasets.
• The headline comparison spans different hardware. The 17.8 times figure is eight A100s against a processor-based framework running eight workers per accelerator on two 64-core hosts, and the paper notes that framework saturates its processors and then stops improving. That is a real limitation of the baseline, but it also means the number partly measures a processor deliberately not scaled out — which is exactly what the industrial deployments avoid by adding hundreds of nodes.
• Half the workloads are synthetic. Two of the four preparation plans follow the framework’s published default. The other two, which are the large ones at 384 and 1,548 operations, were generated by randomly applying operations to more features.
• The optimiser optimises the wrong objective and is then corrected. The integer program maximises merging with no resource constraint, producing kernels that frequently do not fit, which a separate splitting pass then undoes. It works, but it means the globally optimal fusion plan being solved for is not the plan that gets run.
• Neither optimisation is strong alone. Against the multi-process baseline, removing the placement optimisation still gives 1.19 times and removing the fusion gives 1.15 times, against 1.43 for the full system. The argument is that they are complementary, and the individual contributions are modest.
• The plan assumes a stable input distribution. Everything is decided offline. Drift is handled by regenerating the plan, argued to take a few minutes against a drift interval of days to months — an argument the paper makes rather than an experiment it runs.
• The prediction accuracy metric is loose. What is reported is the fraction of samples whose predicted runtime falls within ten percent of the measured one. That fraction is high, between 92.9 and 98.5 percent, but ten percent is not tight and nothing is said about how wrong the remainder are.
• Accelerator memory is not discussed. Industrial embedding tables run to terabytes and memory is the binding constraint in most work on these models. Adding preparation buffers and staged input to devices that are already memory-starved does not appear in the analysis, and the datasets used here are far smaller than the industrial ones motivating the paper.
• The near-ideal result holds because the work happened to fit. Ending within 3.24 percent of a hypothetical no-preparation baseline depends on the preparation fitting inside the leftover, which the authors say held even for their heaviest plan. For cases where it does not, they sketch a hybrid using processors as well, as future work.
• The system it would actually displace is never compared against. The evaluation is against one-node alternatives, not against a disaggregated preparation fleet of the kind the motivation describes.

The evaluation

Check yourself

Sources

The paper is “RAP: Resource-aware Automated GPU Sharing for Multi-GPU Recommendation Model Training and Input Preprocessing” by Zheng Wang, Yuke Wang, Jiaqi Deng, Da Zheng, Ang Li and Yufei Ding, published at ASPLOS 2024, pages 964-979, DOI 10.1145/3620665.3640406. It is open access.

This explainer is written from the full 16-page text. Everything above comes from the paper’s own sections: the background on the three-tier industrial pipeline of storage, preparation and training nodes with the cited figure for their power share, the hybrid parallelism that replicates dense layers and partitions embedding tables and the asymmetric data consumption it produces, and the table of common preparation operations across normalisation, feature generation and utility categories; the motivation section laying out the three design spaces of graph placement, co-running schedule and their joint optimisation, with the analysis of why batch-wise placement causes communication and dependency-wise placement causes imbalance; the cost model section covering the latency-based overhead abstraction with its geometric argument and its validation across three operation types, the demonstration that thread-group count fails as a metric, the gradient-boosted latency predictor with its offline training, and the formula for exposed preparation latency; the horizontal fusion section covering the distinction from vertical fusion, the type and dependency constraints, the conflict between fusion opportunities, the integer programming formulation with its correctness and dependency constraints and its sum-of-squares objective, the resource-aware sharding fallback, and the inter-batch interleaving of host-side data preparation with device-side kernels; the heuristic search combining intra-device scheduling with inter-device placement in four steps from a locality-based initial mapping; and the evaluation covering the baselines and their configurations, the two datasets and four preparation plans with their operation counts, the platform and software versions, the end-to-end results across device counts, the breakdown and optimality analysis, the latency turning-point study with its utilisation table, the placement adaptability study on a skewed graph, and the predictor accuracy table.