ElasticFlow: An Elastic Serverless Training Platform for Distributed Deep Learning

Training platforms make developers guess a GPU count and hold it until the job ends — ElasticFlow asks only for the model and a deadline, admits jobs by the minimum GPU share that provably meets each deadline, and spends leftover GPUs where diminishing returns bite least, meeting 1.5–7.7× more deadlines on a 128-GPU cluster.

Why this problem is worth solving

Training a deep neural network on a cluster today follows the server-centric model: the developer requests instances, wraps code in containers, picks a worker count, and divides the global batch size by it to fit each GPU’s memory. Two different kinds of expertise collide in one person. Tuning the global batch size and learning rate is a machine-learning problem; choosing worker counts and local batch sizes is a systems problem — and most DL developers are not systems engineers.

The second gap is guarantees. Production pipelines retrain models on schedules — the paper’s example is fine-tuning BERT with daily news so a recommendation service updates every day. What the developer actually wants to express is a deadline. No server-centric platform can promise one, because a fixed-size reservation cannot adapt when the cluster gets busy or idle.

What makes deadline scheduling for training genuinely hard is the shape of the scaling curve. Distributed data-parallel training synchronizes gradients among workers every iteration, so throughput grows concavely with GPU count — each added GPU helps less than the last, and per-GPU efficiency falls. Worse, the curve depends on placement: eight workers inside one server ride NVLink; spread across four servers they ride slower inter-server links, yielding a different curve entirely. A deadline scheduler must reason over a family of placement-dependent, non-linear curves per job. (For the inference-side twin of this problem — serverless functions with latency SLOs — see the INFless explainer elsewhere on this site.)

What people did before, and where it broke

Generic cluster managers (Kubernetes, YARN) treat DL jobs as opaque containers — no elasticity, no deadline awareness, low utilization.

DL-specialized schedulers each solved a slice. Gandiva introspects and migrates but keeps the user-specified GPU count. Tiresias reduces completion time with clever queueing — also non-elastic. Themis adds finish-time fairness. Pollux is genuinely elastic, adapting GPU counts to co-optimize throughput and statistical efficiency — but it optimizes average completion time, not promises. Chronus is deadline-aware — it prioritizes SLO jobs — but not elastic, so it cannot exploit idle GPUs to bank time against future load. Earliest-Deadline-First (EDF), the classic real-time policy, fails differently: with non-linear scaling, giving the most urgent job all the GPUs it can use wastes enormous GPU-time (diminishing returns) that later jobs needed — the paper’s motivating example shows EDF missing deadlines an optimal schedule meets comfortably.

The missing combination — elastic and deadline-guaranteed — is exactly the pairing that requires solving the concave-curve admission problem, which is ElasticFlow’s core contribution.

The idea

Make the interface say what users mean. Users mean “train this model, done by Friday.” The serverless interface captures exactly that, and its omissions are the design: by not letting users pin a GPU count, the platform gains the freedom to rescale jobs at any moment — the freedom every guarantee below depends on.

Measure jobs in GPU-time, not GPUs. With concave curves, one GPU is always the most efficient allocation (resource usage = GPUs × time is minimized), but deadlines forbid pure efficiency. The

Admission control is a feature, not a failure. Dropping a job at submission — while it can still be run elsewhere or renegotiated — is strictly kinder than accepting it and silently missing the deadline. Every admitted job’s deadline is guaranteed by construction; EDF-style optimistic acceptance is where the 7.65× gap comes from.

Spend the surplus where curves are steepest. Leftover GPUs are handed out one at a time to the job whose throughput gains most from one more GPU. Under concave curves this greedy rule is provably optimal. The effect compounds: accelerated jobs finish early, GPUs free up early, and future arrivals find a roomier cluster — elasticity converts idle capacity into future admissions. Best-effort and soft-deadline jobs slot into the same framework with infinite deadlines, fed after guaranteed shares.

Erase placement as a variable. Because scaling curves depend on topology, placement threatens to entangle everything. ElasticFlow cuts the knot: power-of-two worker counts plus migration implement

job = {model, hyperparams, max_iters, deadline}
GPU count / duration: platform-owned

Why it works

• The interface withholds GPU counts from users → every job is rescalable at any instant → the scheduler can promise outcomes because it, not the user, owns the means.
• Concave curves make GPU-time the real currency → MSS spends the minimum GPU-time each deadline permits → guaranteed jobs occupy as little of the future as possible, maximizing admissions.
• Progressive filling accounts for already-promised slots → MSS is computed against reality, including time-varying shares → admission decisions are safe, not optimistic.
• Marginal-return-greedy surplus allocation is optimal for concave curves → surplus turns into early finishes → early finishes turn into future capacity, a feedback loop static schedulers never get.
• Power-of-two sizing + migration = buddy allocation over the GPU hierarchy → fragmentation is impossible → placement-dependent curves collapse to one curve per size, keeping admission tractable.

What it costs

• Dropped jobs. Guarantees are bought by refusing unguaranteeable work; users must handle rejection (and the paper leaves anti-gaming quotas and pricing to operators).
• Power-of-two rigidity. Worker counts of 3, 5, or 6 are off the menu; the buddy guarantee is purchased with allocation granularity.
• Scaling and migration overheads. Every rescale checkpoints and relaunches workers; measured and modeled, but nonzero — and frequent bursts mean frequent rescales.
• Profiling trust. MSS is only as good as the scaling curve; curves are profiled per job and adjusted online, but unpredictable jobs (accuracy-based termination) are bounded by max iterations rather than known precisely.
• Data-parallel scope. The design targets DDP-style training; model/pipeline parallelism and its different scaling behavior are outside the evaluation.

The evaluation

The testbed is serious: 16 Azure ND A100 v4 servers — 128 A100 GPUs with NVLink inside servers and 8 HDR InfiniBand adapters each — driven by two months of real job traces from ten of Microsoft’s internal ITP clusters, with six model families (VGG16, ResNet50, Deep Speech 2, Inception V3, BERT, GPT-2). A simulator validated to within 3% of testbed results extends the comparison. The metric is the deadline satisfactory ratio: the fraction of jobs finishing by their deadlines.