Research schedulers for GPU clusters win benchmarks by preempting, resizing, and rewriting users' training jobs — which is exactly why production clusters still run FIFO; Lucid shows that a scheduler that never touches user code (a ~200-second profiled test run, interference-avoiding "indolent" job packing, and duration-predicted priorities, all built on interpretable models) beats the intrusive state of the art by up to 1.3× on job completion time while deciding in milliseconds.
The argument in one glance
-
Problem
GPU clusters are expensive, idle, and badly queued
Companies run multi-tenant GPU clusters for deep learning training, and the numbers are grim: in an Alibaba datacenter, only 16% of GPUs exceed 50% utilization; at Microsoft, nearly 70% of resources go to jobs that ultimately fail or are canceled. Meanwhile most deployed clusters schedule with FIFO, so a short debugging job — the majority of all submissions — can sit behind a multi-day training run for hours. Users wait, GPUs idle, money burns.
-
What people did
Research schedulers: preempt, resize, adapt, optimize
A wave of DL-specific schedulers attacked this with powerful mechanisms: Tiresias preempts and reorders by least-attained-service; Gandiva migrates and packs jobs introspectively; Pollux elastically rescales jobs and even retunes their batch sizes; Gavel solves allocation as an optimization problem; others train black-box ML models to predict what to run where. On benchmarks, they all beat FIFO handily.
-
Why it fell short
None of them survive contact with production
Five deployment killers. Preemption and elasticity require users to import libraries and restructure their training code — Microsoft's own assessment reads that most DNN workloads are not checkpointable or resizable. Framework and CUDA modifications demand permanent maintenance against fast-moving DL stacks. Adaptive batch-size tricks degrade final model accuracy (Pollux costs an EfficientNet over 2% validation accuracy — unacceptable when accuracy is revenue). Solvers don't scale: Gavel needs ~30 minutes for a 2048-job allocation; Pollux takes hours beyond 320 jobs. And black-box models give operators no way to trust, debug, or tune decisions.
-
This paper's idea
Schedule from the outside, with models you can read
Lucid never touches user code, frameworks, or running jobs. Every job first runs ~200 seconds on a small profiling cluster where nvidia-smi-level metrics classify it (a readable decision tree); an "indolent" packing policy colocates only pairs that provably won't interfere; and a transparent additive model predicts each job's duration from the cluster's recurring history to prioritize the queue — no preemption anywhere. Result: up to 1.3× better average completion time than preemptive Tiresias, 1.8–9.1× less queuing, scheduling decisions in under 3 milliseconds for 2048 jobs.
Why this problem is worth solving
A deep-learning cluster scheduler has one core enemy:
a long job holding GPUs makes every
job behind it wait — with no preemption, a multi-day training run
can pin a two-minute debugging job in the queue for hours; the
classic cure, shortest-job-first, needs to know durations in
advance
The workload statistics explain why a gentler design has room to win. Production DL clusters are dominated by small, repetitive, short-lived work: over 95% of jobs fit on a single node, roughly 90% are recurring hyperparameter-search submissions of near-identical jobs, and the largest share of submissions are short debugging runs whose authors want feedback in minutes, not hours. GPUs are chronically underutilized, so there is enormous slack to reclaim by sharing — if interference can be predicted without instrumenting anyone’s code. And recurrence means history is predictive: yesterday’s jobs tell you how long today’s will run. Lucid is a bet that these three observations, exploited carefully, beat preemption’s raw power once preemption’s overheads (62 seconds per checkpoint-restart in Tiresias, an extra 13% queuing overhead) and deployment costs are priced in.
What people did before, and where it broke
- FIFO — what clusters actually run. Yarn, Kubernetes, Slurm defaults. Runtime-agnostic, so head-of-line blocking is unmanaged: average job completion time on the Microsoft Philly trace is 36.85 hours, of which 30.45 hours is queuing.
- Preemptive schedulers (Tiresias).
lets Tiresias approximate shortest-job-first without duration knowledge (discretized least-attained-service). It works — 9.02-hour average JCT on Philly — but is intrusive (user-code changes), and each preemption costs ~62 seconds of checkpoint and cold-start.
checkpointing a running job, releasing its GPUs, and resuming it later — powerful for reordering, but the job must be checkpointable, which in practice means users modify their training code
- Packing schedulers (Gandiva, AntMan, Salus, Horus).
is the obvious answer to 16%-busy GPUs, but every existing packing scheduler decides who shares by intrusive means: modifying frameworks, or converting user models into ONNX graphs.
running two jobs on the same GPU (colocation/GPU sharing, via MPS or MIG) to reclaim underutilized capacity — profitable when jobs don’t contend, poisonous when they do
- Adaptive/elastic schedulers (Pollux). Rescaling workers and retuning batch sizes squeezes efficiency but changes training itself — over 2% accuracy loss on EfficientNet — and its solver stops scaling in the hundreds of jobs.
- Optimization- and ML-based schedulers (Gavel, Metis, and friends). LP solvers take tens of minutes at thousand-job scale; reinforcement-learning schedulers handle dozens of jobs; and random-forest/GBDT predictors are black boxes — operators can’t verify why a decision was made, can’t safely tune knobs, and therefore don’t deploy them.
How Lucid works
Lucid's pipeline — three modules, each with a readable model
-
Non-intrusive Job Profiler new
Every incoming job first runs up to ~200 s on a small, separate profiling cluster while nvidia-smi/DCGM record three hardware metrics: GPU utilization, GPU memory footprint, and memory-bandwidth utilization. No library imports, no code changes. Side effect: 23–55% of jobs (debug runs, crashes) FINISH here and never touch the main cluster. Space-aware ordering (smallest GPU demand first) keeps the profiling queue itself unblocked; time-aware scaling borrows nodes from idle virtual clusters during submission bursts.
-
Packing Analyze Model new
A pruned decision tree maps the profiled metrics (plus an optional user flag for mixed-precision training) to a Sharing Score: Tiny (0) — barely affected by colocation; Medium (1); Jumbo (2) — resource-hungry, share with caution. The tree is small enough to print, 94.1% accurate, and trains in under a second.
-
Affine-Jobpair Binder — indolent packing new
Pack two jobs on the same GPUs only when their Sharing Scores sum to at most 2, memory fits with a hard OOM guard, GPU demands are equal (no stragglers), never more than two jobs, never distributed jobs, and evict a pair if utilization turns unstable. Deliberately lazy — it forgoes risky wins to make sharing safe enough that users accept it. A throughput forecast dials packing down (or off) when the cluster is idle and sharing would only slow jobs.
-
Resource Orchestrator new
A transparent additive model (GA2M) predicts each job's duration from profiled features plus the cluster's recurring-job history (job names bucketized by edit distance, user, GPU demand, submission time). Priority = predicted duration × GPU count — a shortest-job-first variant that also charges for width. Jobs are allocated in priority order, consolidated to minimize fragmentation. No preemption: once placed, a job runs to completion.
-
System Optimizer new
An Update Engine fine-tunes the models on fresh logs (weekly is enough — daily adds 1.6%); a System Tuner lets operators read the models' learned shape functions (e.g., the diurnal submission curve) and adjust knobs with guidance, including monotonicity constraints that both fix nonsense and improve accuracy.
The idea
Interference is predictable from the outside. The enabling measurement: colocation slowdown correlates strongly with the sum of the two jobs’ GPU utilizations. When the pair sums to 100% or less, average packed speed is ~0.92× — near-free sharing. Those utilizations are visible to nvidia-smi without touching the job. So Lucid replaces every prior scheduler’s intrusive introspection (framework hooks, ONNX graph extraction, online trial-colocation) with a three-metric classification learned once — and simplifies the prediction target from “exact packed throughput” to a ternary Sharing Score, which is all the packing decision actually needs. The lazy rule — pack only when scores sum safely — gets 98.1% of its chosen pairs interference-free while still capturing 87% of the packing opportunities.
The profiler is a filter, not a tax. Conventional wisdom says profiling adds queuing delay. Lucid inverts this by exploiting the workload’s shape: most jobs are short debugging runs, so a ~200-second profiled execution is the job for 23–55% of submissions — they finish (or crash) in the profiling cluster, giving their authors feedback in minutes and shrinking the main scheduler’s optimization space. The profiling cluster gets its own scheduling theory: order by GPU demand (space-aware — an 11.6× improvement over naive profiling queues), and elastically resize using the throughput forecast.
Recurrence makes duration predictable; duration makes preemption unnecessary. With ~90% recurring jobs, a new submission usually resembles the same user’s earlier ones. Lucid’s duration model leans on that — matching via profiled features plus edit-distance-clustered job names — and its priority rule (duration × GPUs) approximates shortest-job-first well enough that the head-of-line problem preemption exists to solve mostly dissolves. The ablation is striking: with all modules on, average performance approaches the no-queuing optimal, from a scheduler that never interrupts anything.
Interpretability is a feature that pays for itself. Every model
is chosen to be readable: a pruned decision tree, and
generalized additive model with pairwise
interactions — the prediction is a SUM of simple per-feature shape
functions (plus a few two-feature terms), so every prediction
decomposes into visible per-feature contributions, and each learned
curve can be plotted and sanity-checked
A job’s journey through Lucid
One training job, from submission to completion
SS = Sharing Score (Tiny 0 / Medium 1 / Jumbo 2), predicted by the decision tree. T_prof = profiling time limit (~200 s default). JCT = job completion time (queuing + running). GA2M = the additive, readable regression model. Numbers from the Venus/Saturn (SenseTime) and Philly (Microsoft) trace evaluations.
Step 1 of 7
Submission lands in the profiling queue
The job enters the small, dedicated profiling cluster. Its queue is ordered space-aware — smallest GPU demand first — so a 1-GPU debug run never waits behind an 8-GPU profile. Average profiling-stage queuing: about one minute.
space-aware ordering: up to 11.6× vs naive profiling queues burst? time-aware scaling borrows idle nodes
Why it works
- Colocation slowdown tracks the sum of GPU utilizations, and utilization is observable from outside the process → a ternary Sharing Score from three nvidia-smi metrics suffices for safe packing decisions → the entire intrusive apparatus of prior packing schedulers (framework mods, model-graph extraction) becomes unnecessary for >95% of jobs.
- Most submissions are short debug runs → a time-boxed profiled execution completes them outright → profiling subtracts load from the main cluster instead of adding delay, and debugging users get their feedback 4.1–24.8× sooner.
- ~90% of jobs recur → history plus profiled features predict duration well enough (R² 0.413, best of all models tested) → priority = duration × GPUs approximates SJF, dissolving head-of-line blocking without preemption’s 62-second-per-event overhead and code demands.
- Never modifying the job → final model accuracy is preserved by construction and users need change nothing → the adoption barriers (G1–G3) vanish rather than being mitigated.
- Simple additive/tree models → millisecond inference, minutes of training on millions of samples, and decisions operators can audit → scalability and trust — the two failure modes of solver-based and black-box schedulers — are solved by the same design choice, and the readable shape functions even enable tuning that improves accuracy further.
What it costs
- No mid-run correction. A job whose duration was badly underestimated occupies its GPUs until it finishes — preemptive schedulers can fix such mistakes, Lucid cannot. Its defense is prediction quality plus the tail numbers (P99.9 queuing still 1.4–2.1× better than Tiresias), not a mechanism.
- Profiling infrastructure. A dedicated (if small, elastically borrowed) slice of the cluster, plus a ~200 s delay for jobs that do proceed to the main cluster. Jobs with slow initialization need a longer T_prof or a user-set “long cold-start” flag — a small crack in the zero-user-effort story.
- Conservative packing leaves money on the table. Indolent
packing skips risky-but-profitable colocations (13% of true
opportunities missed) and never packs distributed jobs — the
5% of jobs that use the most GPUs get no sharing benefit.
- Depends on recurrence. The duration model leans on repetitive workloads and same-user history; a cluster of one-off, novel jobs would degrade toward the fallback (average-by-size) predictions.
- Homogeneous view. Lucid schedules one GPU generation and treats GPUs as the only resource; heterogeneous GPU selection and CPU/network awareness are explicitly future work.
- Single objective. Average JCT (plus utilization and feedback latency); fairness and SLO guarantees are not yet goals.
The evaluation
Two layers: a real 32-GPU cluster (4 nodes × 8 RTX 3090) replaying sampled traces, and large-scale simulation over three production traces — SenseTime’s Venus and Saturn and Microsoft’s Philly, spanning thousands of GPUs and weeks-to-months of submissions. The simulator is validated against the physical runs (under 4.6% error on makespan and JCT). On the physical cluster Lucid beats FIFO/SJF/Tiresias on both a static batch (makespan 8.45 vs 11.56 / 11.27 / 9.23 hours) and a Poisson stream (average JCT 3.64 vs 8.17 / 4.59 / 4.03 hours). The headline table below is the simulation at production scale.
Deployed practice vs intrusive SOTA vs Lucid (production traces)
| Property | FIFO (what clusters run) | Tiresias (preemptive, intrusive) | Lucid (non-intrusive) |
|---|---|---|---|
| Average job completion time | |||
| Average queuing delay | |||
| What users must change | |||
| Debugging-job experience | Short jobs wait behind multi-day runs | Better — short jobs gain priority as low-attained-service | |
| Scalability and transparency | Trivially scalable, trivially transparent, poor results | Scales, but decisions come from priority-queue internals plus preemption heuristics |
Check yourself
Check your understanding
1. Prior packing schedulers instrumented jobs (framework hooks, ONNX graph extraction) to predict colocation interference. Lucid uses only nvidia-smi-level metrics. Predict which external signal turned out to predict interference — and how well.
2. Conventional wisdom says a mandatory profiling stage ADDS queuing delay. Why does Lucid's ~200-second profiling stage end up reducing overall latency instead?
3. Lucid never preempts, yet beats the preemptive Tiresias on average JCT. What combination makes preemption unnecessary here?
4. Lucid uses decision trees and GA2M additive models instead of the random forests, gradient boosting, or DNNs of prior ML schedulers. What did interpretability buy, beyond operator trust?