GFS: A Preemption-aware Scheduling Framework for GPU Clusters with Predictive Spot Instance Management

Jiaang Duan, Shenglin Xu, Shiyou Qian, Dingyu Yang, Kangjin Wang, Chenzhi Liao and 12 others

Cloud GPU clusters rent out spare capacity cheaply as preemptible "spot" instances for low-priority jobs, but whenever high-priority jobs surge the provider must evict those spot jobs — and doing it with today's static quotas and simple scheduling wastes work, loses revenue, and leaves GPUs idle; this paper (Shanghai Jiao Tong University and Alibaba — Jiaang Duan, Shiyou Qian, and colleagues) presents GFS, which forecasts each organization's future GPU demand with uncertainty, sizes the spot quota proactively from that forecast, and preempts spot jobs at the lowest cost — cutting evictions and queuing while raising GPU utilization on a cluster of more than ten thousand GPUs.

The argument in one glance

Problem

Renting out spare GPUs cheaply collides with protecting the jobs that own them

A large GPU cluster serving many tenants faces a tension. Some jobs are high-priority (HP) — LLM training and online inference with strict service-level objectives (SLOs) — and are treated as non-preemptible: once they get their GPUs, the cluster will not take them away. But HP jobs rarely use the whole cluster, so a lot of GPU capacity sits idle. To monetize that idle capacity, cloud providers offer it as preemptible "spot" instances at a steep discount (as much as ninety percent below on-demand price) to low-priority (LP) jobs like batch work and experimental training. The catch is that spot capacity is borrowed: the moment an HP job needs GPUs, the provider must evict the spot jobs sitting on them. That eviction is expensive in ways that are easy to miss. Spot GPU jobs are often gang-scheduled and lack checkpoints, so an eviction throws away all the work done so far; and providers typically promise spot jobs a "guaranteed duration," so a job preempted before that duration cannot even be billed. Meanwhile the LP job goes back into a queue and waits. So the provider is caught between two goals that pull against each other: honor HP SLOs (which means always being ready to reclaim GPUs) and keep spot jobs alive long enough to be useful and billable (which means not reclaiming too aggressively) — all while keeping overall GPU utilization high.
What people did

Set a fixed spot quota and place jobs first-fit

The common approach is to carve out a static spot quota — a fixed number of GPUs the cluster is willing to lend to LP jobs — and schedule everything with simple first-fit placement, assigning each job the first available GPUs that fit. HP jobs get priority; whatever capacity is left up to the quota is offered to spot jobs. When an HP job arrives and no free GPUs remain, the scheduler preempts whichever spot jobs are in the way.
Why it fell short

Static quotas and blind placement cause evictions in surges and idle GPUs otherwise

A fixed quota cannot match a demand that swings over time, and the scheduler has no idea what is coming. The paper's analysis of a one-month trace from a production cluster of 10,365 GPUs makes the failure concrete. During HP demand surges, spot eviction rates spike — and because spot jobs are un-checkpointed and gang-scheduled, each eviction discards real work, and evicting before the guaranteed duration means the provider cannot charge for it. At quieter times the opposite happens: without a forecast of HP demand, the static quota over-commits GPUs to spot (the paper measures a 41.2 percent over-commitment of spot quotas), creating conflicts and, elsewhere, leaving GPUs underused. First-fit placement makes it worse by fragmenting the cluster and picking arbitrary spot jobs to evict, ignoring how costly each particular preemption is. The result is a cluster that simultaneously evicts too much when it is busy and wastes capacity when it is not, missing HP SLOs and lowering overall utilization. What was missing was a scheduler that anticipates HP demand, sizes the spot quota to it, and preempts thoughtfully rather than blindly.
This paper's idea

Forecast demand, size the quota to it, and preempt at minimum cost

GFS is a preemption-aware scheduling framework that manages spot instances proactively through a closed loop of three components. First, a GPU Demand Estimator (GDE) forecasts future GPU demand — not a single number but a probability distribution — for each tenant. Its model, OrgLinear, is a lightweight hierarchical time-series forecaster that decomposes demand into shared periodic patterns (hourly-to-weekly cycles) and organization-specific residuals, because different organizations have distinct business calendars (one may drop demand 35.7 percent on weekends while another does not), and it quantifies uncertainty rather than giving a bare point estimate. Second, a Spot Quota Allocator (SQA) turns that forecast into a spot quota that changes over time: through an eviction-aware feedback loop it combines the predicted HP demand (hence the anticipated idle GPU inventory), real-time cluster conditions, historical eviction rates, and queuing time to set how many GPUs to lend to spot jobs, so the quota shrinks ahead of a surge and grows when GPUs will be free. Third, a Preemptive Task Scheduler (PTS) prioritizes placing HP jobs on genuinely idle GPUs, and only when it must preempt does it choose the spot jobs whose preemption costs the least. The three run as a closed loop: predictions drive quotas, quotas and real evictions feed back to refine predictions. Deployed in production, GFS reports a 33.0 percent reduction in spot eviction rates, a 44.1 percent cut in spot queuing delays, up to a 22.8 percent higher GPU allocation rate, and — on a cluster of more than ten thousand GPUs — roughly 459,715 US dollars in monthly benefit.

Why this problem is worth solving

GPUs are the scarcest, most expensive resource in modern datacenters, so even small improvements in how a cluster lends out and reclaims spare capacity translate into large, direct savings.

A single large GPU cluster represents an enormous capital investment, and every idle GPU is money not being earned. Spot instances exist precisely to recover that value — lending idle GPUs to cheap, preemptible jobs — but the mechanism only pays off if it is managed well. Badly managed, it produces the worst of both worlds: high-priority jobs still miss their deadlines, spot jobs are evicted so often that their discarded work and lost billing eat the savings, and GPUs still sit idle at the wrong moments. Because the resource is so costly and the cluster so large, the difference between good and bad spot management is measured in real money at scale — the paper puts it at roughly 459,715 US dollars per month on a ten-thousand-GPU cluster. Improving it is therefore not a marginal tuning problem but a first-order efficiency lever for anyone operating GPUs at scale, which is now a large and growing share of the industry.

The core difficulty is that the decision is about the future, and the future is uncertain and organization-specific.

The reason static quotas and first-fit fail is that both react to the present while the cost is set by what happens next: you lend GPUs to a spot job now, and whether that was wise depends on whether an HP surge arrives before the job finishes. So the problem is fundamentally predictive — to lend safely, you must anticipate HP demand. But two things make naive forecasting inadequate. First, demand is heterogeneous across organizations: different tenants have different business calendars and periodicities, so a single cluster-wide model misses the structure that actually drives demand. Second, a point prediction is not enough for a risk decision; sizing a spot quota is about reliability, so you need to know not just the expected demand but its uncertainty, to leave the right safety margin. GFS is built around exactly these two needs — per-organization decomposition and uncertainty quantification — because getting the forecast right, and knowing how much to trust it, is what lets the quota be aggressive when it is safe and cautious when it is not. Everything downstream (the quota, the preemption choices) is only as good as that probabilistic, tenant-aware view of what is coming.

What people did before, and where it broke

Set a static spot quota. Fix how many GPUs the cluster will lend to spot jobs. Where it broke. A fixed number cannot track demand that swings over time — it over-commits when demand is low (the paper measures 41.2 percent over-commitment) and evicts heavily when demand surges.
Place jobs with first-fit scheduling. Assign each job the first available GPUs that fit. Where it broke. It fragments the cluster and, when it must preempt, picks arbitrary spot jobs without regard to how costly each particular eviction is.
React to eviction and SLO violations after they happen. Adjust only once evictions spike or SLOs are missed. Where it broke. Reacting is too late — the un-checkpointed work is already lost and the HP job has already queued; the damage is done before the scheduler responds.
Forecast demand with a single cluster-wide point model. Predict total GPU demand with one time-series model. Where it broke. It ignores organization-specific periodicities and business calendars, and a point estimate gives no uncertainty, so the quota cannot leave a principled safety margin.
The gap. No scheduler forecast per-organization demand with uncertainty, sized the spot quota proactively from that forecast, and preempted spot jobs at minimum cost — all in one closed loop.

How GFS works

A closed loop: forecast demand, size the spot quota, preempt at minimum cost

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Decide under uncertainty by forecasting demand and how much to trust the forecast. GFS’s foundation is that sizing a spot quota is a risk decision about the future, so it predicts each tenant’s demand as a probability distribution and uses the uncertainty to set a safe margin, not a bare guess. Managing shared resources by forecasting demand and explicitly reasoning about the uncertainty of that forecast, rather than reacting to the present, is the same approach the work on uncertainty-aware resource management for multi-stage services (Aquatope) covered elsewhere on this site takes, provisioning ahead of demand with a principled account of what might go wrong. GFS applies it to GPU spot quotas, where the uncertainty margin is what keeps the quota aggressive when safe and cautious when a surge is likely.

Lend out idle capacity to cheap work, and take it back gracefully when the owner needs it. The whole spot mechanism is a lease of unused GPUs to low-priority jobs, reclaimed the instant high-priority jobs return — GFS’s job is to make that lending and reclaiming efficient rather than wasteful. Harvesting resources that would otherwise sit idle for opportunistic use, then returning them to their primary owner on demand, is the same principle the work on memory-harvesting VMs in cloud platforms covered elsewhere on this site is built on, turning slack capacity into value without hurting the owner. GFS harvests idle GPUs for spot jobs and reclaims them through minimum-cost preemption, so the reclaim disturbs the borrowers as little as possible.

Schedule a shared GPU cluster so priority jobs are protected without starving the rest. At bottom GFS is a cluster scheduler mediating between jobs of different priority, trying to meet the important ones’ SLOs while keeping the cheap ones alive and the cluster busy. Scheduling deep-learning jobs on a shared GPU cluster to honor priorities and utilization together, rather than treating placement as an afterthought, is the same problem the work on a non-intrusive, interpretable deep-learning cluster scheduler (Lucid) covered elsewhere on this site addresses, arbitrating a contended GPU fleet among competing jobs. GFS adds the spot dimension — forecasting, dynamic quotas, and cost-aware preemption — to that scheduling problem.

Following a demand surge through GFS

Protecting high-priority SLOs during a surge without needless spot evictions

A trace, drawn from the paper, of how GFS's closed loop handles an approaching high-priority demand surge.

Step 1 of 5

GDE forecasts an approaching HP surge

OrgLinear predicts each organization's future GPU demand as a distribution, decomposing shared periodic cycles from organization-specific patterns. It sees that a particular tenant's demand is about to rise sharply, and reports both the expected demand and its uncertainty.

The forecast is per-organization, because business calendars and periodicities differ.

Why it works

Forecasting lets the quota move before the surge, not after. Static quotas react only once evictions spike → but predicting HP demand ahead of time → so the spot quota shrinks before capacity is reclaimed, avoiding the mass eviction entirely.
Per-organization decomposition captures the real structure of demand. One cluster-wide model misses tenant-specific cycles → but decomposing shared periodic bases from organization-specific residuals → so the forecast reflects each tenant’s actual business calendar and is far more accurate.
Uncertainty turns a guess into a risk-sized decision. A point estimate gives no basis for a safety margin → but predicting a distribution and its uncertainty → so the quota can be aggressive when demand is confidently low and cautious when a surge is plausible.
Minimum-cost preemption spares the expensive evictions. First-fit evicts arbitrary spot jobs → but weighing each candidate’s lost work and guaranteed-duration breakage → so GFS discards the least value when it must preempt at all.
The closed loop compounds the gains. A one-shot prediction drifts as conditions change → but feeding real evictions and cluster state back into the estimator and allocator → so forecasts and quotas stay calibrated and improve over time.

What it costs (trade-offs and scope)

The gains depend on demand being forecastable. GFS’s advantage comes from anticipating HP demand; workloads that are genuinely erratic, with little periodicity or organizational structure, give the forecaster less to work with, so the proactive quota has less edge over a reactive one.
It targets a specific setting: preemptible spot jobs on a non-preemptible HP cluster. The framework assumes the HP/LP priority split, guaranteed-duration spot semantics, and the ability to preempt spot jobs; environments without that structure would need adaptation.
Uncertainty-aware, per-organization forecasting adds machinery. OrgLinear’s hierarchical decomposition and the closed-loop allocator are more complex than a static quota, and require historical per-tenant traces and continual feedback to stay calibrated — an operational cost a fixed quota does not have.
Minimum-cost preemption is only as good as the cost model. Choosing the cheapest victims depends on estimating each spot job’s lost work and duration risk; if those estimates are wrong, the preemption choices degrade, so the cost model matters.
Reported benefits are from one company’s production clusters. The headline figures (33.0 percent fewer evictions, up to 22.8 percent higher allocation, the monthly dollar benefit) come from specific production and trace-driven settings; the exact numbers reflect that environment’s workload mix and pricing, and would differ elsewhere.

The evaluation

Managing spot instances on a multi-tenant GPU cluster

Property	Static quota plus first-fit	Reactive adjustment	GFS
Spot quota
Demand forecasting
Preemption choice
Spot eviction rate
GPU allocation rate

A comparison of spot-instance management on multi-tenant GPU clusters. By forecasting each organization's future GPU demand as a distribution (OrgLinear), sizing the spot quota proactively from that forecast, and preempting spot jobs at minimum cost, GFS reports a 33.0 percent reduction in spot eviction rates, a 44.1 percent cut in spot queuing delays, up to a 22.8 percent higher GPU allocation rate, and roughly 459,715 US dollars in monthly benefit on a production cluster of more than ten thousand GPUs. Exact figures and baselines should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2509.11134, of the ASPLOS 2026 paper). The described system is the paper’s: GFS is a preemption-aware scheduling framework for multi-tenant GPU clusters that manages spot instances proactively through three components in a closed loop. The GPU Demand Estimator (GDE) uses OrgLinear, a lightweight hierarchical time-series model that decomposes demand into shared periodic patterns and organization-specific residuals and quantifies uncertainty, to predict a probability distribution of future GPU demand per organization. The Spot Quota Allocator (SQA) uses an eviction-aware feedback loop that combines predicted demand, real-time cluster conditions, historical eviction rates, and queuing time to set dynamic spot quotas with guaranteed durations. The Preemptive Task Scheduler (PTS) prioritizes non-preemptive placement of high-priority tasks and, when preemption is necessary, chooses spot jobs to evict by preemption cost. The paper reports, from real-world production implementation and trace-driven simulations on a cluster of 10,365 GPUs, a 33.0 percent reduction in spot eviction rates, a 44.1 percent decrease in spot queuing times, up to a 22.8 percent improvement in GPU allocation rate, and roughly 459,715 US dollars in monthly benefit on a cluster of more than ten thousand GPUs (with additional simulation results such as reduced high-priority queuing time). Specific numbers, such as the 41.2 percent spot-quota over-commitment and the 35.7 percent weekend demand drop, are the paper’s measurements of its production trace. The step-by-step walkthrough above is a plausible illustration of the described mechanism. Exact model details, the SQA and PTS algorithms, and the full evaluation should be read from the paper; the source code is reported to be available.

Background

Why running big shared GPU clusters keeps turning into a forecasting problem

It helps to see GFS within a broad pattern: as GPU clusters grow into the most expensive, most contended resource in a datacenter, operating them efficiently increasingly means anticipating demand rather than merely reacting to it. The central difficulty of a shared cluster is that many tenants with different priorities and rhythms compete for the same scarce accelerators, and every allocation decision made now is really a bet about what will be needed soon. Spot instances sharpen this: they exist to recover idle value, but the value is only recovered if the lending is timed against a demand that has not happened yet. So the answers converge on the same ingredients — model each tenant’s demand and its periodic structure, reason about the uncertainty, and use that to provision ahead and reclaim gracefully. That GFS needs a per-organization forecaster, a feedback-driven quota, and a cost-aware preemptor to do this well shows how much machinery careful cluster efficiency now requires. This concern for operating a large shared GPU fleet so that competing jobs coexist efficiently is the same terrain the work on the design and operation of shared machine-learning clusters covered elsewhere on this site occupies, wrestling with how to keep an expensive cluster both busy and fair. GFS’s contribution is to show that, with an uncertainty-aware demand forecast driving dynamic quotas and minimum-cost preemption, a provider can cut evictions and queuing while raising utilization — turning spot management from a source of waste into real money at scale.

If you remember only three things

1. Multi-tenant GPU clusters lend idle capacity to cheap, preemptible "spot" jobs (low-priority), running on GPUs not currently used by non-preemptible high-priority jobs. But existing schedulers use a static spot quota and first-fit placement with no forecast of high-priority demand, so during demand surges spot jobs are evicted en masse — discarding their un-checkpointed, gang-scheduled work and, when evicted before a guaranteed duration, becoming un-billable — while at quieter times quotas over-commit (the paper measures 41.2 percent over-commitment) and GPUs sit idle. The result hurts both high-priority SLOs and overall utilization.
2. GFS is a preemption-aware framework that manages spot instances proactively via a closed loop of three parts. A GPU Demand Estimator forecasts each organization's future demand as a probability distribution using OrgLinear, a hierarchical model that separates shared periodic cycles from organization-specific patterns and quantifies uncertainty. A Spot Quota Allocator turns that forecast, plus real-time cluster state, historical eviction rates, and queuing time, into a dynamic spot quota that shrinks ahead of surges and grows when GPUs will be free. A Preemptive Task Scheduler places high-priority jobs on idle GPUs first and, when it must preempt, evicts the spot jobs whose preemption costs the least.
3. Deployed in production and evaluated with trace-driven simulations on a 10,365-GPU cluster, the paper reports GFS cuts spot eviction rates by 33.0 percent, spot queuing delays by 44.1 percent, and raises the GPU allocation rate by up to 22.8 percent, yielding roughly 459,715 US dollars in monthly benefit on a cluster of more than ten thousand GPUs. The trade-offs: the gains depend on demand being forecastable and on the HP/LP spot setting, the uncertainty-aware per-organization forecasting and closed-loop allocator add machinery and require per-tenant history, and minimum-cost preemption is only as good as its cost model. The reported numbers reflect one operator's production environment.

Check yourself

Check your understanding

1. Why is evicting a GPU "spot" job so costly for both the provider and the low-priority user?
2. What is the fundamental limitation of using a static spot quota with first-fit scheduling?
3. How does GFS's GPU Demand Estimator (using the OrgLinear model) forecast demand, and why in that particular way?
4. When GFS must preempt spot jobs to make room for a high-priority job, how does it decide which to evict, and what were the reported production results?

Sources

The paper is “GFS: A Preemption-aware Scheduling Framework for GPU Clusters with Predictive Spot Instance Management” by Jiaang Duan, Shenglin Xu, Shiyou Qian, Dingyu Yang, Kangjin Wang, Chenzhi Liao, Yinghao Yu, Qin Hua, Hanwen Hu, Qi Wang, Wenchao Wu, Dongqing Bao, Tianyu Lu, Jian Cao, Guangtao Xue, Guodong Yang, Liping Zhang, and Gang Chen (Shanghai Jiao Tong University, Alibaba Group, and Zhejiang University), published at ASPLOS 2026, DOI 10.1145/3760250.3762231. It is available on arXiv as 2509.11134.

This explainer is written from the paper’s full text (the authors’ arXiv copy). GFS is a preemption-aware scheduling framework for multi-tenant GPU clusters that manages spot instances proactively through three components in a closed loop: a GPU Demand Estimator (GDE) using OrgLinear, a lightweight hierarchical time-series model that decomposes demand into shared periodic patterns and organization-specific residuals and quantifies uncertainty, to predict a per-organization probability distribution of future GPU demand; a Spot Quota Allocator (SQA) that uses an eviction-aware feedback loop over predicted demand, real-time cluster conditions, historical eviction rates, and queuing time to set dynamic spot quotas with guaranteed durations; and a Preemptive Task Scheduler (PTS) that prioritizes non-preemptive placement of high-priority tasks and, when preemption is necessary, evicts spot jobs by minimum preemption cost. Evaluated via production implementation and trace-driven simulation on a 10,365-GPU cluster, the paper reports a 33.0 percent reduction in spot eviction rates, a 44.1 percent decrease in spot queuing times, up to a 22.8 percent improvement in GPU allocation rate, and roughly 459,715 US dollars in monthly benefit on a cluster of more than ten thousand GPUs. Numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper, and the source code is reported to be available.