SpotServe: Serving Generative Large Language Models on Preemptible Instances

Spot GPU instances cost a fraction of on-demand ones but can vanish on 30 seconds' notice — SpotServe serves large language models across them anyway, reshaping the parallel layout on the fly, migrating weights and attention state instead of reloading them, and committing generation progress token by token so no work is ever redone.

Why this problem is worth solving

Generative large language models (LLMs) produce text one token at a time, and each token touches every parameter of the model. The parameters alone outgrow single machines: the paper’s smallest test model (OPT-6.7B) needs 25 GB, the largest (LLaMA-30B) 111.8 GB — a minimum of 4 and 16 GPUs respectively on the NVIDIA T4 hardware used in the evaluation. GPT-3 class models need more than 16 A100 GPUs, above $66 per hour for a single inference pipeline. For most organizations this makes LLM serving on regular cloud instances prohibitively expensive.

Clouds already sell the solution — with a trap inside. Spot instances (AWS spot, Azure spot VMs) rent the cloud’s spare capacity at up to 90% below on-demand prices; on the paper’s test hardware the gap is 1.9 versus 3.9 dollars per hour. The trap: the cloud may preempt the instance whenever a paying customer needs the capacity, giving only a grace period — about 30 seconds — to wrap up. Spot instances built a whole ecosystem of cheap serving for small models. Nobody had made them work for models that must be spread across many machines, which is exactly where the cost pressure is worst.

What people did before, and where it broke

How distributed LLM inference works. A served LLM combines three kinds of parallelism, written in this paper as a configuration (D, P, M): D data-parallel pipelines each independently serving requests; each pipeline split into P pipeline stages (consecutive layers on different instances, passing activations stage to stage); each stage sharded across M GPUs by tensor parallelism (every GPU holds a slice of each weight matrix, synchronized by collective communication). During generation, each request also accumulates a

Spot serving for small models. MArk reroutes requests around preempted replicas; Cocktail absorbs preemptions inside a model ensemble; other systems duplicate computation so a lost instance never loses work. All assume the unit of failure equals the unit of serving — one independent replica per instance.

Why that assumption dies with LLMs. Pipeline dependencies and tensor collectives have no fault tolerance: preempt one instance and every other instance in its pipeline hangs idle. A multi-GPU machine can host slices of several pipelines, so one preemption can break many at once — the paper calls this chain crashing. Rerouting now drops an entire pipeline’s worth of capacity, and redundancy would mean duplicating a 16-GPU deployment. Training-side systems (Varuna, Bamboo) do adapt parallelism to preemptions, but training tolerates lost iterations; serving has live requests with latency targets and per-request state. Existing LLM serving engines (FasterTransformer, Orca, DeepSpeed) simply assume dedicated instances.

The idea

SpotServe’s answer to preemption has three layers, one per challenge.

Treat the parallel configuration as a live variable. With spot instances, the number of available machines N changes constantly — and the request arrival rate changes too. A configuration that is optimal at twelve instances may be infeasible at nine and wasteful at fifteen. The parallelization controller re-solves the choice on every change: among configurations whose throughput covers the observed arrival rate, pick the one with the lowest inference latency, preferring fewer instances at a tie (money). If no configuration can cover the arrival rate, maximize throughput instead — controlled degradation rather than an unbounded queue. The search runs online in under a second because every configuration’s latency was profiled offline. SpotServe also keeps a couple of spare instances as a buffer and can mix in on-demand instances when spot capacity dries up.

Migrate state; never restart. Changing configuration used to mean restarting every engine and reloading weights from storage — minutes of downtime. SpotServe observes that most of the bytes the new configuration needs are already sitting in GPU memory, just possibly on the wrong machine. The context daemon keeps weights and KV caches alive across engine restarts. The device mapper then decides which physical GPU takes which logical position (d, p, m) in the new layout — formalized as bipartite matching where an edge’s weight is the size of the overlap between what the GPU holds and what the position needs, and solved optimally by the Kuhn-Munkres algorithm. What cannot be reused is transferred GPU-to-GPU, in pipeline-stage order, so migration overlaps with inference: stage 0 can serve while stage 2 is still receiving weights.

Commit progress at the token, not the request. Here the paper exploits what LLMs are. Autoregressive decoding produces one token per iteration, and the KV cache is a complete checkpoint of progress so far — no snapshot format needed. So an interrupted request need not restart: migrate its KV cache and continue from the next token. The interruption arranger runs this as a

config: (2, 2, 8)
grace period: 30 s countdown starts

Why it works

• Model parallelism couples instances → one preemption stalls whole pipelines → the only fix that preserves capacity is changing the parallel shape itself → SpotServe makes (D, P, M) a runtime decision re-solved in under a second.
• Weights and caches already live in GPU memory across the cluster → restarting from storage throws that away (2+ minutes for a 120B model) → matching GPUs to new positions by reusable bytes turns reconfiguration into a small, optimally-chosen set of GPU-to-GPU transfers that fit inside a 30-second grace period.
• Decoding is autoregressive → the KV cache is a byte-exact record of progress → committing per token is free, and resuming costs one cache transfer instead of re-decoding hundreds of tokens → tail latency stops being dominated by recomputation.
• Migration follows pipeline-stage order → early stages serve while late stages transfer → reconfiguration downtime shrinks from “system restart” to “one stage’s transfer time.”
• The grace period is spent, not wasted → the doomed instance decodes to the last safe second before preemption, and waits before acquisitions → every second of cheap compute is harvested.

What it costs

• A latency premium versus on-demand serving. The 54% cost saving comes with average latency below 18% higher and P99 tail latency about 90% higher than an all-on-demand deployment — acceptable for many workloads, not for hard real-time ones.
• Standby resources. A dedicated on-demand CPU coordinator, a small pool (about two) of spare instances, and optional on-demand GPU mixing when spot capacity vanishes.
• Memory and bandwidth for migration. Context transfers need communication buffers in GPU memory (mitigated by the memory-aware planner) and consume network bandwidth during reconfigurations.
• Offline profiling. Latency and migration-cost models must be profiled per model and per instance type before deployment.
• Scope limits. One GPU instance type at a time (no heterogeneous or cross-cloud mixing), a latency-first objective rather than strict SLO guarantees, and reactive handling only — SpotServe responds to preemption notices; it does not predict them.

The evaluation

SpotServe was evaluated on AWS g4dn.12xlarge instances (4 NVIDIA T4 GPUs each), replaying two 20-minute segments of a real 12-hour spot availability trace — one calm, one turbulent — optionally mixed with on-demand instances, plus a rescaled production workload trace from Microsoft Azure Functions for fluctuating request rates. Three models were served: OPT-6.7B, GPT-20B, and LLaMA-30B (25 to 111.8 GB of weights, needing 4 to 16 GPUs minimum). Since no distributed LLM spot serving system existed, both baselines were built by the authors on the same FasterTransformer engine, representing the two prior philosophies.

The ablation assigns credit cleanly: the parallelization controller matters most under load (removing it inflates tail latency 179% on the turbulent trace, because the system stops adapting throughput to the queue), the migration planner is next (1.4× / 3.1× on the two traces, and its memory-aware scheduling cut GPT-20B’s minimum from 16 to 12 GPUs), and the interruption arranger contributes a further 29% tail reduction by saving the KV caches of in-flight requests.