TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms

Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Esha Choukse, Haoran Qiu, Rodrigo Fonseca and 2 others

A datacenter's cooling towers and power feeders are provisioned for the single hottest, hungriest rack — so the entire building's capacity is wasted on rows that never reach that peak, and one bad rack of LLM GPUs can force everyone else to be throttled; TAPAS (from Microsoft Azure and UIUC) reclaims that slack for LLM inference clusters by exploiting three things traditional schedulers ignore — that inference splits into millisecond prefill and decode phases with opposite thermal/power profiles, that provider-managed SaaS models expose knobs (batch size, parallelism, quantization, model size) that trade a little quality for real degrees and watts, and that opaque tenant IaaS VMs do not — placing, routing, and reconfiguring so no aisle overheats and no row trips, cutting peak temperature 17% and row power 23% to unlock up to 40% more capacity in the same building.

The argument in one glance

Problem

Cooling and power are provisioned for a peak almost nobody hits

An LLM inference cluster runs on GPUs that each dissipate kilowatts — an NVIDIA H100 server's thermal design power reaches 10.2 kW — and the two resources hardest to add to a datacenter are the cooling that carries that heat away and the power feeders that deliver it. Operators provision both for the WORST case: enough cooling for the hottest possible aisle, enough power for the hungriest possible row, with the limits divided evenly down the hierarchy for simplicity. But real rows are wildly uneven — in production, half of all rows draw 28% less peak power than the hungriest one. The capacity sized for that hungry row and never used elsewhere is stranded: paid for, cooled for, and idle.
What people did

Oversubscribe, the way clouds always have

The classic reclaim is oversubscription: provision for the AVERAGE rather than the sum of peaks, pack in more racks to use the slack, and cap power or throttle when reality briefly exceeds the budget. Decades of thermal-aware and power-aware scheduling place and cap CPU VMs to make this safe — spread synchronous power spikes apart, steer hot VMs to cool servers, shed load when a feeder nears its limit.
Why it fell short

CPU-era scheduling is blind to how LLM inference actually behaves

Those techniques were built for coarse, slowly-varying CPU workloads, and LLM inference breaks three of their assumptions at once. Timescale: inference alternates between a compute-heavy PREFILL phase (processing the whole prompt in parallel) and a memory-heavy DECODE phase (generating tokens one at a time) — millisecond phases with opposite thermal and power signatures a coarse scheduler cannot see. Controllability: provider-managed SaaS inference has knobs — batch size, tensor parallelism, quantization, model size, GPU frequency — that trade result quality for temperature and watts, while opaque tenant IaaS VMs expose nothing. Co-location: every aisle mixes controllable SaaS and untouchable IaaS, so a scheme must extract all its slack from the half it can actually influence, without ever touching the half it cannot.
This paper's idea

Exploit the phases, the knobs, and the asymmetry — at three timescales

TAPAS (Microsoft Azure Research and UIUC) is the first thermal- and power-aware scheduler built for LLM inference. It acts at three timescales on a shared insight: SaaS workloads are malleable, IaaS are not, so extract slack from SaaS while shielding IaaS. At VM-arrival time it PLACES: opaque hot IaaS VMs go to the coolest servers, flexible SaaS VMs to warmer ones, balancing IaaS/SaaS across every aisle and row so no hotspot forms. At request time (every few minutes) it ROUTES: inference requests avoid any VM whose aisle would exceed airflow, whose row would exceed power, or whose server would overheat. At iteration time (per-inference) it RECONFIGURES: an instance nearing a limit walks down a hierarchy of knobs — frequency first (free, instant), quantization and model-size last (they cost quality) — to shed heat and watts with minimal harm. Result on production Azure traces: peak temperature down 17%, row power down 23%, tail latency held, unlocking up to 40% more capacity and cutting thermal/power throttling events by 97%/99%.

Why this problem is worth solving

The cost of an AI datacenter is dominated by two things it is physically hard to buy more of: the electricity feeding the GPUs and the cooling removing their heat. Both are provisioned once, for the worst case, and that worst-case provisioning is where the money leaks.

selling or provisioning more capacity than the infrastructure could satisfy if every consumer peaked at once — betting, correctly almost always, that they will not. A datacenter provisions cooling for the hottest aisle and power for the hungriest row; since real rows are uneven (half draw ~28% below the peak row), the provider can safely add racks into that slack, so long as it can quickly cap or throttle in the rare moment demand actually approaches the limit. More oversubscription means more compute sold from the same building — directly lower total cost of ownership — but only if the throttling stays rare and controlled

is the lever, and its safety depends entirely on being able to shave a peak on demand without wrecking the workload. For CPU clouds, that meant capping frequencies and shifting VMs. For LLM inference, the paper shows, the old shaving tools are both too blunt and blind to a richer set of options.

What makes inference different — and exploitable — is that its work is not uniform.

the two phases of generating an LLM response. PREFILL processes the entire input prompt in a single parallel pass — compute-bound, hammering the GPU cores, hot and power-hungry in a burst. DECODE then emits output tokens one at a time, each token needing a fresh sweep of the model weights from memory — memory-bandwidth- bound, which shifts the thermal load toward the memory subsystem and changes how each configuration knob affects temperature and power. The phases alternate at millisecond granularity, so a scheduler that samples every few minutes sees only their blur

gives a fine-grained scheduler signals a coarse one cannot use, and the configuration knobs give it actuators a CPU-era scheme never had. The catch is the tenant split: only the provider’s own SaaS inference is adjustable; a customer’s IaaS GPU VM is an opaque box the provider must cool and power but cannot reconfigure. TAPAS’s whole design is organized around turning that asymmetry from a constraint into a strategy — squeeze the flexible half, protect the rigid half.

What people did before, and where it broke

Thermal-aware CPU scheduling (hot-VM-to-cool-server, airflow balancing). Correct instinct, wrong resolution: it steers whole VMs at placement time and cannot see or exploit inference’s millisecond prefill/decode phases or the quality/heat knobs inside a SaaS model.
Power-oversubscription frameworks (Flex, SmartOClock, SmoothOperator). Safely pack CPU racks by placing workloads and shedding or capping under strain — but they reason about coarse, homogeneous power patterns, not phase-varying GPU inference with a quality dimension.
LLM power work (POLCA, μ-Serve). Closer — these target LLM clusters and use parallelism or predictive scheduling for power. But they optimize power alone, not the JOINT thermal-and-power problem across a real datacenter’s aisle airflow and row feeders, and not the SaaS/IaaS co-location that dictates what is actually controllable.
The gap. No prior scheme unifies placement, routing, and per-iteration reconfiguration; reasons about cooling AND power together at aisle/row/server granularity; or builds its policy around the controllable-SaaS / opaque-IaaS asymmetry that governs every shared aisle.

How TAPAS works

Three actuators at three timescales, all shielding IaaS

Offline profiling

At datacenter bring-up, TAPAS models the layout, each server's inlet temperature and GPU temperature, server and GPU power versus load, and per-aisle airflow — via piecewise-polynomial regression (mean error under 1°C, and it generalizes to unseen values where random forests overfit). When a new LLM is onboarded, it profiles how every configuration knob shifts that model's temperature, power, and goodput, yielding a per-model Pareto frontier. Refined weekly from the prior week's load patterns.
↓ →
PLACE — the VM Allocator (arrival time) new

Rule-based (after Protean): first a VALIDATOR filters out any aisle that would exceed airflow or any row that would exceed power if this VM landed there; then a PREFERENCE rule steers opaque IaaS VMs to the coolest servers and flexible SaaS VMs to the warmest (since SaaS can be turned down later if a server heats up); then a BALANCE rule avoids over-concentrating IaaS or SaaS in any aisle/row, so every aisle keeps enough controllable SaaS load to shave later.
↓ →
ROUTE — the Load Balancer (every ~5 min) new

SaaS endpoints span many VM instances, so requests can be steered. TAPAS estimates, per aisle, the airflow the VMs demand; per row, the aggregate power; per server, the projected GPU temperature — and refuses to route a request to any instance that would push its aisle, row, or server past a limit. Among safe instances it then applies ordinary goals (reuse a customer's KV cache, concentrate for energy, spread for performance).
↓ →
RECONFIGURE — the Instance Configurator (per iteration) new

When placement and routing are not enough, each SaaS instance walks a hierarchy of knobs to fit its thermal/power/performance limits at maximum goodput. Ordered by cost: GPU FREQUENCY first (instant, no quality loss), then batch size and tensor parallelism (performance trade-offs, no quality loss), and only as a LAST RESORT model quantization and size — because those cost result quality AND require a multi-second model reload. Lightweight and cached; runs every LLM iteration.
↓ →
Phase- and knob-awareness new

The knobs act differently by phase, and TAPAS exploits it: lower tensor parallelism cuts total server power most in prefill but raises the hottest GPU's temperature in decode; smaller batches lower compute heat but, in decode, raise MEMORY temperature via more frequent fetches. So the optimal setting depends on whether the bottleneck right now is temperature or power, and which phase is running — resolution a coarse scheduler structurally lacks.
↓ →
Oversubscription & emergencies new

The reclaimed slack lets operators add racks into existing rows (a simulator sizes it). On a cooling or power failure, TAPAS recomputes each aisle's available airflow and each row's power, routes requests away from constrained servers, and turns down SaaS configs; only if that is insufficient does it fall back to hard-capping IaaS VMs. SaaS absorbs the emergency first; IaaS is touched last.

Evaluated on one-week production A100 traces (~1,000 servers, thousands of GPUs, 50/50 IaaS/SaaS) via a validated discrete-time simulator, plus a scaled real-cluster experiment (two rows, 80 servers, one hour) whose peak-power reduction matched simulation within 4% absolute. SaaS runs Llama2 on vLLM; IaaS uses real production power readings.

The idea

One resource is malleable, the other is sacred — build the whole policy on that. The SaaS/IaaS split is not a nuisance TAPAS tolerates; it is the axis the design turns on. Provider SaaS inference can be placed in warm servers, routed away from strained rows, and dialed down knob by knob, because the provider owns it and users bought a quality-of-service target, not a specific GPU frequency. Tenant IaaS VMs are opaque and contractually untouchable, so TAPAS shields them — coolest servers at placement, and hard-capped only in a genuine emergency after every SaaS lever is spent. The emergency table makes the philosophy vivid: in a thermal emergency the baseline uniformly caps everyone (IaaS performance down 22%), while TAPAS holds IaaS at zero impact and pays with 6% SaaS quality instead. The scheme spends the resource it is allowed to spend.

Match the actuator’s timescale to the disturbance’s. Datacenter thermal and power problems arrive at every timescale, so TAPAS answers at three. Slow structural imbalance (which rack is chronically hot) is fixed once, at placement. Medium drift (a row trending toward its power cap over minutes) is smoothed by routing, recomputed every few minutes. Fast phase transitions (prefill’s compute burst, decode’s memory pressure, millisecond to millisecond) are handled by per-iteration reconfiguration inside the instance. The ablation shows why all three are needed: each alone buys about 12%, pairs buy more, and only the full stack reaches 17% temperature and 23% power — because each actuator catches disturbances the others are too slow or too coarse to reach.

Order the knobs by what they cost, and pay in that order. Reconfiguration is a hierarchy, not a switch. GPU frequency is free (no quality loss, instant) so it goes first; batch size and parallelism trade performance but not quality, so they go next; quantization and model-size shrink quality and need a slow reload, so they are the last resort, used “only when necessary.” This is graceful degradation as an explicit cost-ordered ladder — spend the cheapest slack first, reach for the expensive slack only when the cheap runs out — the same disciplined-fallback shape this site keeps meeting, here governing how a cloud sheds heat without shedding the quality its customers are paying for.

A row approaches its power cap

Steering an LLM cluster away from a throttle

A row of GPU servers mixing tenant IaaS VMs and provider SaaS Llama2 endpoints. Numbers from the production-trace evaluation and the emergency table.

Step 1 of 5

Placement already set the stage

When these VMs arrived, TAPAS put the opaque IaaS VMs on the coolest servers and the flexible SaaS endpoints on warmer ones, and kept every aisle from over-concentrating either type. So this row has controllable SaaS load to work with — the placement rule deliberately preserved a lever for later, rather than packing all the adjustable work into one corner.

IaaS to cool servers because they cannot be turned down; SaaS to warm because they can
validator rule had already rejected any placement that would exceed aisle airflow or row power

Why it works

Cooling and power are provisioned for worst-case peaks that real aisles and rows rarely hit → shaving those peaks on demand converts stranded headroom into sellable capacity → up to 40% more compute from the same building, the direct TCO win.
SaaS inference is controllable and IaaS is opaque → extracting all slack from SaaS while shielding IaaS keeps the untouchable tenants whole → emergencies cost SaaS a little quality instead of costing IaaS real performance.
Disturbances span timescales from structural to millisecond → placement, minute-scale routing, and per-iteration reconfiguration each catch the band the others miss → the full stack reaches 17%/23% where any single actuator reaches ~12%.
The knobs affect temperature and power differently by phase and by which resource is the bottleneck → phase-aware reconfiguration picks the right knob for prefill versus decode, temperature versus power → heat and watts drop with minimal performance or quality loss.
Knobs are ordered by cost (frequency free, quantization expensive) → TAPAS spends cheap slack first and quality- costing slack last → degradation is graceful and quality SLOs hold under all but genuine emergencies.

What it costs

Quality is the emergency currency. Under normal load TAPAS holds quality; in a real thermal or power emergency it accepts up to 12% quality impact (6% in the paper’s thermal case) by routing some requests to smaller or quantized models — a deliberate, bounded trade, but a trade.
SaaS-fraction dependent. The flexibility IS the SaaS workload: an all-IaaS aisle leaves only placement to work with (limited gains), while all-SaaS reaches the full 23% temperature and 28% power reductions. A cloud thin on provider-managed inference sees less.
Profiling and modeling debt. Every LLM must be profiled per hardware, the datacenter thermally modeled, and the models refreshed weekly; accuracy of the placement and routing decisions rests on those profiles.
Reconfiguration latency. Frequency changes are instant, but switching model size or quantization means a multi-second reload — which is exactly why those knobs sit last and requests are held off an instance mid-transition.
Simulation-heavy evidence. The headline scale numbers come from a discrete-time simulator (validated within 4% against a two-row, one-hour real cluster); full-scale production deployment behavior is inferred, not shown.

The evaluation

One-week production A100 traces (~1,000 servers, thousands of GPUs, 50/50 IaaS/SaaS) through a discrete-time simulator validated against a scaled real-cluster run, against a thermal- and power-oblivious baseline plus every combination of the three components.

Reclaiming datacenter slack for LLM inference

Property	Oblivious baseline (traditional placement + routing)	CPU-era / power-only schemes (Flex, POLCA, μ-Serve)
What it can see and control
Peak temperature & row power	Reference (provisioned worst case)	Partial — power-only schemes miss thermal, and vice versa
Capacity & throttling		Some power headroom reclaimed
Performance & quality (normal load)	Held, but low utilization	Held
Behavior in a thermal/power emergency		Power capping, tenant-agnostic

Modeling: piecewise-polynomial regression (mean error <1°C, generalizes better than random forests). Real-cluster validation matched simulation within 4% absolute on peak power. Ablation: Place slightly leads Route and Config individually (it balances both IaaS and SaaS), but only the holistic combination reaches 17%/23%.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

TAPAS’s leverage is proportional to how much of the fleet is controllable SaaS inference: an all-IaaS aisle leaves only placement, and only an all-SaaS aisle reaches the full 23%/28% reductions. The gains under duress are paid in quality — up to 12% in a real emergency — a bounded and deliberate trade, but one that presumes customers accept occasional smaller-model answers to protect tail latency and tenant isolation. The approach carries standing profiling and weekly-modeling debt, and its headline scale numbers come from a discrete-time simulator — carefully validated within 4% against a two-row real cluster, but a simulator nonetheless. And it targets air-cooled datacenters; the authors argue the insights transfer to liquid cooling, but that is asserted, not shown.

Background

Worst-case provisioning, the datacenter floor edition

TAPAS is the newest entry in the site’s most persistent thread: worst-case provisioning wastes what per-member adaptation recovers. The warm-water-cooling page prices it in coolant valves, the inference-serving page in model variants, the memory page in PIM dosage — and TAPAS prices it in the cooling towers and power feeders themselves, sized for a peak that half the rows never reach. It shares the serving page’s exact substrate (accuracy/quality scaling on SaaS LLMs) but aims it at a new objective: not meeting demand under fixed hardware, but reclaiming thermal and power slack under a fixed building. Two further rhymes. The three-timescale actuator stack (place / route / reconfigure) is the timescale-split control this site keeps finding — periodic optimizer above, fast local mechanism below — as in the warm-water and quantum-crosstalk pages. And the cost-ordered knob ladder (frequency free, quantization expensive) is the graceful- fallback discipline of the best-effort-with-diagnosed-fallback family, here deciding how a cloud sheds heat without shedding the quality its customers pay for.

If you remember only three things

1. Datacenter cooling and power are provisioned for worst-case aisle/row peaks that real infrastructure rarely hits (half of rows draw ~28% below the hungriest), stranding capacity — and traditional CPU-era thermal/power scheduling cannot reclaim it for LLM inference because it is blind to inference's millisecond prefill (compute-bound) versus decode (memory-bound) phases, to the quality-versus-heat knobs only provider-managed SaaS models expose, and to the mix of controllable SaaS and untouchable IaaS tenants in every aisle.
2. TAPAS builds its whole policy on the SaaS-malleable / IaaS-sacred asymmetry and acts at three matched timescales: PLACE opaque hot IaaS VMs on cool servers and flexible SaaS on warm ones (balancing both across aisles/rows), ROUTE inference requests every few minutes away from any aisle/row/server nearing an airflow/power/thermal limit, and RECONFIGURE per-iteration down a cost-ordered knob ladder (GPU frequency free and first; quantization and model-size quality-costing and last) — with each actuator catching disturbances the others are too coarse or slow to reach.
3. On production Azure traces (validated within 4% against a real two-row cluster): peak temperature down 17%, peak row power down 23%, P99 latency and quality held under normal load, unlocking up to 40% more capacity and cutting thermal/power throttling 97%/99% — and in emergencies it protects tenant IaaS at zero performance impact by spending up to 12% SaaS quality, with the honest caveats that the leverage scales with the SaaS fraction and the headline numbers are simulator-driven.

Check yourself

Check your understanding

1. TAPAS treats SaaS and IaaS workloads fundamentally differently — steering, dialing down, and even hard-capping SaaS while shielding IaaS. Predict why this asymmetry is the organizing principle of the whole design, not just an implementation detail.
2. TAPAS acts at three distinct timescales — placement (VM arrival), routing (every ~5 minutes), reconfiguration (every LLM iteration). Why not just use the most powerful one everywhere?
3. When an instance must shed heat or power, TAPAS adjusts GPU frequency first and model quantization/size last. Predict the principle behind this ordering and why it matters for the quality guarantee.
4. TAPAS's effectiveness ranges from limited (all-IaaS aisle) to full 23%/28% reductions (all-SaaS aisle). A cloud architect is deciding whether to deploy it. Predict what determines the payoff and what the deployment implies.

Sources

ACM DL: TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms — ASPLOS 2025 (open access)
arXiv preprint 2501.02600