Characterizing Power Management Opportunities for LLMs in the Cloud

Microsoft measured how language models actually draw power and found inference clusters using only 80% of what was provisioned — so POLCA safely packs 30% more servers under the same power contract, with two frequency-capping tripwires as the safety net.

Why this problem is worth solving

A datacenter’s power budget is a physical contract: generators, transformers and utility agreements fix how many megawatts the building may draw. Within it, every server is provisioned — allocated a slice of budget equal to its worst-case consumption — because exceeding the total is not a performance problem but a safety event: uninterruptible power supplies (UPSes) tolerate a 133% overload for only about 10 seconds before the facility must shed load or trip.

GPU servers make this provisioning brutally expensive. A DGX A100 draws around half to 60% of its power in its eight GPUs, each provisioned at its full 400-watt rating — and measured peak GPU power can exceed even the aggregate GPU rating by up to 500 watts per server. If real workloads rarely approach these peaks, the gap between provisioned and used power is capacity paid for and locked away — at exactly the moment when GPU capacity is the scarcest resource in the industry and a new datacenter takes years to build.

Power oversubscription reclaims that gap: deploy more servers than the budget nominally allows, betting that aggregate draw stays under the limit, with throttling as the backstop. CPU fleets have done this for years. For GPU fleets serving large language models, two questions were open: is there actually headroom — do these workloads leave provisioned power unused? — and can the bet be enforced safely given the cloud’s weak GPU controls? This paper answers both, with measurements first and a system second.

What people did before, and where it broke

CPU-era oversubscription rests on two supports that crumble under GPUs.

Fast, unified controls. A CPU server exposes in-band interfaces (like Intel RAPL) that cap the whole server’s power quickly and reliably. In the cloud, GPUs run in direct-device-access mode — the customer’s virtual machine owns the driver, so the provider cannot use fast in-band tools. What remains is the out-of-band path through the board management controller: per-GPU frequency caps and power caps that take up to 40 seconds to take effect, and one fast lever — the hardware power brake, which lands in 5 seconds but slams every GPU to 288 MHz, nearly halting work. An enforcement toolkit of slow nudges and one sledgehammer.

Known workload behavior. Oversubscription is a bet on statistics, and nobody had published the statistics for LLM clusters. The paper’s characterization fills this in, and its findings are the design:

• Inference has two phases with opposite power profiles. Processing the prompt is parallel and compute-intensive — power spikes, often above the GPU’s rated power, for under a second. Generating tokens is sequential and memory-bound — long, stable, much lower draw. Longer inputs and bigger batches raise the spike; longer outputs just stretch the quiet phase.
• Server spikes do not align. Different requests hit different servers at different moments, so prompt-phase spikes statistically multiplex away: the production inference cluster peaks at only 79–80% of provisioned power, moving diurnally, with worst spikes of 9% in 2 seconds and 11.8% over 40 seconds.
• Training is the opposite. Synchronous training jobs peak at 97% and swing by up to 37.5% within 2 seconds — thousands of GPUs dipping and surging together at every iteration boundary. Headroom: ~10%, and the swings themselves menace the power infrastructure.
• Frequency capping beats power capping. Power caps are reactive — sub-second prompt spikes sail through before enforcement reacts. Frequency caps are proactive and, better, superlinear: up to 20% of peak power reclaimed for at most 7% performance loss (GPT-NeoX gives up 13% power for zero loss). Larger models pay more; BLOOM-176B is the worst case.

The conclusions write themselves: separate inference clusters from training clusters (mixing them lets training’s swings preclude inference’s opportunity), oversubscribe the inference clusters, and enforce with frequency caps — if the slow controls can be made safe.

The idea

Size the thresholds to outrun your own slowness

POLCA’s central design problem is arithmetic under latency: caps take up to 40 seconds to land, and production data says row power can climb at most 11.8% in 40 seconds. So the upper threshold T2 sits at 89% — even if power spikes maximally the instant T2 trips, it reaches at most ~100% just as the caps arrive, and the 10-second UPS deadline is never tested (the power brake itself lands in 5 seconds if the 100% line is ever crossed anyway). The lower threshold T1 sits at 80%, ten points below T2, so that capping cheap work early usually prevents T2 from tripping at all.

Cap by priority, in stages

Workloads declare one of two priorities, and the cloud’s allocator mixes them within every row (the evaluation uses summarization as low priority, search as high, chat split evenly). At T1, all low-priority GPUs are frequency-capped to 1275 MHz — the A100’s base frequency, the minimum the cloud already promises, chosen because anything lower breaks the low-priority latency SLO. At T2, low-priority drops further to 1110 MHz; only if power still climbs do high-priority GPUs get a nearly-free trim to 1305 MHz. Uncapping happens 5% below each threshold — hysteresis, so the system does not flap between capping and uncapping.

The frequency-not-power choice inherits the characterization directly: frequency caps act on both inference phases proactively, their cost is superlinear-cheap, and unlike power caps they cannot be outrun by a sub-second prompt spike.

Live inside the cloud’s constraints

Every control flows out-of-band — power manager to rack manager to board management controller to GPU — never through the customer’s VM, and provider settings override customer settings. Policy parameters (T1, T2, frequencies) are retuned periodically from power traces plus capping history as models evolve; robustness to short-term drift (a model update that raises power 5%) is evaluated explicitly, and POLCA holds its SLOs where simpler policies break.

row power: 74% | caps: none
server-level spikes: frequent, uncorrelated, absorbed

Why it works

Because prompt-phase spikes last under a second and arrive uncorrelated across servers, they multiplex away at the row level — so the row, not the server, is the honest unit of power truth, and it runs at only ~80% of provision. → Because frequency capping is proactive and superlinear — up to 20% of peak power for at most 7% performance — cheap enforcement exists, provided it is triggered in time. → Because the worst 40-second power climb is 11.8% and T2 sits 11% below the ceiling, the slow out-of-band path always wins the race against the surge; the two-stage, priority-ordered policy spends almost all of the capping cost on work that tolerates it. → Because training clusters obey none of these statistics — 97% peaks, coordinated 37.5% swings — the same measurements that justify oversubscribing inference justify separating inference from training, and using capping in training clusters only to tame swings. → Therefore a provider adds 30% more servers per inference row under an unchanged utility contract — capacity that arrives in weeks instead of the years a new datacenter takes — while the emergency brake never fires.

What it costs

• Low-priority work absorbs the policy. The design deliberately spends its capping budget on low-priority workloads — up to 50% p99 latency degradation is inside their SLO. Customers must accept (and price) that tier.
• It needs a priority mix. If a row skews too high-priority, there is too little cheap work to cap at T1, and high-priority p99 SLOs start to break — the allocator must actively maintain the blend.
• Thresholds are workload-derived. T1, T2 and the frequencies come from profiling; a model update that shifts power patterns erodes margins until POLCA is retuned (it survives a simulated 5% drift; larger shifts need reconfiguration).
• Training clusters are excluded. The headroom story is inference-only; for training, capping merely softens swings — oversubscription there remains, in the authors’ word, tricky.
• The evaluation is a simulation. A discrete-event simulator replays six weeks of production power traces (replicated within 3% error using open-source models, BLOOM-176B as the capping-sensitive worst case) on 40 DGX-A100 servers — faithful, but not a live deployment.

The evaluation

Six weeks of power telemetry from a production Azure LLM inference cluster, replicated with open-source models (mean absolute percentage error under 3%), replayed in a discrete-event simulator of a 40-server DGX-A100 row. Week one tunes the policy; weeks two through six evaluate. Workload mix: summarization (low priority), search (high), chat (half and half), all on BLOOM-176B — the model the characterization found most performance-sensitive to capping.