Going Green for Less Green: Optimizing the Cost of Reducing Cloud Carbon Emissions

Walid A. Hanafy, Qianlin Liang, Noman Bashir, Abel Souza, David Irwin and Prashant J. Shenoy

Shifting batch jobs to low-carbon hours is the standard demand-side climate lever, but nobody had priced it — concentrating demand into green windows idles prepaid reserved instances and spikes on-demand usage, so a 36% carbon cut can cost 68% more money and 5.3× the completion time (in clean-grid Sweden, 4% carbon for 76% cost) — GAIA makes the three-way trade explicit with policies that maximize carbon saved per unit of waiting and stay work-conserving on reserved capacity, doubling carbon savings per percentage of cost increase while cutting performance overhead 26%.

The argument in one glance

Problem

The grid's carbon content swings — and batch jobs can surf it

Electricity's carbon intensity varies 1.7–3.37× within a day (solar noon vs gas-fired evening) and up to 9× across regions. Batch workloads — ML training, HPC, analytics — are temporally flexible: run them when the grid is green and the same computation emits a fraction of the CO₂. With datacenter PUE already near its floor of 1.0, this demand-side shifting is one of the few remaining levers cloud USERS control.
What people did

Chase the greenest hours with suspend-resume

Carbon-aware schedulers like Wait Awhile and Ecovisor take the direct approach: given a job's length and deadline, pick the lowest-carbon-intensity time slots in the window, suspending execution whenever the grid turns dirty and resuming when it cleans up. Measured purely in grams of CO₂, this is optimal.
Why it fell short

Nobody priced it — and the price is brutal

Cloud cost efficiency rests on reserved instances: prepaid capacity at ~40% of on-demand price, worth it only if kept busy. Carbon-chasing concentrates demand into green windows — idling prepaid reserved capacity during dirty hours, then meeting the green-hour spike with expensive on-demand instances. Measured on a California trace: 36% carbon savings costs 68% MORE money and 5.3× the completion time. In Sweden's clean, stable grid: 4% carbon savings for 76% more cost and 4.9× the time. Prior systems also assume known job lengths, per-job deadlines, and interruptible execution — none of which production batch schedulers actually have.
This paper's idea

Price the trade-off; then optimize inside it

GAIA is a carbon-aware layer for cloud batch schedulers (built on AWS ParallelCluster/Slurm) that treats carbon, performance, and cost as one three-way trade. Its Carbon-Time policy maximizes carbon saved PER unit of added completion time — refusing delays that buy little; its RES-First variant is work-conserving on reserved instances — an idle prepaid server always runs a job immediately; Spot-First puts short jobs on 20%-price spot instances. No job lengths, no deadlines, no suspensions required — just queue-level bounds. Result: double the carbon savings per percentage of cost increase, with 26% less performance overhead than existing policies.

Why this problem is worth solving

Demand-side carbon optimization has quietly become the systems community’s favorite sustainability lever, and for good reason:

grams of CO₂-equivalent emitted per kilowatt-hour of grid electricity — a time-varying mix signal. Hydro, nuclear, wind, and solar drive it down; coal and gas drive it up. Services like Electricity Maps publish real-time values and forecasts per region

is visible in real time, batch jobs are movable, and the technique needs no new hardware. The prior wave of work — including Ecovisor, covered elsewhere on this site — proved the mechanism works.

This paper’s contribution is to point out what that wave ignored: money, and specifically the structure of cloud pricing.

cloud capacity purchased on 1–3-year contracts at a steep discount (~40% of on-demand price here) — paid in full whether used or not. Their economics depend entirely on utilization: every idle reserved hour raises the effective price of the hours actually used

are how cost-conscious organizations buy compute; on-demand fills spikes;

deeply discounted surplus capacity (~20% of on-demand price) that the provider can revoke at any moment — attractive for work that can tolerate losing progress

fill gaps cheaper still. A carbon-aware scheduler that reshapes demand without understanding this portfolio doesn’t just slow jobs down — it silently repriced the whole cluster. And the Sweden example shows the failure can be nearly total: in a grid that is already clean and stable, aggressive carbon-chasing pays 76% more for a 4% carbon reduction. If carbon-aware computing is to be adopted by anyone with a budget, the three-way trade-off has to be measured, exposed, and optimized — which is exactly the gap GAIA fills.

What people did before, and where it broke

Suspend-resume carbon chasers. Wait Awhile picks the lowest-carbon slots summing to the job’s length within a deadline; Ecovisor runs jobs only while carbon intensity is below a threshold. Both achieve the best raw carbon numbers — and the worst everything else: they need the job length and a per-job deadline up front, require suspend/resume support, and are aggressive by construction — they’ll delay a job to the last permissible moment for any nonzero carbon gain, no matter how small, fragmenting demand and maximizing waiting.
Cost-aware cloud batch scheduling. A separate literature optimizes the reserved-vs-on-demand mix (e.g., waiting for a reserved instance to free up rather than paying for on-demand). Its core trick — spreading demand smoothly over time — is the exact opposite of carbon-chasing, which concentrates demand into green windows. The two objectives collide head-on, and no prior scheduler mediated the collision.
Unrealistic scheduler assumptions. Production batch systems (Slurm and kin) know a job’s queue — short or long, an upper bound — not its length; users rarely give meaningful deadlines; and application-agnostic checkpoint/restart is notoriously hard in HPC settings. Policies requiring exact lengths, deadlines, and interruptibility don’t deploy as-is.

How GAIA works

GAIA: a carbon-aware layer over a cloud batch scheduler

Standard job submission

Users submit to length-bounded queues exactly as in Slurm today — a short queue (≤2 hrs here) and a long queue. No job lengths, no deadlines, no workflow changes. The administrator sets system-wide maximum waits per queue (6 hrs short, 24 hrs long): a guarantee, not a user burden.
↓ →
Carbon Information Service

Third-party feeds (Electricity Maps) supply per-region real-time carbon intensity and forecasts — prior work shows the forecasts are accurate enough to plan against.
↓ →
Carbon-aware start-time policies new

For a job arriving at time t, choose a start within [t, t+W): Lowest-Slot starts at the single greenest slot (no length knowledge needed); Lowest-Window minimizes total footprint over the queue's AVERAGE job length; Carbon-Time maximizes carbon-saved-per-completion-time — the ratio policy that refuses delays yielding little carbon per hour of waiting. Jobs then run uninterrupted to completion.
↓ →
Work-conserving reserved use (RES-First) new

Prepaid capacity is never left idle on purpose: an arriving or queued job starts IMMEDIATELY if a reserved instance is free; otherwise it waits for its carbon-optimal start and takes an on-demand instance — unless a reserved instance frees up first, in which case it runs early. Cost efficiency first, carbon within it.
↓ →
Spot integration (Spot-First / Spot-RES) new

Short jobs execute their carbon-aware schedule on 20%-price spot instances, restarting on-demand if evicted (progress assumed lost, as in HPC practice); long jobs stick to RES-First. The combination (Spot-RES) covers each purchase option's sweet spot.
↓ →
Accounting + deployment

Carbon (energy × intensity per slot), cost (usage-based for on-demand/spot, prepaid for reserved), and waiting are tracked per job. Implemented in ~2k lines of Python on AWS ParallelCluster with PySlurm, plus a year-scale simulator; open source.

From UMass Amherst (ASPLOS 2024) — the same group behind Ecovisor, here effectively auditing the cost of its own earlier approach. GAIA is scheduler-agnostic by design and currently runs jobs uninterruptibly; suspend-resume is named future work.

The idea

Carbon savings must be priced in the currencies users already pay. The paper’s analytical core is the demand-reshaping argument: cost-aware scheduling spreads demand (to keep prepaid capacity busy), carbon-aware scheduling concentrates it (into green windows) — so carbon-chasing doesn’t merely add waiting, it degrades the utilization that reserved-instance economics rest on, raising the effective price of every computed hour. The reserved-capacity sweep makes the structure legible: with few reserved instances (below the demand floor), carbon optimization is nearly free; in a middle regime, each added reserved instance trades carbon for cost; with excess capacity, flexibility is gone and money is wasted regardless. GAIA doesn’t dictate a point on this curve — it exposes the curve and delivers the best carbon available at whichever point the operator picks (e.g., 15 rather than 18 reserved instances: 4% more cost for 10% carbon savings, still 55% cheaper than pure on-demand).

Optimize the ratio, not the raw number. Existing policies minimize grams of CO₂, full stop — which is why they delay jobs for arbitrarily small gains (the paper circles time slots with near-identical carbon intensity that Wait Awhile agonizes between). Carbon-Time instead maximizes carbon saved per unit of completion time: a delay must earn its waiting. This single change captures 80–90% of the aggressive policy’s savings at 20–30% less waiting — and it degrades gracefully in stable grids, where the right answer is mostly “don’t bother waiting,” the answer the Sweden example shows pure carbon-chasers get catastrophically wrong.

Realism is a feature, not a compromise. GAIA’s policies need only what Slurm actually knows: queue-level length bounds and historical averages. The surprising empirical result is how little the missing information costs — Lowest-Window, knowing only the queue average and never suspending anything, lands within 3% of Ecovisor and 16% of Wait Awhile on carbon. The job-length analysis explains why: half of all carbon savings comes from mid-length (3–12 hr) jobs, which are long enough to matter but short enough to shift; sub-hour jobs (half the trace!) contribute 10%, and multi-day jobs just 7.5% — they straddle the grid’s ~24-hour carbon cycle, so shifting barely helps. Coarse knowledge suffices because the savings live where coarse knowledge is good enough.

A job’s journey through GAIA

One long-queue job under Spot-RES (GAIA's combined policy)

W = queue's max waiting time (24 hrs for long jobs). J-avg = queue's historical average job length. CST = carbon saved per completion time, the Carbon-Time objective. CI = grid carbon intensity. Reserved = prepaid at 40% of on-demand price; spot = 20%, revocable.

Step 1 of 6

Arrival: queue, not questionnaire

The job arrives in the long queue. GAIA learns only what Slurm would know: it must start within 24 hours, and long-queue jobs historically average some length J-avg. No deadline, no length declaration, no user configuration.

guarantees: start ≤ W after arrival, run uninterrupted
assumptions match production batch schedulers

Why it works

Reserved instances are prepaid → an idle reserved hour is money already spent producing nothing → work-conservation (run immediately on idle prepaid capacity) is simultaneously the cost-optimal and, per dollar, the carbon-rational move; carbon optimization applies only to demand that must be bought anyway.
Pure carbon minimization accepts any delay for any gain → the Carbon-Time ratio makes waiting a currency that carbon must purchase at a decent exchange rate → savings concentrate where they’re cheap (deep CI valleys nearby) and evaporate where they’re not (stable grids), automatically de-escalating in exactly the situations where aggression was disastrous.
Carbon savings are dominated by mid-length jobs (3–12 hrs = ~50% of savings; sub-hour jobs = 10%; multi-day jobs = 7.5%, straddling the grid’s diurnal cycle) → queue-average length estimates suffice → policies deployable on real schedulers give up only 3–16% of the carbon that clairvoyant, interruptible policies achieve.
Spot instances discount execution without dictating WHEN it happens → short jobs can follow the carbon-aware schedule at 20% price (full carbon savings + 17% cost savings), while eviction risk compounding with runtime cleanly assigns long jobs to reserved/on-demand — each purchase option is used where its economics work.
The trade-off is exposed, not hidden → operators choose reserved capacity knowing the three regimes (free savings / genuine trade / pure waste) → small deliberate moves (18→15 reserved instances) buy meaningful carbon at explicit, chosen cost.

The evaluation

A real prototype on AWS ParallelCluster (week-long, 1,000-job Alibaba-PAI sample, 5× expedited, c7gn instances at actual AWS pricing: reserved 40%, spot 20% of on-demand) plus a year-scale simulator replaying 100k-job traces from three production sources — Alibaba-PAI (ML), Azure-VM (cloud), and LANL Mustang (HPC/MPI) — against 2022 hourly carbon intensity from six regions spanning low/medium/high average intensity and stable/variable profiles. Baselines cover both prior camps: carbon-agnostic NoWait and cost-aware AllWait-Threshold on one side, suspend-resume Wait Awhile and Ecovisor (with their length/deadline/interruptibility advantages granted) on the other.

Carbon-agnostic vs carbon-only vs three-way-aware scheduling

Property	Cost-aware, carbon-agnostic (NoWait / AllWait-Threshold)	Suspend-resume carbon chasers (Wait Awhile / Ecovisor)
What they need from users and systems	Nothing new
Carbon emissions
Cost (hybrid cluster, reserved at mean demand)
Performance (waiting / completion time)	NoWait: none; AllWait-Threshold: highest waiting
The exchange rate that matters	No carbon per cost — the origin point	Poor: much carbon, purchased at extreme cost and delay; in stable grids, almost none at any price

Prototype: AWS ParallelCluster, 1k-job week-long Alibaba-PAI sample, South Australia CI, 9 reserved instances unless noted. Simulator: year-long 100k-job traces (Alibaba-PAI, Azure-VM, LANL Mustang), 2022 hourly Electricity Maps CI for six regions; perfect CI forecasts assumed; reserved = 40%, spot = 20% of on-demand price; spot evictions lose all progress. Regional range: 27.5% carbon savings in variable South Australia down to 1% in stable Kentucky.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

GAIA optimizes operational carbon only, taking grid CI feeds as ground truth and assuming perfect forecasts (defensible per prior accuracy studies, but an assumption). Its savings are hostage to geography: 27.5% in swingy South Australia, 1% in coal-steady Kentucky — and the normalized-vs-total distinction cuts deeper, since equal percentage savings mean very different grams across regions. The prototype is small (1k jobs, ≤4-CPU instances, 5× time acceleration); the year-scale results are simulation. Uninterruptibility is both realism and limitation — suspend-resume would add savings and is explicitly future work, as are geographic shifting and checkpointing-aware spot policies. And the reserved-instance economics assume instances can be powered off when idle (no energy/carbon charge), plus a specific 40%/20% pricing ratio; different contracts move the trade-off points, if not the structure.

Background

The sustainability cluster grows a price tag

This site’s sustainability papers form a neat progression: Carbon Explorer weighs supply-side options (renewables, batteries, shifting) for operators; Ecovisor virtualizes the energy system so applications can adapt themselves; Junkyard Computing reuses old phones to attack embodied carbon. GAIA — from the same UMass group as Ecovisor — is the reckoning installment: it audits the demand-side technique the community (its own authors included) championed, and finds the cost dimension missing from every prior analysis. The deeper pattern is a classic systems lesson wearing green: single-objective optimization silently spends the objectives you didn’t measure. The fix is likewise classic — turn the hidden costs into an explicit multi-objective trade-off, then let the ratio (carbon per dollar, carbon per hour of waiting) rather than the raw metric drive decisions. And the job-length finding — savings live in the 3–12-hour middle, not the short-job bulk or multi-day tail — is a portable insight for anyone building carbon-aware anything.

If you remember only three things

1. Carbon-aware temporal shifting has a hidden bill: concentrating batch demand into green windows guts the utilization that prepaid reserved-instance economics depend on, so the state-of-the-art carbon chasers bought 36% carbon savings for 68% more cost and 5.3× the completion time in California — and in clean, stable Sweden, 4% carbon for 76% cost — while assuming job lengths, deadlines, and interruptibility that production schedulers don't have.
2. GAIA's three moves are all about exchange rates rather than raw grams: Carbon-Time maximizes carbon saved per unit of completion time (refusing delays that don't earn their waiting — 80–90% of aggressive savings at 20–30% less delay), RES-First is work-conserving on prepaid capacity (an idle reserved instance always runs a job now), and Spot-First discounts the carbon-aware schedule itself by 5× for short jobs — needing only queue-level bounds, because half of all carbon savings comes from 3–12-hour jobs where coarse estimates suffice.
3. The measured outcome across a real AWS prototype and year-long trace simulations: double the carbon savings per percentage of cost increase and 26% less performance overhead than prior carbon-aware policies, with the full carbon-performance-cost curve exposed — including its geographic honesty: 27.5% savings in variable South Australia, 1% in stable Kentucky, and a warning that regions with equal percentage savings can differ wildly in actual grams removed.

Check yourself

Check your understanding

1. Delaying jobs to greener hours seems like it should cost nothing but time. Predict how carbon-aware scheduling ends up RAISING cloud costs by 68%.
2. In Sweden, aggressive carbon-aware scheduling delivered 4% carbon savings for 76% more cost and 4.9× longer completion times. What is the root cause, and how does GAIA's Carbon-Time policy avoid this failure mode?
3. Half the jobs in the trace are shorter than an hour. How much of the total carbon savings do they contribute, and which jobs actually carry the savings?
4. Spot instances cost 20% of on-demand and don't constrain WHEN a job runs — seemingly perfect for carbon-aware scheduling. Why does GAIA restrict them to short jobs only?