Societal infrastructure in the age of Artificial General Intelligence

Generative AI is demanding compute faster than ever, right as the old ways of making chips automatically faster and more efficient have run out of steam — so in this ASPLOS 2024 keynote (Amin Vahdat, Google) the argument is that the next generation of computing infrastructure cannot come from general-purpose scaling anymore; it must be increasingly specialized and co-designed with the algorithms and software that run on it, all while treating security and sustainability as first-order concerns rather than afterthoughts.

The argument in one glance

Problem

AI wants more compute than ever, just as the easy efficiency gains have run out

This is a keynote talk, not a research paper, and its framing is a tension. On one side, emerging Generative AI services are placing unprecedented demand on compute: training and serving large models consumes orders of magnitude more processing, memory, and data movement than the workloads that came before. On the other side, the architectural patterns that the industry leaned on for decades to keep making computing more efficient have stalled. For a long time, each new generation of chips got automatically faster and more energy-efficient because transistors kept shrinking, and general-purpose processors kept getting better without anyone having to rethink the whole stack. That free ride has largely ended. So the field faces a widening gap: demand is exploding upward exactly as the usual supply of efficiency has flattened. The talk sets out to ask what the next generation of computing infrastructure must look like to close that gap — not as an abstract exercise, but grounded in recent, concrete experience building large-scale systems at Google, spanning networks, accelerators, and servers.
What people did

Keep scaling general-purpose computing the way we always have

The default response, and the one that worked for a generation, is to keep relying on general-purpose scaling: buy more of the same kind of servers, wait for the next process node to make them faster, and treat the hardware, the algorithms, and the software as separate layers that each improve on their own schedule. Under this model, infrastructure is largely uniform and interchangeable, efficiency comes mostly from the underlying silicon getting better, and concerns like security and energy use are handled as add-ons layered on top of a design that was optimized for raw performance.
Why it fell short

General-purpose scaling can no longer keep pace with the demand

That approach breaks precisely under the new conditions. Because the automatic, technology-driven efficiency gains have slowed, throwing more identical general-purpose hardware at the problem yields diminishing returns while costs — in dollars, in power, in carbon — climb steeply. Generative AI's appetite grows faster than uniform scaling can satisfy, so the gap keeps widening. Treating the layers as independent also leaves large efficiencies unclaimed: hardware built without knowledge of the specific algorithms it will run, and algorithms written without regard to the hardware's strengths, both settle for far less than a jointly-designed system could achieve. And handling security and sustainability as afterthoughts becomes untenable at this scale, where a single design choice can move enormous amounts of energy or expose an enormous attack surface. The keynote's claim is that continuing on the general-purpose path is not enough; the infrastructure itself has to change shape.
This paper's idea

Specialize, co-design with software, and make security and sustainability first-order

The course the talk charts has a few connected commitments. First, computing must become increasingly specialized: rather than one general-purpose design for everything, infrastructure should include hardware tailored to the shapes of the workloads that dominate — the domain-specific accelerators, purpose-built networks, and tuned servers that a company like Google has been building across its fleet. Second, that specialization only pays off when hardware is co-optimized with the algorithms and software that run on it: the model, the compiler, the runtime, and the silicon are designed together so each is shaped to the others, unlocking efficiencies no single layer could reach alone. Third and fourth, this must all happen while fundamentally focusing on security and sustainability — not bolting them on afterward, but treating them as first-order design constraints, because at the scale of societal AI infrastructure the energy footprint and the trustworthiness of the system are as central as its speed. The talk uses recent examples at Google, from networks to accelerators to servers, to illustrate both the challenges and the opportunities of building computing this way for the age of increasingly capable AI.

Why this problem is worth solving

The premise that makes this keynote urgent is that two long-running trends have crossed, and the crossing is structural rather than temporary.

For decades, computing improved on autopilot. As transistors shrank, each generation of chips packed in more of them and ran them faster at similar power, so software got faster simply by waiting for new hardware — and general-purpose processors, the same CPUs running everything, kept riding that curve. Two physical trends powered this: the steady doubling of transistor density, and the property that shrinking transistors kept their power density roughly constant. Both have slowed dramatically. Transistors no longer get cheaper and more efficient at the old pace, and pushing clock speeds or core counts on general-purpose designs now costs disproportionate power. The consequence is that the industry can no longer count on the substrate to deliver efficiency for free. This is why the talk treats the moment as an inflection point: it is not that one product needs tuning, but that the entire mechanism the field relied on to meet growing demand has weakened, right as demand from Generative AI is surging. When the automatic supply of efficiency stalls and demand accelerates, the only way to close the gap is to change how systems are designed — which is the whole subject of the talk.

The second reason it matters is scale: at the size of global AI infrastructure, choices about specialization, energy, and security stop being technical details and become societal ones.

The title’s phrase — societal infrastructure — is deliberate. When AI services are woven into everyday life, the computing that powers them becomes infrastructure in the same sense as roads or the power grid: something many people depend on, whose efficiency and reliability have consequences far beyond any one company. That reframing raises the stakes on every design decision. Specialization is not just a performance trick but a way to keep the whole enterprise economically and physically feasible as demand grows. Energy use is not a line item but a sustainability question with real environmental weight, because datacenters at this scale consume enormous power and carbon. And security is not a feature but a foundation, since infrastructure this widely relied upon is both a bigger target and a bigger risk if it fails. This is why the keynote insists that security and sustainability be first-order concerns co-equal with performance: at societal scale, an architecture that is fast but wasteful or insecure is not actually fit for purpose. The value of getting the direction right is correspondingly large — it shapes what the next decade of computing can afford to do.

What people did before, and where it broke

Ride transistor scaling for free efficiency. Assume each process generation makes general-purpose chips faster and more efficient automatically. Where it broke. The density and power-density trends that powered this have slowed sharply, so the automatic gains the industry planned around no longer arrive on schedule.
Scale by adding more uniform general-purpose servers. Meet rising demand by buying more of the same interchangeable hardware. Where it broke. With per-unit efficiency gains stalled, uniform scaling yields diminishing returns while dollar, power, and carbon costs climb — and Generative AI’s demand grows faster than this can keep up.
Design hardware, algorithms, and software as separate layers. Let each layer improve independently on its own timeline. Where it broke. Hardware ignorant of its workloads and algorithms ignorant of their hardware both leave large joint efficiencies unclaimed that only co-design can capture.
Treat security and energy as add-ons. Optimize for performance first, then layer on protection and efficiency measures. Where it broke. At societal AI scale, energy footprint and attack surface are too consequential to retrofit; afterthought handling leaves systems wasteful or exposed.
The gap. No path built purely on general-purpose scaling can meet exploding AI demand within acceptable cost, energy, and security limits — which is why the talk argues the infrastructure itself must change shape.

The shape of the argument

From general-purpose scaling to specialized, co-designed, secure, sustainable infrastructure

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Specialize instead of scaling one general-purpose design. The talk’s first commitment is that, with automatic scaling stalled, efficiency now comes from building hardware tailored to the workloads that matter rather than more of the same uniform servers. Turning to purpose-built, domain-specific hardware to get performance and efficiency that a general-purpose design cannot is the same specialization strategy the work on software-oriented acceleration for heterogeneous systems-on-chip (Cohort) covered elsewhere on this site pursues, building tailored accelerators rather than leaning on general-purpose cores. The keynote generalizes that instinct to a whole fleet, from networks to accelerators to servers.

Design the hardware and the software together. The second commitment is that specialization only pays off when the silicon is co-optimized with the algorithms, compilers, and runtimes that run on it, so each is shaped to the others rather than evolving in isolation. Jointly optimizing the model and the hardware so that the algorithm is designed with the machine in mind is the same co-design principle the work on hardware-optimized neural architecture search (HyperScale) covered elsewhere on this site is built on, searching for algorithms that fit the hardware they will run on. The talk elevates this from a single model-hardware pair to a design discipline for the whole infrastructure stack.

Put sustainability and security at the center, not the margins. The third commitment is that energy, carbon, and security must be first-order design constraints — because at societal AI scale the environmental footprint and the trustworthiness of the system are as consequential as its speed. Treating a processor’s carbon and sustainability as a quantity to design against from the start, rather than a cost discovered afterward, is the same sustainability-first framing the work on a first-order carbon model for processor design (FOCAL) covered elsewhere on this site adopts, making environmental impact a design input. The keynote pairs that sustainability focus with an equally fundamental focus on security, arguing both belong at the core of how future infrastructure is built.

Walking through the talk’s reasoning

How the keynote moves from the problem to the charted course

A walkthrough of the argument as stated in the talk's abstract. This is a keynote; the elaboration below is standard background on the trends it names, not additional claims from the talk.

Step 1 of 5

Name the inflection point

The talk opens by identifying a crossing: Generative AI is placing unprecedented demand on compute at the same time that the established architectural patterns for improving efficiency have stalled. The demand curve and the supply-of-efficiency curve are moving in opposite directions.

This tension is the premise everything else responds to.

Why it works

Specialization recovers efficiency that scaling no longer supplies. Automatic per-generation gains have stalled → but tailoring hardware to the dominant workloads → so efficiency comes from fit-to-purpose design rather than from the substrate improving.
Co-design unlocks joint gains isolated layers miss. Separately optimized hardware and algorithms each settle for less → but shaping the silicon, model, and software to one another → so the system reaches efficiencies no single layer could achieve alone.
First-order security suits infrastructure everyone relies on. Bolted-on protection leaves a large attack surface exposed → but making security foundational to the design → so trustworthiness scales with how widely the infrastructure is depended upon.
First-order sustainability matches the scale of the footprint. Afterthought efficiency wastes energy at enormous scale → but folding carbon and power into the core objectives → so the environmental cost is designed down rather than discovered late.
Grounding in real fleets makes the direction credible. Abstract predictions are easy to doubt → but drawing the argument from Google’s networks, accelerators, and servers → so the charted course rests on concrete large-scale experience.

What it costs (trade-offs and scope)

This is a vision keynote, not a results paper. It charts a direction and cites experience; it does not present a single system, benchmark, or measured result, so there is nothing here to reproduce or quantify.
Specialization trades flexibility for efficiency. Hardware tailored to today’s dominant workloads is, by construction, less general — a real risk if the workloads that matter shift, which the tension between specializing and staying adaptable makes a genuine cost.
Co-design raises engineering coupling. Designing hardware, algorithms, and software together is powerful but harder to organize than letting each layer evolve independently, demanding coordination across teams and longer joint design cycles.
Security and sustainability as first-order goals can compete with raw speed. Elevating them to co-equal constraints means sometimes not choosing the fastest option, a trade the talk argues is necessary at societal scale but a trade nonetheless.
The specifics live in the talk, not the abstract. The concrete Google examples, the exact challenges, and the detailed opportunities were presented in the keynote itself; this page can only convey the argument the abstract states, elaborated with standard background on the named trends.

The comparison

Two models for meeting computing demand

Property	General-purpose scaling	The AGI-era course
Source of efficiency
Relationship between layers
Role of security
Role of sustainability
Fit for surging AI demand

The contrast the keynote draws. General-purpose scaling met demand for decades by riding transistor improvements and treating hardware, algorithms, and software as separate layers, with security and energy handled afterward. The talk argues that with those automatic gains stalled and Generative AI demand surging, infrastructure must instead be increasingly specialized, co-optimized with algorithms and software, and designed with security and sustainability as first-order concerns. This reflects the abstract's stated direction, not a measured comparison.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This is an ASPLOS 2024 keynote talk by Amin Vahdat (Google), and this explainer is based only on its published abstract and TLDR — there is no full paper, dataset, or set of results behind it, so this page is marked a draft. The talk’s stated thesis is what is reported here: computing is at an inflection point where Generative AI places unprecedented demand on compute while established architectural patterns for improving efficiency have stalled; the talk discusses the likely needs of the next generation of computing infrastructure, drawing on recent examples at Google spanning networks, accelerators, and servers; and it charts a course in which computing must be increasingly specialized and co-optimized with algorithms and software, while fundamentally focusing on security and sustainability. Everything beyond that — the specific Google systems shown, particular numbers, and the detailed challenges and opportunities discussed live — was part of the talk itself and is not contained in the abstract. The background provided here about the end of automatic transistor scaling, domain-specific specialization, hardware-software co-design, datacenter energy, and security is widely-known context that helps explain the trends the talk names; it is not additional content from the talk, and none of it should be read as specific claims Vahdat made. For the actual arguments, examples, and any figures, the keynote itself is the source.

Background

Why this keynote sits at the center of a whole field's response to AI's compute demand

It helps to see this talk as a high-level framing of concerns that a large body of computer-systems research is working out in detail. The end of easy transistor scaling and the surge of AI compute have pushed the field toward exactly the directions the keynote names. Specialization has produced a wave of domain-specific accelerators; co-design has produced techniques that shape models and hardware to each other; and the sheer energy footprint of AI has made sustainability and power management central research topics rather than niche ones. The keynote’s value is in tying these threads into one argument about what societal-scale infrastructure must become, from the vantage point of an organization operating some of the largest such infrastructure in the world. The particular pressure it dwells on — the unprecedented compute and energy demand of Generative AI in the datacenter — is the same terrain the work on characterizing power-management opportunities for large language models in the cloud (power management for LLMs in the cloud) covered elsewhere on this site occupies, studying how to run these workloads within real energy limits. Where that work measures one facet in depth, this keynote sketches the whole landscape and argues for the shape of the response: specialize, co-design, and hold security and sustainability at the center.

If you remember only three things

1. This is an ASPLOS 2024 keynote by Amin Vahdat of Google, and only its abstract is available — there is no full paper or results. Its framing is an inflection point: Generative AI is placing unprecedented demand on compute at exactly the moment when the architectural patterns that delivered efficiency for decades — the automatic gains from shrinking transistors and general-purpose scaling — have stalled. Demand is surging while the usual supply of efficiency has flattened, opening a widening gap that conventional scaling cannot close.
2. The course the talk charts has connected commitments. Computing must become increasingly specialized, with hardware tailored to dominant workloads rather than uniform general-purpose servers. That specialization must be co-optimized with the algorithms and software that run on it, so the layers are designed together and reach efficiencies none could achieve alone. And all of this must proceed while fundamentally focusing on security and sustainability as first-order design concerns rather than afterthoughts — because at the scale of societal AI infrastructure, energy footprint and trustworthiness are as central as speed. The talk grounds the argument in recent Google experience across networks, accelerators, and servers.
3. Because this is a vision talk, the takeaway is a direction rather than a technique: the era of getting faster and more efficient for free from general-purpose scaling is over, and meeting AI's demand within acceptable cost, energy, and security limits requires reshaping the infrastructure itself. The background offered here about transistor scaling, domain-specific accelerators, co-design, and datacenter energy is standard context for the trends the talk names — the specific Google examples, numbers, and detailed opportunities were part of the keynote and are not in the abstract.

Check yourself

Check your understanding

1. What is the central tension the keynote identifies?
2. What course does the keynote chart for future computing infrastructure?
3. Why does the keynote treat security and sustainability as first-order concerns rather than add-ons?
4. How should this page be read, given what it is based on?

Sources

The talk is “Societal infrastructure in the age of Artificial General Intelligence,” a keynote by Amin Vahdat (Google), presented at ASPLOS 2024, DOI 10.1145/3620666.3655589.

This explainer is written only from the talk’s published abstract and TLDR (obtained via the Semantic Scholar API); there is no full paper, and no open-access text was available, so this page is marked a draft. The abstract states the talk’s thesis: computing is at an inflection point where Generative AI places unprecedented demand on compute while established architectural patterns for efficiency have stalled; the talk discusses the likely needs of the next generation of computing infrastructure using recent examples at Google from networks to accelerators to servers; and it charts a course in which computing must be increasingly specialized and co-optimized with algorithms and software, while fundamentally focusing on security and sustainability. The background here about the end of transistor scaling, domain-specific specialization, hardware-software co-design, datacenter energy, and security is standard, widely-known context for the trends the talk names, provided to make the argument legible; it is not additional content from the talk and should not be read as specific claims made in the keynote. For the actual examples, arguments, and any figures, the keynote itself is the source.