Title: An AI Stack: From Scaling AI Workloads to Evaluating LLMs

Large language models have strained the world's compute and raised hard questions about whether their answers are any good, and no single tool fixes that — it takes a whole stack, from software that spreads AI work across thousands of machines, to engines that serve models fast enough to be affordable, to a fair way to measure which model is actually better; in this ASPLOS 2026 keynote, Ion Stoica (UC Berkeley) tours that stack through four open-source projects he helped build — Ray for scaling AI workloads, vLLM and SGLang for high-throughput LLM inference, and LMArena for benchmarking — and draws lessons for what comes next.

The argument in one glance

Problem

LLMs strained the world's compute and raised doubts about their own outputs

This is a keynote talk, and its framing is the wave of systems challenges that large language models (LLMs) set off. Their sudden usefulness enabled a rush of new applications, but that success came with costs the abstract names directly. First, demand: training and running these models consumes enormous compute, intensifying GPU shortages and making it expensive and hard to scale AI workloads across the machines available. Second, trust: LLMs raised concerns about the accuracy of their outputs — they can be confidently wrong — so knowing which model is actually good, and how good, became a real problem. These are not one problem but several intertwined ones, spanning very different layers of the system: how to spread AI computation across many machines efficiently, how to serve a model to users fast and cheaply once trained, and how to measure a model's quality in a way that reflects real use. No single tool addresses all of this. The talk's premise is that meeting these challenges takes a whole stack of infrastructure, built layer by layer, and it presents that stack through a set of widely-used open-source projects the speaker helped create.
What people did

Reach for general-purpose distributed tools and existing benchmarks

The default response to each layer, before purpose-built systems existed, was to reuse what was already around. For scaling, that meant general distributed-computing frameworks not designed for the irregular, mixed workloads of modern AI. For serving, it meant running models on inference setups adapted from earlier deep-learning serving, without the memory and batching tricks LLM generation specifically needs. And for evaluation, it meant static, fixed-answer benchmarks — quiz-like test sets — inherited from earlier NLP. Each of these was serviceable but not shaped to what LLMs actually demand.
Why it fell short

Generic tools miss what AI scale, LLM serving, and honest evaluation each require

Each borrowed approach falls short in its own way. General distributed frameworks struggle with AI's mix of training, data processing, reinforcement learning, and serving — heterogeneous tasks with complex dependencies that a framework built for simpler batch jobs handles awkwardly, leaving performance and programmer productivity on the table. Inference stacks not built for LLMs waste the scarce GPUs: LLM generation produces tokens one at a time and accumulates a large, growing key-value cache of past context, and a serving engine ignorant of this manages memory poorly and batches requests inefficiently, so throughput is far below what the hardware could deliver — precisely the wrong outcome amid a GPU shortage. And static benchmarks fail to capture LLM quality honestly: fixed test sets can be memorized or gamed, they saturate as models improve, and they do not reflect the open-ended, subjective ways people actually use chatbots, so a high score need not mean a genuinely better model. The gap, across all three layers, is that LLMs are different enough from prior workloads that the general-purpose tools do not fit, and getting AI to work at scale, cheaply, and measurably requires infrastructure designed for it.
This paper's idea

Build an AI stack: scale it, serve it fast, and evaluate it fairly

The talk answers each layer with a purpose-built, open-source system, together forming an AI stack. For scaling AI workloads, it presents Ray, a distributed framework that gives a unified way to express and run the diverse tasks of modern AI — training, tuning, data processing, reinforcement learning, and serving — across a cluster, so heterogeneous AI pipelines scale without hand-built plumbing. For serving LLMs efficiently, it presents two high-throughput inference engines: vLLM, known for managing the key-value cache far more efficiently so many more requests fit on a GPU at once, and SGLang, which speeds up LLM programs by reusing shared computation across requests and co-designing the way complex, multi-step LLM applications are expressed and executed. For evaluating LLMs accurately, it presents LMArena, a platform that benchmarks models by having real people compare their outputs head to head and aggregating those judgments into a ranking, capturing human preference in a way static tests cannot. The talk concludes with the lessons drawn from building these systems and directions for future research. Taken together, the projects illustrate a thesis: the practical success of LLMs rests not on any one breakthrough but on a stack of infrastructure — for scaling, for serving, and for evaluation — each layer engineered for what AI specifically needs.

Why this problem is worth solving

The systems layer is where the promise of LLMs is either realized or squandered, because the models are only as useful as the infrastructure that can run and judge them affordably.

It is tempting to see the model as the whole story and the systems around it as plumbing, but the reverse is closer to the truth for whether LLMs actually work in practice. A capable model is inert without a way to train it across enough hardware, serve it to users within acceptable cost and latency, and know whether it is any good. Each of these is a hard systems problem sharpened by scale. Scaling matters because the compute required is so large that inefficiency translates directly into more GPUs, more money, and, during a shortage, work that simply cannot be done. Serving efficiency matters for the same reason from the other end: the marginal cost of every answer a deployed model gives is set by how many requests a GPU can handle at once, so a serving engine that doubles throughput halves the cost of running the model, which at global scale is enormous. And evaluation matters because progress is only as trustworthy as its measurement: if you cannot tell which model is better, you cannot improve, choose, or deploy responsibly. So the infrastructure is not secondary to the AI revolution; it is a large part of what made it possible and what governs how far it can go. That is why a keynote surveying this stack is really surveying the load-bearing structure beneath the whole enterprise.

The unifying insight is that these three concerns are one problem seen at different layers, and treating them as a coherent stack — rather than isolated tools — is what makes the whole practical.

The scaling, serving, and evaluation challenges look separate, but they are bound together by the same underlying pressures and reinforce one another. Efficient serving reduces the compute you must scale; better scaling makes it feasible to run the large evaluations that reveal which models to serve; honest evaluation tells you which models are worth the cost of scaling and serving in the first place. Seen this way, the right response is not a single clever system but a layered stack where each layer has a clean role and composes with the others: a distributed substrate to run anything at scale, inference engines that make the running cheap, and an evaluation platform that closes the loop by telling you what is actually good. This stack view also explains why open-sourcing these systems mattered so much — a shared, composable stack lets the whole community build on each layer rather than reinventing it, which is how infrastructure like this comes to underpin an entire field. The keynote’s value is in showing these projects not as a list but as a coherent architecture for making AI workloads scalable, affordable, and trustworthy, and in extracting the lessons that generalize beyond any one of them. Recognizing the stack, and building each layer to fit AI’s real demands, is the through-line.

What people did before, and where it broke

General distributed-computing frameworks. Run AI on frameworks designed for simpler batch or data jobs. Where it broke. Modern AI mixes training, tuning, reinforcement learning, data processing, and serving with complex dependencies, which general frameworks handle awkwardly, hurting both performance and productivity.
Inference stacks not built for LLMs. Serve LLMs on engines adapted from earlier deep-learning serving. Where it broke. They ignore that LLM generation is token-by-token with a large growing key-value cache, so they manage GPU memory and batching poorly and deliver throughput far below the hardware’s potential.
Static, fixed-answer benchmarks. Measure LLM quality with quiz-like test sets from earlier NLP. Where it broke. They can be memorized or gamed, saturate as models improve, and do not reflect the open-ended, subjective ways people use LLMs, so scores need not track real quality.
Treating the layers in isolation. Solve scaling, serving, or evaluation one at a time with ad-hoc tools. Where it broke. The concerns are intertwined, so isolated fixes miss the leverage of a coherent, composable stack where each layer supports the others.
The gap. There was no purpose-built, open-source stack spanning distributed execution, efficient LLM inference, and rigorous evaluation, each shaped to what AI workloads specifically need.

The layers of the AI stack

From LLM challenges to a layered, open-source AI stack

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Scale AI workloads across a cluster with a unified framework. The stack’s foundation is a distributed system that lets the many heterogeneous tasks of modern AI run at scale without hand-built infrastructure. Providing an elastic, distributed platform that runs diverse deep-learning workloads across a cluster, so that training and related jobs scale smoothly, is the same distributed-AI-platform goal the work on an elastic platform for distributed deep learning (ElasticFlow) covered elsewhere on this site pursues, spreading AI work across machines under a common system. The talk presents Ray as this foundational layer for scaling AI computation of all kinds.

Serve LLMs with high throughput to make them affordable. The stack’s serving layer is about squeezing the most requests out of each scarce GPU, so the cost per answer falls. Building a high-throughput inference serving system that maximizes how many requests the hardware can handle at once is the same serving-efficiency goal the work on a high-throughput inference serving system (Proteus) covered elsewhere on this site is built on, driving up utilization to lower the cost of serving. The talk’s vLLM and SGLang pursue this for LLMs specifically, through efficient key-value-cache management and computation reuse across requests.

Evaluate models rigorously so progress is trustworthy. The stack’s top layer measures quality in a way that resists gaming and reflects real use, closing the loop on which models are worth running. Building a rigorous, systematic way to evaluate a system so that its true quality is exposed rather than assumed is the same evaluation-first discipline the work on generating diverse test cases to evaluate deep-learning compilers (NNSmith) covered elsewhere on this site embodies, testing thoroughly instead of trusting a narrow benchmark. LMArena brings that rigor to LLMs by grounding evaluation in large-scale human comparison rather than static tests.

Walking through the AI stack

How the four projects address the LLM systems challenges

A walkthrough of the talk's stack as described in its abstract. Descriptions of the projects reflect their widely-known public design, not additional claims from the talk.

Step 1 of 5

The challenge: scale, cost, and trust

LLMs strain compute (intensifying GPU shortages), cost a great deal to serve, and raise concerns about output accuracy. The talk organizes its response around these intertwined challenges, addressing each with a purpose-built open-source project.

Scaling, serving, and evaluation are the three layers in play.

Why it works

A unified distributed substrate fits AI’s heterogeneity. General frameworks handle mixed AI tasks awkwardly → but a framework built for training, serving, and data together → so diverse AI pipelines scale without bespoke plumbing.
Efficient serving turns scarce GPUs into more answers. Naive LLM serving underuses the hardware → but managing the key-value cache and batching for how LLMs generate → so throughput rises and the cost per answer falls.
Reusing shared work across requests compounds the savings. Recomputing common context for every request wastes cycles → but reusing shared computation across LLM calls → so serving gets cheaper still on real, repetitive workloads.
Human comparison measures what static tests miss. Fixed benchmarks saturate and can be gamed → but ranking models by large-scale head-to-head human preference → so evaluation reflects genuine, open-ended quality.
Composing the layers makes the whole practical. Isolated tools miss the leverage between concerns → but a coherent stack where scaling, serving, and evaluation reinforce each other → so AI becomes scalable, affordable, and trustworthy together.

What it costs (trade-offs and scope)

This is a keynote, not a results paper. It surveys a body of work and its lessons; it does not present a single new system or measured evaluation, so there is nothing here to reproduce or quantify.
It reflects one group’s stack. The projects are those the speaker helped build, so the tour is a coherent but particular view of the AI stack, not a neutral survey of every alternative in each layer.
The layers still involve real trade-offs. Efficient serving trades implementation complexity for throughput, human-preference evaluation trades the repeatability of static tests for realism, and a general distributed framework trades some specialization for breadth — the talk’s framing highlights the wins more than these costs.
Evaluation by human preference has its own limits. Ranking by crowd comparison captures subjective quality but can reflect stylistic preferences and is harder to control than fixed benchmarks; it complements rather than fully replaces targeted testing.
This explainer is built from the abstract only. The talk’s specific lessons, examples, and future directions were delivered live; the project descriptions here are widely-known public knowledge about Ray, vLLM, SGLang, and LMArena, provided as context, not additional content from the talk.

The comparison

Meeting the LLM systems challenges

Property	Borrowed general-purpose tools	The AI stack in the talk
Scaling AI workloads
Serving LLMs
Evaluating LLMs
How the pieces relate
Availability

The contrast the keynote draws. LLMs created intertwined challenges of scaling, serving, and evaluation that borrowed general-purpose tools address poorly. The talk presents a purpose-built, open-source AI stack: Ray for scaling AI workloads across a cluster, vLLM and SGLang for high-throughput LLM inference, and LMArena for human-preference-based benchmarking — layers that compose into a coherent whole. This reflects the abstract's stated framing, not a measured comparison; project descriptions are widely-known public knowledge.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This is an ASPLOS 2026 keynote talk by Ion Stoica (University of California, Berkeley), and this explainer is based only on its published abstract — there is no full paper or set of results behind it, so this page is marked a draft. The abstract’s stated content is what is reported as the talk’s own: LLMs have enabled new applications, intensified GPU shortages, and raised concerns about output accuracy, and the talk presents several projects the speaker worked on to address these challenges — Ray (a distributed framework for scaling AI workloads), vLLM and SGLang (two high-throughput inference engines for LLMs), and LMArena (a platform for accurate LLM benchmarking) — concluding with lessons learned and future directions. The descriptions of what each project does — Ray as a unified distributed framework for diverse AI tasks; vLLM as a high-throughput LLM serving engine known for efficient key-value-cache management; SGLang as a fast serving system that reuses shared computation across requests and co-designs how LLM programs are expressed; and LMArena (formerly known as Chatbot Arena) as a platform that ranks LLMs via large-scale head-to-head human preference comparisons — are widely-known public facts about these well-documented open-source systems, provided here to make the abstract legible. They are not additional content from the talk, and none of the specific lessons, examples, numbers, or future directions the speaker presented live are contained in the abstract or reproduced here. For the actual arguments and any figures, the keynote itself is the source. Background on distributed computing, LLM serving, key-value caches, and benchmarking is standard context.

Background

Why the AI revolution is as much a systems story as a modeling one

It helps to see this keynote as a statement about where the hard, load-bearing work of the AI era actually happens. The public narrative centers on models and their capabilities, but each leap in what models can do has been matched by, and often gated by, advances in the infrastructure that trains, serves, and measures them. Distributed frameworks made it possible to marshal thousands of accelerators toward a single workload; inference engines drove the cost of running a model down by large factors, turning demos into deployable products; and evaluation platforms gave the field a shared, harder-to-game way to tell whether progress was real. This pattern — that a capability locked behind cost and uncertainty becomes practical only when the surrounding systems catch up — recurs throughout computing, and it is especially stark for LLMs because their appetite for compute and their opacity are both extreme. This effort to make demanding machine-learning workloads efficient and deployable through better systems is the same terrain the work on accelerating serverless LLM inference (Medusa) covered elsewhere on this site occupies, lowering the cost and friction of running large models. The keynote’s distinctive contribution is to step back and present the whole picture at once — scaling, serving, and evaluation as one stack — and to argue, through a set of influential open-source projects, that the systems layer is not a supporting act but a large part of the main event.

If you remember only three things

1. This is an ASPLOS 2026 keynote by Ion Stoica of UC Berkeley, and only its abstract is available — there is no full paper or results. Its framing is that large language models set off a wave of intertwined systems challenges: their success intensified GPU shortages and made scaling and efficient serving essential, while concerns about the accuracy of their outputs made trustworthy evaluation essential. These span very different layers — spreading AI computation across many machines, serving a trained model to users cheaply and quickly, and measuring a model's real quality — and no single tool addresses them, so meeting them takes a whole stack of purpose-built infrastructure.
2. The talk presents that stack through four widely-used open-source projects the speaker helped build. For scaling AI workloads, Ray, a distributed framework that gives a unified way to run the diverse tasks of modern AI — training, tuning, data processing, reinforcement learning, serving — across a cluster. For serving LLMs efficiently, two high-throughput inference engines: vLLM, known for managing the growing key-value cache far more efficiently so many more requests fit on a GPU, and SGLang, which reuses shared computation across requests and co-designs how complex LLM programs are expressed and executed. For evaluating LLMs accurately, LMArena, which benchmarks models by having real people compare their outputs head to head and aggregating those judgments into a ranking, capturing human preference in a way static tests cannot. The talk closes with lessons learned and future directions.
3. The through-line is that the practical success of LLMs rests not on any single breakthrough but on a coherent stack of infrastructure — for scaling, serving, and evaluation — each layer engineered for what AI specifically needs and composing with the others, and each open-sourced so the whole community can build on it. Because this is a keynote, the takeaway is a viewpoint rather than a technique: the systems layer is a large part of what made LLMs deployable and governs how far they can go. The project descriptions here are standard public knowledge about Ray, vLLM, SGLang, and LMArena, offered as context; the specific lessons, examples, and future directions the speaker gave live are not in the abstract and are not reproduced here.

Check yourself

Check your understanding

1. What intertwined challenges does the keynote say large language models created?
2. Which projects does the talk present, and what layer of the stack does each address?
3. Why does the talk favor human-preference benchmarking (LMArena) over static benchmarks for LLMs?
4. What is the keynote's overarching message, and how should this page be read?

Sources

The talk is “An AI Stack: From Scaling AI Workloads to Evaluating LLMs,” a keynote by Ion Stoica (University of California, Berkeley), presented at ASPLOS 2026, DOI 10.1145/3779212.3795612.

This explainer is written only from the talk’s published abstract; there is no full paper, so this page is marked a draft. The abstract states the talk’s framing: LLMs have enabled new applications, intensified GPU shortages, and raised concerns about output accuracy, and the talk presents several projects the speaker worked on to address these challenges — Ray (a distributed framework for scaling AI workloads), vLLM and SGLang (two high-throughput inference engines for LLMs), and LMArena (a platform for accurate LLM benchmarking) — concluding with lessons learned and future directions. The descriptions in this explainer of what each project does are widely-known public facts about these well-documented open-source systems (Ray as a unified distributed framework; vLLM as a high-throughput LLM serving engine with efficient key-value-cache management; SGLang as a fast serving system that reuses shared computation across requests; and LMArena, formerly Chatbot Arena, as a human-preference-based LLM ranking platform), provided as context to make the abstract legible; they are not additional content from the talk, and the speaker’s specific lessons, examples, and future directions are not in the abstract. Background on distributed computing, LLM serving, and benchmarking is standard context. For the actual arguments and any figures, the keynote itself is the source.