Towards End-to-End Optimization of LLM-based Applications with Ayo

A real LLM application is a pipeline of parts — retrievers, embedders, rerankers, database lookups, and several model calls — and the frameworks that wire them together run each part as an opaque black box in sequence, so the non-model work can be more than half the latency and nothing overlaps; Ayo (CUHK) throws away the module as the unit and rebuilds each query as a graph of fine-grained primitives, exposing enough structure to run independent pieces in parallel, stream a model's output the moment a sentence is ready, and batch across the whole application rather than one call at a time.

The argument in one glance

Problem

The model is only part of what a real LLM application does

A retrieval-augmented question-answering app embeds the query, searches a vector database, reranks results, and only then calls the model — often several times. Measured end to end, the non-model work can exceed half the latency, and it is largely ignored.
What people did

Wire the parts together as a chain of modules

Orchestration frameworks let a developer compose an application from high-level modules — indexing, query expansion, synthesis — each running on its own engine, one after another.
Why it fell short

A module is a black box that runs to completion, so nothing overlaps

Each module hides its internal operations and finishes before the next begins, so optimization is confined inside a module and the dependencies that would let independent work overlap are invisible. And scheduling, cut off from the application, tunes each request alone rather than the whole workflow.
This paper's idea

Make the unit a fine-grained primitive and expose the whole graph

Decompose every module into primitive operations — embed, search, prefill, decode — and build each query as a dataflow graph of them. The graph reveals what is independent, so pieces run in parallel, a model streams its output as it emerges, and batching can serve the application rather than one call.

Why this problem is worth solving

A modern LLM application is a workflow, not a model call.

Because a language model’s training data goes stale and it cannot reach the outside world, real applications surround it with other components. The dominant pattern is retrieval-augmented generation: the user’s question is turned into an embedding vector, that vector searches a database of document chunks, the most relevant chunks are retrieved and often reranked, and only then does the model synthesize an answer from them. Advanced versions add more model calls — expanding the query into several variants to search more broadly, or generating a context summary for each chunk before it is indexed. An agent adds tool calls, a proxy model to decide whether a search is even needed, a judge model to check it. Across these workflows the model is one component among many, and measured end to end the embedding, searching, reranking and database work frequently accounts for more than half the total latency — in one measured application, more than half of it.

The frameworks that assemble these workflows treat each stage as sealed.

The standard way to build such an application is to chain high-level modules — an indexing module, a query-expansion module, a synthesis module — each a black box that runs on its own backend engine and completes before the next one starts. This has two consequences that cost performance. First, optimization cannot cross a module boundary: the internal operations of a module, and the dependencies between one module’s operations and another’s, are invisible, so nothing that is actually independent can be made to overlap. Second, the frontend that decides the order of modules is separated from the backend engines that run them, so when an engine schedules its work it has no idea what application the requests belong to or how they relate — it can only optimize each request’s own latency, which is often the wrong objective for the workflow as a whole.

What people did before, and where it broke

Chain the workflow as a sequence of modules. The standard approach in every popular orchestration framework: compose the application from high-level task modules, each running to completion on its engine. Where it broke. A module is a black box, so optimization is confined within it and the dependencies that would let independent operations overlap are hidden; the whole workflow runs essentially serially even where it need not.
Optimize the model inference itself. Enormous effort has gone into faster prefilling, decoding and memory management for the model call. Where it broke. It leaves the non-model half of the latency untouched, and even a perfect model engine cannot overlap the embedding and searching that a serial chain runs before and after it.
Manually parallelize the independent modules. A more advanced framework variant inspects the pipeline and runs modules with no dependency between them concurrently, and caches prompt prefixes to avoid recomputing shared instructions. Where it broke. It parallelizes at module granularity, so it can overlap indexing with query embedding but cannot break inside a module — it cannot start the model on the parts of a prompt already available while waiting for the retrieved context, nor stream a query-expansion model’s output chunk by chunk.
Group requests into agents. Agentic frameworks bundle several modules into an agent and connect agents by a predefined graph. Where it broke. The granularity is even coarser; there is no attempt to optimize across agents or the sub-tasks inside them.
Let each engine schedule its own requests. The backend engine batches whatever requests it receives to optimize their latency. Where it broke. With no application context, it treats every request identically — a fixed batch size, per-request latency as the goal — when the right objective is often the whole batch’s completion time, or prioritising the requests on the workflow’s critical path.
The gap. Every layer optimized within its own boundary — the module within itself, the engine within one request — and the boundaries were exactly where the overlap and the right scheduling objective lived. Nobody had made the whole workflow visible as one structure.

How a workflow stops being a chain of black boxes

From opaque modules in sequence to a graph of primitives

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Changing the unit of orchestration from the module to the primitive is what makes the structure visible. A module is opaque by construction — indexing is “indexing”, a single node that runs and returns — and everything the optimizer would want to know is sealed inside it: that indexing is really embedding then ingesting, that the embedding of the documents has nothing to do with the prefilling of the instruction, that a query-expansion model emits its rewritten queries one at a time. Decomposing each module into primitive operations and laying them out as a dataflow graph turns all of that from hidden to explicit. Once the graph exists, the optimizations are almost forced by it: independent branches are visibly independent and run in parallel; a node’s output that is produced incrementally can visibly be streamed to its successor; a prompt whose parts arrive at different times can visibly be prefilled in the pieces that are ready. The reported effect on one worked example is an end-to-end time cut from 4.1 seconds to 2.4 purely by rearranging what runs when. Replacing a coarse black-box unit with a fine-grained one so that a joint optimizer can finally see across the old boundaries is the same move the schedule-language work covered elsewhere on this site makes when it separates a model’s definition from its execution so optimizations can address the parts.

The two seams that block optimization — inside a module and between frontend and backend — are the same seam, and one representation closes both. The first seam hides the operations within a module from each other; the second hides the application from the engine that executes its requests. The dataflow graph closes the first directly, by making the operations nodes. It closes the second because the graph is what the scheduler now dispatches: when a request reaches an engine it carries which query it belongs to, which primitive produced it, and how deep in the graph that primitive sits. So the engine can stop optimizing each request’s own latency — the objective it was stuck with for lack of any other information — and start optimizing what the application actually cares about. Forty-eight embedding requests from one indexing step should be batched for their total completion time, not per-request latency, which alone is worth a third off that step; a tree of dependent model calls should be batched by depth. Giving a low-level scheduler the high-level context it needs to pursue the right objective, rather than the locally reasonable wrong one, is the same correction the length-aware serving work covered elsewhere on this site makes when it feeds the whole batch’s future memory demand into an admission decision that was otherwise made per request.

Streaming a model’s output the instant a coherent piece exists is the optimization that only a fine enough graph can express. The sharpest example is a model call whose job is to rewrite one query into several. Under module orchestration that call is a black box that returns when all the rewrites are done, and only then does the next module — embedding, searching — begin. But the rewrites emerge one at a time during decoding, and each finished one is immediately useful: the moment the first rewritten query is complete it can be sent to be embedded and searched while the model is still generating the second. Expressing this needs the decode annotated as splittable and a parser watching its structured output to forward each complete piece, which is only possible when the decode is its own primitive with the successor primitive explicitly downstream of it. The same logic runs in reverse on the input side: the parts of a prompt that are known early are prefilled early. That the interesting parallelism lives in the interactions rarely exposed — here, between the middle of one operation and the start of the next — is the same lesson the multi-stage GPU-service work covered elsewhere on this site draws when it finds the microservice boundaries themselves are where the latency hides.

Following one advanced-RAG query through the optimizer

What happens to a document-QA query with query expansion and refine synthesis

The developer defined the workflow as modules. The optimizer rebuilds it per query as a primitive graph and transforms it.

Step 1 of 5

Build the primitive graph from the module template and the query

The indexing module becomes embed and ingest primitives; query expansion and synthesis each become prefill and decode primitives. Every node carries metadata — inputs, outputs, batch behaviour, and flags like batchable or splittable.

The graph is per query, because the actual input data and configuration — how many chunks, which synthesis mode — determine its exact shape. Results of optimized subgraphs are cached to keep the per-query cost small.

Why it works

A module hides exactly the structure the optimizer needs. Optimization cannot cross a boundary it cannot see through, and a module seals its internal operations and their cross-module dependencies → so decomposing modules into primitive nodes in a dataflow graph makes those operations and dependencies explicit → and independent branches, streamable outputs and splittable prompts all become visible opportunities the module hid.
False dependencies are what force serial execution. A module chain implies an order between things that are not actually data-dependent → so pruning edges down to real data dependencies detaches genuinely independent branches → letting the retrieval path and the instruction-prefill path run at once instead of one after the other.
Model outputs and inputs are both incremental, so both can overlap their neighbours. A decode emits coherent pieces one at a time and a prompt’s parts become ready at different times → so a splittable decode streams each finished piece to its successor immediately and a prefill computes the parts that are ready while others are still pending → turning two serial steps into overlapping ones, which the coarse black box could never express.
The engine can pursue the right objective once it knows the application. Cut off from the workflow, an engine can only minimise each request’s own latency, which is often wrong — forty-eight requests from one indexing step want minimum total completion time, not minimum per-request time → so dispatching the graph node with its query and depth attached gives the engine the context it lacked → and it can batch for the application’s actual goal, worth about a third off that embedding step.
Depth is a cheap proxy for how much a request matters. Requests differ in how much they advance their query, but comparing them exactly is expensive → so batching by topological depth prefers the deeper, more contributive nodes and keeps the request correlations within a primitive intact → improving end-to-end latency without an expensive critical-path computation.

What it costs (trade-offs and scope)

The workflow must be defined up front and does not change per query. The whole approach assumes a fixed workflow template the developer designs in advance; the paper lists dynamic workflows, where the structure is decided at runtime, as out of scope and future work — so an agent that plans its own steps on the fly is not covered.
Splitting a prefill is a real slowdown, bet against a parallelism win. Decomposing one complete prefill into partial and full prefills is measured at 3 to 12 percent slower for that operation; it pays off only because the exposed parallelism saves more elsewhere, which is a bet that holds on these workflows but is not free.
It requires modifying the backend engines. Supporting decomposed primitives — partial prefill, split decode with a structured-output parser — means changing engine-side mechanisms, so this is not a drop-in layer over unmodified inference servers; the coupling to the backends is acknowledged.
The headline speedup is against the authors’ own baselines. The comparison points are a Ray-based distributed reimplementation of a popular framework and an enhanced variant the authors built, plus one agent framework — reasonable given that few systems target this exact problem, but the up-to-2.09-times figure is relative to reconstructions rather than a deployed competitor.
The gains vary widely by workflow, and the honest low end is modest. The speedup ranges from 2.09 times on the most complex advanced-RAG app down to 1.06 times on the contextual-retrieval workflow, where a single lightweight extra model call leaves little to parallelize. The big numbers come from the apps with the most exploitable structure.
Topology-aware batching is a heuristic, not an optimum. Depth is used as a proxy for importance precisely because the true critical path is expensive to compute; the paper notes exploiting real critical-path information as future work, so the scheduling is a reasonable approximation rather than the best achievable.
Co-located applications are handled independently, leaving value unclaimed. When several apps share the infrastructure and the same model, each is optimized in isolation; cross-application opportunities like sharing the model’s cache are explicitly not exploited.
The per-query optimization adds overhead that only caching keeps small. Graph optimization runs per query and is only 1.3 to 3 percent of latency when subgraph results are cached; communication overhead from the central scheduler is 3.1 to 6.2 percent. The overheads are modest but real and depend on the cache being effective.
Streaming a decode needs its output to be cleanly parseable. The splittable optimization relies on the model emitting structured output — a parser extracting complete pieces — so it applies to calls whose output divides into distinct parts, not to free-form generation where no piece is complete until the end.
Scope is a handful of workflows on specific hardware. Five application patterns, a few models up to thirty billion parameters, on a particular GPU testbed. The generality of the primitive decomposition across a wider range of applications is argued from a GitHub survey rather than demonstrated broadly.
Prefix caching is in the baseline, so part of the model-side win is not new. The stronger baseline already caches shared prompt prefixes, so the contribution is specifically the parallelization and pipelining structure, not the prompt-caching idea.

The evaluation

Three ways to run an LLM application workflow

Property	Chain of black-box modules	Modules with manual parallelism and prefix caching
What the optimizer can see
Can it overlap inside a module
What the engine optimizes for
End-to-end speedup
Cost	None beyond the framework	Manual parallelization effort

Five real workflow patterns — search-augmented generation, an agent, naive and advanced RAG document QA, and a production contextual-retrieval workflow — with models up to thirty billion parameters, against a distributed framework reimplementation, an enhanced variant, and an agent framework.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The result is a real end-to-end win but with a spread worth reading in full: up to 2.09 times on the most structured advanced-RAG workflow, and as little as 1.06 times on the contextual-retrieval app where a single lightweight extra model call leaves almost nothing to overlap — the big numbers come from the apps with the most exploitable parallelism. The baselines are the authors’ own: a Ray-based distributed reimplementation of a popular framework, an enhanced variant they built with manual parallelism and prefix caching, and one agent framework — reasonable because few systems target this problem, but the comparison is against reconstructions rather than a deployed competitor, and the stronger baseline already includes prompt caching, so the contribution is specifically the parallelization and pipelining. Two design costs are acknowledged: splitting a prefill is 3 to 12 percent slower for that operation, paid back only by the parallelism it unlocks, and the whole scheme requires modifying the backend engines to support decomposed primitives, so it is not a drop-in layer. The batching is a depth-based heuristic because the true critical path is expensive to compute; co-located applications are optimized independently, leaving cross-app cache sharing unclaimed; and the streaming optimization needs cleanly structured model output to parse. The largest scope limit is that the workflow must be fixed in advance — dynamic, self-planning workflows are explicitly future work — and the per-query optimization overhead stays small (1.3 to 3 percent) only when subgraph caching is effective.

Background

Why the model stopped being the whole story

It is worth being clear about why end-to-end workflow optimization became worth a paper when so much effort already goes into the model. For a while the model call genuinely was the workflow — you sent a prompt and got an answer, and everything was in the forward pass. Retrieval-augmented generation changed that, because the fix for a model’s two structural weaknesses, stale knowledge and no access to the outside world, is to wrap it in machinery that fetches fresh, relevant context at query time: embed the question, search a database, rerank, and only then generate. Each of those steps runs on its own engine with its own cost, and as the pattern matured the wrapping grew — multiple retrievals, query rewriting, per-chunk summarization, judge models, tool calls. The measured consequence is that the non-model work now routinely rivals or exceeds the model work in latency, so optimizing only the forward pass optimizes a shrinking fraction of the whole. This is the same shift the long-sequence transformer accelerator work covered elsewhere on this site responds to from the hardware side, noticing that as inputs grow the operation mix changes and the part everyone provisioned for stops dominating. Here the change is in the software architecture: the application became a distributed dataflow program that happens to contain a language model, and it needs to be scheduled like one.

If you remember only three things

1. A real LLM application is a pipeline of retrievers, embedders, rerankers, database lookups and several model calls, and the non-model work can be more than half the end-to-end latency — but orchestration frameworks chain these as opaque modules that each run to completion, so optimization is trapped inside a module, the dependencies that would allow overlap are invisible, and the engine that runs the requests has no idea which application they serve.
2. Decomposing every module into fine-grained primitives — embed, search, prefill, decode — and building each query as a dataflow graph exposes the structure both seams hid, and the optimizations then follow from the graph: prune false dependencies so independent branches run in parallel, prefill the parts of a prompt available early while retrieval runs, and stream a model's output the instant a coherent piece emerges rather than waiting for the whole decode.
3. Because the scheduler now dispatches graph nodes carrying their query and depth, the engine can batch for the application rather than per request — grouping by query and favouring deeper, more critical nodes — cutting one worked example from 4.1 to 2.4 seconds and delivering up to 2.09 times end-to-end speedup, though the gain falls to 1.06 times on workflows with little exploitable structure and the backends must be modified to support the decomposed primitives.

Check yourself

Check your understanding

1. Why does module-based orchestration prevent overlapping independent work?
2. How does the fine-grained graph fix the scheduling seam between frontend and backend?
3. What does streaming a splittable model decode accomplish, and why does it need the fine-grained graph?
4. Which limitation most tempers the headline result?

Sources

The paper is “Towards End-to-End Optimization of LLM-based Applications with Ayo” by Xin Tan, Yimin Jiang, Yitao Yang and Hong Xu, published at ASPLOS 2025, pages 1302-1316, DOI 10.1145/3676641.3716278. The lead authors are at the Chinese University of Hong Kong; the system is named Ayo in the published version (Teola in the preprint).

This explainer is written from the full arXiv text, arXiv:2407.00326. Everything above comes from the paper’s own sections: the motivation showing the non-model latency breakdown across five real workflows and a GitHub survey of their prevalence, the primer on LLM prefilling and decoding, and the two identified limitations of module-based orchestration — the confinement of optimization within a module and the mismatch between request-level engine scheduling and application-level performance, illustrated with the embedding-batching and tree-synthesis examples; the design of the primitive-level dataflow graph with its per-node metadata, the worked comparison reducing an example from 4.1 to 2.4 seconds, and the architecture of the graph optimizer and two-tier runtime scheduler; the four optimization passes — dependency pruning, stage decomposition, prefilling split, and decoding pipelining — with the optimization procedure and a worked advanced-RAG example; the runtime scheduling with its graph scheduler, engine schedulers, blind-versus-topology-aware batching comparison and the depth-based batching algorithm; and the evaluation covering the baselines (a distributed framework reimplementation, an enhanced parallel-and-caching variant, and an agent framework, each under per-invocation and throughput-oriented engine scheduling), the per-application speedups, the co-location and ablation studies, and the overhead breakdown for graph optimization, communication, and prefill decomposition, together with the stated limitations on dynamic workflows, backend coupling, critical-path exploitation, and cross-application optimization.