Bat: Efficient Generative Recommender Serving with Bipartite Attention

Jie Sun, Shaohang Wang, Zimo Zhang, Zhengyu Liu, Yunlong Xu, Peng Sun and 4 others

A new style of recommender treats picking what to show you as a generative sequence task, which is powerful but as compute-heavy to serve as the prompt-processing stage of a large language model; the usual trick of caching and reusing a shared prompt prefix barely helps, because each query mixes a highly personal user profile with freshly-retrieved items that rarely repeat; this paper (Zhejiang University and collaborators) observes that user and item tokens are order-agnostic, so it can flexibly make either the user or the item the reusable prefix, and builds Bat around that to raise serving throughput up to 1.6 times while cutting computation up to 58 percent.

The argument in one glance

Problem

A powerful new recommender is expensive to serve, and the usual caching trick barely helps

Generative recommenders are a promising new alternative to the deep-learning recommendation models that have long powered feeds and product suggestions. Instead of the traditional feature-crunching approach, they treat recommendation as a generative, sequence-modeling task built on attention, much like a language model. That gives them more expressive power, but it also makes them expensive to serve: their inference is compute-bound, resembling the prefill stage of large language model inference, where the whole input has to be processed in one heavy pass. A standard way to cut that cost in language models is prefix caching: if many queries share a common opening, the attention state (the key-value cache) for that shared prefix can be computed once and reused, saving redundant work. The trouble is that this reuse hardly pays off for generative recommenders. Each of their queries pairs a highly personalized user profile with a set of items retrieved in real time, and both parts change from query to query, so there is little common prefix to share across queries — and the cache reuse, and thus the savings, stay small.
What people did

Cache the user as the prefix and reuse it

The conventional way to apply prefix caching to a generative recommender is to treat the user profile as the prompt prefix. The idea is that a user's profile is fixed for the duration of a session, so its key-value cache could be built once and reused across the many items scored for that user, avoiding recomputation of the user part on every query.
Why it fell short

The user prefix is too personal and the items too fresh to reuse widely

Making the user the prefix runs into the two defining properties of generative recommenders. First, user profiles are highly personalized: every user is different, so a user-based prefix cache is useful only within that one user's requests and cannot be shared across the many different users the system serves, which limits how much redundant computation it can remove overall. Second, items are retrieved in real time and vary from query to query, so the item portion, which is not the prefix, must be recomputed each time. The result is that the reusable part is narrow and the recomputed part is broad. Fixing the prefix to always be the user leaves a large amount of shareable structure on the table — in particular, the fact that a popular item may appear in the candidate lists of enormous numbers of users, which a user-as-prefix scheme can never exploit. The abstract's point is that a single, fixed choice of what to cache is the wrong constraint.
This paper's idea

Let either the user or the item be the reusable prefix, since their order does not matter

Bat is a serving system for generative recommenders built on one key observation from the abstract: the semantics between user and item tokens are permutation-invariant — their relative order in the sequence does not change the result. That means the system is free to choose which one to place first as the cacheable prefix. Bat introduces Bipartite Attention, an attention mechanism that adaptively selects either the user or the item as the prompt prefix without compromising accuracy, unlocking new opportunities for key-value cache reuse — for instance making a hot item the prefix so its cache is shared across the many users who see it. Around this, Bat co-designs a disaggregated cache pool that manages user-prefix and item-prefix caches separately, a hot-replicated cold-sharded placement that keeps the added item-cache memory small, and a hotness-aware scheduler that chooses prefixes under memory limits — raising throughput up to 1.6 times while cutting computation up to 58 percent.

Why this problem is worth solving

Generative recommenders inherit the serving cost of language models, so the techniques that make language-model serving affordable are the natural place to look — and the most important one does not transfer.

Recommendation has traditionally been served by deep-learning recommendation models, which cross features and look up embeddings. Generative recommenders instead cast the task as a generative, attention-based sequence problem, closer in spirit to a language model, which the abstract notes makes them more capable but computationally expensive at inference. Their inference is compute-bound and resembles the prefill stage of large language model inference — the phase where the entire input prompt is processed together in a heavy, parallel pass to build up the model’s internal state, as opposed to the lighter, memory-bound step-by-step generation that follows. Because so much of the cost is this prefill-like processing of the input, the classic optimization is to avoid redundant processing across queries. In language models this is done with prefix caching: the attention state, held in a key-value cache, for a shared opening segment is computed once and reused by every query that begins the same way, so the shared work is not repeated. For generative recommenders that are essentially all prefill, cutting redundant prefill work is exactly where the wins are — if only the reuse could be made to apply.

The reason the standard reuse fails is specific to what a recommendation query is made of.

A generative recommender query is not a single free-form prompt; it is composed of two very different parts, a user profile and a set of candidate items. The abstract identifies why this defeats ordinary prefix caching. The user profile is highly personalized — it encodes an individual’s history and preferences, so it differs from user to user, which means a cache keyed on the user prefix is reusable only across that one user’s own requests and never shared across the population. The items, meanwhile, are retrieved in real time, assembled fresh for each query, so the item side is constantly changing. If the convention is to put the user first and cache that, then the only thing ever reused is the per-user prefix, and everything item-related is recomputed. This misses a large source of potential sharing: unlike users, items are not unique to one query — a popular item can appear in the candidate lists of a great many users at once, so caching around items could be reused far more broadly than caching around users. The limitation is not prefix caching itself but the rigid decision to always make the user the prefix, and relaxing that decision is the opportunity Bat pursues.

What people did before, and where it broke

Serve generative recommenders without reuse. Run each query’s compute-bound, prefill-like inference from scratch. Where it broke. It repeats redundant computation across queries, and generative recommenders are expensive enough that this redundancy is a serious cost.
Apply language-model prefix caching directly. Reuse the key-value cache for a shared prompt prefix across queries. Where it broke. Generative recommender queries have little common prefix, because the user profile is personalized and the retrieved items change, so cross-query reuse is limited.
Use the user profile as the prefix. Cache the user’s key-value state and reuse it across the items scored for that user. Where it broke. A user-based prefix is reusable only within one user’s requests and cannot be shared across users, and it never exploits that a popular item recurs across many users’ queries.
The gap. No approach let the system choose, per query, whether the user or the item should be the reusable prefix, so as to maximize key-value cache reuse — nor managed the resulting caches efficiently under memory limits. Doing that, enabled by an attention mechanism that keeps accuracy while allowing the swap, is what Bat is described as contributing.

How Bat works

Choosing the reusable prefix per query and managing the resulting caches

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Since order does not matter, stop forcing the user to be the prefix. The conceptual pivot the abstract describes is the observation that in a generative recommender the semantics between user tokens and item tokens are permutation-invariant: swapping which comes first does not change what the model computes. That single fact dissolves the constraint that had been quietly limiting reuse. If the user must always be the prefix, then only per-user state is ever cached, and the item side — where the real cross-query sharing lives, because a hot item recurs across countless users — can never be reused. By recognizing that either can be the prefix, Bat turns a fixed decision into an adaptive one. Rethinking the attention computation so that what gets treated as the reusable, cacheable part is chosen to maximize sharing, rather than fixed by convention, is the same attention-level optimization instinct the work on optimized dataflow for mitigating attention bottlenecks covered elsewhere on this site pursues, reshaping how attention is computed to relieve the cost that dominates these models.

Make the whole serving system revolve around that flexible choice. A flexible prefix is only useful if the surrounding system can act on it, so Bat is co-designed end to end around the choice. Bipartite Attention supplies the mechanism to select user or item as prefix without hurting accuracy; a disaggregated cache pool then manages user-prefix and item-prefix caches as distinct components, since they have different reuse patterns; and a hotness-aware scheduler decides, under memory limits, which prefix each query should use to get the most reuse. Building an inference-serving system whose scheduling and resource decisions are organized around the specific structure of the workload, rather than treating the model as a generic black box, is the same serving-system co-design the work on constraint-aware resource scheduling for large language model inference covered elsewhere on this site embodies, tuning how requests are handled to the shape of the computation. The payoff the abstract reports is up to 1.6 times the throughput of the user-as-prefix baseline, at up to 58 percent less computation.

Keep the new item caches cheap by treating popular and rare items differently. Enabling item prefixes introduces a cost — item caches consume memory, and there are many items — so Bat manages that memory with a hot-replicated cold-sharded placement. Hot items, accessed by many queries, are replicated so any request finds them locally with little communication, which is worth their memory footprint; cold items, rarely accessed, are sharded so each lives in only one place, minimizing memory at the price of occasional remote access. Splitting a large body of recommendation data into a small, hot, widely-shared portion handled one way and a long, cold tail handled another is the same skew-aware placement strategy the work on statistical-feature-based memory optimization for recommendation covered elsewhere on this site applies, letting the extreme popularity skew of recommendation workloads dictate where each piece of data should live. This is what lets Bat add item-side reuse without paying an unaffordable memory bill.

Following a query through Bat

Serving a query that includes a popular item

An illustrative trace, at the level the abstract describes, of how Bat chooses a reusable prefix for a generative-recommender query. Specific internal details beyond the abstract should be confirmed against the full paper.

Step 1 of 5

A query arrives: a user profile plus retrieved items

The system receives a generative-recommender query combining one user's personalized profile with a set of items retrieved in real time for that user, and must score them with a compute-bound, prefill-like pass.

Under a fixed user-as-prefix scheme, only the user part could be reused, and only for this user.

Why it works

Permutation invariance makes the prefix a free choice. A fixed user-as-prefix scheme is forced to cache only per-user state → but the abstract’s observation that user and item token semantics are permutation-invariant means order does not affect the result → so the system can choose whichever of user or item yields more reuse without changing accuracy.
Items are shared across users in a way user profiles are not. A user prefix is reusable only within one user’s requests → but a popular item recurs across the candidate lists of many users → so making a hot item the prefix lets one cached computation serve many queries, a reuse a user-as-prefix scheme cannot reach.
Separating the two cache types lets each be managed well. User-prefix and item-prefix caches have different access and reuse patterns → but a disaggregated pool holds them as distinct components → so each can be placed and evicted according to its own behavior rather than a single one-size policy.
Hot-replicated cold-sharded placement keeps the added memory small. Item caches add memory and there are many items → but replicating only hot items while sharding cold ones stores widely-used data locally and rare data compactly → so item-side reuse is enabled without an unaffordable memory or communication cost.
Hotness-aware scheduling turns the flexibility into hits. Having a choice of prefix is only useful if exercised well under memory limits → but scheduling that favors prefixes whose caches are hot and resident maximizes actual reuse → so the adaptive prefix selection translates into the reported throughput and compute gains.

What it costs (trade-offs and scope)

This account is based on the abstract only. The full paper was not accessible when this page was written, so the mechanisms are described at the level the abstract states; the internal design of Bipartite Attention, the exact cache-pool management, the placement thresholds, and the scheduling policy are not detailed here and should be read from the paper.
Enabling item prefixes costs memory. The abstract is explicit that introducing item caches incurs additional memory overhead; the hot-replicated cold-sharded placement is designed to minimize it, but item-side caching is a memory investment that a user-only scheme avoids, and its benefit depends on item popularity skew.
The gains depend on reusable structure existing. The wins come from popular items recurring across many users; workloads where candidate items are nearly unique per query, or where hot items are rare, would offer less item-side reuse for Bat to exploit.
Accuracy rests on the permutation-invariance claim. Bipartite Attention is stated to select prefixes without compromising accuracy, grounded in the permutation-invariance of user and item token semantics; how exactly this is preserved, and for which generative-recommender architectures it holds, are details to confirm in the full paper.
The reported numbers are for a specific setting. The up-to-1.6-times throughput and up-to-58-percent computation reduction are relative to the conventional user-as-prefix approach on the datasets evaluated; the baselines, workloads, and hardware, and how the benefit varies across them, are things to verify in the full paper.

The evaluation

Reusing computation when serving generative recommenders

Property	No reuse	User-as-prefix caching
What can be reused
Exploits popular items	No
Cache management	None	Single user cache
Item-cache memory cost	n/a
Reported result	Baseline compute

From the paper's abstract. Bat adaptively selects the user or the item as the cacheable prefix via Bipartite Attention, manages user- and item-prefix caches in a disaggregated pool with hot-replicated cold-sharded item placement, and schedules prompts by hotness, improving serving throughput by up to 1.6 times and reducing total computation by up to 58 percent over the conventional user-as-prefix approach. Baselines, datasets, and hardware details are in the full paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This page is written from the paper’s abstract; the full text was not accessible at the time of writing, so it stays at the level the abstract supports and fills in only standard, clearly-labeled background about prefix caching and key-value reuse. The claim structure is: generative recommenders are compute-bound to serve like language-model prefill, and prefix caching helps little because their personalized user profiles and real-time items give queries little common prefix; Bat observes that user and item token semantics are permutation-invariant, introduces Bipartite Attention to adaptively make either the user or the item the cacheable prefix without hurting accuracy, and co-designs a disaggregated cache pool, a hot-replicated cold-sharded item placement, and a hotness-aware scheduler, reporting up to 1.6 times throughput and up to 58 percent less computation over user-as-prefix. Treat the specific mechanism internals, the permutation-invariance justification, the datasets, and the baselines as things to verify against the full paper, since they are not established from the abstract alone; the query trace in the step-through is illustrative. Within these limits, the core idea is clear and appealing: exploit that a popular item recurs across many users to make the item, not just the user, a source of cache reuse.

Background

Why serving recommenders efficiently is such high-stakes work

It helps to see why a few tenths of a factor in recommender serving throughput matters. Recommendation is one of the highest-volume workloads in all of computing: every feed refresh, product page, and video suggestion issues queries that must be answered in milliseconds, billions of times a day, across enormous fleets of accelerators. As the models behind these recommendations grow more powerful — shifting from feature-crossing networks toward the generative, attention-based designs this work targets — they also grow far more expensive to run, and at this scale even a modest reduction in compute per query is an enormous saving in machines and energy. That pressure is why so much systems effort goes into wringing efficiency out of recommendation inference specifically, treating it as a first-class serving problem rather than a generic one. Attacking the cost of serving recommendation models at scale, by exploiting the particular structure of how these models compute, is the same objective the work on a compiler approach to accelerating recommendation model inference covered elsewhere on this site pursues, and Bat continues it into the new generative era. As generative recommenders adopt the machinery of language models, the techniques that made language-model serving affordable — and the ways they must be adapted, as here, to the peculiarities of recommendation — become central to keeping these systems economical.

If you remember only three things

1. Generative recommenders are a powerful new alternative to traditional deep-learning recommendation models, but they are compute-bound to serve, resembling the prefill stage of large language model inference, and the standard optimization of prefix caching — computing a shared prompt prefix's key-value cache once and reusing it — helps little, because their queries pair a highly personalized user profile with items retrieved in real time, so there is little common prefix across queries and the conventional user-as-prefix scheme can reuse only per-user state, never exploiting that a popular item recurs across many users.
2. Bat is a serving system built on the observation that the semantics between user and item tokens are permutation-invariant, so the order does not affect the result: it introduces Bipartite Attention, which adaptively selects either the user or the item as the cacheable prompt prefix without compromising accuracy, unlocking key-value cache reuse such as making a hot item the prefix so its cache is shared across all the users who see it, and co-designs a disaggregated cache pool that manages user-prefix and item-prefix caches separately, a hot-replicated cold-sharded item-cache placement that minimizes the added memory while keeping communication low, and a hotness-aware prompt scheduler that picks prefixes under memory limits.
3. The abstract reports that Bat improves serving throughput by up to 1.6 times and reduces total computation by up to 58 percent over the conventional user-as-prefix approach on the evaluated recommendation datasets; this page is written from the abstract because the full text was unavailable, so the internal design of Bipartite Attention, the cache management and placement details, the permutation-invariance justification, the datasets, and the baselines should be confirmed against the full paper, and the durable idea to retain is that letting either the user or the item be the reusable prefix, rather than fixing it to the user, turns popular items into a broad new source of cache reuse.

Check yourself

Check your understanding

1. Why does prefix caching help little when serving generative recommenders?
2. What is the key observation behind Bat, and what mechanism does it enable?
3. How does Bat manage the extra memory that item caches introduce?
4. What did Bat report achieving, and what should you keep in mind about this summary?

Sources

The paper is “Bat: Efficient Generative Recommender Serving with Bipartite Attention” by Jie Sun, Shaohang Wang, Zimo Zhang, Zhengyu Liu, Yunlong Xu, Peng Sun, Bo Zhao, Bingsheng He, Fei Wu, and Zeke Wang (Zhejiang University, The University of Hong Kong, Alibaba Group, Aalto University, and the National University of Singapore), published at ASPLOS 2026 (Volume 2), DOI 10.1145/3779212.3790131. It is released under a Creative Commons Attribution license.

This explainer is written from the paper’s abstract; the full open-access text was not accessible at the time of writing. The claims traceable to the abstract are: that generative recommenders have emerged as promising alternatives to traditional deep-learning recommendation models but remain computationally expensive in inference, exhibiting compute-bound characteristics similar to the prefill stage of large language model inference; that prefix caching can reduce redundant computation by reusing previously constructed key-value caches, but the unique properties of generative recommenders — highly personalized user profiles and real-time item retrieval — make cache reuse across queries challenging and the computational savings limited; that the key observation is that the semantics between user and item tokens are permutation-invariant; that Bat proposes Bipartite Attention, an attention mechanism enabling adaptive selection of either the user or the item as the prompt prefix without compromising accuracy, unlocking new opportunities for key-value cache reuse; that Bat co-designs a disaggregated key-value cache pool to proactively manage user-prefix and item-prefix caches as separate components; that, because introducing item caches incurs additional memory overhead, Bat develops a hot-replicated cold-sharded item cache placement strategy that minimizes memory usage while maintaining low communication overheads; that Bat introduces a hotness-aware prompt scheduling strategy to optimize prefix selection under memory constraints; and that experiments on multiple recommendation datasets show Bat improves serving throughput by up to 1.6 times over the conventional user-as-prefix approach while reducing total computation by up to 58 percent. Background explanations of prefill-like inference, prefix and key-value caching, and hot-versus-cold data placement are standard domain context provided to make the abstract’s claims understandable, and the query trace in the step-through is illustrative. The internal design details, the accuracy justification, the datasets, the baselines, and the hardware should be confirmed against the full paper, which is available through the ACM Digital Library at the DOI above.