MoDM: Efficient Serving for Image Generation via Mixture-of-Diffusion Models

Text-to-image diffusion models face a hard choice — small models are fast but low quality, big models look great but take tens of seconds — and serving millions of image requests makes that expensive; this paper (University of Michigan and Intel Labs — Yuchen Xia, Souvik Kundu, Nishil Talati, and colleagues) presents MoDM, which caches finished images (not model-internal features, so they work across model families), retrieves one that matches a new prompt, and has a small fast model lightly repaint it — reserving the big slow model only for genuinely new prompts, cutting serving time 2.5 times with no quality loss.

The argument in one glance

Problem

Generating images with diffusion models is slow, and you must trade quality for speed

Text-to-image diffusion models — the technology behind tools like Stable Diffusion, DALL-E, and Adobe Firefly — have made it possible to turn a written prompt into a photorealistic image, and demand is enormous (Firefly alone has generated billions of images). But generating one image is expensive. A diffusion model works by starting from random noise and iteratively "de-noising" it into an image over many steps — typically around fifty — and each step runs the entire model, so a single image can take tens of seconds. That is far slower than older one-shot generators, and it makes serving the flood of incoming prompts costly. Worse, there is a built-in trade-off between latency and quality: a small diffusion model is fast but produces lower-quality images, while a large model produces beautiful images but is slow. A serving system therefore seems forced to choose — be quick and disappoint users with mediocre images, or be slow and expensive but look good. At the scale these services operate, neither choice is acceptable: you cannot make every user wait tens of seconds on a big model, and you cannot ship low-quality results from a small one. The problem is how to serve high-quality images fast, at scale, without paying the full cost of a large model on every request.
What people did

Cache intermediate diffusion features to skip some of the work

A natural way to speed diffusion up is caching: reuse work from previous generations instead of redoing it. Prior systems cache the model's internal intermediate state — latent caching stores the partway-denoised noisy features, feature caching stores activations, and some store image patches. A system like NIRVANA caches these intermediate latents so that a new, similar prompt can start partway through the de-noising process rather than from scratch, saving some of the fifty steps.
Why it fell short

Model-specific caches lock you to one model and leave gains on the table

Caching intermediate features has two limitations that cap its benefit. First, the cached content is model-specific: an intermediate latent or activation only makes sense inside the exact model that produced it, so the cache ties the whole serving system to a single model. You cannot reuse a cached feature across different diffusion model families, which forecloses using a mixture of models. Second, relying on a single model limits the potential gains and the flexibility to optimize inference — you are stuck with that one model's latency-quality point for every request, whether it hit or missed the cache. And because the retrieval in these systems matches a new prompt to cached content by comparing the prompt text to the prompt that produced the cached feature (text-to-text similarity), it does not directly measure whether the cached image actually suits the new prompt. So prior caching helps, but it cannot exploit the very latency-quality trade-off the problem is about, because it is welded to one model and one representation. What was missing was a cache that is model-agnostic, retrieved by how well its content matches the prompt, and usable to route different requests to differently-sized models.
This paper's idea

Cache finished images, retrieve by prompt match, and repaint with a small model

MoDM is a caching-based serving system that dynamically balances latency and quality using a mixture of diffusion models, and its founding move is what it caches: final generated images, not model-internal intermediate features. A finished image is model-agnostic, so it can be retrieved and reused across multiple diffusion model families, and it can be matched to a new prompt by how well the image aligns with that prompt (text-to-image similarity, measured with a CLIP score), which the authors show tracks user intent better than the text-to-text matching of prior work. On a cache hit — a new prompt closely matches a cached image — MoDM retrieves that high-quality image, applies controlled noise to partially re-noise it, and refines it with a small, fast model for fewer than the full set of de-noising steps. Because the small model starts from a good image and only edits it toward the new prompt, the result keeps the cached image's quality while enjoying the small model's low latency. On a cache miss — a genuinely new prompt — MoDM runs the large model for full quality, and adds the result to the cache for future reuse. Around this, a Request Scheduler categorizes requests into hits and misses, retrieves cached images, and maintains the cache over time, while a Global Monitor watches request rates and hit/miss distributions to dynamically allocate GPU resources and schedule the right models, keeping throughput high and meeting service-level objectives (SLOs) under varying load. MoDM reports reducing average serving time by 2.5 times while retaining image quality.

Why this problem is worth solving

Image generation is a massive, fast-growing, and expensive workload, so serving it more efficiently saves large amounts of compute across the industry.

Text-to-image generation has become a cornerstone of creative tools, and the volume is staggering — services report billions of generated images and exponentially rising prompt submissions. Each of those images costs tens of seconds of heavy GPU computation, because diffusion is inherently iterative, so the aggregate cost of serving this demand is enormous. Any technique that delivers the same quality with a fraction of the work therefore has a large, direct payoff in hardware, energy, and user-facing latency, multiplied across an immense number of requests. And because the latency-quality trade-off is fundamental to diffusion, a system that can escape it — giving users high-quality images at low latency rather than forcing a compromise — improves both the economics for the provider and the experience for the user at once. As demand keeps climbing, the efficiency of the serving system, not just the quality of the model, becomes the thing that determines whether image generation is affordable at scale.

The key realization is that what you cache determines what you can do with it, and finished images are far more reusable than model internals.

The reason prior diffusion caching could not fully exploit the latency-quality trade-off is a choice buried in what it stored. An intermediate feature — a partway-denoised latent or an activation — is meaningful only inside the specific model that produced it, so caching it chains the system to that one model, and matching new prompts against it can only be done indirectly through the prompts that created it. MoDM’s insight is that a final image has neither limitation. It is just an image, so it is model-agnostic: any model, of any size, can take it as a starting point and refine it, which is exactly what makes a mixture of models possible — small models for the easy, cache-hit cases and large models for the hard, cache-miss ones. And an image can be matched to a prompt directly, by measuring how well the image depicts what the prompt asks for, which retrieves better-suited content than comparing prompt texts. This is why the representation choice cascades into the whole design: caching final images is what lets MoDM retrieve by relevance, reuse across model families, and route requests to the right-sized model, turning a fixed trade-off into a lever the serving system can pull per request. The rest of the system — controlled re-noising, the scheduler, the monitor — exists to turn that reusable cache into fast, high-quality, SLO-meeting service.

What people did before, and where it broke

Serve every request with a large diffusion model. Run the full model, about fifty de-noising steps, for each image. Where it broke. Each generation takes tens of seconds, which is far too slow and expensive at the scale these services operate.
Serve with a small model for speed. Use a faster, smaller diffusion model. Where it broke. Small models produce noticeably lower-quality images, disappointing users.
Cache intermediate latents or activations (NIRVANA). Store partway-denoised features so similar prompts skip early steps. Where it broke. The cached content is model-specific, tying serving to one model and foreclosing a mixture of models.
Retrieve cached content by text-to-text similarity. Match a new prompt to the prompt that generated a cached feature. Where it broke. Comparing prompt texts does not directly measure whether the cached image actually suits the new prompt, so retrieval is less well-aligned with intent.
The gap. No system cached a model-agnostic artifact retrievable by prompt relevance and usable to route requests across differently-sized models, so none could turn the latency-quality trade-off into an advantage.

How MoDM works

Caching final images to serve high quality fast with a mixture of models

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Retrieve a relevant past result and adapt it instead of generating from scratch. MoDM’s core is to find a cached image that matches the new prompt and lightly repaint it, rather than run the full expensive generation. Serving a request by retrieving relevant prior content and reusing it to shortcut the heavy computation, instead of recomputing everything, is the same approach the work on accelerating retrieval-augmented generation covered elsewhere on this site takes, letting retrieved material carry much of the load so the model does less. MoDM applies this to image generation, where the retrieved artifact is a cached image and the shortcut is starting a small model from it rather than from noise.

Route each request to the right-sized model. MoDM uses a mixture of models — a small fast model for cache hits and a large accurate model for cache misses — choosing per request which to run. Dynamically selecting among differently-capable models for each request, so cheap work goes to a cheap model and hard work to an expensive one, is the same idea the work on a dynamic scheduler for real-time multi-model ML workloads (DREAM) covered elsewhere on this site develops, matching each request to an appropriate model rather than using one for all. MoDM’s model-agnostic image cache is precisely what makes this multi-model routing possible.

Allocate resources and schedule to meet latency targets under changing load. MoDM’s Global Monitor watches the request rate and cache behavior and reallocates GPUs and models to keep throughput up and hit SLOs. Managing GPU serving resources adaptively to satisfy quality-of-service targets under varying, multi-stage load is the same concern the work on QoS-aware, resource-efficient multi-stage GPU serving (Astraea) covered elsewhere on this site addresses, treating SLO satisfaction as the objective the allocator must meet. MoDM applies it to a diffusion pipeline whose per-request cost depends on whether it hit the cache.

Following a request through MoDM

Serving a cache-hit and a cache-miss prompt

A trace, drawn from the paper, of how MoDM routes two requests through its mixture of models.

Step 1 of 5

A prompt arrives and is matched against the cache

The Request Scheduler takes an incoming prompt and searches the cache of final images for one whose content best matches the prompt, using text-to-image (CLIP) similarity rather than comparing prompt texts.

Caching images is what makes relevance-based retrieval possible.

Why it works

Final images are reusable where features are not. Intermediate latents only work inside one model → but a finished image is model-agnostic → so it can be retrieved and refined by any model, enabling a mixture of models.
Relevance retrieval finds better-suited content. Matching prompt texts is indirect → but scoring how well a cached image depicts the new prompt → so retrieval returns images that actually fit user intent, keeping quality high on hits.
Repainting a good image is cheap and high-quality. Generating from noise takes the full de-noising process → but starting a small model from a retrieved image with controlled noise → so few steps suffice and the small model’s speed comes without its usual quality loss.
Routing sends hard cases to the big model only. Using the large model for everything is slow → but reserving it for cache misses while hits use the small model → so full quality is paid for only when genuinely needed.
The monitor keeps it stable under changing load. Per-request savings alone do not guarantee SLOs → but dynamically allocating GPUs and models to the hit/miss mix and request rate → so throughput stays high and latency targets are met as conditions vary.

What it costs (trade-offs and scope)

The gains depend on the cache being hit. MoDM’s speedup comes from cache-hit requests using the small model; a workload of mostly novel prompts that miss the cache falls back to the large model, so the benefit scales with how much prompt overlap the traffic has.
Retrieval quality is critical. If retrieval returns an image that does not truly match the prompt, refining it can produce a wrong or lower-quality result; the approach leans on the text-to-image similarity being a good judge of relevance.
Controlled re-noising must be tuned. How much noise to add to a retrieved image, and how many refinement steps to run, trades quality against latency; too little re-noising may leave the wrong content, too much loses the cached quality’s benefit.
Maintaining and searching the image cache has cost. Storing final images and searching them by similarity consumes memory and lookup time, and the cache must be maintained over time as prompts drift, overhead a stateless server does not have.
Reported gains are workload- and setup-specific. The 2.5-times average serving-time reduction is measured on particular models, datasets, and request patterns; the exact improvement depends on the model sizes, the cache hit rate, and the load.

The evaluation

Serving text-to-image diffusion requests

Property	Large model for all	Intermediate-feature cache (NIRVANA)	MoDM
What is cached
Reuse across model families
Retrieval basis
Model routing
Average serving time

A comparison of ways to serve text-to-image diffusion. By caching final images (model-agnostic, retrievable by text-to-image relevance) and using a mixture of models — a small fast model to repaint retrieved images on cache hits, a large model on misses — with a scheduler and a global monitor for SLO-aware GPU allocation, MoDM reports reducing average serving time by 2.5 times while retaining image quality. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2503.11972, of the ASPLOS 2026 paper). The described system is the paper’s: MoDM is a caching-based serving system for text-to-image diffusion models that dynamically balances latency and image quality through a mixture of diffusion models. Its key design choice is to cache final generated images rather than model-specific intermediate features, which makes the cache model-agnostic (reusable across diffusion model families) and enables retrieval by text-to-image similarity (using a CLIP score) rather than the text-to-text similarity of prior work such as NIRVANA. On a cache hit, MoDM retrieves the matching image, applies controlled noise, and refines it with a smaller, faster model for fewer than the full de-noising steps, preserving quality while reducing latency; on a cache miss, it uses a large model for full quality and adds the result to the cache. A Request Scheduler categorizes hits and misses, retrieves cached images, and maintains the cache; a Global Monitor analyzes request rates and hit/miss distributions to dynamically allocate GPU resources and schedule models, ensuring high throughput and SLO satisfaction under varying request rates. The paper reports MoDM reduces average serving time by 2.5 times while retaining image quality. The background above on diffusion de-noising, the roughly fifty steps, latent/feature caching, and CLIP-based similarity is standard context provided to make the approach intuitive. The step-by-step walkthrough is an illustration of the described mechanism. Exact retrieval, re-noising, scheduling, and monitor details, and the full evaluation and quality metrics, should be read from the paper.

Background

Why serving generative models keeps becoming a problem of reuse and routing

It helps to see MoDM within a broad pattern: as generative models get more capable and more expensive to run, and as demand for them explodes, the efficiency battle shifts from making a single inference faster to serving many requests cleverly — and the two recurring weapons are reuse and routing. Reuse means noticing that much of the incoming work resembles work already done, and finding a cheap way to recycle a prior result instead of recomputing it from nothing; routing means recognizing that not every request needs the most expensive model, and sending easy cases to cheap paths while reserving the costly path for the hard ones. MoDM is a clean marriage of the two: it reuses finished images by retrieving relevant ones from a cache, and it routes cache hits to a small model and misses to a large one. What makes this possible is a careful choice of what to remember — final images, which are general enough to reuse and match by relevance — rather than a form of memory chained to one model. This effort to serve expensive generative models efficiently at scale, under real latency and load constraints, is the same terrain the work on serving generative large language models on preemptible instances (SpotServe) covered elsewhere on this site occupies, engineering the serving system rather than only the model. MoDM’s contribution is to show that, for text-to-image diffusion, caching final images and mixing model sizes turns the fundamental latency-quality trade-off into a knob the serving system can turn per request, cutting average serving time substantially without sacrificing the images users see.

If you remember only three things

1. Text-to-image diffusion models generate an image by iteratively de-noising over about fifty steps, each running the full model, so one image takes tens of seconds — expensive at the scale these services run (billions of images). They also impose a latency-quality trade-off: small models are fast but lower quality, large models are high quality but slow. Prior caching speeds diffusion up by storing intermediate features (latents, activations), but that cached content is model-specific, so it ties the serving system to a single model, forecloses using a mixture of models, and its retrieval matches prompt texts rather than checking whether a cached image actually suits the new prompt.
2. MoDM's founding move is to cache final generated images instead of model internals. A finished image is model-agnostic, so it can be retrieved and reused across diffusion model families, and matched to a new prompt by how well the image depicts it (text-to-image similarity via CLIP), which aligns with user intent better than text-to-text matching. On a cache hit, MoDM retrieves the matching image, applies controlled noise, and refines it toward the new prompt with a small fast model for fewer than the full de-noising steps — keeping the cached image's quality at the small model's low latency. On a cache miss, it runs the large model for full quality and adds the result to the cache. A Request Scheduler manages hits, misses, retrieval, and cache upkeep, and a Global Monitor dynamically allocates GPUs and schedules models to meet SLOs under varying load.
3. The paper reports MoDM reduces average serving time by 2.5 times while retaining image quality, turning the diffusion latency-quality trade-off into a per-request lever: cheap small-model repainting for cache hits, full large-model generation only for genuinely new prompts. The trade-offs: the speedup depends on the cache being hit (novel-prompt-heavy traffic falls back to the large model), retrieval quality is critical, the controlled re-noising must be tuned to balance quality against latency, and maintaining and searching the image cache adds memory and lookup cost. The reported gains reflect the specific models, datasets, and request patterns evaluated.

Check yourself

Check your understanding

1. Why is serving text-to-image diffusion models expensive, and what trade-off does it force?
2. What was the key limitation of prior diffusion caching (like NIRVANA) that cached intermediate features?
3. What is MoDM's founding design choice, and why does it unlock a mixture of models?
4. How does MoDM serve a cache-hit request, and what does it achieve overall?

Sources

The paper is “MoDM: Efficient Serving for Image Generation via Mixture-of-Diffusion Models” by Yuchen Xia, Divyam Sharma, Yichao Yuan, Souvik Kundu, and Nishil Talati (University of Michigan and Intel Labs), published at ASPLOS 2026, DOI 10.1145/3760250.3762220. It is available on arXiv as 2503.11972.

This explainer is written from the paper’s full text (the authors’ arXiv copy). MoDM is a caching-based serving system for text-to-image diffusion models that dynamically balances latency and image quality through a mixture of diffusion models. Its key design choice is caching final generated images rather than model-specific intermediate features, making the cache model-agnostic (reusable across diffusion model families) and enabling text-to-image similarity retrieval (via CLIP score) instead of the text-to-text retrieval of prior work (NIRVANA). On a cache hit, MoDM retrieves the matching image, applies controlled noise, and refines it with a smaller model for fewer de-noising steps, preserving quality at low latency; on a cache miss, it uses a large model for full quality and adds the result to the cache. A Request Scheduler manages hits/misses, retrieval, and cache maintenance, and a Global Monitor dynamically allocates GPU resources and schedules models to meet SLOs under varying request rates. The paper reports MoDM reduces average serving time by 2.5 times while retaining image quality. Numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.