CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory

Jiashun Suo, Xiaojian Liao, Limin Xiao, Li Ruan, Jinquan Wang, Xiao Su and 1 other

A Collaboration-of-Experts application chains hundreds of small specialized models to reach accuracy a single model cannot, but on a memory-limited edge device the experts do not all fit in GPU memory, so they are shuffled in and out from CPU memory and SSD — and that switching, not the actual computation, ends up dominating latency; this paper (Beihang University — Jiashun Suo, Xiaojian Liao, and colleagues) presents CoServe, which exploits expert dependency to schedule requests and manage experts so each expert is loaded far less often, reaching 4.5 to 12 times higher throughput on real intelligent-manufacturing workloads.

The argument in one glance

Problem

Hundreds of expert models beat one big model, but they do not fit in memory

A growing idea in machine learning is that many small, specialized models can beat one giant general model on a specific task. Collaboration-of-Experts (CoE) puts this to work: instead of a single monolithic large language model, it routes each input through several experts, each trained for one narrow job, and combines their answers. For precision-critical uses this is decisive — automatic circuit-board quality inspection, for instance, needs about 99.9 percent detection accuracy that no single model reaches, but a collaboration of experts can. The catch is memory. A real circuit-board inspection application uses over 300 experts (about 13 billion parameters, roughly 60 gigabytes), and it must run on an edge device for low latency and privacy. A typical edge GPU has only about 12 gigabytes of memory, nowhere near enough to hold all the experts. So the experts are kept in CPU memory or on SSD and loaded into GPU memory on demand as inference needs them. That loading — moving an expert from a slower tier into the GPU — is called expert switching, and it turns out to be the whole problem: switching an expert in from SSD can account for over 90 percent of inference latency, so the machine spends most of its time shuffling models around rather than computing.
What people did

Borrow expert-management tricks from Mixture-of-Experts systems

The natural starting point is to reuse the techniques that existing multi-expert systems already use. The state-of-the-art CoE system, Samba-CoE, handles requests first-come, first-served and, when GPU memory is full, evicts whichever expert was least recently used (LRU) — the same kind of history-based policy that Mixture-of-Experts (MoE) serving systems rely on to decide which experts to keep resident.
Why it fell short

History-based policies ignore the dependency structure unique to CoE

These borrowed policies leave most of the switching on the table, because they throw away information that CoE inference actually has. First, requests are served in arrival order, so two requests that both need the same expert can sit far apart in the queue; after the first finishes, the expert may be evicted to make room, only to be switched back in for the later request — a switch that simply reordering the queue would have avoided. Second, evicting by least-recently-used considers only past usage, but CoE systems know more: each expert comes with a pre-assessed usage probability, and experts have dependencies (a subsequent expert runs only after the preliminary expert it depends on has produced its output). An LRU policy blind to these can evict exactly the expert about to be needed, or keep a subsequent expert whose preliminary expert is not even loaded yet. Third, the best split of scarce memory between storing experts and holding the intermediate results of a batch depends on the device, and on whether the work runs on the CPU or the GPU, which differ sharply in compute and memory. What was missing was a serving system that exploits expert dependency — the property that makes CoE distinctive — to avoid unnecessary switching, and that tunes itself to heterogeneous edge hardware.
This paper's idea

Make scheduling and eviction dependency-aware, and profile the device

CoServe is a CoE serving system for heterogeneous CPU and GPU with limited memory, whose central idea is to exploit expert dependency during both request scheduling and expert management so that experts are switched far less often. It has three parts. Dependency-aware request scheduling assigns each incoming request to the CPU or GPU executor queue that minimizes the total inference time, and reorders requests so that those needing the same expert are grouped together within a window — so the expert is loaded at most once for the whole group instead of repeatedly. Dependency-aware expert management replaces LRU with a two-stage eviction policy: first evict subsequent experts whose preliminary expert is not present (they cannot run yet anyway), then evict remaining experts in ascending order of usage probability, keeping the experts most likely to be needed. An optional offline profiler measures each device once, producing a performance matrix (how batch latency scales, and the maximum useful batch size) and choosing the memory split between expert storage and batch intermediates. Because these decisions use the dependency and usage information that is unique to CoE, CoServe cuts switching dramatically and delivers 4.5 to 12 times higher throughput than the state-of-the-art system on real intelligent-manufacturing workloads.

Why this problem is worth solving

Collaboration-of-Experts unlocks accuracy that matters in the real world, but only if it can actually run where it is needed.

Some applications have hard accuracy floors that a single model cannot clear. Automatic circuit-board inspection is a clean example: it demands roughly 99.9 percent detection accuracy, and a monolithic model falls short, while a collaboration of many narrow experts — each trained to catch one kind of defect or check one kind of component — can reach it. This is why CoE is attractive for precision-critical, high-stakes tasks. But the same design that gives the accuracy creates the deployment problem: reaching that accuracy takes hundreds of experts, tens of gigabytes of parameters in total, far more than an edge GPU can hold. And these applications often must run at the edge, not in a cloud, for latency and for privacy (factory image data may not leave the premises). So the experts have to be stored in slower tiers — CPU memory, SSD — and streamed into the GPU as inference calls for them. Making that work efficiently is what stands between CoE’s accuracy and its practical use.

The reason it runs slowly is that the cost has moved from computing to moving models, and the movement is largely avoidable.

On a memory-constrained device, only a handful of the experts fit in GPU memory at once, so inference constantly loads experts in from CPU memory or SSD. Measurements show this dominates: switching an expert from SSD into the GPU can be over 90 percent of inference latency, and even on a unified-memory device where CPU and GPU share memory, moving an expert from the CPU side still costs over 60 percent, partly because AI frameworks reorganize the data on the way. So the device spends most of its time not on the actual defect-detection math but on shuttling model weights across tiers. The crucial observation is that much of this switching is unnecessary. If two requests that both use the same expert were processed back to back, the expert would be loaded once and reused, not evicted and reloaded. If eviction respected which experts are actually likely to be used next — information CoE has in the form of usage probabilities and dependencies — the wrong expert would not be thrown out just before it is needed. The switching cost is real, but a large fraction of it is an artifact of scheduling and eviction policies that ignore what the system already knows.

What people did before, and where it broke

Serve requests first-come, first-served (Samba-CoE). Process requests strictly in arrival order. Where it broke. Two requests needing the same expert can be far apart in the queue, so the expert is evicted between them and must be switched back in — a reload that reordering would have avoided.
Evict experts by least-recently-used (LRU). When GPU memory is full, drop whichever expert was used longest ago. Where it broke. LRU looks only at past usage and ignores CoE’s known usage probabilities and dependencies, so it can evict an expert that is about to be needed, or keep a subsequent expert whose preliminary expert is not even loaded.
Reuse Mixture-of-Experts management methods. Apply MoE expert-scheduling and caching policies directly to CoE. Where it broke. MoE routers are trained into a single model and predict poorly here; the methods rely on historical statistics with low accuracy, so they evict the wrong experts.
Fix a single memory split and batch size. Allocate memory between expert storage and batch intermediates once, uniformly. Where it broke. The best split depends on the device and on whether work runs on the CPU or the GPU, which differ greatly in compute and memory, so one fixed choice underuses some hardware and thrashes on others.
The gap. No CoE serving system exploited expert dependency — the property that distinguishes CoE from MoE — to cut unnecessary switching, nor tuned its memory and batching to heterogeneous edge devices.

How CoServe works

Turning dependency information into fewer expert switches

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Exploit a structural property of the workload to skip redundant loading. CoServe’s foundation is that CoE inference has structure prior systems threw away — which experts tend to be used, and which depend on which — and that this structure predicts what to keep near the compute so scarce fast memory is not wasted loading things that will not run. Reading a model’s own usage structure to decide what to bring into fast memory, instead of blindly loading and evicting, is the same undertaking the work on fast on-device large-model inference by exploiting neuron co-activation (NeuraLink) covered elsewhere on this site pursues, letting the workload’s predictable pattern drive what is resident. CoServe applies this at the granularity of whole experts, using dependency and usage probability to avoid loading experts that cannot yet run or are unlikely to be needed.

Manage a small fast memory as a predictive cache across tiers. The heart of CoServe’s expert management is deciding, when GPU memory fills up, which experts to evict to CPU memory or SSD and which to keep — a tiered-memory caching problem driven by predictions of future use rather than by raw history. Treating a limited fast tier as a cache and using predictions of what will be needed to move data across tiers before it stalls the computation is the same principle the work on tensor migration and prefetching in unified memory (DeepUM) covered elsewhere on this site is built on, moving data across a memory hierarchy according to anticipated need. CoServe’s two-stage eviction is exactly such a prediction-driven policy, replacing least-recently-used with usage probabilities and dependency order.

Group and batch identical work to amortize a costly setup. CoServe’s request scheduling collects requests that use the same expert and runs them together as a batch, so the one expensive act — loading that expert into the GPU — is paid once and then reused across many inputs. Batching many requests that share the same heavy resource so that a costly one-time cost is spread over lots of useful work is the same idea the work on batched large-model inference on heterogeneous accelerators (NeuPIMs) covered elsewhere on this site develops, amortizing an expensive setup across a full batch. CoServe does this by reordering requests behind others that use the same expert and splitting them into hardware-sized batches, so each loaded expert serves as much work as possible.

Following a stream of requests through CoServe

Scheduling and serving CoE requests with fewer switches

A trace, drawn from the paper, of how CoServe processes incoming inference requests across CPU and GPU executors.

Step 1 of 5

Requests arrive, each needing a particular expert

A stream of inference requests comes in (for example, one per component image on a circuit board). Each request needs a specific expert, and the experts are spread across GPU memory, CPU memory, and SSD.

Whether an expert is already resident determines whether a costly switch is needed.

Why it works

Grouping same-expert requests turns many loads into one. Requests using one expert can be scattered through the queue → but arranging them adjacently means the expert is loaded once for the whole group → so repeated switches of the same expert disappear.
Dependency-aware eviction keeps the right experts resident. Least-recently-used evicts by history alone → but CoServe evicts by usage probability and dependency order → so the experts most likely to be needed next stay in memory and are not reloaded.
Evicting stuck subsequent experts frees memory for free. A subsequent expert whose preliminary expert is absent cannot run yet → so evicting it loses no usable work → and it frees space for an expert that can run now, wasting nothing.
Balancing parallel CPU and GPU queues raises aggregate throughput. The slowest executor queue sets the total time → but assigning each request to minimize the maximum queue time balances the heterogeneous executors → so both CPU and GPU stay busy and total throughput rises.
Offline profiling matches memory and batch size to the device. The best split of memory and the best batch size differ per device and processor → but a one-time profile measures latency scaling and the useful batch limit → so CoServe picks an allocation that neither thrashes nor underuses the hardware.

What it costs (trade-offs and scope)

It depends on CoE’s dependency and usage information. CoServe’s advantage comes from knowing expert dependencies and pre-assessed usage probabilities, which CoE provides through its explicit routing module; systems without this structure (for example Mixture-of-Experts models, whose routers are trained into one model) cannot supply the same signals.
The offline profiler must be run per device. Getting the performance matrix and memory split right requires profiling each device configuration once; this is a one-time cost, but it must be repeated whenever the hardware changes, and the online decisions lean on its accuracy.
Its objective is throughput, not per-request latency. Grouping and batching requests that share an expert maximizes overall throughput, which suits workloads with a batch deadline (all images analyzed within a time window), but it can delay any individual request relative to strict arrival order.
The overheads are small but real. The paper reports request scheduling adds under 3 percent and expert management under 0.2 percent of task time; these are modest, but they are extra work the baseline does not do, justified only because they cut far more switching.
It is validated on one real application. Because public CoE models and datasets are scarce, CoServe is evaluated on a real circuit-board inspection workload; the authors argue the design carries over to any CoE with a routing module and experts, but broad generality across CoE applications is not demonstrated.

The evaluation

Serving a 300-expert CoE application on a memory-limited edge device

Property	Samba-CoE (FCFS + LRU)	MoE-style management	CoServe
Request ordering
Eviction policy
Expert switches
Device and memory tuning
Throughput vs state of the art

A comparison of CoE serving strategies on a memory-limited edge device. By exploiting expert dependency in both request scheduling and expert management, and by profiling each device to tune memory and batching, CoServe loads each expert far less often and reaches 4.5 to 12 times higher throughput than the state-of-the-art Samba-CoE on real intelligent-manufacturing workloads.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy of the ASPLOS 2025 paper). The system described is the paper’s: CoServe is a CoE model serving system for heterogeneous CPU and GPU with limited memory. Its key idea is to exploit expert dependency — a property unique to CoE inference — during scheduling and expert management to reduce unnecessary expert switching. It has three components: dependency-aware request scheduling (assign each request to the executor queue that minimizes total inference time, reorder to group requests using the same expert, and split into batches up to a maximum executable batch size), dependency-aware expert management (a two-stage eviction that first drops subsequent experts lacking a present preliminary expert, then evicts by ascending usage probability), and an optional offline profiler that produces a per-device performance matrix and chooses the memory allocation between expert storage and batch intermediates. The paper reports that switching from SSD can exceed 90 percent of inference latency, that scheduling overhead is under 3 percent and expert-management overhead under 0.2 percent of task time, and that CoServe achieves 4.5 to 12 times higher throughput than the state-of-the-art Samba-CoE on a real circuit-board inspection workload. Exact algorithms, the latency model constants, the ablations, and full results should be read from the paper.

Background

Why serving giant models from small memories is a recurring systems problem

It helps to see CoServe within a broader pattern in machine-learning systems: models keep growing faster than the fast memory available to run them, so a large part of the engineering becomes deciding what to keep close to the compute and what to fetch from a slower tier just in time. Whether the pieces are the layers of one network, the embedding tables of a recommender, or, as here, hundreds of independent experts, the recurring question is the same — the working set does not fit, so success hinges on predicting the near future and staging data across a hierarchy of GPU memory, CPU memory, and storage without letting the transfers stall the computation. The naive answers (serve in arrival order, evict by recency) waste bandwidth because they do not use what the workload knows about itself. The effort to serve a model far larger than fast memory by streaming its parts from storage at high bandwidth is the same terrain the work on maximizing SSD bandwidth for huge embedding models (MaxEmbed) covered elsewhere on this site occupies, treating storage as an extension of memory for an oversized model. CoServe adds the insight that CoE inference carries explicit dependency and usage information, and that scheduling and eviction which respect that structure can eliminate most of the data movement rather than merely speeding it up.

If you remember only three things

1. Collaboration-of-Experts (CoE) inference chains hundreds of small specialized experts to reach accuracy a single model cannot (about 99.9 percent for circuit-board inspection), but the experts far exceed edge GPU memory, so they are offloaded to CPU memory and SSD and switched in on demand. That switching dominates: loading an expert from SSD can be over 90 percent of inference latency. Prior systems serve requests first-come-first-served and evict by least-recently-used, ignoring the dependency and usage structure that is unique to CoE and that makes most switching avoidable.
2. CoServe exploits expert dependency in three ways. Dependency-aware request scheduling assigns each request to the CPU or GPU queue that minimizes total inference time and reorders requests so same-expert ones are grouped and batched, loading each expert at most once per group. Dependency-aware expert management replaces LRU with two-stage eviction: first drop subsequent experts whose preliminary expert is absent, then evict by lowest usage probability. An offline profiler measures each device to set the max batch size and the memory split between expert storage and batch intermediates.
3. On a real intelligent-manufacturing (circuit-board inspection) workload, CoServe reaches 4.5 to 12 times higher throughput than the state-of-the-art Samba-CoE, with scheduling overhead under 3 percent and expert-management overhead under 0.2 percent of task time. The trade-offs: it relies on CoE's explicit dependency and usage information (not available in Mixture-of-Experts), the profiler must be re-run per device, its objective is throughput rather than per-request latency, and it is validated on one application, though the authors argue the design generalizes to any CoE with a routing module.

Check yourself

Check your understanding

1. Why is expert switching, rather than computation, the main bottleneck for CoE inference on edge devices?
2. What is "expert dependency," and why does using it beat the least-recently-used (LRU) policy?
3. How does CoServe's dependency-aware request scheduling reduce the number of expert loads?
4. Predict: a CoServe deployment moves to a new edge device with a different GPU and SSD, but the offline profiler is not re-run. What is likely to happen?

Sources

The paper is “CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory” by Jiashun Suo, Xiaojian Liao, Limin Xiao, Li Ruan, Jinquan Wang, Xiao Su, and Zhisheng Huo (State Key Laboratory of CCSE and School of Computer Science and Engineering, Beihang University), published at ASPLOS 2025, DOI 10.1145/3676641.3715986. An author copy is available at arXiv:2503.02354.

This explainer is written from the paper’s full text (the authors’ arXiv copy). CoServe is an efficient CoE model serving system for heterogeneous CPU and GPU with limited memory. Its key idea is to reduce unnecessary expert switching by exploiting expert dependency, a property of CoE inference overlooked by prior frameworks. CoServe introduces dependency-aware request scheduling (assigning requests to the executor queue that minimizes total inference time, grouping requests that use the same expert, and splitting them into batches up to a maximum executable batch size), dependency-aware expert management (a two-stage eviction that first removes subsequent experts lacking a present preliminary expert, then evicts by ascending usage probability), and an optional offline profiler that automatically finds an optimal resource allocation across processors and devices via a performance matrix. The paper reports that expert switching from SSD can exceed 90 percent of inference latency, that scheduling and expert-management overheads are small (under 3 percent and under 0.2 percent of task time respectively), and that in real intelligent-manufacturing (circuit-board inspection) workloads, CoServe achieves 4.5 to 12 times higher throughput than the state-of-the-art Samba-CoE. Numbers are the paper’s own; exact algorithms, the latency model, the ablations, and full results should be confirmed in the paper.