gShare: Efficient GPU Sharing with Aggressive Scheduling in Multi-tenant FaaS platform

Yanan Yang, Zhengxiong Jiang, Meiqi Zhu, Hongqiang Xu, Yujun Wang, Liang Li and 2 others

Cloud "serverless" platforms now let you run an ML model as a function on a GPU, but they hand out GPUs in coarse chunks tied to each function's own virtual machine, so a small model that needs a sliver of a GPU still reserves far more — leaving GPUs badly under-used and bills high; this paper (China Telecom and Temple University — Yanan Yang, Liang Li, Jie Wu, and colleagues) presents gShare, which slices each GPU finely, lets many tenants transparently time-share it, and aggressively packs their requests together while still meeting latency targets.

The argument in one glance

Problem

Serverless GPUs are handed out in coarse chunks, so they sit mostly idle

Serverless computing (Function-as-a-Service, or FaaS) lets developers run code as small functions that scale automatically and bill only for what they use, and it has become a popular way to serve machine-learning models: send a request, a function spins up, runs the model, returns the answer. To make this fast, cloud providers now offer GPU functions. But there is a mismatch. For CPU and memory, FaaS platforms let you ask for tiny slivers — as little as a twentieth of a CPU core and a hundred-odd megabytes of memory — so you pay for just what your function needs. For GPUs, provisioning is coarse: you get a whole GPU, or at best a gigabyte-granularity slice of its memory, and that allocation is tied to your function's own virtual machine. The trouble is that most served models are small. An analysis of the most popular models finds the great majority need under four gigabytes, and two-thirds need under one gigabyte — far less than a whole GPU. So a small model reserves a big GPU it barely uses, wasting up to 85 percent of the resource, and because the GPU is bound to one function's VM, no other tenant can use the idle capacity. The result is GPUs that sit mostly idle while the provider buys more of them and tenants pay for capacity they never touch.
What people did

Share a GPU by swapping models in and out through a proxy

The natural fix is to let multiple functions share a physical GPU. Existing designs do this by keeping each function's model data in host memory and swapping it onto the GPU when a request arrives, coordinated by a central GPU proxy (a control plane) that different functions talk to. When function A is done, its model is swapped out and function B's is swapped in, so several functions time-share the node's GPUs instead of each monopolizing one. This raises utilization compared to giving every function its own GPU.
Why it fell short

The proxy adds overhead, swapping adds latency, and scheduling is blind

Proxy-and-swap sharing helps but falls short of true efficiency, for three reasons. First, routing every request through a GPU proxy control plane adds system overhead — the paper measures 7 to 61 percent performance fluctuation and about 40 percent more forwarding delay — because the proxy sits between the function and the GPU. Second, swapping a model in and out of the GPU is not free: it can add tens to hundreds of milliseconds of latency to a request, which is a lot for a real-time inference call with a strict latency target (service-level objective, or SLO). Third, the scheduling that decides which function runs when is a simple heuristic that is oblivious to how functions actually behave, so it either leaves the GPU under-utilized (too cautious) or overloads it and misses SLOs (too greedy). Underneath all this, the GPU is still coupled to the CPU-oriented function model — allocated and managed as if it were just another part of a function's VM — which is the wrong abstraction for a resource meant to be shared. What was missing was a way to slice GPUs finely, decouple them from the per-function VM, and schedule the sharing intelligently enough to pack tenants tightly without breaking their latency targets.
This paper's idea

Slice the GPU finely, time-share it transparently, and pack requests aggressively

gShare is a GPU function management policy for VM-based multi-tenant FaaS that makes GPU sharing both fine-grained and efficient. It rests on two insights: GPUs should be provisioned finely, like CPUs, to match what each model actually needs; and the GPU should be decoupled from the CPU-oriented function paradigm so it can be shared flexibly across tenants. To realize this, gShare provides an elastic vGPU (virtual GPU) abstraction over a microVM-based FaaS environment, using VM-passthrough virtualization for near-native performance. Its first mechanism is user-transparent vGPU switching and sharing: gShare keeps each function's model data in host memory and, using checkpoint/restore of that model data, swaps it onto the physical GPU on demand and switches which tenant's vGPU currently owns the hardware — all invisibly to the user — so many functions time-share one GPU. Its second mechanism is an aggressive request-scheduling policy: gShare formulates GPU sharing as a constrained optimization problem (meeting every function's latency SLO while minimizing GPU usage) and, because solving it exactly does not scale, uses an efficient online scheduling algorithm that reaches about 75 percent of the optimal solution while deciding 10 to 100 times faster. "Aggressive" means it packs tenants onto each GPU tightly, betting that bursty, small model-serving functions leave enough idle time to co-locate safely. The paper reports gShare cuts GPU usage by 43 to 63 percent versus a keep-alive baseline while meeting more than 95 percent of latency targets, and reduces cloud cost by 24 to 58 percent versus the state of the art while maintaining better performance — saving a typical cluster hundreds of thousands of dollars a year in GPU procurement.

Why this problem is worth solving

GPUs are the costliest resource in the cloud and serverless is where they are provisioned most wastefully, so fixing GPU sharing in FaaS is a large, direct cost lever.

Serverless model serving is now a mainstream cloud use case — major providers all offer GPU functions — and GPUs are the most expensive hardware in the datacenter. The combination is unfortunate: serverless workloads are exactly the small, bursty, elastic functions that use only a fraction of a GPU, yet the platforms hand out GPUs so coarsely that most of that expensive silicon sits idle. Every idle GPU is capital the provider paid for and is not earning from, and every over-provisioned function is a tenant paying for capacity they do not use. Because the resource is so costly and the waste so large (up to 85 percent over-provisioning), even a moderate improvement in how finely and flexibly GPUs are shared translates into major savings on both sides — the paper puts it at hundreds of thousands of dollars a year in procurement for a single typical cluster. Making serverless GPU sharing efficient is therefore not a niche tuning problem but a first-order economic win for a fast-growing slice of cloud computing.

The difficulty is that fine sharing and strict latency pull against each other, and the GPU is stuck in the wrong abstraction.

Two forces make this hard. The first is that packing more tenants onto a GPU to raise utilization directly threatens their latency: model-serving requests have strict service-level objectives, and every function you co-locate competes for the same GPU, so aggressive sharing risks queuing delays and swap overheads that blow the SLO. A cautious scheduler avoids that by under-packing, which wastes the GPU; a greedy one over-packs and misses deadlines. The right answer has to be aggressive but SLO-aware — pack tightly precisely up to the point where deadlines are still met — which is a genuine optimization problem, not a fixed rule. The second force is that the GPU has been bolted onto the CPU-oriented function model: a function’s GPU is treated as part of its VM, provisioned and bound like its CPU and memory. That coupling is why capacity cannot flow to whoever needs it. So a real solution must both decouple the GPU into a fungible, finely divisible resource and schedule its sharing intelligently against latency targets. gShare needs both a new abstraction (the elastic vGPU) and a new scheduler (the aggressive, SLO-aware optimizer), because neither alone fixes the mismatch.

What people did before, and where it broke

Provision a whole GPU (or a coarse memory slice) per function. Give each GPU function its own GPU or gigabyte-granularity allocation, tied to its VM. Where it broke. Most served models are small (two-thirds under one gigabyte), so they reserve far more than they use — up to 85 percent over-provisioning waste — and the idle capacity cannot be reused.
Share a GPU by swapping models through a central proxy. Keep models in host memory and swap them onto the GPU via a GPU proxy control plane. Where it broke. The proxy adds system overhead (7 to 61 percent performance fluctuation, about 40 percent more forwarding delay) because it sits between the function and the GPU.
Swap models in and out on demand. Move a model onto the GPU when its request arrives, off when done. Where it broke. Swapping adds tens to hundreds of milliseconds per request, which is severe for real-time inference with strict latency SLOs.
Schedule sharing with a simple heuristic. Use a behavior-blind rule to decide which function runs when. Where it broke. Being oblivious to function behavior, it either under-utilizes the GPU (too cautious) or overloads it and violates SLOs (too greedy).
The gap. No design sliced GPUs finely, decoupled them from the per-function VM, and scheduled the sharing aggressively yet SLO-aware enough to pack tenants tightly without missing deadlines.

How gShare works

Fine-grained, transparent GPU sharing with an SLO-aware aggressive scheduler

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Turn one expensive GPU into many finely-shared slices across tenants. gShare’s founding move is to stop giving each function a whole GPU and instead virtualize the GPU into elastic vGPUs that many tenants share, matching each model’s modest real needs. Multiplexing a scarce, costly accelerator among many co-located tenants so its capacity is actually filled, rather than dedicating it to one, is the same undertaking the work on high-performance multi-tenant deep-learning services (Veltair) covered elsewhere on this site pursues, co-locating workloads on shared hardware to drive up utilization. gShare does this for serverless GPU functions, with a fine-grained vGPU abstraction that lets a small model take only the slice it needs.

Fit the mechanism to the serverless setting of small, bursty, on-demand functions. gShare is built for FaaS specifically: functions appear and vanish, models are small, and demand is spiky, so it keeps models in host memory and streams them onto the GPU only when a request arrives. Designing a serving system around the elastic, request-driven, latency-sensitive nature of serverless inference is the same problem the work on a native serverless system for low-latency, high-throughput inference (INFless) covered elsewhere on this site addresses, tailoring the platform to how serverless model serving actually behaves. gShare adds the GPU-sharing dimension that a serverless inference platform needs to be cost-efficient.

Decide how tightly to pack by solving a constrained optimization, not guessing. gShare’s aggressive scheduler is not a fixed rule; it writes the sharing decision as an optimization — minimize GPU usage subject to latency SLOs — and solves it online because exact solving is too slow. Casting a resource-allocation decision under strict deadlines as a constrained optimization and solving it efficiently, rather than relying on a hand-tuned heuristic, is the same approach the work on constraint-aware resource scheduling for inference (ExeGPT) covered elsewhere on this site takes, letting a solver find the schedule a simple rule cannot. gShare’s online algorithm reaches about 75 percent of the optimal packing at a fraction of the solve time, which is what makes aggressive-yet-safe co-location practical.

Following a request through gShare

Serving a small model on a GPU shared with other tenants

A trace, drawn from the paper, of how gShare admits and runs a request under aggressive, SLO-aware sharing.

Step 1 of 5

A request arrives for a tenant's small model

A function invocation comes in for a model that needs only a fraction of a GPU. Rather than reserving a whole GPU, gShare treats the request as a candidate to run on a vGPU sharing a physical GPU with other tenants' functions.

Most served models are small enough to co-locate several per GPU.

Why it works

Fine slicing matches supply to demand. A whole GPU per small model wastes most of it → but virtualizing the GPU into elastic vGPUs sized to each model → so tenants reserve only what they need and the over-provisioning waste collapses.
Decoupling frees idle capacity to flow. Binding the GPU to one function’s VM strands its idle time → but managing the GPU as a separate, fungible resource → so many tenants share one physical GPU and no capacity is locked to an idle function.
Passthrough plus host-memory models avoids the proxy tax. A GPU proxy on the request path adds overhead and delay → but VM-passthrough virtualization with model checkpoint/restore from host memory → so functions run at near-native speed while still time-sharing the GPU.
The aggressive scheduler packs up to the SLO limit. A blind heuristic either under-fills or overloads the GPU → but optimizing GPU usage subject to each function’s latency SLO → so tenants are co-located as tightly as their deadlines allow, and no tighter.
An online solver makes the optimization practical. Solving the sharing problem exactly is too slow at scale → but an online algorithm reaching about 75 percent of optimal, 10 to 100 times faster → so near-optimal packing decisions are made fast enough to run live.

What it costs (trade-offs and scope)

Aggressive packing leaves less headroom for surprises. Filling GPUs up to the SLO limit means a burst or a mispredicted runtime can push co-located tenants past their deadlines; gShare meets more than 95 percent of targets, but the aggressive stance inherently trades some safety margin for utilization.
The online scheduler is approximate. It reaches about 75 percent of the optimal solution, not the optimum, in exchange for deciding fast enough to be usable; the exact constrained-optimization solver becomes infeasible as the cluster scales, so some packing efficiency is given up for speed.
Transparent switching relies on model checkpoint/restore. Sharing works by restoring model data onto the GPU on demand; the approach depends on that restore being fast enough, and the paper notes practical limits (for example, constraints of current GPU checkpoint/restore tooling around remapping memory addresses).
It targets small, bursty model-serving functions. The gains come from the fact that most served models are small and demand is spiky, leaving idle time to co-locate; workloads that saturate a whole GPU continuously offer little room to share and benefit less.
It assumes a VM-based multi-tenant FaaS platform. gShare’s vGPU abstraction and VM-passthrough virtualization are built for a microVM-based environment; platforms with a different isolation or GPU-access model would need adaptation.

The evaluation

Managing GPU functions in a multi-tenant serverless platform

Property	Whole-GPU per function	Proxy-and-swap sharing	gShare
GPU allocation granularity
GPU coupled to function VM
Sharing overhead
Scheduling
GPU usage and cost

A comparison of GPU function management in multi-tenant serverless platforms. By virtualizing each GPU into fine-grained elastic vGPUs, decoupling the GPU from the per-function VM, switching tenants transparently via model checkpoint/restore, and packing requests with an aggressive SLO-aware online scheduler, gShare reports a 43 to 63 percent reduction in GPU usage versus a keep-alive baseline while meeting more than 95 percent of latency targets, and a 24 to 58 percent cloud-cost reduction versus the state of the art with better performance. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ open-access copy hosted at Temple University, of the ASPLOS 2026 paper). The described system is the paper’s: gShare is a GPU function management policy for VM-based multi-tenant FaaS platforms that provides a fine-grained GPU virtualization solution and decouples the GPU from the CPU-oriented function management paradigm to enable flexible cross-tenant sharing. It provides an elastic vGPU abstraction over a microVM-based environment using VM-passthrough virtualization, and a user-transparent vGPU switching and sharing technique that keeps model data in host memory and uses checkpoint/restore to swap models onto the physical GPU on demand, so functions time-share the GPU. It formulates GPU sharing as a constrained optimization (minimize GPU usage subject to latency SLOs) and uses an online scheduling algorithm that the paper reports reaches about 75 percent of the optimal solution while deciding 10 to 100 times faster than an exact solver. The paper reports gShare reduces GPU usage by 43 to 63 percent versus a keep-alive baseline while meeting more than 95 percent of user latency targets, and reduces cloud cost by 24 to 58 percent versus the state-of-the-art method while maintaining better function performance (also described as outperforming the state of the art by 1.8 to 2.7 times in both performance and cost), saving a typical cluster hundreds of thousands of dollars a year in GPU procurement. Specific measurements, such as the up-to-85-percent over-provisioning and the 7-to-61-percent proxy overhead, are the paper’s analysis of existing platforms. The step-by-step walkthrough above is an illustration of the described mechanism. Exact algorithms, the vGPU switching implementation, and the full evaluation should be read from the paper.

Background

Why the cloud keeps working to make expensive shared resources actually shared

It helps to see gShare within a broad pattern: the economics of the cloud depend on packing many tenants onto expensive shared hardware, and every new, costly resource goes through the same maturation — first handed out in coarse, dedicated chunks, then gradually virtualized, sliced finely, and multiplexed as the waste becomes intolerable. CPUs and memory went through this long ago, which is why FaaS can hand out a twentieth of a core; GPUs are now going through it, lagging because they are harder to virtualize and share safely. The recurring challenge is that sharing a resource among tenants with strict performance targets is a scheduling problem with real tension — pack too loosely and the resource is wasted, too tightly and someone’s deadline breaks — so the answer is always some blend of a finer abstraction and a smarter, SLO-aware scheduler. gShare is a clean instance for GPUs in serverless. This effort to manage a shared resource efficiently across many tenants while honoring each one’s service-level agreement is the same terrain the work on efficient resource management for shared microservices with service-level guarantees (ERMS) covered elsewhere on this site occupies, squeezing utilization out of contended infrastructure without violating tenants’ targets. gShare’s contribution is to show that, with fine-grained vGPUs, transparent time-sharing, and an aggressive but SLO-aware scheduler, serverless GPUs can be packed far tighter than today’s coarse provisioning allows — turning idle, over-reserved accelerators into real savings for provider and tenant alike.

If you remember only three things

1. Serverless (FaaS) platforms increasingly serve ML models on GPUs, but they provision GPUs coarsely — a whole GPU or a gigabyte-granularity memory slice, bound to each function's CPU-oriented VM — even though most served models are small (two-thirds under one gigabyte). So small models over-reserve capacity (up to 85 percent waste), and the idle capacity, locked to one function's VM, cannot be reused. Existing sharing designs swap models onto the GPU through a central proxy, but the proxy adds overhead (7 to 61 percent performance fluctuation, about 40 percent more forwarding delay), swapping adds tens to hundreds of milliseconds of latency, and their behavior-blind scheduling either under-utilizes the GPU or violates latency SLOs.
2. gShare makes GPU sharing fine-grained and efficient for VM-based multi-tenant FaaS. It provides an elastic vGPU abstraction over a microVM environment with VM-passthrough virtualization (near-native speed), and decouples the GPU from the per-function VM so it becomes a fungible shared resource. A user-transparent vGPU switching technique keeps models in host memory and uses checkpoint/restore to swap them onto the GPU on demand, letting many tenants time-share one GPU without a heavyweight proxy. And an aggressive scheduler formulates GPU sharing as a constrained optimization — minimize GPU usage subject to each function's latency SLO — solved by an online algorithm that reaches about 75 percent of optimal while deciding 10 to 100 times faster than an exact solver, packing tenants as tightly as their deadlines allow.
3. The paper reports gShare cuts GPU usage by 43 to 63 percent versus a keep-alive baseline while meeting more than 95 percent of latency targets, and reduces cloud cost by 24 to 58 percent versus the state of the art with better performance, saving a typical cluster hundreds of thousands of dollars a year in GPU procurement. The trade-offs: aggressive packing leaves less headroom for bursts, the online scheduler is approximate (about 75 percent of optimal), transparent switching depends on fast model checkpoint/restore, and the gains assume small, bursty model-serving functions on a VM-based FaaS platform. The reported numbers reflect the paper's specific workloads and setup.

Check yourself

Check your understanding

1. Why do GPUs sit mostly idle in today's serverless (FaaS) platforms?
2. What were the problems with the existing approach of sharing a GPU by swapping models through a central proxy?
3. What does gShare's "aggressive" scheduling actually do, and how does it avoid missing latency targets?
4. What does gShare achieve, and how does it enable fine-grained sharing without a heavyweight proxy?

Sources

The paper is “gShare: Efficient GPU Sharing with Aggressive Scheduling in Multi-tenant FaaS platform” by Yanan Yang, Zhengxiong Jiang, Meiqi Zhu, Hongqiang Xu, Yujun Wang, Liang Li, Jiansong Zhang, and Jie Wu (China Telecom Cloud Computing / China Telecom Cloud Technology, and Temple University), published at ASPLOS 2026, DOI 10.1145/3779212.3790168. It is open access under CC BY, with an author copy hosted at Temple University.

This explainer is written from the paper’s full text (the Temple-hosted open-access copy). gShare is a GPU function management policy for VM-based multi-tenant FaaS platforms that provides fine-grained GPU virtualization and decouples the GPU from the CPU-oriented function management paradigm to enable flexible cross-tenant sharing. It offers an elastic vGPU abstraction over a microVM environment via VM-passthrough virtualization, and a user-transparent vGPU switching and sharing technique that keeps model data in host memory and uses checkpoint/restore to swap models onto the GPU on demand, so functions time-share the hardware. It formulates GPU sharing as a constrained optimization (minimize GPU usage subject to latency SLOs) solved by an online algorithm reaching about 75 percent of optimal while deciding 10 to 100 times faster than an exact solver. The paper reports gShare reduces GPU usage by 43 to 63 percent versus a keep-alive baseline while meeting more than 95 percent of latency targets, and reduces cloud cost by 24 to 58 percent versus the state of the art with better performance. Numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.