Dilu: Enabling GPU Resourcing-on-Demand for Serverless DL Serving via Introspective Elasticity

Cunchi Lv, Xiao Shi, Zhengyu Lei, Jinyue Huang, Wenting Tan, Xiaohui Zheng and 1 other

Serverless platforms run deep-learning models on GPUs, but they hand each function a whole GPU or a fixed slice of one, so as bursty workloads ebb the GPUs sit half-empty — fragmentation of 15 to 94 percent that quietly eats the savings serverless promised; this work from the Chinese Academy of Sciences builds Dilu, which adds a second scaling dimension beyond adding and removing instances — it continuously resizes the GPU compute slice each instance gets, packs complementary workloads onto the same card, and reacts to bursts in milliseconds — cutting fragmentation by 10 to 46 percent and raising inference throughput up to 1.8 times.

The argument in one glance

Problem

Serverless GPU serving wastes enormous fractions of every GPU

Serverless computing is attractive for deep learning because it auto-scales and bills only for what runs. But GPUs are expensive and lumpy, and serverless DL systems leave 15 to 94 percent of them idle. A function's real GPU demand rises and falls with bursty traffic, drops to nothing during the communication phases of distributed training, and lingers nearly unused while a container is kept alive between sparse requests — yet the GPU stays reserved the whole time.
What people did

Give each function a whole GPU, or a fixed static slice of one

The simplest scheme hands each function an entire GPU by pass-through, which is clean but wastes most of it. To do better, systems use the GPU's sharing mechanism to give each instance a fixed slice — a set percentage of the compute units — so several functions share a card. Either way the allocation is decided up front and held constant, sized generously to be safe.
Why it fell short

Static slices and instance-only scaling cannot follow real demand

A fixed slice, chosen high to guarantee quality, is over-provisioned most of the time and cannot be handed to a neighbor when idle, nor stretched past its limit to absorb a burst. And classic serverless scales in only one dimension: it adds or removes whole instances in response to load. That coarse, reactive, all-or-nothing scaling never reclaims the compute sitting idle inside an instance, and its slow, cold-start-heavy deployment of large models violates latency targets under bursts.
This paper's idea

Add a second scaling dimension: continuously resize each instance's GPU slice

Dilu introduces introspective elasticity — a two-dimensional co-scaling that pairs the usual horizontal scaling (adding and removing instances) with a new vertical one (continuously resizing the GPU compute slice an instance holds). It profiles each function's real minimum and cost-effective quotas, collocates complementary workloads to fill fragments, and reacts to bursts in milliseconds by shifting compute between collocated instances — so the slow horizontal scaling can be lazy. The result cuts fragmentation by 10 to 46 percent and lifts inference throughput up to 1.8 times.

Why this problem is worth solving

The waste comes from a mismatch between how GPUs are handed out and how deep-learning work actually uses them, so it is worth being precise about the resource.

A GPU is made of many streaming multiprocessors (its compute units) plus a fixed pool of memory, and a deep-learning function runs as a stream of CUDA kernels dispatched onto those units. The trouble is that a function almost never uses the whole GPU steadily: training alternates compute-heavy forward and backward passes with communication phases that use no compute at all; inference load swings with traffic and sits idle between requests; and large models may need lots of memory but only part of the compute, or vice versa. When a scheduler gives a function an entire GPU, or a fixed static slice sized for its peak, the parts it is not using at any instant — call them fragments — are stranded, reserved but idle. Across a cluster these fragments add up to most of the hardware. GPU sharing tries to help by letting several functions occupy one card, but if each still gets a fixed slice decided in advance, the fragments simply move around rather than disappear.

The reason existing systems cannot fix this is that they can only pull one lever.

Elasticity — growing and shrinking resources with demand — is what makes serverless efficient, but it has two possible dimensions. Horizontal scaling adds or removes whole function instances: when traffic rises you launch another copy, when it falls you kill one. This is what classic serverless does, and it is coarse and slow — each instance is a full deployment, and for a large deep-learning model, spinning one up incurs a heavy cold start. Vertical scaling instead resizes the resource a single instance holds: give it more or fewer GPU compute units without changing the number of instances. Horizontal scaling can eliminate fragments between instances (external fragments) but is blind to the compute sitting idle inside an instance (internal fragments); vertical scaling can reclaim that internal idle compute but cannot, by itself, add capacity for a sustained surge. Neither alone suffices. The compute a training worker frees during its communication phase, or an inference instance frees at low load, can only be reclaimed by resizing — and a burst can only be met without a cold start if that resizing is fast. The opportunity is to use both dimensions together.

What people did before, and where it broke

Exclusive whole-GPU allocation. Give each function its own GPU by hardware pass-through. Where it broke. Most of the GPU sits idle most of the time, the most wasteful option of all — utilization far below the reserved capacity.
Static slices via the GPU multiprocessing service. Let several instances share a card, each capped at a fixed percentage of compute units. Where it broke. The slice is decided up front and cannot shrink to give a neighbor idle compute or grow past its cap to absorb a burst, so fragments persist and sudden load is dropped.
Spatio-temporal sharing on top of static multiprocessing. Combine spatial slices with time-sharing to squeeze more onto a GPU. Where it broke. It still rests on the static sharing mechanism, so adjusting a partition takes significant time and cannot react quickly to serverless-scale fluctuations.
Priority time-sharing (opportunistic jobs). Let a high-priority job run first and grant a low-priority collocated job leftover time by trial. Where it broke. It protects the important job but nearly starves the collocated one, and its cautious, speculative sharing adapts poorly to highly hybrid, fluctuating deep-learning workloads.
Horizontal-only serverless autoscalers with keep-alive. Add and remove instances reactively, keeping some warm and pre-warming from workload prediction. Where it broke. It cannot touch internal fragments, and short-term unpredictability plus the slow deployment of big models still cause cold starts and latency violations; the keep-alive instances themselves waste GPU.
The gap. Every approach either allocated coarsely and statically, or scaled in only the horizontal dimension, so none could reclaim the compute idling inside an instance or resize fast enough to meet a burst without a cold start. Doing both — fine-grained on-demand resizing coordinated with instance scaling — is exactly what serverless GPU serving needs.

How Dilu works

Two-dimensional co-scaling for GPU resourcing-on-demand

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Give elasticity a second dimension: resize the slice, do not just count instances. Classic serverless can only add or remove whole instances, a coarse lever that leaves the compute idling inside each instance stranded. Dilu’s core move is to add vertical scaling alongside horizontal — continuously growing and shrinking the fraction of a GPU’s compute units an instance holds, in fine steps rather than whole-GPU jumps, and coordinating the two dimensions so that the fast, fine vertical adjustment handles moment-to-moment fluctuation while the slow, coarse horizontal one handles sustained shifts. GPU provisioning stops being a discrete integer count of cards and becomes a continuous quantity. Splitting a resource-adjustment problem into distinct coordinated dimensions and tuning a resource continuously instead of in coarse all-or-nothing steps is the same discipline the memory-resizing work covered elsewhere on this site follows, separating the decisions of what to resize and when so each can be made smoothly on its own terms.

Defragment by collocating workloads whose demands complement. A single deep-learning function rarely saturates both the compute and the memory of a GPU at the same instant — a trainer floods the compute units but pauses to communicate, an inference server needs memory but only bursts of compute — so a card devoted to one function is mostly fragments. Dilu’s scheduler treats placement as a packing problem: it collocates functions whose demands fill different parts of the same GPU, so the idle compute of one becomes the working compute of another, and it favors pairing instances with similar load patterns so a shared card is not repeatedly disrupted. Squeezing more useful work onto a shared resource by filling the capacity that would otherwise sit idle under bursty, uneven demand is the same instinct the serverless orchestration work covered elsewhere on this site acts on, reusing capacity already spun up rather than letting it idle while new demand waits.

Profile the real need cheaply, instead of padding the quota. The reason systems over-provision is that they guess a safe-but-high quota; the reason they do not measure the true need is that measuring every possible setting is slow. Dilu profiles each function’s actual requirement with a guided search — a binary search on the compute slice for training, and for inference a climb guided by a throughput-per-compute-unit efficacy metric that exploits the diminishing returns of adding more compute — pinning the minimum quota that still meets the quality target in a handful of trials rather than an exhaustive sweep. This turns each function’s previously hidden internal fragments into a known, reclaimable amount. Estimating a needed quantity efficiently with a principled metric, instead of brute-forcing every possibility, is the same economy the hardware-verification-coverage work covered elsewhere on this site practices, replacing exhaustive measurement with a guided estimate that reaches the answer far sooner.

Following a bursty inference function through Dilu

Serving an inference function under a traffic burst

Tracing one function shows profiling, collocation, and co-scaling together.

Step 1 of 5

Profile its two quotas

An inference function is deployed with a latency target. Profiling has already found its request slice — the minimum share of GPU compute units that meets the target at normal load — and a larger limit slice for bursts, along with the batch size that gives the best throughput per unit of compute.

The guided search finds these in a handful of trials, not an exhaustive sweep of every setting.

Why it works

A second scaling dimension reclaims the compute idle inside an instance. Classic serverless only adds or removes whole instances, so the compute units a function leaves idle stay wasted → but resizing an instance’s GPU slice up and down lets that idle compute be handed to a neighbor → so utilization rises without the coarse jumps of instance-level scaling alone.
Collocating complementary workloads fills the fragments. A single function rarely uses all of a GPU’s compute and memory at once, leaving parts idle → but pairing functions whose demands differ, a compute-bound trainer with a bursty inference, packs their needs into one card → so fewer GPUs sit half-empty and deployment density climbs.
Software token control gives fine-grained slices without driver support. GPU drivers are closed, so the system cannot partition the compute units directly → but intercepting each instance’s kernels and releasing them only up to granted tokens throttles compute in software at a few-millisecond granularity → so instances get continuous fractional slices that static sharing cannot.
Fast vertical scaling lets horizontal scaling be lazy. Reacting to a burst by launching an instance incurs a slow, cold-start-heavy deployment that violates latency targets → but instantly granting more tokens to the pressured instance handles the spike in milliseconds, buying time → so the expensive scale-out is delayed and often avoided entirely.
Cheap profiling finds the real need instead of over-provisioning. Guessing a safe-but-high quota wastes GPU, while exhaustively measuring every setting is slow → but a guided search — binary search for training, an efficacy-metric climb for inference — pins the minimum quota in a handful of trials → so allocation tracks real demand rather than a padded guess.

What it costs (trade-offs and scope)

The fine-grained control is indirect and leans on the vendor’s sharing tools. Because GPU drivers are closed, Dilu cannot partition compute natively; it intercepts CUDA kernels and throttles them with tokens, and profiling still uses the vendor’s thread-percentage sharing — a workaround rather than true hardware partitioning.
Collocation causes interference, and quality rests on tuned knobs. Sharing a GPU means instances contend; the oversubscription caps, token budget, burst-window thresholds, and limit-to-request ratio are all set empirically, and the sensitivity study shows too-tight or too-loose settings hurt.
Priority-based scaling favors latency-sensitive work over collocated jobs. The emergency scale-up throttles neighbors to protect a pressured inference instance, so collocated training does slow — a few percent typically, more under heavy contention.
The inference profiling assumes a well-behaved efficacy surface. The guided search assumes throughput-per-compute-unit behaves smoothly; the authors note some models may not conform, though the search stays safe because it only accepts settings meeting the quality target.
The kernel interception adds overhead and framework coupling. Intercepting every CUDA launch via library preloading and polling for tokens every few milliseconds carries a small cost, claimed under one percent, and ties the system to CUDA frameworks.
The largest-scale results are simulated. The cluster gains at thousand-node scale come from simulation; the real testbed is five nodes and twenty GPUs.
Isolation is only container-strong. Multiple tenants share a physical GPU with container isolation and restart-based fault tolerance; stronger GPU-sharing protection is left to future work.

The evaluation

GPU provisioning for serverless deep-learning serving

Property	Exclusive	MPS (static share)	Horizontal-only	Dilu
Allocation granularity			Fixed slice, scale instances
Scaling dimensions	Instance count only	None, static quota
Reclaims idle in-instance compute	No	No
GPU fragmentation	Highest, 15 to 94 percent		High	Cut 10 to 46 percent
Inference throughput and cold starts	Baseline	Modest gain		Up to 1.8 times, 91 percent fewer cold starts

GPU provisioning schemes for serverless deep-learning serving, evaluated on a five-node cluster of A100 GPUs (and simulated at thousand-node scale) across vision, NLP, and LLM models with Azure production traces. Dilu cuts GPU fragmentation by 10 to 46 percent, raises inference throughput up to 1.8 times and training up to 1.1 times, reduces SLO-violation rates by 11 to 71 percent and cold starts by up to 91 percent, and reaches 97 percent of exclusive-mode throughput while using half the GPUs.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

Dilu is a thorough, well-engineered system: it identifies that serverless GPU serving wastes 15 to 94 percent of its GPUs to coarse static allocation and one-dimensional scaling, and adds a genuine second dimension — continuously resizing each instance’s GPU compute slice, coordinated with instance scaling — plus complementary collocation and cheap profiling, cutting fragmentation by 10 to 46 percent and lifting inference throughput up to 1.8 times while nearly matching exclusive mode on half the GPUs. Read the gains with their conditions. The fine-grained control is indirect, built by intercepting CUDA kernels and throttling with tokens because GPU drivers are closed, and it still rests on the vendor’s sharing mechanism. Collocation means interference, and quality-of-service depends on several empirically tuned knobs; the priority mechanism protects latency-sensitive work by slowing collocated training. The wide benefit ranges are workload-dependent, the largest-scale numbers are simulated on a five-node real testbed, and isolation is only as strong as containers. The durable idea is the reframing of GPU elasticity itself: provisioning need not be a whole-instance, all-or-nothing decision — make it a continuous quantity you can resize in milliseconds, and most of the stranded capacity comes back.

Background

Why serverless keeps fighting the same enemy on GPUs

It helps to see Dilu as the latest round in a long fight. The promise of serverless is that you never think about servers: the platform scales your function to demand and bills only for what runs. But that promise collides with a stubborn reality — starting a function from nothing is slow, and keeping it ready costs resources that sit idle. On CPUs this shows up as the cold-start tax; on GPUs it is worse, because the hardware is far more expensive and lumpier, so both the cost of a cold start and the cost of keeping a GPU warm are magnified, and a badly-packed GPU wastes a resource that costs many times a CPU. Every serverless-GPU system is, in some form, trying to keep these costly accelerators busy without paying ruinous cold starts. The serverless function-warming work covered elsewhere on this site attacks the same enemy from the cold-start side, deciding when and where to keep functions ready; Dilu attacks it from the utilization side, reclaiming the compute a warm-but-idle instance is wasting. The broader lesson is that on scarce, expensive accelerators, the efficiency of a serverless platform is decided less by raw speed than by how finely it can share and reshape the hardware around fluctuating demand.

If you remember only three things

1. Serverless deep-learning serving on GPUs wastes 15 to 94 percent of its hardware because GPUs are allocated coarsely — a whole card or a fixed static slice per function — while real demand fluctuates with bursty traffic, drops to zero during the communication phases of distributed training, and idles under keep-alive; classic serverless makes this worse by scaling in only one dimension (adding and removing whole instances), which is blind to the compute sitting idle inside an instance and whose slow deployment of large models causes cold starts.
2. Dilu introduces introspective elasticity, a two-dimensional co-scaling that adds vertical scaling (continuously resizing the fraction of a GPU's compute units an instance holds) to the usual horizontal scaling, built from three parts: cheap guided profiling (binary search for training, a throughput-per-compute efficacy climb for inference) that finds each function's minimum request and cost-effective limit quotas; resource-complementary scheduling that collocates workloads whose demands fill different parts of a GPU to defragment it; and an adaptive co-scaler that, because GPU drivers are closed, throttles compute in software by intercepting CUDA kernels and issuing time tokens — fast-scaling a pressured instance up in milliseconds when its kernel-launch cycle lengthens, so the slow, cold-start-prone horizontal scale-out can be lazy.
3. On a five-node A100 cluster (and simulated at thousand-node scale) across vision, NLP, and LLM models with Azure traces, Dilu cuts GPU fragmentation by 10 to 46 percent, raises inference throughput up to 1.8 times and training up to 1.1 times, reduces SLO-violation rates by 11 to 71 percent and cold starts by up to 91 percent, and reaches 97 percent of exclusive-mode throughput on half the GPUs — but its control is an indirect kernel-interception workaround dependent on the vendor's sharing tool, collocation causes interference governed by empirically tuned knobs, it favors latency-sensitive work over collocated training, and the largest-scale results are simulated.

Check yourself

Check your understanding

1. Why do serverless GPU serving systems waste so much of their GPUs?
2. What is "introspective elasticity," Dilu's core idea?
3. How does Dilu resize a GPU slice in fine steps when GPU drivers are closed-source?
4. Which statement best reflects the limits of Dilu?

Sources

The paper is “Dilu: Enabling GPU Resourcing-on-Demand for Serverless DL Serving via Introspective Elasticity” by Cunchi Lv, Xiao Shi, Zhengyu Lei, Jinyue Huang, Wenting Tan, Xiaohui Zheng, and Xiaofang Zhao, published at ASPLOS 2025, DOI 10.1145/3669940.3707251. The authors are at the Institute of Computing Technology, Chinese Academy of Sciences (and affiliated institutes and Zhongguancun Lab); the system is open-sourced.

This explainer is written from the full 15-page text. Everything above comes from the paper’s own sections: the introduction and motivation establishing GPU fragmentation in serverless DL (15 to 94 percent) and its three sources (overprovisioning, idling during training communication, and keep-alive waste), and the insight of introspective elasticity as a two-dimensional co-scaling; the system design covering multi-factor profiling (binary search for training quotas and the Hybrid Growth Search Strategy with its throughput-efficacy metric for inference, up to 3.3 times faster than the prior method), resourcing-complementary scheduling as a heuristic two-dimensional bin-packing with affinity-first collocation, complementarity-based defragmentation, and oversubscription control, and the adaptive two-dimensional co-scaling with its token-based real-time CUDA kernel manager and interception library for fast vertical scaling (using kernel-launch-cycle time as the contention signal) plus lazy horizontal scaling with sliding-window thresholds; and the evaluation on a five-node cluster of A100 GPUs and a simulated thousand-node cluster, reporting 10 to 46 percent defragmentation, up to 1.8 times inference and 1.1 times training throughput, 11 to 71 percent SLO-violation-rate reduction, up to 91 percent cold-start reduction, 97 percent of exclusive throughput on half the GPUs, and 30 percent cost reduction over exclusive allocation at scale.