INFless: a native serverless system for low-latency, high-throughput inference

A serverless platform rebuilt around machine-learning inference — it groups requests into batches inside the platform, runs them on mixed CPU and GPU hardware, and sizes every instance individually, so the same cluster answers two to five times more requests per second without breaking latency promises.

Why this problem is worth solving

Machine-learning inference means using an already-trained model to answer a request — classify an image, detect fraud, match a question to an answer. It is different from training, which builds the model in the first place. Inference runs inside interactive websites, so it inherits their deadlines.

The paper grounds this in one production example: China’s largest local life service website, which runs more than 600 inference models in its core business. The models screen advertisements, answer customer questions, and detect fraud, serving millions of requests every minute — about 1.9 billion requests per day. For 90% of these models, the answer must arrive within 200 milliseconds, and for some within 50.

A service level objective, written SLO from here on, is such a promised deadline. “This function answers within 200 milliseconds” is an SLO.

Hosting these models on rented virtual machines works, but the owner must size the cluster, keep it running through the night, and pay for every idle hour. Serverless computing promises to remove exactly that burden: the developer uploads a function — a small program that handles one request — and the platform finds machines, starts copies, scales them with the load, and bills only for use. A running copy of a function, with its own resources, is called an instance.

So serverless looks like the natural home for inference. The paper measures what actually happens when you try, on Amazon Web Services Lambda, with eleven models from the MLPerf benchmark suite and from production services.

What people did before, and where it broke

Attempt one: run models on a general serverless platform. Four measured problems appeared.

1. No accelerators. Inference is heavy arithmetic, and large models need a graphics processing unit (GPU) — a chip built for exactly this kind of parallel arithmetic. Commercial serverless platforms offer only ordinary processors (CPUs). The paper measures models such as Bert-v1, ResNet-50 and VGGNet taking longer than 200 milliseconds per request even at the platform’s maximum memory setting. No configuration meets the SLO.
2. Resources come in the wrong shape. These platforms give a function CPU power in proportion to its configured memory. Inference needs much computation and little memory, so developers must buy memory they do not use to obtain the processor share they do. Serving the SSD object-detection model within 200 milliseconds required an allocation of 1,792 megabytes while the model actually used 427 — more than four times the real need. More than half of all allocated function memory was over-provisioned.
3. One request, one instance. The platforms map each incoming request to its own instance. Inference could share: a GPU multiplies matrices for eight inputs almost as fast as for one. The paper shows that grouping requests four at a time cut the number of function launches by 72% and the number of instances created by 35%.
4. No way to state the deadline. No commercial platform lets the developer declare a latency SLO, so the platform cannot schedule toward it.

Attempt two: add a batching layer on top. A batch is a group of requests executed together in one run of the model. Systems such as BATCH and MArk kept the commercial platform underneath and added their own buffer service in front of it, aggregating requests into batches before submission. The paper calls this the on-top-of-platform design, OTP from here on.

OTP batching helps — about 30% more throughput than plain Lambda — but it breaks in three ways. The buffer layer runs on its own dedicated server that someone must pay for and maintain. It cannot see the queuing and scheduling delays inside the platform, so it cannot promise an end-to-end deadline. And it cannot touch resource allocation: when the efficient batch size or the right hardware changes with the load, an outside layer has no lever to pull. Underneath, the platform still scales uniformly — every instance of a function identical — so it cannot exploit the leftover slivers of capacity on partly used servers.

The gap, then, is not one missing feature. It is that batching, resource allocation, scheduling, and cold-start management need to be decided together, and only the platform itself stands in a position to decide them.

The idea, one mechanism at a time

INFless is built from four mechanisms. Each one answers a specific failure from the motivation.

The latency budget

Every request’s total latency has three parts, and INFless manages each:

latency = cold-start time + batch waiting time + execution time

A cold start is the delay of preparing a fresh instance — pulling the runtime, loading the model into memory — before the first request can run. For inference it often exceeds the work itself, because models and their serving libraries are large. The last mechanism below exists to make this term zero for most requests. The middle term is the time a request sits in a batch queue waiting for company. The last term is the model actually computing.

Built-in, non-uniform batching

Batching moves inside the platform, and it stops being one-size-fits-all.

Every instance owns a private batch queue. An arriving request waits in the queue until the batch is full or a timeout fires; the batch then executes as one run. Because the platform itself manages the queue, it knows the waiting time and can keep the total within the SLO.

Non-uniform means each instance gets its own batch size and its own resource slice — even two instances of the same function can differ. A big instance with batch size 8 and a generous GPU slice can absorb a busy stream. A small instance with batch size 1 and a slice of CPU can mop up a trickle — or fit exactly into the leftover capacity of an almost-full server.

The safe load for an instance follows from arithmetic. If a batch of size b executes in time t_exec and the deadline is t_slo, the instance can accept at most b / t_exec requests per second — faster than that and batches pile up. It must also receive at least b / (t_slo − t_exec) requests per second — slower than that and a batch cannot fill before the deadline forces it out. INFless keeps each instance’s arrival rate inside this range, and requires t_exec to be at most half the SLO so the range is never empty. For a 200-millisecond SLO and a 50-millisecond batch of four, the range is 28 to 80 requests per second.

Combined operator profiling

The scheduler above needs t_exec for any candidate configuration — every batch size, every CPU and GPU slice. Measuring every model under every configuration would be far too slow for a site that deploys hundreds of models.

The escape is a structural fact about neural networks. A model is a graph of operators — primitive computations such as matrix multiplication (MatMul) or two-dimensional convolution (Conv2D). Across the paper’s eleven models there are more than 1,000 operator calls but only 71 distinct operators. And a handful dominate: matrix multiplications take 76% of the execution time of the LSTM-2365 text model, and Conv2D alone takes more than 95% of ResNet-50’s.

So INFless profiles operators, once, into a database — execution time as a function of input size, batch size, CPU share and GPU share. To predict a whole model, it parses the model’s graph and combines the entries: operators in a chain add their times; parallel branches take the maximum. The paper calls this combined operator profiling. Predictions land within 10% of measured latency on average (8.6% error for ResNet-50, 7.8% for MobileNet, 9.74% for LSTM-2365), and INFless pads every prediction by 10% so that an underestimate does not become a missed deadline.

Scheduling as bin packing, solved greedily

Now the platform must choose, for each function: how many instances, each with what batch size, what CPU share, what GPU share, on which server. The paper writes this as an optimization problem — minimize the total hardware used, subject to every SLO holding and no server overflowing — and notes it is at least as hard as bin packing, a classic problem with no known fast exact solution.

INFless uses a greedy algorithm instead. It tries the largest batch size first, since batching contributes the most throughput. For each candidate batch size it asks the profiler which resource slices would meet the SLO. Among the feasible choices it picks the instance-and-server pair with the best resource efficiency: requests served per unit of hardware, divided by the fragmentation it would leave behind on the server. High score means “serves a lot and wastes little of the server it lands on.” Scheduling one instance takes about half a millisecond; even 10,000 concurrent requests schedule in under one second.

Cold starts: the long-short term histogram

The best existing keep-alive policy — the hybrid histogram policy from Microsoft’s serverless study — watches one window of a function’s idle times and derives two numbers: how long to wait before pre-warming an instance (loading it in anticipation), and how long to keep an idle instance alive before evicting it.

Production inference traffic defeats a single window, because it has two personalities at once. Over days it is periodic — a daily rhythm of user activity. Over minutes it is bursty — sudden jumps and drops. A long observation window smooths away the bursts and wastes resources when load suddenly falls. A short window never sees the daily rhythm and causes cold starts every morning.

INFless’s long-short term histogram policy simply keeps both: one histogram of idle times over the last hour, one over the last day. Each histogram yields a pre-warm window (from its 5th percentile) and a keep-alive window (from its 99th percentile), and the final windows are a weighted average of the two, half-and-half by default. Compared with the hybrid histogram policy, this cuts the cold-start rate by 21.9% — about one fifth of the remaining cold starts — while also cutting idle resource waste by 24.3%.

function: resnet-50
latency_target: 200ms   <- the SLO, declared

Why it works

Because batching lives inside the platform, the platform sees and controls all three parts of latency — cold start, waiting, execution — so it can promise their sum. → Because every instance carries its own batch size and resource slice, the scheduler can build big instances where traffic is heavy and small ones that fit into leftover server capacity, so less hardware sits idle. → Because operator profiles predict execution time within about 10% without running the model, the scheduler can search many configurations cheaply and still trust that its choices meet the deadline. → Because the two-horizon histogram sees both the daily rhythm and the bursts, instances are warm when the wave arrives and released when it recedes, so cold starts and idle waste fall at the same time. → Therefore the same cluster serves several times more requests per second, while missing fewer deadlines than platforms that serve less.

The through-line is co-design: each mechanism feeds the next. The profiler is only useful because the scheduler has freedom to vary configurations; the scheduler can only promise deadlines because batching is internal; internal batching only stays warm because the histogram plans ahead.

What it costs

• The developer must know the deadline. The whole machine keys off a declared SLO. A function without a meaningful latency target gains little.
• Execution must fit in half the deadline. The batching arithmetic requires execution time at most half the SLO. A model that barely fits its deadline cannot be batched at all and falls back to batch size 1.
• Prediction padding is paid waste. Every latency estimate is inflated by 10% to absorb profiling error. That headroom is real capacity reserved and not used.
• History cannot predict a first. If a spike exceeds anything in the recorded histograms, cold starts happen anyway. The paper points to faster-startup techniques from other work as the remedy, not its own policy.
• The framework is not universal. The prototype serves models compiled for TensorFlow (with converters for Keras and PyTorch), shares GPUs using a specific NVIDIA partitioning technology, and modifies one platform, OpenFaaS. Porting the ideas elsewhere is engineering left to the reader.

The evaluation

The testbed is an 8-machine cluster with 16 NVIDIA RTX 2080Ti GPUs, connected by 10-gigabit Ethernet. Workloads are two real applications from the production website: a vehicle-trading service (SSD, MobileNet, ResNet-50; SLO 200 milliseconds) and a question-answering robot (TextCNN-69, LSTM-2365, DSSM-2389; SLO 50 milliseconds). Request arrivals replay Microsoft Azure Functions’ published 7-day production trace in its three patterns — sporadic, periodic and bursty. A simulator running INFless’s real scheduling code scales the evaluation to 2,000 servers and 10,000 instances.

Two baselines: OpenFaaS+, the same underlying platform with GPU support added but no batching and a fixed 300-second keep-alive; and BATCH, the state-of-the-art on-top-of-platform batching system, re-implemented on OpenFaaS and extended with the same GPU support, so the comparison isolates design rather than hardware access.

An ablation confirms where the gains come from. Disabling built-in batching costs the most throughput — a drop of 45.6% on the vehicle-trading workload and 60% on question answering. Disabling operator-profile prediction (by padding estimates 100% instead of 10%) costs about 35%. Disabling fragment-aware scheduling costs 21.9% on the vehicle workload but only 7% on question answering, whose small models barely fragment servers in the first place. In the large-scale simulation, INFless’s placement leaves about 15% of server capacity stranded as unusable fragments, far less than the baselines.