VELTAIR: towards high-performance multi-tenant deep learning services via adaptive compilation and scheduling

Serving many models on one CPU, the right schedule changes with load and the right machine code changes with neighbor noise — VELTAIR compiles each layer once into a small Pareto family from locality-greedy to parallelism-heavy, then picks code versions and layer-block sizes on the fly, landing within 10% of interference-free speed.

Why this problem is worth solving

Inference serving has an occupancy problem. Training saturates machines; serving arrives one query at a time, and a single small-batch inference cannot use 64 cores. The paper’s measurement: an AMD Threadripper 3990X serves around 300 queries per second on MLPerf vision models — one machine, many customers’ worth of capacity. So providers co-locate: multiple models, multiple tenants, one CPU. (This paper deliberately targets general-purpose hardware — the CPUs many companies actually serve on — where resource boundaries are softest.)

Co-location’s tax is

The paper’s framing insight is that two decisions govern this tax, and both are conventionally frozen. The schedule — at what granularity work claims cores — is fixed per system. And the code — the loop tiling, unrolling, and parallelization of every layer, produced by compilers like TVM (see the TensorIR and AStitch explainers elsewhere on this site) — is auto-tuned once, in isolation, on a quiet machine. Multi-tenancy invalidates the quiet machine. VELTAIR is, by its authors’ account, the first system to treat compilation itself as a serving-time knob.

What people did before, and where it broke

Multi-tenant DNN schedulers. PREMA (temporal multiplexing with preemption), AI-MT (overlapping compute- and memory-heavy layers), and Planaria (spatially splitting a systolic array) advanced accelerator sharing; PARTIES and similar QoS managers partition resources for conventional interactive services. All hold both granularity and code fixed. Some (AI-MT, Planaria) even split layers into sub-layers — which the analysis here shows is overly fine: layer-wise is already conflict-dominated.

The granularity dilemma, measured. Scheduling whole models (first-come-first-served) allocates each model’s peak requirement for its whole runtime; since per-layer needs are jagged, cores idle — a 47% gap versus the per-layer minimum at 75% load. Scheduling single layers tracks the jagged profile precisely but pays a conflict rate an order of magnitude higher (every layer boundary is a reallocation, and concurrent tenants’ reallocations collide) plus per-layer scheduling overhead. Fixed blocks of layers sit between — but the best block size moves with load: 6 layers wins at 150 QPS, 11 at 200. No static choice survives a bursty day.

The compilation blind spot. Auto-tuned kernels (TVM outperforms even Intel’s MKL-DNN library here) are searched on an idle machine. The search rewards big tiles that live in the L3 cache — maximum locality. Under a noisy neighbor, that bet inverts: the cache is contended, big-tile code stalls, and a differently-compiled version of the same layer — smaller tiles, more parallelism — wins. The optimal binary is a function of interference level, which is unknowable at compile time.

The idea

Interference tolerance is a compilable property. The two code versions that win at the extremes differ systematically: the quiet-machine winner uses large tiles (locality — data parked in the shared L3) and moderate parallelism; the noisy-machine winner uses small tiles and maximal parallelism (little cache footprint to lose, more cores to hide latency). These are not two tricks but two ends of the classic parallelism-locality trade-off, and the intermediate points form a

Conflict-prone layers should start blocks. The scheduling insight is about shape: a model’s per-layer core requirement is jagged, and conflicts concentrate at the spikes — layers demanding far more cores than the model’s average. The block-formation algorithm walks the layers once, splits at every spike (requirement > average + threshold), and flattens each block internally by letting the spike borrow schedule slack from its cheaper neighbors while the block still meets its share of the QoS budget. Blocks claim cores rarely (low conflict, like model-wise) but claim only what they need (low waste, like layer-wise).

Let spare capacity set the aggression. The threshold — how much a block may exceed the average before being split — is not tuned but computed: idle cores (total minus sum of active models’ averages) are distributed proportionally to each model’s appetite. Heavy load → few idle cores → threshold near zero → many small, frugal blocks; light load → generous thresholds → big blocks that sprint. The system’s granularity breathes with its load, which is exactly what the static block-size study said was needed.

one pass ≈ 20 min vs 5 passes ≈ 2 h
kept: ≤5 dominant versions/layer (3% of layers need 5)

Why it works

• Interference changes which loop structure is optimal → compiling one binary per layer bakes in a bet about neighbors → a small Pareto family per layer (locality → parallelism) lets the runtime re-bet every block, at code-selection cost only.
• The auto-scheduler already explores the parallelism-locality space → harvesting the frontier during one pass is nearly free → five interference-specialized versions cost one search, not five.
• Conflicts concentrate at core-requirement spikes → splitting blocks exactly at spikes and smoothing within them gives model-wise conflict rates at layer-wise efficiency → the false dilemma between waste and collisions dissolves.
• The right granularity depends on spare capacity → deriving the threshold from idle cores makes block size track load automatically → no operator-tuned constant to go stale.
• Both runtime algorithms are linear scans plus a linear proxy → under 0.1 ms per model → adaptivity never becomes its own interference source.

What it costs

• Code storage. Up to five binaries per layer — mitigated by keeping only dominant versions (most layers need 2–3; only 3% need five).
• A proxy, not a measurement. Interference level comes from a linear model over observable system state; mispredictions pick suboptimal versions (bounded by the versions’ own robustness).
• CPU-centric mechanics. The knobs (tiles vs threads, L3 behavior, core counts) are general-purpose-hardware concepts; porting to GPUs/accelerators with hard partitions changes the trade-off space.
• QoS model. Per-layer core requirements derive from profiled latencies against per-model QoS budgets; new models need that profiling pass.
• Latency-critical focus. Best-effort co-location is sketched as an extension, not evaluated.

The evaluation

Seven MLPerf-derived models spanning weight classes — light (EfficientNet, MobileNet-V2, Tiny-YOLOv2), medium (ResNet-50, GoogLeNet), heavy (SSD, BERT-Large) — arrive as Poisson streams on a 64-core AMD Threadripper 3990X. Baselines: PREMA-style temporal multiplexing, Planaria-style layer-wise spatial scheduling (adapted to CPU), plus VELTAIR ablations (adaptive scheduling only, adaptive compilation only, full).