Tally: Non-Intrusive Performance Isolation for Concurrent Deep Learning Workloads

GPUs in deep-learning clusters sit mostly idle — one study found median utilization of just 4 percent — because a latency-critical inference job cannot fill a whole GPU, yet packing a background training job onto the same GPU to soak up the idle time wrecks the inference job's response time, since a GPU cannot interrupt a running kernel; this paper (Toronto, Stanford, Vector) builds Tally, a transparent layer that rewrites each background kernel's low-level code so it can be sliced up or paused between thread blocks, letting it yield the GPU in microseconds when a high-priority kernel arrives — cutting the latency hit from around 189 percent to 7 percent while keeping most of the throughput.

The argument in one glance

Problem

GPUs sit idle, but sharing them to fill the gaps wrecks the important job's latency

GPUs are the expensive heart of deep-learning clusters, yet they are badly underused — one production study found median GPU utilization of only about 4 percent. The reason is that many workloads cannot fill a GPU on their own: a latency-critical inference service only does work when requests arrive, and request rates are bursty, so the GPU is idle much of the time. The obvious fix is GPU sharing: run a best-effort background job, such as a training run, on the same GPU to soak up the idle time. Studies suggest this could cut the GPUs a cluster needs by half. But there is a catch. The inference service has a strict latency target measured in milliseconds, and when a background kernel is occupying the GPU and an inference kernel arrives, the inference kernel has to wait — and modern GPUs can only switch between tasks at the boundary of a whole kernel, with no way to interrupt one mid-flight. Many training kernels run for a long time, so that wait can blow the latency target badly, in practice slowing tail latency by up to twenty times.
What people did

Use existing GPU sharing mechanisms

There are established ways to share a GPU. Time-slicing rotates whole kernels through the GPU. NVIDIA's Multi-Process Service, and its priority variant, let multiple processes' kernels run at once. And research systems co-locate training and inference, some inserting hooks into the deep-learning framework to control scheduling. All of them aim to reclaim the idle GPU time that a single latency-critical job leaves on the table.
Why it fell short

They either ruin isolation, or demand intrusive changes and special workloads

These mechanisms fail the two tests production needs. First, isolation: because they still switch at the granularity of entire kernels, a long-running best-effort kernel holds the GPU while the high-priority kernel waits, so tail-latency overheads run to roughly 190 to 350 percent across the state-of-the-art systems — far outside a millisecond service-level agreement. Second, deployability: many of the systems that do better are intrusive, requiring modifications to user code or to the deep-learning framework itself, which imposes heavy integration and maintenance costs and limits flexibility. Worse, some demand that workloads have special properties — for example that every kernel be idempotent, safe to re-run — which real, heterogeneous production workloads do not satisfy. So operators are left choosing between sharing that breaks their latency targets and sharing that they cannot practically deploy.
This paper's idea

Transparently rewrite kernels so they can be sliced or paused between thread blocks

Tally is a transparent layer between the application and the GPU that needs no changes to user code or frameworks. Its key move exploits a fact about the GPU programming model: a kernel is a collection of thread blocks that are guaranteed to run independently. So Tally rewrites each best-effort kernel's low-level device code, through automatic transformation passes, to add two scheduling primitives at the thread-block level. Slicing splits a kernel into smaller sub-kernels of a few blocks each; preemption turns a kernel into a small pool of worker blocks that pull work from a counter and can stop and later resume. Either way, a background kernel can yield the GPU within microseconds when a high-priority kernel arrives. A priority-aware scheduler, guided by on-the-fly profiling, picks the primitive and settings that protect the latency target while keeping throughput high.

Why this problem is worth solving

The waste is enormous and the fix is obvious, which is exactly why the obstacle to the fix matters so much.

Deep-learning clusters are built from fleets of GPUs costing enormous sums — a single hyperscale AI cluster can run to hundreds of thousands of GPUs and hundreds of millions of dollars — yet those GPUs are routinely underutilized, with one production study reporting median utilization around 4 percent. The cause is structural: training jobs stall on CPU-side data loading, and inference jobs only compute when requests arrive, with request rates that spike far above their average, so a GPU provisioned for the peak is idle most of the time. GPU sharing attacks this by co-locating multiple low-utilization jobs on one GPU, and simulations on production traces suggest it could cut the GPUs a cluster needs by around half, more at peak. That is a large enough prize — in money and in energy — that the reason it is not already standard practice is worth understanding. The reason is that naive sharing destroys the very thing latency-critical jobs are guaranteed, and doing sharing without that destruction, and without rewriting everyone’s code, is the open problem Tally takes on.

The heart of the difficulty is that GPUs give you no way to reclaim the chip quickly once a kernel is running.

When a high-priority kernel arrives and the GPU is busy with a best-effort kernel, the question that decides everything is how fast the GPU can be freed for the high-priority work — the paper calls this the turnaround latency. Modern GPUs offer no primitive to terminate a running kernel; scheduling switches happen only at the boundary between whole kernels. So the turnaround latency is essentially the remaining runtime of whatever best-effort kernel is executing, and many deep-learning training kernels run for a long time. Measured on a real model, scheduling at the granularity of a whole training iteration gives turnaround latencies of seconds, and even switching at the granularity of individual kernels gives around ten milliseconds — while a latency-critical service needs the GPU back in well under a millisecond. That gap is why kernel-granularity sharing inflates tail latency so badly. To hit a millisecond target, the turnaround latency has to drop into the microseconds, which means finding a way to interrupt or subdivide a kernel that the GPU itself does not natively provide, and doing so without asking the workload to be written any differently.

What people did before, and where it broke

Time-slicing whole kernels. Rotate the GPU between processes at kernel granularity. Where it broke. A long best-effort kernel holds the GPU until it finishes, so a waiting high-priority kernel suffers huge turnaround latency; tail-latency overhead reaches into the hundreds of percent.
Multi-Process Service and its priority variant. Let kernels from multiple processes run concurrently on the GPU. Where it broke. Concurrent execution still cannot evict a running best-effort kernel, so co-execution interference remains and inflates high-priority tail latency by up to twenty times.
Framework-integrated co-location systems. Insert scheduling hooks into the deep-learning framework to interleave training and inference. Where it broke. They are intrusive — they require modifying user code or the framework, imposing integration and maintenance costs and limiting flexibility.
Thread-level or idempotence-based schemes. Get finer control by relying on properties of the kernels, such as requiring every kernel to be idempotent so it can be safely restarted. Where it broke. Real production clusters host heterogeneous workloads that do not meet such constraints, so these mechanisms are not generally applicable.
The gap. No system delivered all three of what production needs at once: non-intrusive deployment, robust performance isolation for latency-critical tasks, and generality across diverse workloads. Getting microsecond-scale turnaround by transparently transforming kernels at the thread-block level is what Tally contributes.

How Tally works

A transparent layer that makes any kernel yield the GPU in microseconds

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Add the scheduling control the GPU lacks by rewriting the kernel, not the application. Tally’s framing is that the missing capability — interrupting or subdividing a running kernel — can be injected transparently, below the level the user ever sees. It is a virtualization layer that intercepts the application’s GPU calls with library interposition, so no user code and no framework changes are required; the workload runs as if talking directly to the GPU. Because GPU programs register their device code with the driver at startup, Tally’s server can capture that low-level code and transform it before it runs, then recompile. Slipping between the application and the hardware to change how low-level code behaves, without the application knowing, is the same transparent-rewriting strategy the work on dynamic binary translation covered elsewhere on this site is built on, intercepting and rewriting a program’s own instructions to give them behavior the original code never had.

Exploit thread-block independence to slice or pause at a granularity the hardware never offered. The technical key is a property of the GPU programming model: a kernel is a grid of thread blocks that are guaranteed to run independently, in any order. That independence is what makes block-level rescheduling safe. Slicing uses it to break a kernel into small sub-kernels, fixing up the block indices with an offset so the pieces together do exactly the original work. Preemption uses it to recast the kernel as a small pool of persistent worker blocks that fetch block indices from a counter, so the kernel’s progress is captured in that counter and it can halt at a block boundary and resume — with a unified-synchronization pass added to keep early-returning threads from diverging and hanging the GPU. Rewriting a program’s low-level instructions through automatic passes to insert new control behavior, while provably preserving what the code computes, is the same discipline the work on binary translation for weak memory architectures covered elsewhere on this site applies, transforming instruction streams under a correctness obligation rather than trusting the original form.

Turn the isolation-versus-throughput dial automatically, per kernel, with profiling. Slicing and preemption each have costs — extra launches for one, synchronization overhead for the other — and which is better, and with what parameters, varies kernel by kernel. So Tally does not fix a policy; its scheduler dispatches high-priority kernels immediately and preempts best-effort ones, then a transparent profiler measures each best-effort kernel under candidate configurations, estimates its turnaround latency, and selects the configuration that keeps that latency under a threshold while maximizing best-effort throughput. The threshold is the knob that trades the inference job’s isolation against the training job’s speed, and driving it to microsecond turnaround is what protects the millisecond target. Scheduling work at microsecond granularity so that a latency-critical task is served almost immediately without starving background work is the same balance the work on microsecond-scale blind scheduling covered elsewhere on this site strikes, reclaiming idle time at a grain fine enough that the urgent task barely waits.

Following a request through Tally

Protecting an inference kernel co-located with a long training kernel

A trace, following the paper's setup, of how Tally keeps a latency-critical inference kernel fast while a best-effort training job fills the idle GPU.

Step 1 of 5

Training fills the idle GPU, but its kernels are long

An inference service leaves the GPU idle between bursts of requests, so Tally runs a best-effort training job to use that time. The training kernels, however, run for a long time — at whole-kernel granularity the GPU would take milliseconds to seconds to free.

The inference service needs the GPU back in well under a millisecond to meet its latency target.

Why it works

Thread-block independence makes block-level rescheduling correct. A kernel’s blocks are guaranteed to run in any order without interacting → but that means the blocks can be regrouped into sub-kernels or pulled from a counter by worker blocks without changing the result → so Tally can slice and preempt any kernel while preserving its exact computation.
Progress captured in a counter makes preemption cheap and resumable. Normally there is no way to save and restore a running kernel → but recasting it as persistent workers pulling from a global block index puts all progress in that index → so the kernel can stop at a block boundary and later resume from exactly where it left off, losing only the current block.
Block-boundary yielding turns milliseconds of turnaround into microseconds. Whole-kernel switching leaves the high-priority kernel waiting out a long best-effort kernel → but stopping between blocks, or between small slices, frees the GPU almost immediately → so turnaround drops from milliseconds or seconds to tens of microseconds, small against a millisecond target.
Transforming device code keeps it non-intrusive and general. Getting fine control by editing frameworks or requiring idempotent kernels excludes most real workloads → but rewriting the intercepted low-level code with automatic passes needs no user changes and assumes only block independence → so it applies across heterogeneous production workloads.
Per-kernel profiling picks the right trade-off instead of guessing. Slicing and preemption have different costs that vary by kernel → but profiling each best-effort kernel across configurations and estimating its turnaround latency reveals the best choice → so the scheduler holds isolation under the threshold while maximizing best-effort throughput, rather than committing to one policy everywhere.

What it costs (trade-offs and scope)

It gives up some best-effort throughput for isolation. Tally deliberately keeps turnaround latency low, which constrains how aggressively background kernels run, so it attains around 80 percent of the throughput of the most aggressive sharing system; the strong isolation is not free, it is paid for in background speed.
Scheduling granularity is a thread block, not a thread. Preemption and slicing act at block boundaries, so a single very long-running individual block cannot be interrupted mid-flight; the authors note thread-level scheduling would isolate more strongly but find block-level sufficient in practice, leaving a residual worst case.
It depends on profiling and a tuned threshold. Choosing per-kernel configurations requires runtime profiling (amortized by caching per work configuration), the turnaround estimate for preemption is a heuristic approximation, and the turnaround-latency threshold is a configurable default set empirically — a knob that must be chosen to balance the two workloads.
It relies on intercepting and recompiling device code. Tally captures the low-level PTX kernels registered with the driver and transforms and recompiles them; this transparent path works broadly but assumes the device code is available to intercept and transformable, and adds startup and forwarding machinery, mitigated to near-native but not eliminated.
The priority model is specific. Tally targets one high-priority task co-located with best-effort tasks; it enforces that priority structure rather than arbitrary fairness among equal-priority tenants, and the evaluation centers on latency-critical inference alongside best-effort training.

The evaluation

Co-locating latency-critical inference with best-effort training

Property	Time-slicing / MPS	Framework-intrusive sharing	Tally
Switching granularity
Tail-latency overhead
Intrusiveness
Workload requirements
Best-effort throughput		Varies

From the paper's evaluation on six training and six inference workloads, with inference traffic driven by the Microsoft Azure Function Trace 2021. Tally incurs only 7.2 percent average overhead on the 99th-percentile latency of high-priority inference co-located with best-effort training, versus 252.3, 345.0, 195.5, and 188.9 percent for Time-Slicing, MPS, MPS-Priority, and TGS respectively, while retaining over 80 percent of TGS's system throughput.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The contribution is a practical answer to a real production bind: how to reclaim wasted GPU time by co-locating jobs without breaking the latency guarantees that make GPU sharing unacceptable today, and without asking anyone to rewrite their code. Tally’s insight is that the fine-grained scheduling control GPUs lack can be injected by transparently rewriting each kernel’s low-level code, exploiting the guaranteed independence of thread blocks to slice a kernel into sub-kernels or turn it into resumable worker blocks, with a unified-synchronization pass making preemption safe on any kernel and a profiling-guided scheduler choosing the right configuration per kernel. The result is strong: tail-latency overhead of 7.2 percent versus roughly 190 percent for the state of the art, at over 80 percent of the throughput. Read it within scope. Isolation is bought with about 20 percent of best-effort throughput; scheduling is at block, not thread, granularity, so a single long block is a residual worst case; it depends on runtime profiling, a heuristic turnaround estimate, and a tuned threshold; and it assumes device code that can be intercepted and transformed. Within those bounds it is a clean demonstration that transparent kernel transformation can give production GPU sharing the isolation it has been missing.

Background

Why GPU sharing keeps coming back

It helps to see why this problem is perennial. GPUs are the most expensive, most contended resource in a modern machine-learning cluster, and the economics of keeping them busy are relentless — every idle GPU-hour is money and energy spent for nothing. Yet the workloads that run on them are exactly the kind that leave gaps: inference services that only compute when requests arrive and must answer fast, and training jobs that stall waiting on data. So the field keeps returning to the same idea, packing more than one job onto a GPU, and keeps hitting the same wall, that co-located jobs interfere and the latency-sensitive one suffers. The enduring challenge is serving the urgent job with the quality it demands while a background job harvests the leftover capacity, and doing so without heroic per-application engineering. That balancing act, protecting the service-level agreement of a latency-critical task while sharing the accelerator with less urgent work, is the same goal the work on quality-of-service-aware, resource-efficient GPU serving covered elsewhere on this site pursues, squeezing utilization out of a GPU without letting the important requests slip. Tally advances it by making the sharing mechanism itself transparent and general, so the isolation no longer costs a rewrite.

If you remember only three things

1. Deep-learning clusters waste GPUs — one study found median utilization around 4 percent — because latency-critical inference jobs cannot keep a GPU busy, but co-locating a best-effort training job to use the idle time wrecks the inference job's tail latency, since a GPU can only switch tasks at whole-kernel boundaries and many training kernels run long, so an arriving high-priority kernel waits out a long one; existing sharing systems either accept this (tail-latency overhead of roughly 190 to 350 percent) or fix it only by intrusively modifying user code and frameworks and requiring special workload properties like idempotent kernels.
2. Tally is a transparent client-server virtualization layer, needing no user-code or framework changes, that injects the fine-grained scheduling control GPUs lack by rewriting each best-effort kernel's low-level device code — exploiting that a kernel's thread blocks are guaranteed to run independently — with two block-level primitives: slicing splits a kernel into small sub-kernels (fixing block indices with an offset), and preemption recasts it as a pool of persistent worker blocks pulling from a global counter so it can stop at a block boundary and resume, made safe on any kernel by a unified-synchronization pass that stops early-returning threads from diverging and hanging the GPU.
3. A priority-aware scheduler dispatches high-priority kernels immediately and preempts best-effort ones, while a transparent profiler measures each best-effort kernel across slicing and preemption configurations and picks the one that keeps turnaround latency under a threshold while maximizing throughput; on six training and six inference workloads with Azure-trace inference traffic, Tally holds 99th-percentile inference latency overhead to 7.2 percent (versus 188.9 percent for the state-of-the-art TGS) while retaining over 80 percent of its throughput — trading roughly 20 percent of best-effort throughput for isolation, scheduling at block rather than thread granularity, and depending on profiling and a tuned threshold.

Check yourself

Check your understanding

1. Why does naively co-locating a best-effort training job with a latency-critical inference job hurt the inference job so much?
2. What is Tally's key mechanism, and why is it non-intrusive?
3. Why does Tally need a "unified synchronization" transformation, and what does it do?
4. What did Tally achieve, and what does it trade off?

Sources

The paper is “Tally: Non-Intrusive Performance Isolation for Concurrent Deep Learning Workloads” by Wei Zhao, Anand Jayarajan, and Gennady Pekhimenko (University of Toronto, Stanford University, Vector Institute, and CentML), published at ASPLOS 2025 (Volume 1), DOI 10.1145/3669940.3707282.

This explainer is written from the full text of the authors’ arXiv version (arXiv:2410.07381). Everything above comes from the paper’s own sections: the introduction and background on GPU underutilization in production deep-learning clusters (median utilization around 4.2 percent, bursty inference request rates spiking far above average, hyperscale clusters costing on the order of half a billion dollars) and the promise of GPU sharing (simulated reductions of about 50 percent in required GPUs); the three criteria a production sharing system must meet — non-intrusiveness, performance isolation, and generality — and why existing systems (time-slicing, Multi-Process Service and its priority variant, and framework-intrusive or idempotence-requiring co-location systems) fail them, with kernel-granularity switching causing tail-latency slowdowns up to twenty times; the concept of turnaround latency and the measurement that whole-iteration and kernel-level scheduling give seconds and around ten milliseconds respectively, versus the sub-millisecond requirement; the design of Tally as a transparent client-server virtualization layer using library interposition, with a kernel transformer, transparent profiler, and priority-aware scheduler; the kernel transformations — slicing (partitioning thread blocks into sub-kernels with a block-offset substitution for blockIdx), preemption (based on the persistent thread block paradigm, with worker blocks pulling block indices from a global counter, a preemption flag, and resumability), and the unified synchronization pass that redirects all synchronization and return instructions to a single point to prevent divergent-synchronization stalls; the priority-aware scheduling algorithm with profile-guided resource provisioning that estimates turnaround latency for candidate slicing and preemption configurations (with the preemption turnaround approximated as kernel latency times worker blocks divided by total blocks) and selects configurations under a configurable turnaround-latency threshold (default about 0.0316 milliseconds); the non-intrusive virtualization implementation using library interposition, server-side interception and recompilation of registered device code, shared-memory client-server channels, and client-side caching of frequent trivial API calls; and the evaluation on six training and six inference workloads with inference traffic from the Microsoft Azure Function Trace 2021, reporting Tally’s 7.2 percent average overhead on 99th-percentile inference latency versus 252.3, 345.0, 195.5, and 188.9 percent for Time-Slicing, MPS, MPS-Priority, and TGS, while retaining over 80 percent of TGS’s system throughput.