NotebookOS: A Replicated Notebook Platform for Interactive Training with On-Demand GPUs

When you train a model interactively in a Jupyter or Colab notebook, the platform reserves a GPU for your whole session even though you only run short training cells now and then — so the expensive GPU sits idle more than 81 percent of the time; this paper (George Mason University and Adobe — Benjamin Carver, Yue Cheng, and colleagues) presents NotebookOS, which replicates each notebook kernel across several servers, hands a GPU to a replica only while a cell is actually running, and oversubscribes the servers, saving over 1,187 GPU hours in 17.5 hours of real workloads while keeping the experience snappy.

The argument in one glance

Problem

Notebooks reserve a GPU for the whole session but barely use it

Interactive notebooks — Jupyter, Google Colab, and the like — are how most people do machine-learning work today: you write code in cells, run a cell, look at the result, tweak, and run again. A dominant use of this is what the paper calls interactive deep learning training (IDLT): debugging a model, tuning hyperparameters, iterating on ideas, each time running a short training cell to test a change. The catch is how the GPU is provided. To feel responsive, these platforms reserve a GPU for the entire lifetime of your notebook session — from the moment it starts until it shuts down — so that whenever you run a cell, a GPU is right there. But IDLT does not use the GPU steadily. Users spend most of their time editing, reading, and debugging code that needs no GPU at all, and only occasionally fire off a short training task. The paper's production analysis makes this stark: reserved GPUs sit idle more than 81 percent of the time, around 70 percent of reserved GPUs are never used at all during their session's lifetime, and only about 15 percent of reserved GPUs are ever actively utilized. GPUs are scarce and expensive — one cluster studied runs roughly 12,000 GPUs at about 18.3 million dollars a month — so reserving them to sit idle is enormously wasteful.
What people did

Keep a GPU pinned to each session, or hand tasks to a batch scheduler

There are two conventional ways to run these workloads. The first is what Jupyter and Colab do: bind a GPU to the notebook session for its whole life, so interactivity is guaranteed because the GPU is always waiting. The second is what some managed platforms do: put a batch GPU scheduler behind the notebook, so each submitted task is queued and scheduled onto GPUs like a normal cluster job, and GPUs are not permanently pinned to idle sessions. Both aim to get training cells onto GPUs; they differ in whether the GPU is dedicated up front or acquired per task.
Why it fell short

Reservation wastes GPUs; batch scheduling wrecks interactivity

Each conventional option fails on the dimension the other protects. Reserving a GPU per session is responsive but ruinously wasteful: since the GPU is idle over 81 percent of the time, you are paying for a scarce accelerator to do nothing most of the session, which is why utilization is so low and cost so high. Routing tasks to a backend batch scheduler recovers utilization — GPUs are no longer pinned to idle sessions — but it destroys the interactivity that makes notebooks worth using: every task now waits on queuing delays and on-demand container startup before it can run, and even short delays are painful when a user is iterating rapidly, expecting a cell to start training the instant they hit run. Traditional GPU cluster schedulers make this worse because they are built for long-running batch jobs and optimize for throughput and job-completion time, treating each task as an independent request and ignoring that notebook users work inside a long-lived, stateful session. What was missing was a way to get the utilization of on-demand allocation without paying the latency of acquiring a GPU at submission time — to have a GPU ready the instant a cell runs, without reserving one that then sits idle.
This paper's idea

Replicate each kernel, and bind a GPU only while a cell runs

NotebookOS is a notebook platform built specifically for IDLT. Its core idea is to decouple GPU allocation from the notebook session. Instead of one kernel pinned to one reserved GPU, each logical notebook kernel becomes a distributed kernel of three replicas placed on different GPU servers, kept in sync so any replica can take over. A GPU is bound to a replica only while a cell is actively executing, and released the instant the task finishes, so GPUs are never held idle. Because a session's tasks arrive far apart — the paper measures a median gap of five minutes between tasks, and 75 percent of gaps under eight minutes — NotebookOS oversubscribes each server, packing many replicas onto shared GPUs on the safe bet that few will want a GPU at the same moment. Replication is what preserves interactivity under oversubscription: when a cell runs, an executor-election protocol quickly picks a replica that has a free GPU right now, so the odds of an immediate start are high. Small kernel state (Python global variables) is kept identical across replicas using the Raft consensus protocol off the critical path, while large GPU objects (model weights, datasets) are checkpointed asynchronously to a distributed data store so another replica can pick up where the last left off. Under sustained load, NotebookOS migrates replicas and automatically scales the cluster out. On real production workloads, this saved over 1,187 GPU hours in 17.5 hours of IDLT while significantly improving interactivity.

Why this problem is worth solving

Interactive notebooks are the default way people build ML models, and GPUs are the scarcest, most expensive resource they consume — so wasting GPUs on idle notebook sessions is a first-order cost.

Notebook-based interactive development is nearly universal in ML and AI, and increasingly that development involves training deep-learning models right inside the notebook. To keep it responsive, providers reserve a GPU for each session. But the reservation model collides with how interactive work actually behaves: it is bursty and mostly idle, with users spending far more time editing and thinking than running GPU tasks. The paper’s production numbers show reserved GPUs idle more than 81 percent of the time and only about 15 percent of reserved GPUs ever actively used — so the overwhelming majority of an extremely expensive resource is being paid for to do nothing. At the scale of a real cluster (roughly 12,000 GPUs, around 18.3 million dollars a month), even a modest improvement in utilization is worth an enormous amount of money and, given how scarce GPUs are globally, frees capacity for others. That is why a platform that keeps notebooks feeling instant while actually using GPUs only when there is work to do is valuable: it attacks a huge, structural source of waste in modern AI infrastructure without asking users to change how they work.

The heart of the difficulty is a genuine tension: on-demand allocation is efficient but slow to acquire, while reservation is instant but wasteful.

The obvious fix for wasted reservations is to allocate GPUs on demand — grab one when a cell runs, release it after. That is exactly what a backend batch scheduler does, and it does recover utilization. But acquiring a GPU at submission time is not free: it means queuing behind the scheduler and often starting a fresh container, which adds seconds of delay before training even begins. For an interactive user hitting run and expecting immediate feedback, that delay is exactly the experience notebooks exist to avoid. So the two goals — high utilization and instant responsiveness — pull in opposite directions under the conventional designs, and picking either one sacrifices the other. NotebookOS’s insight is that you do not have to choose if you change the unit of allocation. By replicating each kernel across several servers and oversubscribing GPUs, it keeps many candidate replicas ready, so that when a cell runs there is very likely a replica with a GPU free at that instant — giving the immediacy of reservation. And because a GPU is bound only during the actual execution and released afterward, and servers are oversubscribed on the strength of long gaps between tasks, it gets the utilization of on-demand allocation. Resolving that tension, rather than trading one side for the other, is what makes the problem both hard and worth solving.

What people did before, and where it broke

Reserve a GPU for the whole notebook session (Jupyter, Colab). Pin one GPU to the session so a cell can always run instantly. Where it broke. IDLT uses the GPU in short bursts, so the reserved GPU is idle over 81 percent of the time — extremely low utilization at very high cost.
Route tasks to a backend batch GPU scheduler. Do not pin GPUs to sessions; schedule each submitted task onto GPUs on demand. Where it broke. Acquiring a GPU per task adds queuing and container-startup delays, so interactivity — the whole point of a notebook — is lost.
Use a traditional GPU cluster scheduler. Manage tasks with schedulers built for large clusters. Where it broke. They optimize throughput and job-completion time for long-running batch jobs and treat each task as independent, a poor fit for short, latency-sensitive tasks inside a long-lived, stateful session.
Make users manage their own GPUs. Have users request and release GPUs and start and stop sessions manually. Where it broke. To avoid the hassle of shutting down, users leave notebooks idling until the provider reclaims them, which is exactly what drives utilization so low.
The gap. No design delivered both the instant responsiveness of a reserved GPU and the efficiency of allocating GPUs only when there is actually work — for stateful, long-lived notebook sessions.

How NotebookOS works

From one reserved GPU per session to a replicated kernel with GPUs bound on demand

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Oversubscribe the GPUs, betting on the gaps between tasks. NotebookOS packs far more notebook kernel replicas onto each server’s GPUs than could all run at once, because it knows from the workload that tasks are short and arrive minutes apart, so at any moment only a few replicas actually want a GPU. Deliberately promising more of a scarce resource than physically exists, and getting away with it because the demand is bursty and rarely peaks all at once, is the same principle the work on a resource allocator for oversubscribing burstable cloud resources (Audible) covered elsewhere on this site is built on, turning idle headroom into served capacity. NotebookOS applies this statistical-multiplexing bet to GPUs across replicated notebook kernels, which is what lets it reclaim the utilization that reservation throws away.

Allocate the GPU only when there is actual work, like a serverless system. Rather than holding a GPU for the session, NotebookOS binds one to a replica only while a cell is executing and frees it immediately afterward, so a GPU is consumed strictly in proportion to real training work. Decoupling an expensive resource from a long-lived session and instead allocating it elastically, only for the duration of the work that needs it, is the same on-demand model the work on an elastic serverless platform for distributed deep-learning training (ElasticFlow) covered elsewhere on this site pursues, sizing resource use to the workload rather than to a reservation. NotebookOS brings that elasticity to interactive notebooks while, through replication, avoiding the cold-start latency that usually comes with acquiring a resource on demand.

Replicate the session state across machines so any copy can take over instantly. For the on-demand bet to feel instant, several replicas must always be ready to run the session’s next task, which means each replica must hold the session’s state — so NotebookOS keeps the small CPU-side state (Python globals) identical across replicas with the Raft consensus protocol and asynchronously checkpoints the large GPU state to shared storage. Keeping a service available and correct by maintaining consistent replicated copies through a consensus protocol, so that no single machine is a bottleneck or a single point of failure, is the same foundation the work on microsecond-scale replicated fault tolerance over disaggregated memory (uBFT) covered elsewhere on this site rests on, using replication and agreement to make a stateful service both fast and dependable. NotebookOS uses state-machine replication to make a notebook kernel a movable, highly available, multi-homed thing rather than one fragile process.

Following a training cell through NotebookOS

What happens when a user runs a training cell

A trace, drawn from the paper, of how a single interactive training task is served without a reserved GPU.

Step 1 of 5

The user runs a training cell

The user hits run on a cell that trains or fine-tunes a model. The request goes to the Jupyter Server and then to NotebookOS's Global Scheduler, which routes it toward the three replicas of this session's distributed kernel.

No GPU has been sitting reserved for this session while the user was editing.

Why it works

Bursty, spread-out tasks make oversubscription safe. Reserving a GPU per session assumes constant demand that is not there → but tasks are short and arrive minutes apart, so few replicas want a GPU at once → so many replicas can safely share oversubscribed GPUs without usually colliding.
Binding a GPU only during execution eliminates idle holding. A session-long reservation pays for the GPU during all the editing and debugging → but binding a GPU strictly for the seconds a cell runs and releasing it after → so GPU-time is consumed in proportion to real work, recovering the wasted 81 percent.
Replication buys back the responsiveness on-demand usually loses. Acquiring a GPU per task would normally add queuing and cold-start delay → but keeping three ready replicas and electing one that already has a GPU free → so a cell starts almost immediately, as if a GPU had been reserved.
Keeping state in sync lets any replica serve the next cell. On-demand execution across machines only works if every replica has the session’s state → but Raft replicates the small CPU state and async checkpointing moves the large GPU state → so the session is not tied to one process and any replica can take over.
Migration and auto-scaling handle the moments demand does peak. Oversubscription can occasionally be caught out when many tasks arrive together → but migrating replicas to lighter servers and scaling the cluster out under load → so interactivity holds even when the statistical bet is temporarily wrong.

What it costs (trade-offs and scope)

Replication multiplies resource and coordination overhead. Running three replicas per kernel and keeping them consistent uses extra CPU, memory, network, and storage, and adds the machinery of a consensus protocol; the paper notes five replicas cost much more without real benefit and two are not supported by Raft, so it settles on three.
Oversubscription is a bet that can be lost. The efficiency rests on tasks rarely peaking together; if many co-located replicas demand GPUs at the same instant, some tasks must wait, migrate, or trigger scaling, so worst-case interactivity depends on how well the bet and the fallback mechanisms hold.
Large GPU state must be moved to switch replicas. Because big objects like model weights are checkpointed to a distributed data store rather than held on every GPU, having a different replica take over involves moving that state, which has its own time and bandwidth cost that the design works to keep off the critical path.
It targets interactive, bursty training, not long batch jobs. NotebookOS is designed for IDLT — short, sporadic, latency-sensitive tasks; a continuously GPU-bound, long-running training job would not benefit from oversubscription and dynamic binding the way bursty interactive work does.
Reported gains come from specific production workloads. The over-1,187-GPU-hours saved in 17.5 hours and the interactivity improvements are measured on particular real-world traces and a prototype-plus-simulation evaluation; the exact savings depend on how idle and bursty a given deployment’s workload is.

The comparison

Serving interactive deep-learning training in notebooks

Property	Reserve a GPU per session	Backend batch scheduler	NotebookOS
When a GPU is held
GPU utilization
Interactivity
Session state model
Behavior under load

A comparison of ways to run interactive deep-learning training in notebooks. Reserving a GPU per session is responsive but leaves GPUs idle over 81 percent of the time; a backend batch scheduler recovers utilization but adds queuing and cold-start delays. NotebookOS replicates each kernel across servers, binds a GPU only during active cell execution, and oversubscribes on the strength of long gaps between tasks — getting both high utilization and near-instant responsiveness, and saving over 1,187 GPU hours in 17.5 hours of real IDLT. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2503.20591, of the ASPLOS 2026 paper). The described system is the paper’s: NotebookOS is a GPU-efficient notebook platform for interactive deep learning training (IDLT). It represents each logical Jupyter kernel as a distributed kernel of three replicas on different GPU servers, synchronizes small CPU-side state (Python globals) across them using the Raft consensus protocol for state-machine replication off the critical path, and asynchronously checkpoints large GPU objects (model parameters, datasets) to a pluggable distributed data store (the paper mentions Redis, AWS S3, and HDFS). It binds GPUs to replicas only during active cell execution (“dynamic GPU binding”), oversubscribes server GPUs by leveraging the high inter-arrival times of IDLT tasks, and uses a lightweight executor-election protocol to pick a replica with a free GPU; it also supports replica migration and automatic cluster scaling under load. The workload figures (reserved GPUs idle over 81 percent of the time, about 15 percent of reserved GPUs actively used, median task inter-arrival time of five minutes, 75 percent of tasks finishing within five minutes) come from the paper’s production-trace analysis, and the headline result — over 1,187 GPU hours saved in 17.5 hours of real-world IDLT with improved interactivity — is from its prototype-and-simulation evaluation against reservation and batch-scheduling baselines. The three-replica choice is the paper’s (five is costlier without benefit; two is unsupported by Raft). Background on Jupyter notebooks, kernels, and cells is standard context. Exact protocol details, the scheduling policy, and the full evaluation should be read from the paper. NotebookOS is open-source.

Background

Why so much systems work is about reclaiming what was reserved but idle

It helps to see NotebookOS within a broad and recurring pattern in cloud and cluster systems: an expensive resource gets reserved to guarantee it is there when needed, but the actual demand is bursty and intermittent, so most of the reserved capacity sits idle — and a great deal of systems progress comes from safely reclaiming that idle capacity without breaking the guarantee. The reservation exists for a real reason (responsiveness, isolation, predictability), so you cannot simply remove it; the art is to keep the guarantee where it matters while giving the idle capacity to someone else, using statistical multiplexing, on-demand allocation, and mechanisms to recover gracefully when the bet occasionally fails. NotebookOS is a clean instance for GPUs and interactive notebooks: it keeps the felt responsiveness of a reserved GPU through replication and oversubscription, but consumes GPU time only for real work and migrates or scales when demand spikes. This drive to harvest reserved-but-unused capacity and hand it to useful work — while protecting the tenants it was reserved for — is the same terrain the work on harvesting unused memory from virtual machines in cloud platforms (memory-harvesting VMs) covered elsewhere on this site occupies, turning stranded reservations into usable resource. NotebookOS’s particular contribution is to show that, for interactive deep-learning training, replicating the kernel and binding GPUs only during execution can reclaim the vast idle time of reserved GPUs while keeping the experience interactive — saving over a thousand GPU hours in less than a day of real use.

If you remember only three things

1. Interactive deep-learning training (IDLT) — debugging models, tuning, and iterating inside Jupyter or Colab notebooks — is a dominant ML workflow, and to stay responsive these platforms reserve a GPU for the whole notebook session. But IDLT uses the GPU only in short, sporadic bursts between long stretches of editing and debugging, so the paper's production analysis finds reserved GPUs sit idle over 81 percent of the time and only about 15 percent of reserved GPUs are ever actively used — extremely low utilization of a scarce, expensive resource (one cluster runs roughly 12,000 GPUs at around 18.3 million dollars a month). Routing tasks to a backend batch scheduler recovers utilization but adds queuing and container-startup delays that destroy the interactivity notebooks exist for.
2. NotebookOS is a notebook platform built for IDLT that decouples GPU allocation from the session. Each logical kernel becomes a distributed kernel of three replicas on different GPU servers, kept consistent with the Raft consensus protocol (small Python globals synced off the critical path; large GPU objects like model weights checkpointed asynchronously to a distributed data store). A GPU is bound to a replica only while a cell is actually executing and released immediately after, and servers are oversubscribed on the strength of long gaps between tasks (median five minutes). When a cell runs, an executor-election protocol picks a replica that has a free GPU right then, so training starts almost instantly — recovering the responsiveness of a reservation without the idle waste — and under load NotebookOS migrates replicas and auto-scales the cluster.
3. On real production workloads, NotebookOS saved over 1,187 GPU hours in 17.5 hours of IDLT while significantly improving interactivity, resolving the usual tension between the efficiency of on-demand allocation and the responsiveness of reservation. The trade-offs: replicating each kernel three ways adds CPU, memory, network, storage, and consensus overhead; oversubscription is a statistical bet that can be lost when many tasks peak together (handled by migration and scaling); switching replicas means moving large GPU state; and the approach targets bursty interactive training, not long continuous batch jobs. The reported savings depend on how idle and bursty a given deployment's workload is.

Check yourself

Check your understanding

1. Why do conventional notebook platforms waste so much GPU capacity on interactive deep-learning training?
2. What is NotebookOS's core design for decoupling GPU allocation from the session?
3. How does NotebookOS keep the experience responsive while still allocating GPUs on demand?
4. What does NotebookOS achieve, and what are the limits of the approach?

Sources

The paper is “NotebookOS: A Replicated Notebook Platform for Interactive Training with On-Demand GPUs” by Benjamin Carver, Jingyuan Zhang, Haoliang Wang, Kanak Mahadik, and Yue Cheng (George Mason University, Adobe, and the University of Virginia), published at ASPLOS 2026, DOI 10.1145/3760250.3762230. It is available on arXiv as 2503.20591, and the system is open-source at github.com/ds2-lab/NotebookOS.

This explainer is written from the paper’s full text (the authors’ arXiv copy). NotebookOS is a GPU-efficient notebook platform for interactive deep learning training (IDLT). It represents each logical kernel as a distributed kernel of three replicas across GPU servers, synchronizes small CPU-side state with the Raft consensus protocol and asynchronously checkpoints large GPU objects to a distributed data store, binds a GPU to a replica only during active cell execution, oversubscribes server GPUs by leveraging the long inter-arrival times of IDLT tasks, and elects a replica with a free GPU per cell; it also supports replica migration and automatic cluster scaling under load. Its production-trace analysis reports reserved GPUs idle over 81 percent of the time and only about 15 percent of reserved GPUs actively used, and its evaluation reports over 1,187 GPU hours saved in 17.5 hours of real IDLT with improved interactivity. Background on notebooks, kernels, and cells is standard context. The numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.