Devices like AR/VR headsets and drones run several ML models at once — hand tracking, eye tracking, translation — in dependent pipelines with hard real-time deadlines (say 60 frames per second) across multiple accelerator chips, and the workload is DYNAMIC on three levels — which tasks are active can change, which model runs next can depend on a previous model's output, and a model can pick a different subset of its own layers at runtime; a schedule computed ahead of time is therefore useless, and prior dynamic schedulers use one fixed algorithm for every situation and ignore energy; this paper (Georgia Tech) builds DREAM, which scores every ready layer-and-accelerator choice with a single "MapScore" folding urgency, speed, fairness, and energy, then continuously RE-TUNES that score's weights to whatever the workload is doing right now — and under overload it proactively drops frames that will miss their deadline anyway and swaps in lighter model variants — cutting a combined deadline-violation-and-energy cost by 32 to 50% over the best prior schedulers, up to 97.6%.
The argument in one glance
-
Problem
Modern devices run many ML models at once, on a deadline, and the workload keeps changing
An AR/VR headset does not run one neural network — it runs several at the same time, in dependent pipelines: hand tracking, eye tracking, face detection, translation, all feeding each other, all on hardware with several accelerator cores, all needing to finish within a real-time deadline (say, 60 frames per second so the display never stutters). Drones and autonomous systems are the same. These real-time multi-model ML (RTMM) workloads are hard enough to schedule when fixed, but their defining feature is that they are DYNAMIC: the set of running tasks changes, the pipeline path is decided at runtime, and individual models change which of their own layers they execute. Deciding which piece of work runs on which accelerator, and when, so that every deadline is met and energy is saved, is the problem.
-
What people did
Compute a schedule, or use a fixed dynamic policy
The classic approach is a STATIC scheduler: analyze the workload offline, compute an optimized schedule that maps each layer to an accelerator and orders everything to meet the deadlines, and run it. When workloads vary, recent DYNAMIC schedulers for ML accelerators — Planaria and Veltair — make decisions at runtime, partitioning the accelerators spatially or blocking layers according to a threshold policy. Both aim to keep the accelerators busy and hit deadlines under multi-model load.
-
Why it fell short
Static plans are invalid, and fixed dynamic policies do not adapt
A static schedule cannot exist for this workload. When the next model to run depends on a previous model's output (a control dependency), or a task appears and disappears, or a model picks a different subset of its layers, the computation graph is simply not known until runtime — so there is nothing valid to pre-compute. Just being dynamic helps a lot: running even a plain First-Come-First-Served policy at runtime instead of statically cuts deadline violations by 52.9%. But the prior DYNAMIC schedulers fall short too: they apply ONE FIXED scheduling algorithm to every situation, do not handle task- and model-level dynamism, and ignore energy — which matters on battery devices. As the workload shifts, a policy tuned for one situation becomes wrong for the next, and no single fixed rule fits them all.
-
This paper's idea
Score every choice, and re-tune the scoring to the live workload
This paper (Georgia Tech — Kim et al.) builds DREAM, a scheduler designed for the dynamism itself. Each time a layer becomes ready, DREAM scores every possible layer-and-accelerator pairing with MapScore, a single number that folds four factors: URGENCY (remaining work divided by slack until the deadline), LATENCY PREFERENCE (put a layer on the accelerator that runs it fastest), STARVATION (do not let light layers wait behind heavy ones), and ENERGY. The pairing with the highest MapScore runs. The key idea is that MapScore's weights are not fixed — DREAM continuously RE-TUNES them to the live workload, optimizing the real objective, UXCost (deadline-violation-rate multiplied by energy), and converging within about five steps as the workload shifts. Under overload it adds two graceful-degradation moves: SMART FRAME DROP proactively discards a frame that will miss its deadline anyway (respecting pipeline dependencies), and SUPERNET SWITCHING swaps in a lighter model variant that still meets accuracy. Across AR/VR, drone, and mobile workloads, DREAM cuts UXCost by 32.2% over Planaria and 50.0% over Veltair (up to 97.6%), with energy savings up to 62.9%.
Why this problem is worth solving
The next generation of interactive devices — headsets, drones, cars — is
defined by running many ML models together under a hard clock, and how
well the system schedules them decides whether the experience is smooth
or unusable. Understanding what makes these workloads special comes first.
A
a
real-time multi-model ML (RTMM) workload runs several neural networks
concurrently, in cascaded pipelines with dependencies, under real-time
deadlines (like 60 frames per second), on multi-accelerator hardware.
What sets it apart is DYNAMISM at three granularities. TASK-level: whole
task pipelines appear and disappear — an AR app switching from a game to a
voice call swaps in an entirely different set of models. MODEL-level: the
pipeline path is a control dependency, so which model runs next is decided
by a previous model’s output and is unknown until that model finishes.
LAYER-level (or operator-level): a single model may be a dynamic network
(a weight-sharing Supernet, an early-exit or skip network) that selects a
different subset of its own layers at runtime. Because of all three, the
exact computation graph does not exist until the moment it runs — which is
precisely what breaks any scheme that needs to know the work in
advance
The mechanism that copes with it, and the paper’s core contribution, is a
score that is itself adaptive. A
MapScore is the single
number DREAM assigns to each candidate layer-and-accelerator pairing to
decide what runs next. It multiplies an URGENCY score (predicted
remaining work divided by slack until the deadline, so near-deadline work
ranks high) by a LATENCY-PREFERENCE score (favoring the accelerator that
runs this layer fastest), then adds weighted STARVATION and ENERGY scores
(so light layers are not starved and energy is respected). The two added
terms carry tunable weights — and the pivotal idea is that DREAM does not
fix those weights but OPTIMIZES them online against the true end-goal,
UXCost, defined as the deadline-violation-rate times the normalized
energy (an energy-delay-product-like cost). A lightweight search samples
weight pairs, interpolates toward the minimum UXCost, and converges within
about five steps, so as the workload changes, the scheduler re-tunes its
own priorities to keep hitting the moving target rather than applying one
fixed rule forever
What people did before, and where it broke
- Static schedulers. Analyze the workload offline and compute an optimized mapping and order. Excellent when the computation graph is known, but RTMM’s control-dependent, task-varying, dynamic-layer workloads have no fixed graph to analyze, so no valid static schedule exists.
- Planaria (deadline-aware spatial partitioning). Dynamically split the accelerator array among tasks by their deadlines. A real dynamic scheduler, but it uses one fixed policy, does not address task- or model-level dynamism, and is not energy-aware.
- Veltair (layer-blocking, threshold scheduling). Group layers and schedule them by a threshold rule at runtime. Again dynamic, but a fixed algorithm applied to every scenario, missing the higher-level dynamism and energy.
- Naive dynamic scheduling (FCFS at runtime). Just react as work arrives. Being dynamic at all already helps (52.9% fewer deadline violations than the static version), which proves the point — but a first-come rule leaves large gains on the table.
- The gap. No scheduler adapted its own policy to the changing workload, handled all three levels of dynamism, and balanced deadlines against energy. That is the gap DREAM fills.
How DREAM schedules a dynamic ML workload
Score every choice, tune the score to the live objective, and degrade gracefully under load
-
The dynamic workload and why static plans fail
Multiple models run concurrently in dependent pipelines under real-time deadlines, and the workload is dynamic at task, model, and layer level, so the computation graph is unknown until runtime. A schedule computed ahead of time is invalid; even running a simple policy dynamically beats running it statically.
-
Score every ready layer-accelerator pairing with MapScore new
Whenever a layer becomes ready, DREAM computes MapScore for each accelerator it could run on — urgency (work over slack) times latency preference (fastest accelerator), plus weighted starvation and energy terms — using offline latency and energy tables, since those are deterministic once an inference launches. The highest-scoring pairing runs.
-
Optimize the true objective, UXCost new
The real goal is not deadlines or energy alone but UXCost, the deadline-violation-rate multiplied by normalized energy (an energy-delay-product-like cost). DREAM tunes MapScore's weights to minimize UXCost, so it will even accept slightly more deadline violations when that lowers total cost.
-
Re-tune the weights online as the workload shifts new
An adaptivity engine watches for workload changes (the active model list, violation and drop rates) and re-optimizes MapScore's weights without blocking execution — sampling weight pairs, interpolating toward the minimum UXCost, and converging within about five steps. When resources are sufficient, it does nothing, adding zero overhead.
-
Skip doomed work under overload new
When a deadline violation is predicted, smart frame drop proactively discards the frame early rather than waste compute on it — but only when the frame sits at a pipeline endpoint (so no downstream work is invalidated), multiple jobs are affected, and the drop rate stays within bounds. It exploits that RTMM's possible execution paths are known in advance.
-
Degrade gracefully with Supernet switching new
Under heavy load, DREAM swaps a model for a lighter variant of a weight-sharing Supernet that still meets an accuracy target (for example, a 96-MFLOP variant at 73.1% top-1 accuracy), cutting the system load enough to meet deadlines while preserving quality — a graceful degradation rather than a missed frame.
The idea
When the workload is dynamic, decide at runtime — and make the policy itself dynamic. The foundational move is to accept that no schedule can be computed ahead of a workload whose graph is unknown until it runs, so decisions must be made online. DREAM goes one level further than prior dynamic schedulers: it recognizes that a single fixed online policy is also wrong, because the best trade-off between urgency, fairness, and energy shifts as tasks appear, paths branch, and load rises, so it re-tunes its own scoring weights to the live workload. This decide-at-runtime, and adapt-the-decision-rule-too, discipline sits squarely on the site’s static-versus-dynamic axis: the on-chip-allocation page plans offline because its problem is fixed, while the dynamic-tiling, runtime-optimization, and elastic-scheduling pages decide at runtime because theirs are not — and DREAM is the extreme case, where even the objective weights are re-optimized on the fly. The lesson is that a dynamic problem is not solved by a clever fixed rule but by a rule that keeps re-fitting itself.
Fold competing objectives into one score, and optimize the true end-cost. Scheduling here juggles deadlines, hardware speed, fairness, and energy — objectives that pull against each other — and DREAM combines them into a single MapScore so one comparable number ranks every choice, exactly the balance-objectives-in-one-price move the site’s heterogeneity-aware scheduler makes to trade completion time against scaling. But the subtler point is WHAT it optimizes: not a proxy but the real end-goal, UXCost, which multiplies deadline violations by energy, so DREAM will knowingly accept a few more violations when that yields lower total cost. Optimizing the true objective rather than a convenient surrogate is the same measure-and-target-the-real-thing discipline the site’s I/O-scheduler and value-profiler pages insist on, and it is what lets DREAM make the right global trade instead of greedily minimizing one metric at another’s expense.
Skip work that cannot help, and degrade gracefully instead of failing. Under overload, the worst outcome is to spend an accelerator on a frame that will miss its deadline anyway and still not finish it. DREAM’s smart frame drop abandons that doomed work early — carefully, only at pipeline endpoints so no dependent work is wasted — which is the same skip-work-that-cannot-contribute discipline the site’s sparsity and dynamic-tiling pages use to avoid computing what will not be used. And Supernet switching trades a little quality for meeting the clock, dropping to a lighter model variant under pressure, the same degrade-gracefully, approximate-under-load move the site’s approximate-query and training-free pages make when full fidelity is unaffordable. Both rest on the fact that RTMM’s dynamism is CONSTRAINED — the possible paths are known and per-layer cost is deterministic once launched — so DREAM can predict a violation and respond before it happens, rather than discover it too late.
Scheduling when the workload keeps changing
From an invalid plan to an adaptive schedule
Scheduling real-time multi-model ML on multiple accelerators. Numbers from the paper's simulation.
Step 1 of 5
Why a pre-computed schedule fails
The pipeline path is chosen at runtime, tasks appear and vanish, and models pick different layers, so the computation graph is unknown until it runs — there is nothing valid to schedule ahead of time. Simply reacting at runtime already helps a great deal, which proves the workload demands a dynamic approach.
running First-Come-First-Served dynamically instead of statically cuts deadline violations by 52.9% the three levels of dynamism (task, model, layer) each make part of the graph unknowable in advance
Why it works
- The computation graph is unknown until runtime (control dependencies, appearing tasks, dynamic layers) → no valid schedule can be computed ahead → deciding online is the only option, and even naive dynamic scheduling beats static by 52.9%.
- The best trade-off among urgency, fairness, and energy shifts as the workload changes → DREAM re-tunes MapScore’s weights online toward the live UXCost minimum → one scheduler fits every situation instead of one fixed rule fitting none.
- Deadlines, hardware speed, fairness, and energy pull against each other → folding them into a single MapScore and optimizing the true UXCost → the scheduler makes the right global trade, even accepting a few violations to cut total cost.
- Spending an accelerator on a frame that will miss its deadline wastes it → smart frame drop abandons doomed frames early, only at pipeline endpoints → compute is freed for work that can still meet its deadline, without invalidating dependents.
- RTMM’s dynamism is constrained — paths are known and per-layer cost is deterministic once launched → DREAM can predict violations and use offline cost tables → it responds before a deadline is missed rather than after.
What it costs (trade-offs and scope)
- It is evaluated in simulation. Results come from modeled multi-accelerator systems and offline latency and energy tables, not a fabricated chip, so they depend on the accuracy of those models and tables.
- It assumes the dynamism is constrained. MapScore relies on offline per-layer cost tables and on the possible execution paths being known and deterministic once launched; a workload with truly unbounded behavior or strongly data-dependent per-layer latency would not fit that assumption.
- The online tuning helps only under contention. It converges in about five steps but has a transient while adapting, and the paper reports zero overhead only when resources are sufficient — the adaptivity is a contention-time mechanism, not a universal free win.
- Frame drop and Supernet switching trade quality for the deadline. Dropping a frame or using a lighter model degrades output quality or accuracy, and DREAM will accept more deadline violations (up to 1.88x in one setting) to win on total UXCost — so meeting deadlines is not absolute, it is balanced against energy and quality.
- It adds scheduler machinery on the critical path. DREAM runs several engines and exposes tunable knobs (drop-rate bounds, the weight-tuning step), and a layer-granularity scheduler must keep its own decision cost small to stay real-time — another moving part that has to be cheap.
The evaluation
Simulation on homogeneous and heterogeneous multi-accelerator systems (4K and 8K processing elements, weight-stationary and output-stationary dataflows, 8 MiB SRAM, 90 GB/s, 700 MHz), on AR/VR, drone, and mobile RTMM workloads (VR gaming, AR call, drone indoor/outdoor, AR social), against FCFS, Planaria, and Veltair, measuring UXCost, deadline violations, and energy.
Scheduling real-time multi-model ML
| Property | Static scheduler | Fixed dynamic (Planaria/Veltair) | DREAM (this work) |
|---|---|---|---|
| Handles the dynamic graph | |||
| Objective | |||
| Adapts to the workload | |||
| Overload handling | |||
| UXCost versus prior best |
Check yourself
Check your understanding
1. DREAM argues that no static schedule can work for real-time multi-model ML, even though static scheduling is well understood. Predict why, and what follows.
2. Prior dynamic schedulers already decide at runtime, yet DREAM outperforms them by re-tuning its scoring weights online. Predict why a fixed online policy is still not enough.
3. DREAM optimizes UXCost (deadline-violations times energy) and will even accept MORE deadline violations in some settings. Predict why targeting this combined cost is better than minimizing deadline violations alone.
4. Under overload, DREAM proactively DROPS a frame that is predicted to miss its deadline, rather than trying to compute it. Predict why dropping doomed work early is the right move, and what makes it safe.
Sources
- DREAM: A Dynamic Scheduler for Dynamic Real-time Multi-model ML Workloads (arXiv:2212.03414) — ASPLOS 2023 (full text read via ar5iv).
- ar5iv HTML rendering — the version read for this explainer.
- arXiv PDF — full paper.