Nazar: Monitoring and Adapting ML Models on Mobile Devices

Wei Hao, Zixi Wang, Lauren Hong, Lingxiao Li, Nader Karayanni, AnMei Dasbach-Prisk and 3 others

Companies push ML models onto millions of phones to run offline, but once out there a model's accuracy can quietly rot — the camera, the weather, the country all differ from the training data — and with no labels from users, operators are blind to it; this paper (Columbia University — Wei Hao, Junfeng Yang, Asaf Cidon, and colleagues) presents Nazar, which watches the models across the whole fleet using only their own confidence, figures out the specific root cause when many devices degrade together, and retrains a targeted fix for each cause, all with no human in the loop.

The argument in one glance

Problem

Once a model is on millions of phones, its accuracy can rot silently

Companies like Google, Meta, Apple, and LinkedIn increasingly run their machine-learning models directly on users' mobile devices, so inference happens locally at low latency and even works offline — for text suggestion, ranking, object detection, speech recognition, and more. The models are trained centrally in datacenters and pushed out to the fleet, and a team of ML operators is responsible for maintaining them. But once a model is running on millions of heterogeneous phones, its accuracy becomes a black box. Accuracy varies enormously across devices and drifts over time, mostly because of data drift: the data a phone actually sees diverges from the data the model was trained on. A self-driving model trained on US streets may not recognize signs in India; a vision model degrades in fog, in noise, or on a phone with a low-quality camera; even the compression used to shrink the model for the device can quietly hurt accuracy on certain classes. The insidious part is that operators cannot see any of this. On a phone, there are no ground-truth labels — users do not tell you whether the model was right — so the accuracy simply drops with no signal reaching the operators. They cannot even detect that something went wrong, let alone fix it. A model can therefore degrade badly across a large slice of the fleet while everyone remains blind to it.
What people did

Use ML algorithms for detecting and adapting to data drift

The machine-learning literature has many algorithms for exactly this: detecting when data drift has occurred, and adapting a model to the new distribution. These techniques can spot that incoming data has shifted and update the model to cope, and there is a rich body of work on both drift detection and drift adaptation. In principle, an operator could apply them to keep on-device models healthy as the world changes around them.
Why it fell short

Those algorithms assume labels, one cause, and a human — none of which hold at scale

The existing algorithms rest on assumptions that break in a real large-scale mobile deployment. Many assume fully-labeled data — but on-device you have none, because users do not label their inputs. Many assume a single source of data drift and adapt the model on every incoming input toward that one shift — but a real fleet has many simultaneous causes of degradation (one region's weather, another's signage, a particular camera model), and adapting indiscriminately to all of them at once gives poor accuracy, because a fix for one cause is wrong for data drifted by a different cause. Some methods need heavy computation or manual inspection of the results, which is untenable across millions of devices with a small operations team. And crucially, none of these algorithms had ever been assembled into an end-to-end system that actually runs continuously in production. So the pieces existed in the abstract, but there was no automatic, label-free, multi-cause, deployable way to keep on-device models accurate. What was missing was a system that could detect degradation without labels, figure out the specific causes across the fleet, and fix each one precisely — all without a human in the loop.
This paper's idea

Detect by confidence, find the shared root cause, and adapt per cause

Nazar is the first end-to-end system for continuously monitoring and adapting on-device models, designed to run fully automatically with no user feedback and no labels. Its guiding observation is that accuracy degradation is usually not random per-device noise but the result of a specific root cause that affects a large group of devices — so if you can find that cause, you can fix it fleet-wide. Nazar has three components in a closed loop. First, detection: it flags data drift by comparing the model's own prediction confidence against a threshold — low confidence signals likely errors. This needs no labels and no surrogate model, is computationally light, and runs on any phone. Second, root cause analysis: once a consistent degradation shows up across many devices, Nazar automatically finds the origin. A classical method, frequent-itemset mining, can tie sets of device/data attributes to the drift, but it produces many overlapping, redundant candidate causes that normally require a human to sort out. Nazar adds two new heuristics — set reduction and counterfactual analysis — that automatically prune the candidates down to a small set of likely root causes by eliminating causes that subsume or overlap each other. Third, by-cause adaptation: instead of adapting the model on every input toward a single assumed drift (as prior work does), Nazar selectively adapts the model to each specific identified cause, using the data suspected to have that cause, in a self-supervised way that needs no labels. The three loop together — an adapted model is redeployed, and its confidence feeds the next round of detection — continuously improving accuracy. On two computer-vision datasets, Nazar consistently beats existing approaches, improving accuracy on a self-driving dataset by an average of 15 percent (and up to 22 percent), and by an average of 19 percent (up to 50 percent) on the drifted data specifically.

Why this problem is worth solving

On-device ML is now everywhere and invisible, so silent accuracy loss degrades products at scale with no one noticing, and no existing tooling catches it.

Running models locally on phones is standard practice because it gives low latency, offline operation, and privacy, and it powers features people use constantly. But that same locality removes the feedback loop operators rely on in the datacenter: there are no labels, no easy way to compute accuracy, and millions of devices each in their own environment. So when a model’s accuracy silently drops on a large group of devices — because of a weather pattern, a common camera, a region the model was never trained for — the product quietly gets worse for those users and no signal reaches the team responsible. At the scale these companies operate, a degradation affecting even a small fraction of the fleet is an enormous number of affected users, and it can persist indefinitely because it is invisible. Beyond quality, these unnoticed accuracy gaps can even open security vulnerabilities. Making on-device model performance observable and fixable, automatically, is therefore not a nicety but a basic requirement for operating ML responsibly at scale — and it is a capability that, before this work, did not exist end-to-end.

The key realization is that degradation has structure — a shared cause across many devices — which is what makes both diagnosis and repair tractable.

If every device degraded for its own idiosyncratic reason, there would be little a central operator could do: no labels, no leverage, millions of separate problems. Nazar’s pivotal observation is that this is not how degradation usually works. Accuracy problems tend to have a specific root cause — a particular environmental condition, a hardware type, a geographic distribution shift — and that cause affects a whole group of devices at once. This changes everything, because it turns an intractable per-device problem into a tractable per-cause one. Detecting a consistent degradation across many devices is a strong signal that a real, shared cause is at work, worth investigating; identifying which shared attributes those degraded devices have in common points at the cause; and fixing the model for that cause repairs the whole group. It also explains why adapting indiscriminately fails: when several causes coexist, a single global adaptation pulls the model in conflicting directions, so the right move is to separate the causes and adapt to each one on its own data. The three components of Nazar are essentially the three steps this insight implies — notice the group degradation without labels, isolate the shared cause cleanly, and adapt by cause — and each is designed to work automatically because a human cannot be in the loop at fleet scale.

What people did before, and where it broke

Retrain and push new model versions periodically. Have operators fine-tune models and redeploy. Where it broke. Operators cannot even detect on-device data drift or verify accuracy, so they do not know when or how to retrain.
Apply drift-detection algorithms that assume labels. Use methods that compare predictions to ground truth. Where it broke. On-device there are no labels, because users do not annotate their inputs.
Adapt to a single assumed drift on every input. Update the model toward one distribution shift as data arrives. Where it broke. A real fleet has many simultaneous causes, and one global adaptation gives poor accuracy because a fix for one cause is wrong for data drifted by another.
Find causes with frequent-itemset mining and inspect them by hand. Tie attribute sets to drift and have a human sort the results. Where it broke. It produces many overlapping, redundant candidate causes, and manual inspection is untenable across millions of devices.
The gap. No end-to-end system detected degradation without labels, isolated the specific shared causes automatically, and adapted the model to each cause, all without human involvement.

How Nazar works

A closed loop: detect drift, find the cause, adapt by cause

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Monitor a deployed model’s health without disturbing it or needing labels. Nazar watches on-device models across the fleet using only their own prediction confidence, a lightweight, label-free signal that reveals when accuracy is slipping. Continuously observing a running machine-learning system to catch problems, using a cheap non-intrusive signal rather than instrumenting or labeling everything, is the same approach the work on fine-grained, non-intrusive LLM training monitoring (Pulse) covered elsewhere on this site takes, reading an available signal to surface trouble without disrupting the workload. Nazar applies it to on-device inference, where prediction confidence is the signal that stands in for the accuracy operators cannot directly measure.

Pinpoint the specific cause among many candidates. When many devices degrade together, Nazar isolates the shared root cause, pruning a thicket of overlapping candidate explanations down to a small, clean set. Attributing a systemic performance problem to its true underlying cause, cutting through correlated and redundant candidates, is the same challenge the work on Shapley-value influence quantification for performance debugging (ShapleyIQ) covered elsewhere on this site tackles, quantifying which factor is actually responsible rather than merely correlated. Nazar’s set reduction and counterfactual analysis do this automatically for on-device data drift, so the fix targets the real cause.

Adapt the model as the world it runs in keeps drifting. Nazar closes the loop by updating the model to the drift it detects, cause by cause, keeping accuracy up as conditions change over time. Treating a deployed learned system as something that must continually adapt to workloads that drift, rather than a fixed artifact, is exactly the recipe the work on whether machine learning for systems has reached an inflection point covered elsewhere on this site names for coping with drift, adapting the model rather than assuming a stationary world. Nazar operationalizes it for mobile ML with self-supervised, per-cause adaptation.

Following a degradation through Nazar

Catching and fixing a fleet-wide accuracy drop with no labels

A trace, drawn from the paper, of how Nazar detects, diagnoses, and repairs an on-device degradation.

Step 1 of 5

Devices start producing low-confidence predictions

On a set of phones, the model's prediction confidence drops below Nazar's threshold — a sign that its inputs have drifted from the training distribution and its answers are likely wrong, even though no labels confirm it.

Confidence is the label-free proxy for accuracy Nazar monitors.

Why it works

Confidence is a label-free stand-in for accuracy. Operators have no on-device ground truth → but comparing the model’s prediction confidence to a threshold → so drift and likely errors are detected cheaply on any phone with no user input.
Group degradation signals a real, shared cause. A single device dropping could be noise → but a consistent degradation across many devices → so Nazar knows to investigate a genuine root cause it can fix fleet-wide.
Pruning yields a clean, actionable cause set. Frequent-itemset mining floods you with overlapping candidates → but set reduction and counterfactual analysis remove causes that subsume each other → so a small set of true causes emerges automatically, no human needed.
By-cause adaptation avoids conflicting fixes. Adapting to all inputs assumes one drift and fails under many → but adapting the model separately to each cause on its own data → so each group is repaired without corrupting the others.
The closed loop compounds the gains. A one-shot fix would drift out of date → but redeploying adapted models and feeding their confidence back into detection → so the three components reinforce each other and accuracy keeps improving.

What it costs (trade-offs and scope)

Confidence is a proxy, not a guarantee. Detecting drift via prediction confidence assumes low confidence tracks low accuracy; a model that is confidently wrong, or uncertain when correct, could mislead the detector, so the signal is a practical heuristic rather than a perfect accuracy measure.
Root cause analysis needs attributes to reason over. Isolating a cause depends on having device and data attributes (hardware type, location, conditions) to mine and prune; causes not reflected in the available attributes are harder to pinpoint.
By-cause adaptation depends on grouping the right data. The fix for a cause is only as good as the data suspected to share that cause; if the grouping is wrong, the adaptation can be mis-targeted, so the quality of the root cause analysis directly bounds the adaptation.
It targets group-level, shared-cause degradations. Nazar’s design exploits that degradations affect large groups; a truly one-off, per-device problem with no shared cause offers little for it to detect or fix.
Reported gains are dataset-specific. The 15-percent-average (up to 22, and 19 on drifted data) improvements come from two computer-vision datasets (a self-driving dataset and a species-classification one) in a specific setup; the exact benefit depends on the workload, the drift, and the fleet.

The evaluation

Keeping on-device ML models accurate as data drifts

Property	Periodic retraining	Standard drift adaptation	Nazar
Detects on-device degradation
Handles multiple drift causes
Root cause identification
Adaptation strategy
Human involvement

A comparison of ways to keep on-device ML models accurate under data drift. By detecting drift from the model's own confidence (no labels), automatically isolating shared root causes with set reduction and counterfactual analysis, and adapting the model by cause in a self-supervised loop, Nazar boosts accuracy over existing approaches — on a self-driving dataset, an average of 15 percent (up to 22) on all data and 19 percent (up to 50) on drifted data — with no human in the loop. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2305.07772, of the ASPLOS 2025 paper). The described system is the paper’s: Nazar is the first end-to-end system for continuously monitoring and adapting machine-learning models on mobile devices without user feedback or labels. Its key observation is that accuracy degradation is often due to a specific root cause affecting a large group of devices. It has three components: detection, using a confidence-threshold method that flags data drift by comparing prediction confidence to a threshold (no labels, lightweight, on-device); root cause analysis, which uses frequent-itemset mining plus two new heuristics, set reduction and counterfactual analysis, to automatically narrow the candidate causes to a small set by eliminating overlapping and subsuming causes; and by-cause adaptation, a self-supervised method that adapts the model to each specific identified cause using the data suspected to have it, rather than adapting on all inputs toward a single assumed drift. The components run in a continuous closed loop. The paper implements Nazar on Amazon AWS and evaluates it on two streaming computer-vision datasets — Cityscapes (self-driving images from European cities) and an ImageNet-derived species-classification dataset — reporting, on Cityscapes, an average 15 percent (up to 22 percent) accuracy boost on all data and an average 19 percent (up to 50 percent) on drifted data over baselines (the abstract summarizes the improvement over existing approaches as up to 19.4 percent). The background above on data drift, on-device inference, and frequent-itemset mining is standard context. The step-by-step walkthrough is an illustration of the described mechanism. Exact algorithms for set reduction, counterfactual analysis, and by-cause adaptation, and the full evaluation, should be read from the paper; the authors state they will open-source the datasets and code.

Background

Why deploying ML shifts the hard problem from training to operating

It helps to see Nazar within a broad pattern: as machine learning moves from the lab into production, and especially onto the vast, uncontrolled population of users’ devices, the hard problem stops being how to train a good model and becomes how to keep it good once it is out in the world. A model is trained once on a fixed dataset, but it then runs for a long time against a world that keeps changing — new conditions, new environments, new hardware — and it has no built-in way to know when it has fallen out of step, especially without the labels that made training possible. Operating ML at scale therefore demands the same disciplines any large system needs: observability to know what is happening, diagnosis to find why something went wrong, and a way to remediate, all working automatically because the scale forbids manual attention. What makes ML distinctive is that these have to work without ground truth and against silent, statistical failure rather than crashes. Nazar is a clean instance of building that operational loop for on-device models, and its emphasis on running fully automatically reflects how central automation is when there are millions of devices and a small team. This concern for making machine learning dependable and operable on the constrained edge devices in people’s hands, where deployment and lifecycle become as hard as accuracy, is the same terrain the work on turbocharging NLP inference at the edge via elastic pipelining (STI) covered elsewhere on this site occupies, contending with the realities of running models out where the users are. Nazar’s contribution is to show that on-device model accuracy can be monitored and repaired automatically, by treating degradation as a shared-cause phenomenon and closing the loop from detection through diagnosis to cause-specific adaptation.

If you remember only three things

1. Companies increasingly run ML models locally on users' mobile devices for low-latency, offline inference, with models trained in datacenters and pushed to the fleet. But once deployed, a model's accuracy degrades unpredictably across devices — mostly from data drift, where the data a phone sees diverges from the training distribution (weather, noise, geography, low-quality sensors, even compression artifacts). Operators are blind to this, because on-device there are no ground-truth labels, so accuracy silently drops with no signal reaching them. Prior drift-detection and adaptation algorithms assume fully-labeled data, or a single drift cause, or need heavy compute or manual inspection, and none had been built into a real end-to-end system.
2. Nazar is the first end-to-end system to continuously monitor and adapt on-device models fully automatically, no labels or user feedback needed. Its guiding insight is that degradation usually stems from a specific root cause affecting a large group of devices, making it tractable to diagnose and fix fleet-wide. It has three components in a closed loop: detection, which flags drift by comparing the model's own prediction confidence to a threshold (label-free, lightweight, on-device); root cause analysis, which uses frequent-itemset mining plus two new heuristics — set reduction and counterfactual analysis — to automatically prune overlapping candidate causes down to a small true set; and by-cause adaptation, which self-supervisedly adapts the model to each identified cause separately (rather than to all inputs toward one assumed drift), so a fix for one cause does not corrupt data from another.
3. Implemented on AWS and evaluated on two computer-vision datasets (Cityscapes self-driving images and an ImageNet-derived species classifier), the paper reports Nazar consistently beats existing approaches, boosting accuracy on Cityscapes by an average of 15 percent (up to 22) on all data and an average of 19 percent (up to 50) on drifted data. The trade-offs: confidence is a proxy for accuracy rather than a guarantee, root cause analysis needs informative device and data attributes to reason over, by-cause adaptation is only as good as the cause grouping, and the design targets group-level shared-cause degradations rather than one-off per-device problems. The reported gains are dataset-specific; the authors will open-source datasets and code.

Check yourself

Check your understanding

1. Why is it hard for ML operators to know when an on-device model's accuracy has degraded?
2. Why did prior drift-detection and adaptation algorithms fail for real large-scale mobile deployments?
3. What is Nazar's guiding insight, and how do its three components follow from it?
4. Why does Nazar adapt "by cause" instead of adapting on all inputs, and what does it achieve?

Sources

The paper is “Nazar: Monitoring and Adapting ML Models on Mobile Devices” by Wei Hao, Zixi Wang, Lauren Hong, Lingxiao Li, Nader Karayanni, AnMei Dasbach-Prisk, Chengzhi Mao, Junfeng Yang, and Asaf Cidon (Columbia University), published at ASPLOS 2025, DOI 10.1145/3669940.3707246. It is available on arXiv as 2305.07772.

This explainer is written from the paper’s full text (the authors’ arXiv copy). Nazar is the first end-to-end system for continuously monitoring and adapting on-device machine-learning models without user feedback or labels, based on the observation that accuracy degradation is often due to a specific root cause affecting a large group of devices. It comprises three components in a closed loop: confidence-threshold drift detection (label-free, lightweight, on-device); automated root cause analysis using frequent-itemset mining plus new set reduction and counterfactual analysis heuristics to prune overlapping candidate causes to a small set; and by-cause self-supervised adaptation that adapts the model to each identified cause using the data suspected to have it, rather than adapting on all inputs toward a single assumed drift. Implemented on Amazon AWS and evaluated on two computer-vision datasets (Cityscapes and an ImageNet-derived species classifier), the paper reports Nazar boosts accuracy over existing approaches by, on Cityscapes, an average of 15 percent (up to 22 percent) on all data and 19 percent (up to 50 percent) on drifted data. Numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.