Segment Only Where You Look: Leveraging Human Gaze Behavior for Efficient Computer Vision Applications in Augmented Reality

Augmented-reality glasses have to figure out which objects are in view by "segmenting" the camera image pixel by pixel, but doing that for the whole high-resolution frame is far too slow on a lightweight headset — this paper (NYU — Tianhua Xia, Haiyu Wang, Sai Qian Zhang) observes that a user only actually looks at one thing at a time, so it uses the headset's eye-tracking to segment only the object you are looking at, co-designing the algorithm, the image sensing, and a small accelerator to cut end-to-end latency by up to 12 times.

The argument in one glance

Problem

Segmenting the whole AR frame is too slow for a headset

Augmented reality works by understanding the real scene in front of the user well enough to overlay digital information onto it, and a foundational step is image segmentation: dividing the camera frame into meaningful parts by identifying and isolating specific objects, pixel by pixel. This is what lets an AR system extract precise digital information from the world — knowing exactly which pixels belong to the cup, the sign, or the person the user is interacting with. The problem is that segmentation is computationally heavy, especially on the high-resolution frames a good AR experience demands, and AR devices are lightweight, battery-powered, and thermally constrained. Running a full segmentation model over the entire frame imposes substantial computation and long processing delays, and in a real-time system that latency directly degrades the experience — the digital overlay lags the real world, breaking the illusion and, at worst, causing discomfort. So AR faces a bind: segmentation is essential to understanding the scene, but doing it over the whole frame at full resolution is exactly the kind of workload these small devices cannot afford. The natural question is whether all of that work is even necessary.
What people did

Run a full-frame segmentation model on the device

The standard approach is to take the AR device's front-facing camera frame and feed it to an image-segmentation neural network that processes the entire scene, producing a labeled map of every object in view. To make this feasible on constrained hardware, systems shrink the model, lower the resolution, or add accelerators, trying to get a whole-frame segmentation to run fast enough. The assumption throughout is that the system should segment everything in the field of view, because it does not know in advance which objects the user cares about.
Why it fell short

Most of that segmentation work is wasted on what you ignore

Segmenting the whole frame spends nearly all of its effort on regions the user is not attending to. Human vision does not take in the whole scene at once: people fixate on one region, focus on a single object of interest, and only then shift their gaze elsewhere. At any given moment, the user is really only engaging with one instance in their field of view, yet the full-frame model painstakingly segments every object, including everything in the periphery that the user is ignoring. Shrinking the model or dropping resolution to fit the device does not fix this — it just makes the wasteful whole-frame computation lower quality, trading accuracy for speed rather than removing the waste. The result is a system that pays for segmenting the entire scene when it only needs the one object the user is looking at. What was missing was a way to exploit where the user is actually looking — information an AR headset already has, because it tracks the eyes — to segment only the instance of interest, and to do so with an algorithm and hardware designed together for that narrower task rather than a generic whole-frame model squeezed onto the device.
This paper's idea

Segment only where you look, co-designed end to end

This work reduces the cost of AR segmentation by leveraging natural human eye dynamics: it segments only where you look (SOLO). Because a user focuses on just one region before shifting perspective, the system uses the headset's gaze information to concentrate segmentation on the single instance of interest instead of the whole frame. It is a co-optimization of the segmentation algorithm and the underlying hardware. The SOLO algorithm is an efficient deep-learning framework that takes both the high-resolution scene image and the user's eye images, and from them segments only the object the user is attending to, rather than everything in view. This is paired with two hardware pieces integrated into the AR device's system-on-chip: saliency-based sensing (SBS), which uses where the user is looking to sense the important part of the scene efficiently, and a dedicated SOLO accelerator, added as a plug-in to the SoC to run the framework fast. By focusing the expensive segmentation on just the gazed-at instance and co-designing the sensing, algorithm, and accelerator around that goal, SOLO significantly lowers the computational cost of segmentation, achieving up to a 12-times reduction in end-to-end latency compared with baselines.

Why this problem is worth solving

Segmentation is the perception step that makes AR possible, and its latency is what separates a convincing overlay from a laggy, unusable one on the small devices people will actually wear.

For augmented reality to feel real, the digital content has to be anchored to the physical world precisely and without perceptible delay, and that requires the device to understand the scene — which objects are where — in real time. Image segmentation is the workhorse of that understanding, isolating objects so the system can attach information to them. But AR’s value depends on running on light, wearable, low-power hardware, and full-scene segmentation is one of the heaviest vision workloads there is, so it collides head-on with the device’s constraints. When segmentation is too slow, the overlay lags the world, the interaction feels broken, and prolonged latency can even cause discomfort — so the perception latency directly caps how usable and comfortable an AR system is. This makes efficient segmentation not a nice-to-have but a gating requirement for practical AR. And because the workload scales with resolution and scene complexity, both of which keep rising, an approach that fundamentally reduces how much segmentation is done — rather than just making a full-frame model marginally faster — is what keeps AR viable as the visual demands grow. Cutting end-to-end latency by an order of magnitude, while still delivering the object the user cares about, is exactly the kind of improvement that moves AR from demo to product.

The key realization is that the user only ever engages one object at a time, and the headset already knows which one, so almost all full-frame segmentation is wasted.

The reason full-frame segmentation is so wasteful is a fact about human perception: we do not process our whole visual field uniformly, but fixate on one small region of interest at a time and shift attention serially. At any instant, the user is interacting with a single instance, and everything else in view is, for that moment, irrelevant to what they are doing. A conventional system cannot exploit this because it does not know where the user is looking — so it hedges by segmenting everything. But an AR headset is unusual: it has eye-tracking cameras, so it does know where the user is looking, in real time. That gaze signal is exactly the information needed to prune the work from the whole scene down to the one instance that matters. The insight of SOLO is to feed that signal into the segmentation itself — taking the eye images alongside the scene image — so the model can concentrate on the gazed-at object and skip the rest, which is where the order-of-magnitude savings come from. The subtlety, and why it is a co-design rather than just a smaller model, is that realizing the savings requires the whole pipeline to cooperate: the sensor should capture the salient region efficiently, the algorithm should be built to segment a single instance from the scene-plus-gaze inputs, and a dedicated accelerator should run it. Matching the computation to what the human is actually attending to, using a signal the device already has, is what turns a heavyweight full-frame task into a lightweight targeted one.

What people did before, and where it broke

Segment the entire frame with a full model. Run a segmentation network over the whole high-resolution scene. Where it broke. It is computationally heavy and slow on constrained AR hardware, adding latency that degrades the real-time experience.
Shrink the model to fit the device. Use a smaller or lower-resolution segmentation network. Where it broke. It still segments the whole frame, just at lower quality, trading accuracy for speed without removing the wasted work.
Add generic acceleration. Throw an accelerator at the whole-frame model. Where it broke. It speeds up the same wasteful computation rather than reducing how much segmentation is actually needed.
Ignore where the user is looking. Treat every object in view as equally important to segment. Where it broke. The user attends to one instance at a time, so segmenting everything spends nearly all its effort on ignored regions.
The gap. No approach used the headset’s available gaze signal to segment only the instance of interest, co-designing the sensing, algorithm, and hardware around that narrower task instead of forcing a generic whole-frame model onto the device.

How SOLO works

From whole-frame segmentation to gaze-targeted, co-designed segmentation

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Do the expensive work only where it matters, not everywhere. SOLO’s foundation is to stop segmenting the whole frame and instead spend computation only on the single instance the user is attending to, pruning away the rest of the scene. Concentrating a costly real-time visual computation on the parts that actually matter and skipping the rest, rather than processing everything uniformly, is the same principle the work on real-time neural rendering with efficiency-aware pruning (MetaSapiens) covered elsewhere on this site is built on, cutting work by not computing what will not be seen or used. SOLO prunes at the granularity of whole objects, keeping only the gazed-at instance.

Co-design the algorithm with dedicated hardware for the AR device. SOLO is not just a lighter model but an algorithm paired with a purpose-built accelerator plugged into the AR SoC, so the targeted segmentation runs efficiently on the constrained headset. Providing architectural support tailored to a specific, heavy visual workload for AR and VR, rather than running it on generic hardware, is the same approach the work on efficient radiance-field rendering via architectural support (GSCore) covered elsewhere on this site takes, co-designing an accelerator with the visual algorithm. SOLO’s accelerator and framework are designed together for gaze-targeted segmentation.

Sense adaptively, capturing only the important region. Through saliency-based sensing, SOLO adapts what the sensor captures and processes to where the user is looking, so it does not pay to fully sense and read out the whole scene. Adapting sensing and sampling to focus effort on the informative regions, rather than uniformly capturing and processing all input, is the same lever the work on exploiting adaptive sampling and data reuse (ASDR) covered elsewhere on this site pulls, reducing cost by sampling where it counts. SOLO’s saliency-based sensing uses gaze to decide where sensing effort should go.

Following a frame through SOLO

Segmenting the object the user is looking at

An illustration, drawn from the paper's abstract, of how SOLO targets segmentation by gaze. Internal mechanism details are the paper's.

Step 1 of 5

The headset captures scene and eye images

The AR device's front camera captures the high-resolution scene, and its eye-tracking cameras capture the user's eyes, giving SOLO both the scene and where the user is looking.

The gaze signal is already available on an AR headset.

Why it works

The user engages one instance at a time. Full-frame segmentation labels every object → but a person fixates on a single region of interest → so only that one instance needs to be segmented at any moment.
Gaze is a signal the headset already has. A generic system cannot tell what the user cares about → but an AR headset tracks the eyes in real time → so the object of interest can be identified without extra sensors.
Targeting the instance removes the wasted work. Shrinking a whole-frame model just lowers quality → but segmenting only the gazed-at object → so the effort spent on ignored regions is eliminated, not merely cheapened.
Adaptive sensing cuts cost before compute. Fully sensing the whole frame is expensive → but saliency-based sensing captures the important region → so data and sensing cost drop ahead of segmentation.
Co-designed hardware runs the targeted task efficiently. A generic accelerator speeds the same wasteful work → but an accelerator built for gaze-targeted segmentation → so the narrow computation executes fast on the constrained device, netting up to 12-times lower latency.

What it costs (trade-offs and scope)

It requires eye tracking and gaze information. SOLO’s savings come from using where the user looks; it depends on the AR device having reliable, real-time eye tracking, without which the targeting cannot work.
It requires hardware integration. The gains rest on saliency-based sensing and a SOLO accelerator plugged into the SoC, so it is a hardware-software co-design, not a software-only model swap.
It segments the instance of interest, not the whole scene. By design, SOLO produces the gazed-at object rather than a full-scene labeling; applications that need every object segmented at once would not be served the same way.
Gaze shifts drive re-segmentation. When the user looks at a new object, the system must segment the new instance; the benefit assumes users engage one instance at a time, as human vision does.
Reported figures are against specific baselines. The up-to-12-times end-to-end latency reduction is measured against particular baselines on the paper’s setup; the exact gain depends on the scene, resolution, hardware, and comparison points.

The comparison

Segmenting the scene for an AR device

Property	Full-frame model	Shrunk full-frame model	SOLO (gaze-targeted)
What is segmented
Uses gaze
Sensing
Hardware
End-to-end latency

A comparison of ways to segment the scene for an AR device. A full-frame model segments every object and is slow; shrinking it just lowers quality while still processing the whole frame. SOLO uses the headset's gaze to segment only the instance the user is looking at, with saliency-based sensing and a dedicated accelerator co-designed for the task, cutting end-to-end latency by up to 12 times. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s abstract and public metadata, not its full text, which was not available at the time of writing. The high-level claims are the authors’: the work reduces the high computational cost of image segmentation in augmented reality by leveraging natural human eye dynamics to segment only where you look (SOLO), co-optimizing the segmentation algorithm with the underlying hardware. The SOLO algorithm is described as an efficient deep-learning framework that takes high-resolution input images and user eye images to segment only the instance of interest; it is integrated with saliency-based sensing (SBS) and a SOLO accelerator, added as a plug-in for the AR device’s system-on-chip, and reported to achieve up to a 12-times reduction in end-to-end latency compared with baselines. The background above on augmented reality, image segmentation, gaze and foveation, eye tracking, and adaptive sensing is standard context provided to make the approach intuitive. The Flow, the step-by-step walkthrough, and the “Why it works” reasoning are illustrations built from the abstract’s description, not detailed accounts from the paper — the exact SOLO network design, how saliency-based sensing works, the accelerator microarchitecture, and the full evaluation should be read from the paper. (A related, separately-titled paper on foveated instance segmentation from an overlapping group exists; this page describes only what the SOLO paper’s abstract states and does not import that other work’s details.) Do not treat specific procedural details here as established beyond what the abstract states.

Background

Why efficient AR and VR keeps coming down to not computing what the user will not perceive

It helps to see SOLO within a broad pattern in augmented and virtual reality systems: the visual workloads are enormous and the devices are tiny, so much of the field’s progress comes from exploiting the limits and habits of human perception to avoid computing what the user will never actually perceive or attend to. The eye is not a uniform camera: it sees fine detail only at the center of gaze, attends to one thing at a time, and tolerates approximation in the periphery and during motion. Systems that recognize this can render, stream, or process far less while the user notices no difference — foveating the rendering, compressing what the eye cannot resolve, reusing frames that barely change, or, as here, segmenting only the object being looked at. The unifying move is to treat human perception as a specification for how much work is actually required, and then to co-design the algorithm and hardware to deliver exactly that. SOLO is a clean instance for scene understanding: it uses the gaze signal an AR headset already produces to shrink a whole-frame segmentation to a single instance. This drive to slash the cost of a heavy AR/VR visual workload by exploiting what the user perceives, and co-designing the system around it, is the same terrain the work on low-bandwidth VR streaming that exploits perceptual redundancy (Pirate) covered elsewhere on this site occupies, cutting cost without cutting the perceived experience. SOLO’s particular contribution is to bring this to segmentation, showing that gaze-targeted, co-designed segmentation can reduce end-to-end latency by up to an order of magnitude while still giving the user the object they care about.

If you remember only three things

1. Augmented reality relies on image segmentation — isolating objects in the camera frame pixel by pixel — to understand the scene and anchor digital overlays to it. But running a segmentation model over the whole high-resolution frame is one of the heaviest vision workloads, and AR devices are lightweight, low-power, and thermally limited, so full-frame segmentation adds substantial computation and processing delay that degrades the real-time experience. Shrinking the model or lowering resolution to fit the device does not fix the underlying waste: it still segments the entire scene, just at lower quality. And nearly all of that effort is spent on regions the user is not attending to, because human vision fixates on one object of interest at a time before shifting gaze.
2. This work reduces the cost by segmenting only where you look (SOLO), exploiting the fact that an AR headset already tracks the user's eyes. The SOLO algorithm is an efficient deep-learning framework that takes both the high-resolution scene image and the user's eye images and segments only the single instance the user is attending to, rather than the whole frame. It is a co-design of algorithm and hardware: saliency-based sensing (SBS) uses the gaze to sense the important region of the scene efficiently, and a dedicated SOLO accelerator, plugged into the AR device's system-on-chip, runs the targeted segmentation fast. By concentrating the expensive computation on just the gazed-at object and co-designing the sensing, algorithm, and accelerator around that goal, SOLO removes the wasted work rather than merely cheapening it.
3. The paper reports that SOLO achieves up to a 12-times reduction in end-to-end latency compared with baselines, while still delivering the object the user cares about. The trade-offs and scope: it depends on the AR device having reliable real-time eye tracking, it requires hardware integration (saliency-based sensing plus the accelerator) rather than being a software-only change, it segments the instance of interest rather than the whole scene (so applications needing every object at once are served differently), and gaze shifts trigger re-segmentation of the new instance. This explainer is based on the paper's abstract; the network, sensing, accelerator, and full evaluation should be confirmed in the paper.

Check yourself

Check your understanding

1. Why is full-frame image segmentation a problem on an AR device?
2. What human-perception insight does SOLO exploit, and why is it useful here?
3. How is SOLO structured, and why is it a co-design rather than just a smaller model?
4. What does SOLO achieve, and what are the limits of the approach?

Sources

The paper is “Segment Only Where You Look: Leveraging Human Gaze Behavior for Efficient Computer Vision Applications in Augmented Reality” by Tianhua Xia, Haiyu Wang, and Sai Qian Zhang (New York University), published at ASPLOS 2026, DOI 10.1145/3779212.3790216.

This explainer is written from the paper’s abstract and public metadata; the full text was not available at the time of writing, so the page is marked a draft. As reported there, the work reduces the computational cost of image segmentation in augmented reality by segmenting only where the user looks (SOLO), leveraging human eye dynamics and co-optimizing the algorithm with hardware. The SOLO algorithm is an efficient deep-learning framework that takes high-resolution scene images and user eye images to segment only the instance of interest, integrated with saliency-based sensing (SBS) and a SOLO accelerator plugged into the AR device’s system-on-chip, achieving up to a 12-times reduction in end-to-end latency versus baselines. Background on AR, image segmentation, gaze and foveation, and adaptive sensing is standard context. The specific numbers are the paper’s own; the mechanism details and full evaluation should be confirmed in the paper.