A self-driving car's perception tries to see everything around it all the time, spending the same effort whether or not it matters to the current driving decision — unlike a human driver, who focuses attention on what the situation demands; this paper (University of Virginia — Wonkyo Choe, Rongxiang Wang, Felix Xiaozhu Lin) presents AnA, which refactors the driving system so the planning module tells perception what it cares about right now (a query interface), an executor turns that into the best set of perception tasks, and a monitor catches misbehaving tasks using driving knowledge, letting perception distinguish light background awareness from focused attention — halving collisions on hard benchmarks while using 44 percent less compute.
The argument in one glance
-
Problem
A self-driving car perceives everything the same, no matter what it is about to do
An autonomous driving system (ADS) is built from modules: a perception module that senses the world (detecting cars, pedestrians, lanes, and signs from cameras and other sensors), and a planning module that decides what the car should do next. Perception is decisive for safety — if the car does not see a hazard, it cannot avoid it. The trouble is that in today's systems, perception runs oblivious to the driving decisions being made. It tries to perceive everything around the car, all the time, with the same uniform effort, regardless of what the current situation actually needs. That is not how humans drive. A human approaching a busy intersection concentrates on the cross-traffic they are about to cut across, while only loosely tracking the parked cars behind them; they direct their attention according to what they are about to do. A machine that spreads its perception evenly either wastes computation on things that do not matter right now, or fails to concentrate enough on the one event that is about to become critical. Perception that ignores the driving decision is both inefficient and, at the worst moment, dangerously unfocused.
-
What people did
Run all perception tasks continuously, and make each one faster
The standard approach is to run the full perception pipeline continuously — every detector and tracker, over the whole scene, every cycle — and to invest in making each perception task individually faster or more accurate, so the system can keep up with real-time deadlines while covering everything.
-
Why it fell short
Uniform perception is not proportional to what driving needs
Running everything all the time, and only speeding up each task, never fixes the underlying mismatch: the effort perception spends is not proportional to what the driving decision requires. Because perception does not know what the planner cares about, it cannot spend more where it matters and less where it does not, so compute is wasted on irrelevant parts of the scene while genuinely important, fast-developing events may not get the concentrated, timely attention they need. Making each task faster raises the ceiling but keeps the same flat allocation. The missing ingredient is a link between planning and perception — a way for the driving decision to guide perception in the moment — and a way for perception to behave like human attention, keeping a light general awareness of the whole scene while focusing intense effort only where the situation demands. Without that, an ADS is stuck being both wasteful and, when it counts, insufficiently attentive.
-
This paper's idea
Let driving decisions steer perception, and separate awareness from attention
AnA refactors the autonomous driving system around two ideas: (1) the system guides its perception using driving knowledge in the moment, so what the planner is about to do shapes what perception focuses on; and (2) perception distinguishes awareness — a light, broad background monitoring of the whole scene — from attention — concentrated, higher-effort perception aimed at what currently matters. To realize this, AnA adds three mechanisms. First, a query interface through which the planning module expresses its interest in perception — in effect, telling perception what it needs to know right now. Second, a query executor that maps each such query to an optimal set of perception tasks, turning a high-level interest into the concrete detectors and trackers that best answer it. Third, a monitor that handles abnormal task executions using driving knowledge, so when a perception task behaves unexpectedly the system responds safely rather than blindly. The authors call the resulting behavior attentive driving, a human-like pattern that makes the system's effort proportional to what the situation requires. On challenging driving benchmarks, AnA responds to adversarial events in time, reducing collisions by 2 times (halving them), and cuts compute usage by 44 percent without compromising driving safety.
Why this problem is worth solving
Perception is where a self-driving car spends much of its computation and stakes its safety, so how that effort is allocated matters enormously. An autonomous driving system is a pipeline: sensors feed a perception module that builds an understanding of the surroundings — where the other vehicles, pedestrians, lanes, and signs are and how they are moving — and that understanding feeds a planning module that decides the car’s next action, which a control module then executes. Perception is the foundation; every downstream decision is only as good as what the car perceived. It is also expensive: running many deep-learning detectors and trackers over high-resolution sensor data, fast enough to react in real time, consumes a large amount of on-board compute and energy, which is limited in a vehicle. Today’s systems run this perception uniformly, processing the whole scene with the same effort on every cycle, independent of what the car is doing. That is safe-sounding but wasteful, and it has a subtler failure mode: because effort is spread evenly, the system may not concentrate enough on the specific, rapidly-developing hazard that is about to matter most. Getting perception’s effort allocation right therefore affects both the compute budget (and thus cost, range, and thermals) and, critically, whether the car reacts to the dangerous event in time. This is why treating perception as a resource to be directed, rather than a fixed uniform load, is worth doing.
The deeper issue is that human driving already shows the answer, and machines were not doing it. Human drivers do not perceive their surroundings uniformly. They maintain a loose, low-effort awareness of the whole scene — a general sense of what is around — while directing intense, focused attention to whatever their current maneuver demands: the gap they are merging into, the pedestrian stepping toward the curb, the car braking ahead. Crucially, this focus is guided by intent: what you are about to do determines where you look. This is efficient (you do not scrutinize everything at once) and safe (you concentrate exactly where risk is concentrating). An autonomous driving system whose perception is oblivious to the planning module cannot do this. The planner knows the car is about to turn left across traffic, but perception does not, so perception cannot shift effort to the oncoming lane. And perception has no notion of awareness versus attention, so it either treats everything as important (wasteful) or applies a fixed budget everywhere (unfocused). Closing this gap means two things: creating a channel so the driving decision can steer perception in situ, and restructuring perception so it can hold light awareness broadly and spend heavy attention narrowly. That is the human-like behavior AnA sets out to give a machine.
What people did before, and where it broke
- Run the full perception pipeline continuously. Execute every detector and tracker over the whole scene every cycle, so nothing is missed. Where it broke. Effort is spread uniformly and is not proportional to the driving decision, so compute is wasted on irrelevant regions while critical, fast-developing events get no extra concentration.
- Speed up or improve individual perception tasks. Invest in making each detector faster or more accurate to meet real-time deadlines. Where it broke. It raises the performance ceiling but keeps the same flat allocation; the mismatch between effort and need remains.
- Keep perception and planning separate. Design perception to produce a general world model, independent of what the planner intends. Where it broke. Perception never learns what the planner cares about right now, so it cannot direct effort toward the maneuver the car is about to make, unlike a human driver guided by intent.
- Treat all perceived objects as equally important. Give the whole scene one uniform level of processing. Where it broke. There is no distinction between broad awareness and focused attention, so the system is simultaneously wasteful on the unimportant and under-focused on the important.
- The gap. No autonomous driving system let the driving decision guide perception in the moment while letting perception separate light, broad awareness from concentrated attention — making effort proportional to what driving actually requires.
How AnA works
Turning a driving decision into focused perception effort
-
-
new
-
new
-
new
-
new
-
new
The idea
Direct perception effort by what the situation needs, like human attention. The core of AnA is to stop treating perception as a uniform load and start treating it as a resource directed by the driving task — keeping broad awareness cheap while spending concentrated attention only where it matters. Scaling perception effort to the difficulty and importance of the moment, rather than running a fixed heavy pipeline all the time, is the same instinct the work on an adaptive runtime for visual-inertial simultaneous localization and mapping covered elsewhere on this site follows, matching work to what the situation actually demands. AnA applies that instinct to a full driving system, using the planner’s intent as the signal for where to focus.
Turn a high-level need into the right set of real-time tasks. AnA’s query executor takes the planner’s expressed interest and maps it to an optimal set of perception tasks — deciding, under real-time constraints, which detectors and trackers to run and where. Dynamically choosing which of many machine-learning tasks to run to satisfy a real-time goal, instead of running a fixed set, is the same problem the work on a dynamic scheduler for real-time multi-model machine-learning workloads covered elsewhere on this site tackles, selecting and scheduling models to meet deadlines. AnA specializes this to perception, where the driving query defines what the optimal task set should accomplish.
Make compute proportional to what driving requires. By focusing effort where it matters and easing off elsewhere, AnA achieves resource proportionality — cutting 44 percent of compute while preserving safety, because the saved work was on things the driving decision did not need. Spending scarce compute in proportion to the value or urgency of the work, rather than uniformly, is the same principle the work on quality-of-service-aware and resource-efficient multi-stage GPU serving covered elsewhere on this site is built on, allocating resources to where the required quality demands them and reclaiming the rest. AnA brings that proportionality to on-vehicle perception, guided by the driving task.
Following a left turn through AnA
Perceiving attentively as the car prepares to turn across traffic
An illustrative walk-through, based on the paper's abstract, of how AnA focuses perception for one maneuver. Exact internal mechanisms should be read from the full paper.
Step 1 of 5
The planner decides to turn left across oncoming traffic
The planning module intends a left turn, which makes the oncoming lane the most safety-critical thing to perceive right now. In a conventional system, perception would not know this and would keep processing the whole scene uniformly.
The driving decision defines what matters, but perception is normally unaware of it.
Why it works
- Intent tells perception where risk is concentrating. The planner knows the maneuver the car is about to make → so a query interface lets it point perception at the relevant part of the scene → so effort lands exactly where the next danger is likely, as with a human driver.
- Separating awareness from attention frees up effort. Not everything needs heavy processing every cycle → but broad awareness can be kept light while attention is concentrated narrowly → so compute is spent proportionally, cutting 44 percent without losing safety.
- Mapping queries to an optimal task set focuses the work. A high-level interest does not name specific detectors → but the query executor selects the best set of perception tasks to answer it → so the system runs the right work rather than all work.
- Concentrated attention improves timeliness on hazards. Uniform perception can miss a fast-developing event → but focusing on the query-highlighted region gives it timely, confident perception → so the car responds to adversarial events in time, halving collisions.
- A monitor keeps the focused scheme safe. Directing effort narrowly risks trusting a misbehaving task → but a monitor catches abnormal executions and applies driving knowledge → so the system responds safely instead of acting on a faulty perception.
What it costs (trade-offs and scope)
- This explainer is from the abstract only. The full paper was not available when this page was written, so the descriptions of the query interface, the query executor, and the monitor are at the level the abstract states; their internal algorithms, the benchmarks, and the exact experimental setup are not reproduced here and should be read from the paper.
- Focusing attention means deliberately under-processing the rest. By design, AnA keeps most of the scene under light awareness; this is what saves compute, but it rests on the queries and awareness correctly capturing everything that could become important, so the quality of that guidance is where the risk lives.
- It adds coupling between planning and perception. The query interface ties perception to the planner’s intent; this is the source of the benefit, but it also makes perception depend on the planner asking the right questions, and the monitor exists precisely because focused, query-driven perception needs a safety net.
- The numbers are on specific benchmarks and baselines. The 2-times collision reduction and 44 percent compute saving are measured on the paper’s challenging driving benchmarks against competitive baselines; results depend on those scenarios, the baselines, and the safety metrics used, and real-world behavior can differ.
- It restructures the driving software, not the sensors or hardware. AnA is a refactoring of how the driving system allocates perception; it does not change the underlying sensors or add hardware, so its gains come from smarter allocation rather than more capable perception primitives.
The evaluation
Allocating perception effort in a self-driving car
| Property | Uniform perception | Faster individual tasks | AnA (attentive) |
|---|---|---|---|
| Perception guided by the driving decision | |||
| Separates awareness from attention | |||
| Effort proportional to what driving needs | |||
| Response to adversarial events | |||
| Compute usage |
Check yourself
Check your understanding
1. What is the core problem AnA identifies with perception in today's autonomous driving systems?
2. What are AnA's three mechanisms?
3. How can AnA both reduce collisions and use less compute at the same time?
4. Predict: the planner signals it is about to change lanes to the right. Under AnA, how does perception respond, and why is a monitor still needed?
Sources
The paper is “AnA: An Attentive Autonomous Driving System” by Wonkyo Choe, Rongxiang Wang, and Felix Xiaozhu Lin (University of Virginia), published at ASPLOS 2025, DOI 10.1145/3669940.3707261. It is licensed CC BY.
This explainer is written from the paper’s abstract only; the full text was not available at the time of writing, so this page is marked as a draft. According to the abstract, in an autonomous driving system (ADS) the perception module is crucial to driving safety and efficiency, but perception in today’s ADS remains oblivious to driving decisions, contrasting with how humans drive. The paper’s idea is to refactor the ADS so that (1) the ADS guides its perception with the driving knowledge in situ, and (2) the perception differentiates between awareness and attention. AnA has three novel mechanisms: (1) a query interface for the planning to express its interest in perception; (2) a query executor that maps queries to an optimal set of perception tasks; and (3) a monitor for handling abnormal task executions with driving knowledge. On challenging driving benchmarks, AnA outperforms competitive baselines: it responds to adversarial events in a timely manner, reducing collisions by 2 times; it reduces compute usage by 44 percent without compromising driving safety. The authors attribute AnA’s efficacy to its attentive driving, a human-like behavior that improves resource proportionality. No numbers or mechanisms beyond the abstract have been introduced here; the internal designs of the three mechanisms, the benchmarks and baselines, and the full experimental methodology should be confirmed in the full paper.