Active Visual Perception: Opportunities and Challenges

Yian Li; Xiaoyu Guo; Hao Zhang; Jiayu Chen; Shuiwang Li; Xiaowei Dai

cs.CVDec 3, 2025

Active Visual Perception: Opportunities and Challenges

Authors: Yian Li, Xiaoyu Guo, Hao Zhang, Jiayu Chen, Shuiwang Li, Xiaowei Dai

Organizations: College of Computer Science and Engineering, Guilin University of Technology, Guilin 541004, China · New Engineering Industry College, Putian University, Putian 351100, China

Abstract

Active visual perception enables an intelligent system to improve perception by controlling how visual evidence is acquired. However, differences in how active perception is defined and evaluated make approaches difficult to compare across tasks. This review clarifies the field's functional scope by distinguishing active observation acquisition from processing fixed inputs. It then organizes representative literature around four questions about what evidence the task requires, which actions can obtain it, how candidate observations should be valued, and how the next sensing action should be selected and executed. New observations update the perceptual state and guide subsequent decisions, forming a closed loop. To evaluate the benefits of active perception, the review examines datasets, interactive environments, physical platforms, and baseline comparisons, and develops an evaluation framework linking improvements in visual evidence to downstream task outcomes while accounting for acquisition costs and risks. The analysis concludes that predicted observation value does not guarantee that useful evidence can be acquired through feasible actions, while task success alone cannot establish the benefit of active observation selection. Further progress requires reliable value estimation and planning over longer decision sequences under physical and resource constraints. It also requires grounding semantic reasoning in acquired evidence and maintaining effective collaboration under imperfect communication, with these capabilities verified through reproducible evaluation of the complete perception and action loop.

Figures & tables

Figure 1: Conceptual framework of active visual perception linking perception targets, sensing actions, decision objectives, decision mechanisms, and application domains.

Review	Primary Scope	Observation Control	Decision Emphasis	Evaluation / Constraints
Bajcsy et al. Bajcsy et al. (2018)	Conceptual foundations of active perception	What, where, when, and how to sense	Task and context dependent sensing	Resource constraints discussed conceptually
Chen et al. Chen et al. (2011)	Active vision in robotic systems	Sensor pose, configuration, and viewpoint planning	Uncertainty and task-oriented sensor planning	Robot configurations, applications, and method evaluation
Zeng et al. Zeng et al. (2020)	View planning for robotic active vision	Primarily sensor viewpoint and view sequence	View quality and planning strategies	Hardware conditions, scanning state, and planning efficiency
Ahmed et al. Ahmed et al. (2023)	Active and collaborative SLAM	Robot trajectory and control action selection	Pose and map uncertainty, information utility	Planning methods, deployment settings, and SLAM performance
This review	Cross-task active observation acquisition	Viewpoint, gaze, sensing, interaction, and coordination	Required evidence, observation value, and action selection	Evidence improvement, task improvement, feasibility, and resource cost

Table 1: Comparison between representative previous reviews and the analytical scope of this review.

Figure 2: Closed-loop process linking perceptual needs, sensing actions, action evaluation, execution, and state updates.

Figure 3: Representative milestones in active visual perception organized by observation control mechanism.

Literature Area	Representative Search Terms
Foundations and prior reviews	Active vision; active visual perception; active sensing; revisiting active perception
Next best view and active reconstruction	Next best view; active 3D reconstruction; active view planning; viewpoint selection for reconstruction
Active SLAM and exploration	Active SLAM; active visual SLAM; information gain for active SLAM; semantic active SLAM; active exploration
Visual search, recognition, and navigation	Active visual search; embodied visual search; active object recognition; active perception for object navigation
Camera control, gaze, and tracking	Active camera control; pan tilt zoom camera tracking; gaze control; active viewpoint tracking; multi-camera tracking
Robotic manipulation and interactive perception	Active perception for robotic manipulation; manipulation under occlusion; interactive perception; active viewpoint selection for manipulation

Table 2: Literature areas and representative search terms used in this review.

Decision Mechanism and Representative Studies	Required Information	Suitable Sensing Decisions	Evaluation Setting and Main Limitation
Rule-based and geometric Frintrop and Jensfelt (2008) ; Rivlin and Rotstein (2000) ; Dunn and Frahm (2009) ; Yang and Waslander (2022)	Designed states, geometric models, visibility, and reachability	Gaze control, camera pose, viewpoint, and platform motion	Simulation evidence. Evaluation is limited to simulation, where performance depends on designed rules, geometric accuracy, and idealized visibility. Physical-platform validation. Not reported; robustness to calibration errors, dynamic occlusion, material effects, and actuation uncertainty remains unverified.
Probabilistic and belief-based Laporte and Arbel (2006) ; Kim and Eustice (2012) ; Kitanov and Indelman (2024)	State beliefs, observation likelihoods, and uncertainty models	Viewpoint, revisit action, path, and sensing sequence	Simulation evidence. Evaluation is limited to simulation, where computation increases with belief dimensionality and planning horizon and depends on assumed noise models. Physical-platform validation. Not reported; robustness to sensor failures, likelihood miscalibration, measurement noise, and real-time constraints remains unverified.
Information-based optimization Mu et al. (2016) ; Lauri et al. (2020) ; Bai et al. (2024)	Calibrated state uncertainty and predicted observation updates	Viewpoint, waypoint, sensor, and measurement selection	Simulation evidence. Evaluation is limited to simulation, where observation values inherit errors from idealized belief and sensor models. Physical-platform validation. Not reported; whether predicted information gain is preserved under sensor noise, model mismatch, and execution delay remains unverified.
Search and receding horizon planning Leung et al. (2006) ; Chen et al. (2020) ; Wang et al. (2023) ; Taioli et al. (2024)	Candidate actions, transition models, and predicted future observations	View sequences, exploration routes, and continuous camera motion	Simulation evidence. Evaluation is limited to simulation, where computation grows with the action space and prediction depth and future observations are often idealized. Physical-platform validation. Not reported; robustness to planning latency, motion uncertainty, dynamic obstacles, and reachability constraints remains unverified.
Supervised and imitation learning Mendoza et al. (2020) ; Xiong et al. (2025) ; Chen et al. (2024)	View labels, expert values, or demonstrations	Viewpoint, gaze, zoom, and camera behavior	Simulation evidence. Evaluation is limited to simulation, where policies inherit training-data coverage limits and may overfit simulated appearance or action discretization. Physical-platform validation. Not reported; robustness to changes in lighting, camera calibration, motion blur, and actuator response remains unverified.
Reinforcement learning Peralta et al. (2020) ; Placed and Castellanos (2020) ; Bartolomei et al. (2021) ; Huang et al. (2023)	Interaction experience, rewards, and environment dynamics	Path, viewpoint, gaze, and task coupled sensing actions	Simulation evidence. Evaluation is limited to simulation, where policies remain sensitive to reward design and may exploit simulator-specific appearance or dynamics. Physical-platform validation. Not reported; sim-to-real transfer, safe exploration, control delay, and reliability under physical disturbances remain unverified.

Table 3: Comparison of representative mechanisms for selecting sensing actions in active visual perception. The reported simulation limitations and the status of physical-platform validation are distinguished.

Task	Representative Resources	Support for Active Evaluation	Main Limitation
Active reconstruction Dunn and Frahm (2009) ; Peralta et al. (2020) ; Chen et al. (2024) ; Dhami et al. (2023)	Synthetic meshes, ShapeNet style objects, Middlebury data, custom CAD models, and real object collections	Accurate geometry and controllable views support coverage, completeness, and view contribution analysis	Viewpoints are often predefined and ignore reachability, collision, and motion error
Active localization and exploration Ahmed et al. (2023) ; Placed et al. (2022) ; Bonetto et al. (2022)	KITTI, Freiburg, Intel Research Lab, MIT Stata Center, custom simulators, and mobile robots	Interactive settings support route, revisit, coverage, and loop closure decisions	Recorded trajectories cannot evaluate action selection, while custom environments limit direct comparison
Search and navigation Ammirato et al. (2017) ; Savva et al. (2019) ; Deitke et al. (2020)	Active Vision Dataset, Habitat, RoboTHOR, and related indoor embodied environments	Policies can change position and gaze and acquire task dependent observations	Results depend on annotations, action discretization, sensor assumptions, and the perception backbone
Active tracking Cai and Medioni (2016) ; Chen et al. (2024) ; Casao et al. (2024)	Controllable camera networks, pan tilt zoom systems, simulated targets, and physical video platforms	Camera orientation, assignment, and target handover can affect future visibility	Custom target dynamics and camera models reduce comparability across studies
Robotic manipulation Jauhri et al. (2024) ; He et al. (2026) ; Liu et al. (2026)	Tabletop simulation, physical manipulators, EFM10, ActiveViewPose 200K, and ActiveManip Bench	Viewpoint and interaction actions can be evaluated together with task execution	Perception, sensing, planning, and control gains remain difficult to separate
Collaborative perception Hu et al. (2022) ; Yang et al. (2023) ; Huang et al. (2024) ; Bai et al. (2024)	V2X-Sim, OPV2V, V2XSet, DAIR-V2X, multirobot maps, and camera networks	Supports complementary observation, message selection, and communication cost analysis	Accurate poses, synchronization, and stable communication can overestimate collaborative gains

Table 4: Representative datasets and interactive environments used in active visual perception.

Evaluation Question	Required Baseline or Measurement	What It Reveals	Common Confounding Factor
Is active sensing necessary?	Passive or fixed observation baseline	Whether changing the observation improves perception or the downstream task	The active method may also use a stronger perception model
Is the selection rule informative?	Random and uniform action baselines	Whether task or uncertainty information improves action choice	Different candidate action sets may favour one method
Does planning horizon matter?	Greedy or immediate utility baseline	Whether reasoning about later observations improves the result	Longer planning may use more computation or privileged prediction
Did the observation itself improve?	Visibility, coverage, uncertainty, valid depth, or pose quality before and after the action	The direct contribution of the acquired evidence	The metric may be poorly calibrated or unrelated to the task
Did the task improve?	Reconstruction, localization, search, tracking, manipulation, or detection performance	Whether the new evidence changes the final task outcome	Control quality and the perception backbone also affect the result
Was the improvement efficient?	View count, path length, action count, time, energy, computation, and communication	The resources required to obtain the reported improvement	A combined score can hide which resource caused the difference

Table 5: Baselines and measurements required for closed loop evaluation of active visual perception.

Panel of Figure 4 — Figure 4: Representative active perception processes in robotic manipulation, collaborative driving, and UAV-assisted disaster rescue. In all subfigures, the yellow arrows point to the target, while the boxes in the lower-right corner of panels (a) and (b) show enlarged local views of the region observed from the current viewpoint. (a) Next-best-view selection for robotic manipulation, where the current and candidate viewpoints are distinguished by viewing cones with different colors; the asterisk marks the target, and the check mark indicates the selected viewpoint. (b) Collaborative driving, where the green dashed arrows indicate the sharing of complementary perceptual information between vehicles and between vehicles and the roadside sensor. (c) UAV-assisted disaster rescue, where the dashed arrows schematically illustrate three candidate search paths for acquiring task-relevant visual observations; the green dashed path indicates the selected search path.

Application	Information Deficit	Controlled Action	Expected Evidence	Dominant Constraints
Robotic manipulation	Hidden grasp region, uncertain pose, or unknown object state	Wrist or head motion, base motion, active-view arm, or object interaction	Clearer affordance, improved pose, or interaction-generated state evidence	Collision, reachability, irreversible action, latency, and task coupling
Autonomous driving	Occluded road user, weak visibility, or incomplete local sensing	Camera orientation, adaptive resolution, modality selection, or collaborative query	Target confirmation, complementary geometry, or remote hazard evidence	Real-time response, safety, weather, synchronization, and pose error
Human-centered systems	Occluded body, uncertain gesture, or missing shared-task evidence	Gaze, camera orientation, robot repositioning, or selective resolution	Clearer pose, attention, intention, or shared-object state	Personal space, privacy, interpretability, consent, and social acceptance
Surveillance and security	Blind spot, target exit, identity ambiguity, or limited resolution	Pan, tilt, zoom, camera assignment, or target handover	Centered target, identity-preserving sequence, or complementary views	Target competition, control delay, privacy, and operator accountability
UAV search and rescue	Unknown layout, hidden victim, or inaccessible region	Flight path, altitude, orientation, revisit, or team allocation	Victim evidence, damage map, route information, or wide-area coverage	Battery, collision, urgency, wind, and communication
Environmental monitoring	Sparse anomaly, changing conditions, or uneven target distribution	Route, viewpoint, sensor schedule, or spatially varying sampling	Focused anomaly evidence and long-term change observations	Energy, weather, revisit fairness, and long-term autonomy

Table 6: Perception–action characteristics of representative application settings.

Figure 5: Interacting challenges in the active visual perception loop.

Challenge	Why It Arises	Consequence for Active Perception	Unresolved Gap
State estimation in incomplete and changing scenes	Partial observations, semantic errors, poorly calibrated confidence, and changing occlusion	The system may select actions from an incorrect or outdated state and reinforce its initial error	State representations that preserve task relevance, temporal validity, and calibrated geometric and semantic uncertainty
Action selection under limited time and resources	Long-term action dependence, large action spaces, and expensive prediction and feasibility checking	Greedy decisions can waste the sensing budget, while more extensive planning may become too slow for the task	Planning methods that adapt their horizon and computational effort to task urgency and available resources
Executable sensing on physical platforms	Ideal viewpoints may ignore reachability, embodiment, motion error, and interference with the task action	An informative viewpoint may be unreachable, unsafe, or detrimental to the subsequent operation	Joint reasoning about observation value, platform dynamics, task feasibility, and transfer from simulation to physical systems
Collaborative perception under imperfect communication	Limited bandwidth, transmission delay, pose disagreement, and agent or sensor failure	Shared evidence may become outdated or misaligned, while poor coordination causes redundant sensing or observation gaps	Distributed policies that coordinate motion, sensing, and message selection while remaining reliable under intermittent communication
Safety, privacy, and system responsibility	Active movement, physical interaction, camera redirection, zoom, and persistent observation can affect people and the environment	The system may cause collision, object damage, privacy intrusion, or unjustified actions under unresolved uncertainty	Risk aware stopping, minimum necessary observation, human oversight, and interpretable reasons for sensing actions
Benchmarking, reproducibility, and fair comparison	Studies use different scenes, action spaces, sensing budgets, perception backbones, and model versions	Reported results are difficult to compare, and the source of a performance gain or failure often remains unclear	Shared closed-loop settings, transparent system configurations, and corresponding simulation and physical evaluations

Table 7: Main challenges and unresolved gaps in active visual perception.

Active Visual Perception

Active visual perception studies how agents choose where, when, and how to look to gather task-relevant evidence efficiently.

See all 47 papers

Jul 17, 2026

An Exam for Active Observers

Human vision is a closed loop: gaze is continuously redirected by intermediate hypotheses rather than a single snapshot. Decades of psychophysics and cognitive science have argued that this active observation is essential for a wide range of tasks. Whether today's multimodal large language models (MLLMs) exercise active observation is an empirical question that current vision-language benchmarks do not answer. We introduce ActiveVision, a benchmark that makes active observation measurable for MLLMs, comprising 17 tasks across 3 categories. Tasks are designed to force repeated visual perception rather than a single static description. Frontier MLLMs collapse on ActiveVision: the highest-scoring model we evaluate, GPT-5.5 at the highest exposed reasoning-effort tier, solves only 10.6% of items and scores zero on 11 of the 17 tasks, and even Claude Fable 5, despite topping most reasoning and coding leaderboards, solves just 3.5%, far behind three human participants who average 96.1%. Furthermore, much of the gap persists even when models write and run their own vision code: such code is unreliable on realistic imagery, and catching its failures itself requires the active perception the models lack. Together, these results indicate that current MLLMs lack robust active visual observation, motivating architectures and training objectives that close the perception-reasoning loop.

Jiarui Zhang, Muzi Tao, Shangshang Wang +3

University of Southern California

VLM Evaluation Multimodal Large Language Models Active Visual Perception Active Vision

Ask PDF

Sep 27, 2026

DeltaSeek: Toward Active Perception in Evolving Construction Environments

Construction environments evolve continuously, causing large geometric changes that degrade static mapping and registration performance. This necessitates active perception, where robots deliberately select sensing configurations to resolve the environment's current state. We present DeltaSeek, an initial framework toward active perception in evolving built environments. While our broader objective is a system that reasons about where, how, and when to observe, this paper addresses a critical prerequisite: how a robot's sensing embodiment constrains the observations it can acquire. We formalize an embodiment's permissible observation set and evaluate with a Husky A300 equipped with a UR5e on an IFC-derived benchmark under chassis-mounted and wrist-mounted RGB-D configurations, scoring observations by geometric visibility and effort by drivable distance. In a room-scale scene with eight controlled changes spanning four observability conditions, exhaustive evaluation over 240 permissible base poses and five arm postures shows that two changes admit no chassis viewpoint whatsoever, while the wrist camera resolves both. For changes observed by both embodiments, the median base travel is

6.0

~m for the wrist camera and

15.2

~m for the chassis camera. These results distinguish sensing limitations from acquisition costs, clarifying whether an observation is impossible or simply requires more travel.

Sanjay Acharjee, Md Nazmus Sakib

Department of Civil Engineering, University of Texas at Arlington, TX 76019, USA.

Robotic Construction Active Visual Perception

Ask PDF

Oct 1, 2026

ActiveWAM: Evidence-Aware Active Vision for World-Action Models

Active vision manipulation requires a policy to control both its camera and its end-effectors, yet camera motion determines which evidence remains visible within finite observation windows. Acquiring a new view can displace task-critical cues, while retaining a view forgoes potentially useful observations. We formulate this as an evidence-aware retain--acquire problem and present ActiveWAM, a unified world--action model that learns observation and manipulation jointly. To this end, we propose training-time inversion which constrains a frozen video prior by task-bearing source evidence and visible temporal changes, eliminating the need for test-time inversion or candidate ranking. At deployment, the policy generates bimanual and pan/tilt actions-including stay and reacquisition behaviors-from view-aware history, and updates context from newly measured RGB observations. Future-video prediction serves as a co-training signal, while action generation requires neither future-video decoding nor optimal viewpoint annotations. We introduce RoboTwin-AV, a 50-task benchmark with executable pan/tilt control and automatically generated demonstrations. ActiveWAM improves TAVIS out-of-distribution success by up to 17.0 percentage points over the strongest baselines, achieves 20.0 additional points over Fast-WAM on RoboTwin-AV, and outperforms it by 26.7 points on real-world physical kitchen tasks.

Renjun Wu, Luzhou Ge, Xuesong Li

Beijing Institute of Technology

World Action Models Multi-Task Robotic Manipulation Robotic Manipulation Active Vision for Robot Manipulation Vision-Based Robot Control Active View Selection+1

Ask PDF

Abstract

Active Visual Perception

Explore similar work

An Exam for Active Observers

DeltaSeek: Toward Active Perception in Evolving Construction Environments

ActiveWAM: Evidence-Aware Active Vision for World-Action Models