In View or Out of View? A Vision-Aware Filter that Infers and Reacts to the Human's Field of View

Mengxue Fu; Ya-Chuan Hsu; Yinlong Dai; Erdem Bıyık; Dylan P. Losey

cs.ROOct 10, 2026

In View or Out of View? A Vision-Aware Filter that Infers and Reacts to the Human's Field of View

Authors: Mengxue Fu, Ya-Chuan Hsu, Yinlong Dai, Erdem Bıyık, Dylan P. Losey

Organizations: LiraLab, Thomas Lord Department of Computer Science, University of Southern California, Los Angeles, CA 90089 · Collaborative Robotics Lab (Collab), Department of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24061

Abstract

During human-robot interaction, sight plays an important role. When the human sees a robot, they can observe the robot's actions and better predict the robot's future behaviors. Conversely, when the human cannot see the robot, they become uncertain about the robot's intent and plans. In this paper, we focus on the human's field of view. Existing human-aware planners either ignore this field of view and assume the human can observe the entire scene, or intentionally design their interactions so the robot is always visible to the human. By contrast, we enable robots to infer what the human has and has not seen during interaction, i.e., the human's field of view. We then apply this estimate as a vision-aware filter on the robot's actions. Given a nominal robot policy and a model of the interaction, we roll out the policy over a short horizon to see how the robot's state updates. We then reweight the actions based on how those actions impact the robot's predicted visibility: higher weights are assigned to actions that bring the robot into (or away from) the human's line of sight. In practice, our framework provides an online estimation and filtering process that repeatedly updates its understanding of the human's field of view and then locally perturbs robot actions based on the human's vision. We evaluate our vision-aware filter on robots that assist humans in collaborative cooking and coordinate with them in driving environments. Across simulations and a 12-participant user study, we find that our vision-aware filter changes both performance and preference: participants completed over three times as many subtasks with in-view assistance while cooking, and had fewer collisions with cars that purposely stayed out of their view while driving.

Figures & tables

Figure 1: Overview of our vision-aware filter. The human (green) drives with an unknown field of view $\theta$ while robots (blue) control the surrounding traffic. Based on the human’s actions the robots infer the human’s current field of view (FOV). More formally, the robots maintain a belief $b_{t}(\theta)$ over what the human can see. The robots then use this belief to modify the actions of a base policy. The robot’s base policy proposes candidate actions (e.g., speed up, slow down), and the robot leverages its model of the human and environment to roll out each candidate action and predict the time the robot will spend in the human’s vision under each FOV hypothesis ( $C_{\theta}$ ). The scores are weighted by the belief ( $\bar{C}$ ). The robot then applies these scores to adjust the action distribution of the base policy, modifying how the robot acts to account for the human’s FOV. Designers can tune whether this steers the robot into or away from the human’s line of sight. Here our approach is used to minimize visibility, driving other robots out of the human’s FOV.

Figure 2: Overcooked environment. Here we show the four kitchen layouts used in our simulation experiments. We visualize the start state for each option. From left to right: back bar, banquet pass, chefs table, and divider. Brick tiles are walls, which block the human’s line of sight; pane tiles are glass, which is impassable but see-through. Each kitchen contains meat, onion, and plate dispensers, a grill, a chopping board, a sink, and a serving window. The green-hat agent is the robot and the blue-hat agent is the human.

	Greedy			Solo			Bayes
$\theta$	Base	Out	In	Base	Out	In	Base	Out	In
$30^{\circ}$	0.43	0.08	0.70	0.40	0.08	0.65	0.20	0.12	0.50
$60^{\circ}$	1.27	1.25	1.48	1.20	1.25	1.45	0.80	1.05	1.10
$90^{\circ}$	1.25	1.30	1.50	1.42	1.28	1.50	0.73	1.02	1.12
$360^{\circ}$	1.70	1.70	1.93	1.55	1.70	1.95	1.02	1.45	1.50

Table I: Overcooked simulation: mean dishes delivered per 400-timestep episode (maximum 3 deliverable orders per episode). Results are averaged over four layouts and 20 seeds. Each row corresponds to a different human FOV

\theta

. Columns correspond to the vision-aware filter: unfiltered (base policy), out-of-view filter (out,

\lambda>0

), in-view filter (in,

\lambda<0

); best is in bold.

	IDM			DQN
$\theta$	Base	Out	In	Base	Out	In
$30^{\circ}$	3.50 $\|$ 1778	4.20 $\|$ 1900	3.90 $\|$ 1711	5.15 $\|$ 1437	2.85 $\|$ 1461	4.10 $\|$ 1284
$60^{\circ}$	0.75 $\|$ 1686	0.40 $\|$ 1907	1.50 $\|$ 1667	3.55 $\|$ 1268	2.50 $\|$ 1232	3.05 $\|$ 1267
$90^{\circ}$	0.50 $\|$ 1702	0.35 $\|$ 1902	1.00 $\|$ 1699	3.40 $\|$ 1237	2.50 $\|$ 1248	3.45 $\|$ 1372

Table II: Highway simulation: mean collisions

|

distance (m) per 60 s session over 20 paired seeds. Each row corresponds to a different human field of view

\theta

. Columns correspond to the different traffic policies and vision-aware filters: unfiltered (base), out-of-view filter (out), in-view filter (in); best in bold (fewer collisions; ties broken by more distance).

Panel of Figure 3 — Figure 3: The user study interfaces. Top: Overcooked, where the participant sees only the region inside their FOV and selects among currently available high-level subtasks from the menu. Bottom: Highway, where the participant steers the ego vehicle and sees only vehicles inside their FOV.

Domain	Metric	Base	Out	In
Overcooked	Subtasks completed	3.17	5.67	11.75
Overcooked	Dishes delivered (total)	0	1	6
Highway	Collisions per session	1.25	1.17	1.42
Highway	Distance (m)	1779	1855	1733

Table III: User study objective results (

N=12

): mean per participant with each robot policy, except dishes delivered, which are totaled across all participants. Columns correspond to the base policy (base), out-of-view filter (out,

\lambda>0

), and in-view filter (in,

\lambda<0

); best is in bold.

Figure 4: User study subjective ratings. Participants responded to a 7-point Likert scale survey after working with each filter type. Error bars show standard error, and an

\star

denotes statistical significance (

p<.05

). We find that people preferred robots that used our vision-aware filter (the colored bars); the correct way to leverage that filter (in-view or out-of-view) depended on the domain.

Figure 5: Forced choice comparison. Our 12 participants ranked the filter type from best to worst in both experimental domains. In Overcooked participants preferred in-view , while in Highway participants preferred out-of-view .

Explore similar work

OA-NBV: Occlusion-Aware Next-Best-View Planning for Human-Centered Active Perception on Mobile Robots
Mar 10, 2026Boxun Hu, Chang Chang, Jiawei Ge +4Active Perception Next-Best-View Planning
Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA · Department of Mechanical Engineering, Johns Hopkins University, Baltimore, MD 21218, USA · Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218, USA +1
HINT-Plan: Human Intention-Aware Robot Task Planning in Context-Rich Environments using Vision Language Models
Sep 15, 2026Yuchen Liu, Luigi Palmieri, Lujun Li +3Human-Robot Collaboration Robot Task Planning
University of Stuttgart, Germany · University of Luxembourg, Luxembourg · Bosch Center for Artificial Intelligence, Germany
Knowing When to Ask: Resolving Uncertainty in Human-Robot Joint Planning via Explicit Dialogue and Implicit Intent Cues
Mar 8, 2026Zeyu Fang, Yuxin Lin, Cheng Liu +6Human-Robot Collaboration Robot Task Planning
Department of Electrical and Computer Engineering, George Washington University · the Department of Electrical and Computer Engineering, Northeastern University · the Department of Mechanical and Aerospace Engineering, George Washington University

Mar 10, 2026J/K move · Enter open · S save

OA-NBV: Occlusion-Aware Next-Best-View Planning for Human-Centered Active Perception on Mobile Robots

Boxun Hu, Chang Chang, Jiawei Ge, Man Namgung+3

Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA · Department of Mechanical Engineering, Johns Hopkins University, Baltimore, MD 21218, USA · Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218, USA+1

Active Perception Next-Best-View Planning Mobile Robotics Active View Selection Robot Motion Planning

Open paper AskPDF arXiv

We naturally step sideways or lean to see around the obstacle when our view is blocked, and recover a more informative observation. Enabling robots to make the same kind of viewpoint choice is critical for human-centered operations, including search, triage, and disaster response, where cluttered environments and partial visibility frequently degrade downstream perception. However, many Next-Best-View (NBV) methods primarily optimize generic exploration or long-horizon coverage, and do not explicitly target the immediate goal of obtaining a single usable observation of a partially occluded person under real motion constraints. We present Occlusion-Aware Next-Best-View Planning for Human-Centered Active Perception on Mobile Robots (OA-NBV), an occlusion-aware NBV pipeline that autonomously selects the next traversable viewpoint to obtain a more complete view of an occluded human. OA-NBV integrates perception and motion planning by scoring candidate viewpoints using a target-centric visibility model that accounts for occlusion, target scale, and target completeness, while restricting candidates to feasible robot poses. OA-NBV achieves over 90% success rate in both simulation and real-world trials, while baseline NBV methods degrade sharply under occlusion. Beyond success rate, OA-NBV improves observation quality: compared to the strongest baseline, it increases normalized target area by at least 81% and keypoint visibility by at least 58% across settings, making it a drop-in view-selection module for diverse human-centered downstream tasks.

	IDM			DQN
$\theta$	Base	Out	In	Base	Out	In
$30^{\circ}$	3.50 $\|$ 1778	4.20 $\|$ 1900	3.90 $\|$ 1711	5.15 $\|$ 1437	2.85 $\|$ 1461	4.10 $\|$ 1284
$60^{\circ}$	0.75 $\|$ 1686	0.40 $\|$ 1907	1.50 $\|$ 1667	3.55 $\|$ 1268	2.50 $\|$ 1232	3.05 $\|$ 1267
$90^{\circ}$	0.50 $\|$ 1702	0.35 $\|$ 1902	1.00 $\|$ 1699	3.40 $\|$ 1237	2.50 $\|$ 1248	3.45 $\|$ 1372

Abstract

Explore similar work

OA-NBV: Occlusion-Aware Next-Best-View Planning for Human-Centered Active Perception on Mobile Robots

HINT-Plan: Human Intention-Aware Robot Task Planning in Context-Rich Environments using Vision Language Models

Knowing When to Ask: Resolving Uncertainty in Human-Robot Joint Planning via Explicit Dialogue and Implicit Intent Cues