Recovering the View: Benchmarking Physical Active Vision for Occlusion Recovery in Robotic Manipulation

Kaijun Luo; Yudi Huang; Qijun Zhong; Xinshuai Song; Yang Liu; Liang Lin

cs.ROSep 29, 2026

Recovering the View: Benchmarking Physical Active Vision for Occlusion Recovery in Robotic Manipulation

Authors: Kaijun Luo, Yudi Huang, Qijun Zhong, Xinshuai Song, Yang Liu, Liang Lin

Organizations: Sun Yat-sen University · University of Electronic Science and Technology of China · X-Era AI Lab · Pengcheng Laboratory

Abstract

Physical active vision allows robots to change their viewpoint when task-relevant observations become unreliable, yet existing manipulation benchmarks provide limited support for studying how policies recover from occlusion during execution. We introduce BAVO-Bench (Bimanual Active Vision under Occlusion), a bimanual active-vision benchmark that systematically controls external visibility through Clean, Stage Occlusion, and Random-time Occlusion conditions, enabling evaluation of both manipulation performance and active visual recovery. Building on this setting, we present A-FAR (Active Future-Aware Recovery), an active-vision policy for joint viewpoint and manipulation control. A-FAR represents moving-camera observations in a unified robot-centric 3D frame and distills relational structure together with its future evolution from a pretrained 4D model, providing the policy with future-aware geometric guidance without requiring future observations at deployment. Experiments across multiple manipulation tasks show that A-FAR improves robustness to both structured and temporally shifted occlusions while maintaining strong performance under clean observations.

Figures & tables

Figure 1: Overview of our motivation, benchmark, and approach. (a) We study physical active vision as closed-loop recovery from external occlusion: using a single active camera as the sole visual sensor, the robot actively changes its viewpoint to recover task-relevant visibility and continue manipulation. (b) We introduce BAVO-Bench , a bimanual active-vision manipulation benchmark that systematically introduces controlled external occlusions to study viewpoint recovery, together with a VR teleoperation pipeline for collecting active-view demonstrations. (c) We present A-FAR , an active-vision policy that canonicalizes moving-view observations in robot-centric 3D and distills D4RT’s 4D relational priors, enabling stable, future-aware control under active viewpoint changes.

Figure 2: BAVO-Bench overview. Top: Five multi-stage bimanual manipulation tasks. Bottom: Representative rollouts under the two occlusion settings. Red-bordered frames mark occlusion events, and the timelines illustrate recovery and manipulation behaviors. Both settings occlude task-relevant targets; Stage Occlusion aligns interventions with predefined task transitions, whereas Random-time Occlusion samples intervention times independently of these transitions.

Figure 3: Overview of A-FAR. Current RGB-D observations are canonicalized into a robot-base point cloud and encoded as current point tokens $H_{t}$ . A state-conditioned WorldQueryFormer predicts future relational tokens $\hat{H}_{t}^{F}$ , which are fused with the current representation for ManiFlow action generation. During training, a frozen D4RT teacher queries the same source points at current and future target times to construct relational targets that supervise the current structure ( $\mathcal{L}_{\mathrm{cur}}$ ) and its temporal evolution ( $\mathcal{L}_{\mathrm{evo}}$ ). The teacher branch and future frames are removed at deployment.

	Clean		Stage Occlusion		Random-time Occlusion
Method	SR (%) $\uparrow$	TPS (%) $\uparrow$	SR (%) $\uparrow$	TPS (%) $\uparrow$	SR (%) $\uparrow$	TPS (%) $\uparrow$
ACT	$10.3{\pm}1.2$	$46.2{\pm}6.8$	$24.7{\pm}2.3$	$58.1{\pm}1.7$	$15.7{\pm}3.2$	$46.6{\pm}0.4$
$\pi_{0.5}$	13.0	55.2	42.0	73.4	4.0	40.8
ManiFlow	$45.7{\pm}1.5$	$70.5{\pm}6.4$	$60.7{\pm}3.1$	$81.5{\pm}3.2$	$15.0{\pm}3.5$	$50.5{\pm}2.7$
A-FAR	$\mathbf{51.3{\pm}4.2}$	$\mathbf{73.9{\pm}5.4}$	$\mathbf{63.7{\pm}2.1}$	$\mathbf{83.9{\pm}0.9}$	$\mathbf{18.7{\pm}1.2}$	$\mathbf{51.9{\pm}1.4}$

Table 1: Main results on BAVO-Bench. Results are macro-averaged over five tasks. ACT, ManiFlow, and A-FAR report mean

\pm

standard deviation over three training seeds;

\pi_{0.5}

is reported from one multi-task training run. Higher is better.

Task	ACT	$\pi_{0.5}$	ManiFlow	A-FAR
Cube	$16.49/0.00$	$\mathbf{32.81}/0.00$	$26.25/2.18$	$25.88/\mathbf{3.44}$
Drawer	$19.71/8.87$	$\mathbf{35.65}/0.00$	$30.47/11.67$	$32.27/\mathbf{12.91}$
Slide	$17.64/\mathbf{5.29}$	$\mathbf{28.72}/2.87$	$22.50/2.99$	$24.08/4.02$
Stack	$20.36/0.00$	$\mathbf{32.94}/0.00$	$28.66/0.00$	$28.93/\mathbf{0.49}$
Sweep	$21.49/0.71$	$\mathbf{41.99}/4.20$	$34.97/5.25$	$37.04/\mathbf{8.04}$
Avg.	$19.14/2.97$	$\mathbf{34.42}/1.41$	$28.57/4.42$	$29.64/\mathbf{5.78}$

Table 2: Active-view analysis under Random-time Occlusion. Each entry reports CVG / Joint . CVG measures camera-attributable visibility recovery (pp), while

\mathrm{Joint}=\mathrm{SR}\times\mathrm{CVG}/100

jointly reflects visibility recovery and task success. Higher is better.

Figure 4: Real-world deployment of A-FAR. Representative rollout under an externally introduced occlusion. From left to right, task-relevant visibility is disrupted during execution, the active camera changes viewpoint to recover the scene, and manipulation proceeds after the target region becomes observable again. The sequence demonstrates closed-loop active-view behavior on our physical manipulation platform.

Variant	C	S	R	Avg.
ManiFlow (base)	44	60	15	41.0
+ Future Evo.	47	64	16	42.3
+ Future Fusion	47	63	18	43.0
A-FAR (base)	50	66	19	44.7
ManiFlow (camera)	26	44	14	28.0
A-FAR (camera)	22	43	16	27.0

Table 3: Ablation study. SR (%) under Clean (C), Stage (S), and Random-time (R) Occlusion.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Objective	Ordered milestones
Cube Handoff	Handoff a cube from the right arm to the left arm and place it on the target plate.	$M_{1}$ : the right gripper contacts the cube; $M_{2}$ : the right arm lifts the cube; $M_{3}$ : both grippers hold the cube during handoff; $M_{4}$ : the left gripper takes over and the right gripper releases; $M_{5}$ : the left arm places the cube on the target plate.
Block Stacking	Move the base block to the central workspace and stack the top block on it.	$M_{1}$ : the right gripper contacts the base block; $M_{2}$ : the base block is placed in the stacking region and released; $M_{3}$ : the left gripper contacts the top block; $M_{4}$ : the top block is lifted into the pre-stacking region; $M_{5}$ : a stable stack is formed with both grippers released.
Drawer Stowing	Open the drawer, stow an object inside, and close the drawer.	$M_{1}$ : the right gripper contacts the drawer handle; $M_{2}$ : the drawer is opened; $M_{3}$ : the left gripper contacts the object; $M_{4}$ : the object is placed and released inside the drawer; $M_{5}$ : the drawer is closed with the object remaining inside.
Slide Retrieval	Slide the object to expose its handle, then retrieve it with the left arm.	$M_{1}$ : the right gripper contacts the blade; $M_{2}$ : the handle is exposed by sliding; $M_{3}$ : the right gripper releases the object; $M_{4}$ : the left gripper contacts the exposed handle; $M_{5}$ : the left arm lifts the object from the support surface.
Dustpan Sweeping	Position the dustpan and sweep the target object into it.	$M_{1}$ : the left gripper contacts the dustpan handle; $M_{2}$ : the dustpan reaches a valid sweeping configuration; $M_{3}$ : the right gripper contacts the broom; $M_{4}$ : the broom contacts the target object during sweeping; $M_{5}$ : the target object reaches the dustpan entrance.

Appendix

Table 4: Task objectives and milestone progression in BAVO-Bench. Each task contains five ordered milestones, with

M_{5}

defining task success.

Task	Initial target	Transitions	Subsequent targets
Cube Handoff	cube	$M_{2},\ M_{4}$	cube $\rightarrow$ plate
Block Stacking	base block	$M_{2},\ M_{4}$	top block $\rightarrow$ stacking region
Drawer Stowing	drawer front	$M_{2},\ M_{3}$	object $\rightarrow$ drawer front
Slide Retrieval	support platform	$M_{1},\ M_{2}$	blade $\rightarrow$ handle
Dustpan Sweeping	dustpan	$M_{2},\ M_{3}$	broom $\rightarrow$ target object

Appendix

Table 5: Stage Occlusion schedule. An initial occlusion is introduced at reset, followed by two stage-aligned target updates.

Figure 5: Rollout visualizations under Clean. Representative sequences for the five benchmark tasks without external visibility intervention. For each task, the top row shows the third-person scene view and the bottom row shows the active-camera view. Frames are ordered from left to right.

Figure 6: Rollout visualizations under Stage Occlusion. Representative sequences for the five benchmark tasks with stage-aligned occlusion events. For each task, the top row shows the third-person scene view and the bottom row shows the active-camera view. Frames are ordered from left to right. The sequences highlight how structured occlusions are introduced at predefined task transitions.

Figure 7: Rollout visualizations under Random-time Occlusion. Representative sequences for the five benchmark tasks with temporally unexpected occlusion events. For each task, the top row shows the third-person scene view and the bottom row shows the active-camera view. Frames are ordered from left to right. The sequences illustrate how visibility disruptions may occur during ongoing manipulation behaviors.

Configuration	Value
Point-cloud size	256
Teacher query points	256
Observation horizon	2
Action horizon	16
Point feature dimension	128
State feature dimension	64

Appendix

Table 6: Main architecture and training settings for A-FAR.

Task	Method	C	S	R
Cube Handoff	ACT	$0.0{\pm}0.0$	$6.7{\pm}7.6$	$0.0{\pm}0.0$
	$\pi_{0.5}$	$5.0$	$30.0$	$0.0$
	ManiFlow	$41.7{\pm}2.9$	$\mathbf{68.3{\pm}5.8}$	$8.3{\pm}7.6$
	A-FAR	$\mathbf{43.3{\pm}14.4}$	$58.3{\pm}2.9$	$\mathbf{13.3{\pm}2.9}$
Drawer Stowing	ACT	$50.0{\pm}5.0$	$56.7{\pm}2.9$	$\mathbf{45.0{\pm}8.7}$
	$\pi_{0.5}$	$25.0$	$75.0$	$0.0$

Appendix

Table 7: Task-level success rate (SR, %). Results are reported under Clean (C), Stage Occlusion (S), and Random-time Occlusion (R). ACT, ManiFlow, and A-FAR are reported as mean

\pm

std over three training seeds;

\pi_{0.5}

is evaluated from a single training run. Best performance for each task and condition is shown in bold.

Task	Method	C	S	R
Cube Handoff	ACT	$33.0{\pm}2.6$	$44.0{\pm}3.6$	$24.0{\pm}6.2$
	$\pi_{0.5}$	$45.0$	$68.0$	$31.0$
	ManiFlow	$\mathbf{70.3{\pm}5.0}$	$\mathbf{87.7{\pm}2.3}$	$45.3{\pm}5.7$
	A-FAR	$69.7{\pm}13.2$	$84.0{\pm}1.7$	$\mathbf{47.7{\pm}6.0}$
Drawer Stowing	ACT	$78.7{\pm}3.1$	$80.7{\pm}2.1$	$\mathbf{77.3{\pm}4.7}$
	$\pi_{0.5}$	$70.0$	$90.0$	$51.0$

Appendix

Table 8: Task-level task progress score (TPS, %). Results are reported under Clean (C), Stage Occlusion (S), and Random-time Occlusion (R). ACT, ManiFlow, and A-FAR are reported as mean

\pm

std over three training seeds;

\pi_{0.5}

is evaluated from a single training run. Best performance for each task and condition is shown in bold.

Figure 8: Real-world Experiement Setup At the home position, the manipulation arm’s gripper points toward the workspace, and the camera is oriented in the same direction.

Explore similar work

ActiveWAM: Evidence-Aware Active Vision for World-Action Models
Oct 1, 2026Renjun Wu, Luzhou Ge, Xuesong LiEfficient World-Action Model Faster-Wam
Beijing Institute of Technology
UniviewVLA: A Unified Multiview Vision-Language-Action Model with World Modeling
Jun 19, 2026Tao Xu, Runhao Zhang, Zhijian Huang +7Diffusion-Based Vision-Language-Actions Diverse Occlusion-And-Revelation Scenarios
1Tongji University · 2Shanghai Innovation Institute · 4Xiaomi EV +1
ActiveArena: Benchmarking and Understanding Active Perception in Robotic Manipulation
Sep 21, 2026Yibo Li, Enshen Zhou, Rui Chen +7Viewpoint-Dependent Active Perception Robotic Manipulation
Beihang University · Beijing Academy of Artificial Intelligence · Peking University +1

Oct 1, 2026, cs.ROJ/K move · Enter open · S save

ActiveWAM: Evidence-Aware Active Vision for World-Action Models

Renjun Wu, Luzhou Ge, Xuesong Li

Beijing Institute of Technology

Efficient World-Action Model Faster-Wam Photometric Supervision World Models

Open paper AskPDF arXiv

Active vision manipulation requires a policy to control both its camera and its end-effectors, yet camera motion determines which evidence remains visible within finite observation windows. Acquiring a new view can displace task-critical cues, while retaining a view forgoes potentially useful observations. We formulate this as an evidence-aware retain--acquire problem and present ActiveWAM, a unified world--action model that learns observation and manipulation jointly. To this end, we propose training-time inversion which constrains a frozen video prior by task-bearing source evidence and visible temporal changes, eliminating the need for test-time inversion or candidate ranking. At deployment, the policy generates bimanual and pan/tilt actions-including stay and reacquisition behaviors-from view-aware history, and updates context from newly measured RGB observations. Future-video prediction serves as a co-training signal, while action generation requires neither future-video decoding nor optimal viewpoint annotations. We introduce RoboTwin-AV, a 50-task benchmark with executable pan/tilt control and automatically generated demonstrations. ActiveWAM improves TAVIS out-of-distribution success by up to 17.0 percentage points over the strongest baselines, achieves 20.0 additional points over Fast-WAM on RoboTwin-AV, and outperforms it by 26.7 points on real-world physical kitchen tasks.

Abstract

Explore similar work

ActiveWAM: Evidence-Aware Active Vision for World-Action Models

UniviewVLA: A Unified Multiview Vision-Language-Action Model with World Modeling

ActiveArena: Benchmarking and Understanding Active Perception in Robotic Manipulation