SaPaVe: Towards Active Perception and Manipulation in Vision-Language-Action Models for Robotics
Authors: Mengzhen Liu, Enshen Zhou, Cheng Chi, Yi Han, Shanyu Rong, Liming Chen, Pengwei Wang, Zhongyuan Wang, +1 more
Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Academy of Artificial Intelligence · School of Software, Beihang University
Active perception and manipulation are crucial for robots to interact with complex scenes. Existing methods struggle to unify semantic-driven active perception with robust, viewpoint-invariant execution. We propose SaPaVe, an end-to-end framework that jointly learns these capabilities in a data-efficient manner. Our approach decouples camera and manipulation actions rather than placing them in a shared action space, and follows a bottom-up training strategy: we first train semantic camera control on a large-scale dataset, then jointly optimize both action types using hybrid data. To support this framework, we introduce ActiveViewPose-200K, a dataset of 200k image-language-camera movement pairs for semantic camera movement learning, and a 3D geometry-aware module that improves execution robustness under dynamic viewpoints. We also present ActiveManip-Bench, the first benchmark for evaluating active manipulation beyond fixed-view settings. Extensive experiments in both simulation and real-world environments show that SaPaVe outperforms recent vision-language-action models such as GR00T N1 and π0, achieving up to 31.25% higher success rates in real-world tasks. These results show that tightly coupled perception and execution, when trained with decoupled yet coordinated strategies, enable efficient and generalizable active manipulation. Project page: https://lmzpai.github.io/SaPaVe
Figures & tables
Figure 2: Overview of SaPaVe . SaPaVe can process RGB images and task instructions and output camera movement and manipulation actions in a decoupled action space. This decoupled design enables the model to achieve active manipulation via a bottom-up, two-stage training strategy: First, large-scale embodiment-agnostic camera control data fosters semantic active perception , which is encoded as prior knowledge in a camera adapter. Second, mixed data together with Universal Spatial Knowledge Injection flexibly incorporate various geometric configurations ( e.g. , absolute depth, camera intrinsics), thereby enhancing spatial precision for active-view execution .
Figure 3: Overview of ActiveViewPose-200K . It is a high-quality dataset comprising 200k image-language and camera movement pairs, enriched with highly detailed semantic annotations to enable semantic camera movement learning.
Figure 4: Overview of ActiveManip-Bench : It is the first simulation benchmark to evaluate active manipulation beyond traditional fixed-view settings. ActiveManip-Bench features 12 richly annotated tasks across 100 objects and 20 diverse scenes.
Method
Val
Test1
Test2
Avg.
Qwen2.5-VL-72B [ 2 ]
63.9
65.1
58.0
62.3
Multi-SpatialMLLM [ 47 ]
72.8
74.3
63.6
70.2
Gemini-2.5-Pro [ 12 ]
73.3
76.5
68.2
72.7
Ours (Stage 1)
85.5
89.1
78.3
84.3
Table 1: The performance of semantic active perception evaluation. We report the success rate (%) compared to current general VLMs and specialized spatial VLMs.
Method
Unoccluded P.a.P
Occluded P.a.P
Out-of-View P.a.P
Unoccluded A.M
Occluded A.M
Out-of-View A.M
Avg.
Fixed Camera
74
46
11
52
27
7
36.17
Fixed Camera + Wrist Camera
83
62
28
66
51
24
52.33
Active Camera + Wrist Camera
86
75
70
74
68
66
73.16
Ours (Active Camera)
85
78
72
76
70
68
74.83
Table 2: Evaluation results for fixed and dynamic cameras in simulation of ActiveManip-Bench. We report the success rate (%) compare to different camera configurations with the same architecture. P.a.P and A.M denote Pick-and-Place and articulated Manipulation.
Method
Occluded
Out-of-View
Occluded
Out-of-View
Avg.
Pick-and-Place
Pick-and-Place
Arti-Manip
Arti-Manip
π0 [ 6 ]
55
45
45
35
45.00
GR00T-N1 [ 5 ]
60
55
50
50
53.75
Ours
90
85
85
80
85.00
Table 3: Performance on active manipulation in real-world settings. We report the success rate (%) compared to the existing VLA models. Our approach achieves the best performance.
Task Name
Object 1
Object 2
Light 1
Light 2
Scene 1
Scene 2
Original
Occluded
85
90
90
95
90
85
90
Pick-and-Place
Out-of-View
85
90
80
90
85
85
85
Pick-and-Place
Occluded
80
85
80
85
85
80
85
Arti-Manip
Table 4: Performance on generalization ability evaluation. We report the success rate (%). Our model demonstrates robust generalization when performing active manipulation across unseen objects, varying lighting conditions, and diverse scenes.
Ablation
Occluded
Out-of-View
Occluded
Out-of-View
Avg.
Pick-and-Place
Pick-and-Place
Arti-Manip
Arti-Manip
w/o. Stage 1
65
55
50
45
53.75
w/o. Stage 2
75
60
70
60
66.25
w/o. D.A.H.
80
70
70
65
71.25
w/o. C.A.
80
75
70
70
73.75
w/o. U.S.K.I.
75
75
65
60
68.75
Table 5: Ablation Study on the effect about training strategy of Stage 1 and Stage2, decoupled action head (D.A.H.), camera adapter (C.A.), and universal spatial knowledge injection (U.S.K.I). We report the avarage success rate (%).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Real-world Execution roll-outs (ego & third view).
Figure 6: Examples of the asset curation process.
Figure 7: Generated synthetic environments using Infinigen integrated with our curated asset library. The scenes feature varying lighting conditions, furniture layouts, and clutter distributions.
Figure 8: Atomic Scene Tasks and Viewpoint Movement Examples.
Figure 9: Two-stage scene task and viewpoint movement example.
Category
Task Name
Success Criteria (Goal State)
Atomic
Pick
Object height form table >5 cm; velocity ≈0 .
Reorient
Orientation error relative to target <15∘ .
OpenDrawer
Prismatic joint distance (opening) >90% of analytical limit.
Table 6: Overview of the 12 Active Manipulation Tasks. The tasks are categorized by horizon length and complexity. Success is rigorously defined by geometric thresholds (position, rotation, joint state) or physical quantities (liquid volume), requiring the robot to maintain the goal state for a stabilization period (e.g., 2 seconds).
Figure 10: We provide the type of the robot, dexterous hand, camera, and servos used in our real-robot experiments.
Figure 11: This diagram illustrates the four tasks in our real-world experiment categories, along with the employed scenarios, assets, and varying lighting conditions.
Figure 12: This image illustrates the active manipulate task performed by the robot in a simulation scenario.
Figure 13: This image illustrates the active manipulate task performed by the robot in a simulation scenario.
Figure 14: This image illustrates the active manipulate task performed by the robot in a real-world scenario.
Figure 15: This image illustrates the active manipulate task performed by the robot in a real-world scenario.
Active perception is essential for robotic manipulation when fixed viewpoints leave task-relevant information occluded or unobserved. However, enabling vision-language-action (VLA) models to reason across changing viewpoints and actively acquire informative observations remains challenging. We present ActiveScale, a framework that advances active perception through coordinated model, data, and hardware designs. Our model augments a VLA with historical video observations and explicit camera-pose supervision, using per-frame pose tokens and a lightweight prediction head to associate observations across viewpoints and support a coherent understanding of the scene. To learn from the camera motion naturally present in human activity, we introduce a scalable human--robot mid-training recipe using 1000 hours of egocentric and robotic data, adapting the model to temporal inputs and pose supervision. We further introduce Active-perception Mobile-manipulation Platform (AMP), a robotic platform that supports active perception and mobile manipulation through single-operator teleoperation, enabling scalable collection of demonstrations that coordinate viewpoint changes and manipulation. Experiments demonstrate improved success rates on active-perception tasks, while ablation studies validate the contributions of camera-pose-aware modeling and egocentric mid-training. Together, these components provide an integrated foundation for studying and developing active perception in robotic manipulation.
Shuai Zhou, Kaisheng Pang, Wenxuan Song +3
Robotics Institute, Carnegie Mellon University · The Hong Kong University of Science and Technology (Guangzhou)
Active perception - the ability of a robot to proactively select viewpoints to acquire task-relevant information - is essential for robust operation in real-world environments. However, existing approaches are typically limited to fixed objectives or constrained settings, and struggle to generalize to open-ended perception intents specified in natural language. We propose I-Perceive, a foundation model for language-conditioned active perception in large-scale indoor environments. Given a query image, a set of context images, and a natural language instruction, I-Perceive predicts a 6D camera pose that fulfills the specified perception intent. The model integrates a vision-language pathway for semantic grounding with a geometric reasoning pathway for multi-view 3D understanding, connected via multi-layer semantic fusion to enable language-conditioned geometric reasoning. To support scalable training, we construct a large-scale dataset of language-viewpoint pairs from both real-world scene-scanning data and simulated environments using an automated pipeline. Extensive experiments demonstrate that I-Perceive significantly outperforms strong baselines on prediction accuracy, viewpoint feasibility, and instructions alignment. The model exhibits strong zero-shot generalization to unseen scenes and instructions, and enables closed-loop active perception, progressively refining viewpoints over sequential interactions.
Yongxi Huang, Zhuohang Wang, Wenjing Tang +3
Shanghai Jiao Tong University · Shanghai Innovation Institute · Beihang University
Vision-Language-Action (VLA) models have emerged as powerful foundations for robotic manipulation, but their reliance on fixed camera configurations during training makes them brittle to changes in camera count or pose during deployment. To overcome these limitations, we propose VersaCamVLA, a camera-configurable framework that decouples camera-set representation from action learning. VersaCamVLA learns a unified scene-token interface that maps an arbitrary, variable set of posed RGB views into fixed-size latent scene tokens. This is achieved via multi-signal target-view prediction and Wrist-Augmented Pose Sampling (WAPS), which leverages natural wrist-camera motion for free pose diversity. At deployment, a lightweight spatial encoder injects these compact scene tokens into a pretrained base VLA as a supplementary visual condition, requiring no explicit 3D sensing or novel-view rendering. Experiments on RoboTwin, LIBERO, and a real-robot platform demonstrate that VersaCamVLA consistently outperforms prior VLA methods and direct multi-view baselines, maintaining robust performance across varying camera counts and unseen camera poses.
Boyao Han, Chen Shi, Jingjing Qian +2
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute · Harbin Institute of Technology, Shenzhen