SaPaVe: Towards Active Perception and Manipulation in Vision-Language-Action Models for Robotics
Authors: Mengzhen Liu, Enshen Zhou, Cheng Chi, Yi Han, Shanyu Rong, Liming Chen, Pengwei Wang, Zhongyuan Wang, +1 more
Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Academy of Artificial Intelligence · School of Software, Beihang University
Active perception and manipulation are crucial for robots to interact with complex scenes. Existing methods struggle to unify semantic-driven active perception with robust, viewpoint-invariant execution. We propose SaPaVe, an end-to-end framework that jointly learns these capabilities in a data-efficient manner. Our approach decouples camera and manipulation actions rather than placing them in a shared action space, and follows a bottom-up training strategy: we first train semantic camera control on a large-scale dataset, then jointly optimize both action types using hybrid data. To support this framework, we introduce ActiveViewPose-200K, a dataset of 200k image-language-camera movement pairs for semantic camera movement learning, and a 3D geometry-aware module that improves execution robustness under dynamic viewpoints. We also present ActiveManip-Bench, the first benchmark for evaluating active manipulation beyond fixed-view settings. Extensive experiments in both simulation and real-world environments show that SaPaVe outperforms recent vision-language-action models such as GR00T N1 and π0, achieving up to 31.25% higher success rates in real-world tasks. These results show that tightly coupled perception and execution, when trained with decoupled yet coordinated strategies, enable efficient and generalizable active manipulation. Project page: https://lmzpai.github.io/SaPaVe
Figures & tables
Figure 2: Overview of SaPaVe . SaPaVe can process RGB images and task instructions and output camera movement and manipulation actions in a decoupled action space. This decoupled design enables the model to achieve active manipulation via a bottom-up, two-stage training strategy: First, large-scale embodiment-agnostic camera control data fosters semantic active perception , which is encoded as prior knowledge in a camera adapter. Second, mixed data together with Universal Spatial Knowledge Injection flexibly incorporate various geometric configurations ( e.g. , absolute depth, camera intrinsics), thereby enhancing spatial precision for active-view execution .
Figure 3: Overview of ActiveViewPose-200K . It is a high-quality dataset comprising 200k image-language and camera movement pairs, enriched with highly detailed semantic annotations to enable semantic camera movement learning.
Figure 4: Overview of ActiveManip-Bench : It is the first simulation benchmark to evaluate active manipulation beyond traditional fixed-view settings. ActiveManip-Bench features 12 richly annotated tasks across 100 objects and 20 diverse scenes.
Method
Val
Test1
Test2
Avg.
Qwen2.5-VL-72B [ 2 ]
63.9
65.1
58.0
62.3
Multi-SpatialMLLM [ 47 ]
72.8
74.3
63.6
70.2
Gemini-2.5-Pro [ 12 ]
73.3
76.5
68.2
72.7
Ours (Stage 1)
85.5
89.1
78.3
84.3
Table 1: The performance of semantic active perception evaluation. We report the success rate (%) compared to current general VLMs and specialized spatial VLMs.
Method
Unoccluded P.a.P
Occluded P.a.P
Out-of-View P.a.P
Unoccluded A.M
Occluded A.M
Out-of-View A.M
Avg.
Fixed Camera
74
46
11
52
27
7
36.17
Fixed Camera + Wrist Camera
83
62
28
66
51
24
52.33
Active Camera + Wrist Camera
86
75
70
74
68
66
73.16
Ours (Active Camera)
85
78
72
76
70
68
74.83
Table 2: Evaluation results for fixed and dynamic cameras in simulation of ActiveManip-Bench. We report the success rate (%) compare to different camera configurations with the same architecture. P.a.P and A.M denote Pick-and-Place and articulated Manipulation.
Method
Occluded
Out-of-View
Occluded
Out-of-View
Avg.
Pick-and-Place
Pick-and-Place
Arti-Manip
Arti-Manip
π0 [ 6 ]
55
45
45
35
45.00
GR00T-N1 [ 5 ]
60
55
50
50
53.75
Ours
90
85
85
80
85.00
Table 3: Performance on active manipulation in real-world settings. We report the success rate (%) compared to the existing VLA models. Our approach achieves the best performance.
Task Name
Object 1
Object 2
Light 1
Light 2
Scene 1
Scene 2
Original
Occluded
85
90
90
95
90
85
90
Pick-and-Place
Out-of-View
85
90
80
90
85
85
85
Pick-and-Place
Occluded
80
85
80
85
85
80
85
Arti-Manip
Table 4: Performance on generalization ability evaluation. We report the success rate (%). Our model demonstrates robust generalization when performing active manipulation across unseen objects, varying lighting conditions, and diverse scenes.
Ablation
Occluded
Out-of-View
Occluded
Out-of-View
Avg.
Pick-and-Place
Pick-and-Place
Arti-Manip
Arti-Manip
w/o. Stage 1
65
55
50
45
53.75
w/o. Stage 2
75
60
70
60
66.25
w/o. D.A.H.
80
70
70
65
71.25
w/o. C.A.
80
75
70
70
73.75
w/o. U.S.K.I.
75
75
65
60
68.75
Table 5: Ablation Study on the effect about training strategy of Stage 1 and Stage2, decoupled action head (D.A.H.), camera adapter (C.A.), and universal spatial knowledge injection (U.S.K.I). We report the avarage success rate (%).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Real-world Execution roll-outs (ego & third view).
Figure 6: Examples of the asset curation process.
Figure 7: Generated synthetic environments using Infinigen integrated with our curated asset library. The scenes feature varying lighting conditions, furniture layouts, and clutter distributions.
Figure 8: Atomic Scene Tasks and Viewpoint Movement Examples.
Figure 9: Two-stage scene task and viewpoint movement example.
Category
Task Name
Success Criteria (Goal State)
Atomic
Pick
Object height form table >5 cm; velocity ≈0 .
Reorient
Orientation error relative to target <15∘ .
OpenDrawer
Prismatic joint distance (opening) >90% of analytical limit.
Table 6: Overview of the 12 Active Manipulation Tasks. The tasks are categorized by horizon length and complexity. Success is rigorously defined by geometric thresholds (position, rotation, joint state) or physical quantities (liquid volume), requiring the robot to maintain the goal state for a stabilization period (e.g., 2 seconds).
Figure 10: We provide the type of the robot, dexterous hand, camera, and servos used in our real-robot experiments.
Figure 11: This diagram illustrates the four tasks in our real-world experiment categories, along with the employed scenarios, assets, and varying lighting conditions.
Figure 12: This image illustrates the active manipulate task performed by the robot in a simulation scenario.
Figure 13: This image illustrates the active manipulate task performed by the robot in a simulation scenario.
Figure 14: This image illustrates the active manipulate task performed by the robot in a real-world scenario.
Figure 15: This image illustrates the active manipulate task performed by the robot in a real-world scenario.