Organizations: Harbin Institute of Technology (Shenzhen), Shenzhen, Guangdong 518055, China · Shandong Jianzhu University, Jinan, Shandong 250101, China
View transformers process multi-view observations to predict actions and have shown impressive performance in robotic manipulation. Existing methods typically extract static visual representations in a view-specific manner, leading to inadequate 3D spatial reasoning ability and a lack of dynamic adaptation. Taking inspiration from how the human brain integrates static and dynamic views to address these challenges, we propose Cortical Policy, a novel dual-stream view transformer for robotic manipulation that jointly reasons from static-view and dynamic-view streams. The static-view stream enhances spatial understanding by aligning features of geometrically consistent keypoints extracted from a pretrained 3D foundation model. The dynamic-view stream achieves adaptive adjustment through position-aware pretraining of an egocentric gaze estimation model, computationally replicating the human cortical dorsal pathway. Subsequently, the complementary view representations of both streams are integrated to determine the final actions, enabling the model to handle spatially-complex and dynamically-changing tasks under language conditions. Empirical evaluations on RLBench, the challenging COLOSSEUM benchmark, and real-world tasks demonstrate that Cortical Policy outperforms state-of-the-art baselines substantially, validating the superiority of dual-stream design for visuomotor control. Our cortex-inspired framework offers a fresh perspective for robotic manipulation and holds potential for broader application in vision-based robot control.
Figures & tables
Figure 1: Deficiencies of prior view transformers for robotic manipulation. (Top) A task requiring the robot to understand the spatial relationships between two bottles before deciding on the placing position. Previous method RVT-2 fails to merge different camera views correctly in 3D, causing wrong block placement. (Bottom) Dynamic adaptation failure during object displacement.
Figure 2: Overview of the proposed cortical policy. Inspired by the dorsal-ventral pathways in visual neuroscience, this architecture implements dual processing streams: a static-view stream for 3D spatial understanding and a dynamic-view stream for end-effector position awareness.
Table 1: Comparison with SOTA methods on RLBench. The “Avg. Success” and “Avg. Rank” columns report the average success rate (%) and the average rank across 18 tasks. Best results are highlighted in bold, and the second best are underlined.
Models
Arch.
Lcgc
Pre.
Heat.
Avg. Success ↑
Avg. Rank ↓
Close Jar
Drag Stick
Insert Peg
Meat off Grill
Open Drawer
Place Cups
Place Wine
Push Buttons
A
Single
✗
–
–
77.5
3.3
93.3 ±1.9
97.3 ±1.9
28.0 ±3.3
100.0 ±0.0
92.0 ±3.3
32.0 ±5.7
84.0 ±9.8
100.0 ±0.0
B
Single
✔
–
–
80.1
2.4
94.7 ±1.9
100.0 ±0.0
25.3 ±5.0
100.0 ±0.0
94.7 ±1.9
21.3 ±1.9
86.7 ±5.0
100.0 ±0.0
C
Dual
✗
✗
✔
77.6
3.0
97.3 ±0.0
98.7 ±0.0
30.7 ±10.0
100.0 ±0.0
88.0 ±3.3
28.0 ±6.5
93.3 ±5.0
100.0 ±0.0
D
Dual
✗
✔
✗
73.3
4.8
90.7 ±1.9
90.7 ±13.2
26.7 ±3.8
100.0 ±0.0
94.7 ±1.9
20.0 ±0.0
82.7 ±8.2
98.7 ±1.9
E
Dual
✗
✔
✔
79.5
3.1
90.7 ±1.9
97.3 ±1.9
29.3 ±1.9
100.0 ±0.0
100.0 ±0.0
46.7 ±7.5
88.0 ±5.7
100.0 ±0.0
F (Ours)
Dual
✔
✔
✔
81.0
1.9
96.0 ±0.0
100.0 ±0.0
38.7 ±6.8
100.0 ±0.0
84.0 ±6.5
24.0 ±3.3
94.7 ±3.8
100.0 ±0.0
Table 2: Ablation study on dual-stream view transformer. All designs contribute to improving performance of Cortical Policy. “Arch.”, “Pre.”, “Heat.” denote model architecture, position-aware pretraining, dynamic-view heatmap, respectively. “Single” means single-stream model with only static viewpoints, “Dual” means dual-stream model integrating dynamic and static viewpoints.
Models
Arch.
Lcgc
Pre.
Heat.
Avg. Success ↑
Avg. Rank ↓
All Perturbations
MO- Color
RO- Color
MO- Texture
RO- Texture
MO- Size
PerAct
–
–
–
–
7.7
7.0
0.0 ±0.0
8.0 ±8.5
5.3 ±5.0
2.0 ±2.0
4.0 ±3.3
16.0 ±17.3
RVT
–
–
–
–
37.7
6.0
3.0 ±5.2
27.0 ±27.3
36.0 ±15.0
50.0 ±38.0
57.3 ±32.7
50.7 ±29.6
RVT-2
Single
✗
–
–
60.5
4.4
15.0 ±17.3
64.0 ±25.6
64.9 ±27.2
93.4 ±2.7
66.2 ±31.8
80.4 ±17.5
Variant B
Single
✔
–
–
63.8
3.3
10.3 ±8.0
69.7 ±28.1
70.2 ±28.0
95.4 ±0.7
71.6 ±30.9
84.0 ±14.2
Variant D
Dual
✗
✔
✗
66.4
2.9
10.0 ±8.2
69.7 ±27.5
72.9 ±29.9
94.0 ±2.0
74.7 ±24.5
83.6 ±16.7
Variant E
Dual
✗
✔
✔
68.7
2.4
8.7 ±15.0
75.0 ±29.7
73.8 ±29.6
96.7 ±0.7
71.1 ±33.4
82.7 ±13.6
Table 3: Results on THE COLOSSEUM. The “Avg. Success” and “Avg. Rank” columns report the average success rate (%) and the average rank across all perturbations on 4 COLOSSEUM tasks.
Figure 4: (a) Training time of Cortical Policy modules , with time cost of 3D supervision generation, dual streams, action head. (b) (Top) Real-world performance comparison. (Bottom) Visualization of the initial and final states for the four real-world tasks. (c) Trajectory visualization for “stack 2 blocks with base displacement” task.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Task Name
Language Template
#of Variations
Variation Type
close jar
“close the jar”
20
color
drag stick
“use the stick to drag the cube onto the target”
20
color
insert peg
“put the ring on the spoke”
20
color
meat off grill
“take the off the grill”
2
category
open drawer
“open the drawer”
3
placement
place cups
“place cups on the cup holder”
3
count
Appendix
Table 4: Summary of the 18 RLBench tasks for multi-task experiments.
Figure 5: Rendered views for 18 RLBench tasks. Three orthographic static cameras and a dynamic camera (defined by the wrist camera’s raw extrinsic parameters) are used to generate image inputs. For each task, the first three lines show the static views (top, front, right), and the last line shows the dynamic view; rendered results of the first (coarse) stage are presented in the left part while that of the second (fine) stage are shown right.
Figure 6: Real-world setup.
Hyperparameters
Value
Batch size
512
Learning rate
5.44×10−3
Optimizer
LAMB
Learning rate schedule
cosine decay
Weight decay
1×10−4
Warm-up steps
2000
Appendix
Table 5: Training Hyperparameters of Cortical Policy.
Figure 7: Comparison of dynamic egocentric views and rendered examples.
Figure 8: RVT-2 behavior comparison. (Top) In the basic task ”stack 2 blocks”, RVT-2 places the first block near the robot arm. (Bottom) In the spatial task ”stack 2 blocks in between the bottles”, RVT-2 attempts to place the first block near the bottles but fails due to imprecision.
Figure 9: Success rate variations with training epochs for compute-control evaluation.
Task Name
Models
Avg. Success ↑
No Perturbations
All Perturbations
MO- Color
RO- Color
MO- Texture
RO- Texture
MO- Size
drag stick
PerAct
17.6
36
0
20
12
4
8
40
RVT
59.2
84
0
24
52
88
88
92
RVT-2
69.8
84.0 ±3.3
0.0 ±0.0
84.0 ±3.3
80.0 ±0.0
90.7 ±1.9
89.3 ±1.9
89.3 ±1.9
Variant B
73.4
85.3 ±5.0
0.0 ±0.0
90.7 ±1.9
89.3 ±1.9
94.7 ±1.9
90.7 ±1.9
92.0 ±0.0
Variant D
78.1
88.0 ±0.0
0.0 ±0.0
92.0 ±0.0
96.0 ±0.0
96.0 ±0.0
92.0 ±0.0
94.7 ±1.9
Variant E
78.3
88.0 ±0.0
0.0 ±0.0
92.0 ±0.0
96.0 ±0.0
97.3 ±1.9
92.0 ±0.0
88.0 ±0.0
Appendix
Table 6: Success rates of different methods under various perturbations of COLOSSEUM.
Models
Parameter Size
Parameter Num
FLOPs
Avg. Success
RVT-2
277.3MB
72.7M
14.98 G
77.5
Baseline (Deeper)
596.6MB
156.4M
25.59 G
78.4
Cortical Policy (Ours)
551.9MB
144.7M
22.37 G
81.0
Appendix
Table 7: Capacity-controlled ablation study on RLBench.
Figure 10: Additional visualization of geometrically consistent keypoints.
Recent advances in robot imitation learning have produced powerful visuomotor policies that manipulate diverse objects from visual inputs. However, monocular observations lack depth information, which is critical for precise manipulation in cluttered or geometrically complex scenes. Explicit depth maps and point clouds are often noisy and fragile in real-world manipulation. We introduce StereoPolicy, a visuomotor policy learning framework that directly leverages synchronized stereo image pairs to improve geometric reasoning without constructing explicit 3D representations. StereoPolicy processes each image with pretrained 2D vision encoders and fuses left-right features through a cross-attention-based Stereo Transformer, capturing spatial correspondence and disparity cues implicitly. The framework integrates with diffusion-based and pretrained vision-language-action (VLA) policies, delivering consistent improvements over RGB, RGB-D, point cloud, and multi-view baselines across three simulation benchmarks and seven real-robot tabletop and bimanual mobile manipulation tasks. Our results show that stereo vision bridges 2D pretrained representations and 3D geometric understanding for robotic manipulation.
Evans Han, Yunfan Jiang, Yingke Wang +6
1Stanford University · 2Northwestern University · 3Lambda, Inc
Recently, end-to-end robotic manipulation models have gained significant attention for their generalizability and scalability. However, they often suffer from limited robustness to camera viewpoint changes when training with a fixed camera. In this paper, we propose VistaBot, a novel framework that integrates feed-forward geometric models with video diffusion models to achieve view-robust closed-loop manipulation without requiring camera calibration at test time. Our approach consists of three key components: 4D geometry estimation, view synthesis latent extraction, and latent action learning. VistaBot is integrated into both action-chunking (ACT) and diffusion-based (π0) policies and evaluated across simulation and real-world tasks. We further introduce the View Generalization Score (VGS) as a new metric for comprehensive evaluation of cross-view generalization. Results show that VistaBot improves VGS by 2.79× and 2.63× over ACT and π0, respectively, while also achieving high-quality novel view synthesis. Our contributions include a geometry-aware synthesis model, a latent action planner, a new benchmark metric, and extensive validation across diverse environments. The code and models will be made publicly available.
This paper tackles spatial perception and manipulation challenges in Vision-Language-Action (VLA) models. To address depth ambiguity from monocular input, we leverage a pre-trained multi-view diffusion model to synthesize latent novel views and propose a Geometry-Guided Gated Transformer (G3T) that aligns multi-view features under 3D geometric guidance while adaptively filtering occlusion noise. To improve action learning efficiency, we introduce Action Manifold Learning (AML), which directly predicts actions on the valid action manifold, bypassing inefficient regression of unstructured targets like noise or velocity. Experiments on LIBERO, RoboTwin 2.0, and real-robot tasks show our method achieves superior success rate and robustness over SOTA baselines. Project page: https://junjxiao.github.io/Multi-view-VLA.github.io/.