Organizations: Harbin Institute of Technology (Shenzhen), Shenzhen, Guangdong 518055, China · Shandong Jianzhu University, Jinan, Shandong 250101, China
View transformers process multi-view observations to predict actions and have shown impressive performance in robotic manipulation. Existing methods typically extract static visual representations in a view-specific manner, leading to inadequate 3D spatial reasoning ability and a lack of dynamic adaptation. Taking inspiration from how the human brain integrates static and dynamic views to address these challenges, we propose Cortical Policy, a novel dual-stream view transformer for robotic manipulation that jointly reasons from static-view and dynamic-view streams. The static-view stream enhances spatial understanding by aligning features of geometrically consistent keypoints extracted from a pretrained 3D foundation model. The dynamic-view stream achieves adaptive adjustment through position-aware pretraining of an egocentric gaze estimation model, computationally replicating the human cortical dorsal pathway. Subsequently, the complementary view representations of both streams are integrated to determine the final actions, enabling the model to handle spatially-complex and dynamically-changing tasks under language conditions. Empirical evaluations on RLBench, the challenging COLOSSEUM benchmark, and real-world tasks demonstrate that Cortical Policy outperforms state-of-the-art baselines substantially, validating the superiority of dual-stream design for visuomotor control. Our cortex-inspired framework offers a fresh perspective for robotic manipulation and holds potential for broader application in vision-based robot control.
Figures & tables
Figure 1: Deficiencies of prior view transformers for robotic manipulation. (Top) A task requiring the robot to understand the spatial relationships between two bottles before deciding on the placing position. Previous method RVT-2 fails to merge different camera views correctly in 3D, causing wrong block placement. (Bottom) Dynamic adaptation failure during object displacement.
Figure 2: Overview of the proposed cortical policy. Inspired by the dorsal-ventral pathways in visual neuroscience, this architecture implements dual processing streams: a static-view stream for 3D spatial understanding and a dynamic-view stream for end-effector position awareness.
Table 1: Comparison with SOTA methods on RLBench. The “Avg. Success” and “Avg. Rank” columns report the average success rate (%) and the average rank across 18 tasks. Best results are highlighted in bold, and the second best are underlined.
Models
Arch.
Lcgc
Pre.
Heat.
Avg. Success ↑
Avg. Rank ↓
Close Jar
Drag Stick
Insert Peg
Meat off Grill
Open Drawer
Place Cups
Place Wine
Push Buttons
A
Single
✗
–
–
77.5
3.3
93.3 ±1.9
97.3 ±1.9
28.0 ±3.3
100.0 ±0.0
92.0 ±3.3
32.0 ±5.7
84.0 ±9.8
100.0 ±0.0
B
Single
✔
–
–
80.1
2.4
94.7 ±1.9
100.0 ±0.0
25.3 ±5.0
100.0 ±0.0
94.7 ±1.9
21.3 ±1.9
86.7 ±5.0
100.0 ±0.0
C
Dual
✗
✗
✔
77.6
3.0
97.3 ±0.0
98.7 ±0.0
30.7 ±10.0
100.0 ±0.0
88.0 ±3.3
28.0 ±6.5
93.3 ±5.0
100.0 ±0.0
D
Dual
✗
✔
✗
73.3
4.8
90.7 ±1.9
90.7 ±13.2
26.7 ±3.8
100.0 ±0.0
94.7 ±1.9
20.0 ±0.0
82.7 ±8.2
98.7 ±1.9
E
Dual
✗
✔
✔
79.5
3.1
90.7 ±1.9
97.3 ±1.9
29.3 ±1.9
100.0 ±0.0
100.0 ±0.0
46.7 ±7.5
88.0 ±5.7
100.0 ±0.0
F (Ours)
Dual
✔
✔
✔
81.0
1.9
96.0 ±0.0
100.0 ±0.0
38.7 ±6.8
100.0 ±0.0
84.0 ±6.5
24.0 ±3.3
94.7 ±3.8
100.0 ±0.0
Table 2: Ablation study on dual-stream view transformer. All designs contribute to improving performance of Cortical Policy. “Arch.”, “Pre.”, “Heat.” denote model architecture, position-aware pretraining, dynamic-view heatmap, respectively. “Single” means single-stream model with only static viewpoints, “Dual” means dual-stream model integrating dynamic and static viewpoints.
Models
Arch.
Lcgc
Pre.
Heat.
Avg. Success ↑
Avg. Rank ↓
All Perturbations
MO- Color
RO- Color
MO- Texture
RO- Texture
MO- Size
PerAct
–
–
–
–
7.7
7.0
0.0 ±0.0
8.0 ±8.5
5.3 ±5.0
2.0 ±2.0
4.0 ±3.3
16.0 ±17.3
RVT
–
–
–
–
37.7
6.0
3.0 ±5.2
27.0 ±27.3
36.0 ±15.0
50.0 ±38.0
57.3 ±32.7
50.7 ±29.6
RVT-2
Single
✗
–
–
60.5
4.4
15.0 ±17.3
64.0 ±25.6
64.9 ±27.2
93.4 ±2.7
66.2 ±31.8
80.4 ±17.5
Variant B
Single
✔
–
–
63.8
3.3
10.3 ±8.0
69.7 ±28.1
70.2 ±28.0
95.4 ±0.7
71.6 ±30.9
84.0 ±14.2
Variant D
Dual
✗
✔
✗
66.4
2.9
10.0 ±8.2
69.7 ±27.5
72.9 ±29.9
94.0 ±2.0
74.7 ±24.5
83.6 ±16.7
Variant E
Dual
✗
✔
✔
68.7
2.4
8.7 ±15.0
75.0 ±29.7
73.8 ±29.6
96.7 ±0.7
71.1 ±33.4
82.7 ±13.6
Table 3: Results on THE COLOSSEUM. The “Avg. Success” and “Avg. Rank” columns report the average success rate (%) and the average rank across all perturbations on 4 COLOSSEUM tasks.
Figure 4: (a) Training time of Cortical Policy modules , with time cost of 3D supervision generation, dual streams, action head. (b) (Top) Real-world performance comparison. (Bottom) Visualization of the initial and final states for the four real-world tasks. (c) Trajectory visualization for “stack 2 blocks with base displacement” task.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Task Name
Language Template
#of Variations
Variation Type
close jar
“close the jar”
20
color
drag stick
“use the stick to drag the cube onto the target”
20
color
insert peg
“put the ring on the spoke”
20
color
meat off grill
“take the off the grill”
2
category
open drawer
“open the drawer”
3
placement
place cups
“place cups on the cup holder”
3
count
Appendix
Table 4: Summary of the 18 RLBench tasks for multi-task experiments.
Figure 5: Rendered views for 18 RLBench tasks. Three orthographic static cameras and a dynamic camera (defined by the wrist camera’s raw extrinsic parameters) are used to generate image inputs. For each task, the first three lines show the static views (top, front, right), and the last line shows the dynamic view; rendered results of the first (coarse) stage are presented in the left part while that of the second (fine) stage are shown right.
Figure 6: Real-world setup.
Hyperparameters
Value
Batch size
512
Learning rate
5.44×10−3
Optimizer
LAMB
Learning rate schedule
cosine decay
Weight decay
1×10−4
Warm-up steps
2000
Appendix
Table 5: Training Hyperparameters of Cortical Policy.
Figure 7: Comparison of dynamic egocentric views and rendered examples.
Figure 8: RVT-2 behavior comparison. (Top) In the basic task ”stack 2 blocks”, RVT-2 places the first block near the robot arm. (Bottom) In the spatial task ”stack 2 blocks in between the bottles”, RVT-2 attempts to place the first block near the bottles but fails due to imprecision.
Figure 9: Success rate variations with training epochs for compute-control evaluation.
Task Name
Models
Avg. Success ↑
No Perturbations
All Perturbations
MO- Color
RO- Color
MO- Texture
RO- Texture
MO- Size
drag stick
PerAct
17.6
36
0
20
12
4
8
40
RVT
59.2
84
0
24
52
88
88
92
RVT-2
69.8
84.0 ±3.3
0.0 ±0.0
84.0 ±3.3
80.0 ±0.0
90.7 ±1.9
89.3 ±1.9
89.3 ±1.9
Variant B
73.4
85.3 ±5.0
0.0 ±0.0
90.7 ±1.9
89.3 ±1.9
94.7 ±1.9
90.7 ±1.9
92.0 ±0.0
Variant D
78.1
88.0 ±0.0
0.0 ±0.0
92.0 ±0.0
96.0 ±0.0
96.0 ±0.0
92.0 ±0.0
94.7 ±1.9
Variant E
78.3
88.0 ±0.0
0.0 ±0.0
92.0 ±0.0
96.0 ±0.0
97.3 ±1.9
92.0 ±0.0
88.0 ±0.0
Appendix
Table 6: Success rates of different methods under various perturbations of COLOSSEUM.
Models
Parameter Size
Parameter Num
FLOPs
Avg. Success
RVT-2
277.3MB
72.7M
14.98 G
77.5
Baseline (Deeper)
596.6MB
156.4M
25.59 G
78.4
Cortical Policy (Ours)
551.9MB
144.7M
22.37 G
81.0
Appendix
Table 7: Capacity-controlled ablation study on RLBench.
Figure 10: Additional visualization of geometrically consistent keypoints.