Geometric structure is essential for robotic manipulation and understanding the 3D world. Most VLA models rely on 2D representations of current observations, while geometry-aware approaches often require additional 3D information. However, geometry inferred independently from individual observations is often discarded rather than accumulated into a persistent task representation. Our key insight is that action generation needs a compact, task-relevant geometric abstraction that can be continuously updated, rather than a complete, renderable 3D reconstruction at every step. We introduce GeoMem-VLA, a VLA framework that internalizes explicit 3D supervision into dedicated geometric tokens and maintains them as temporal memory for action generation, without directly aligning the VLM's native visual embeddings with the teacher feature space. During training, a frozen geometry foundation model supervises learnable GeoPlan tokens through a geometric flow-matching objective over the current scene point map and future end-effector tracks. These tokens are integrated with the memory of observation representations, where historical information is retrieved, adaptively fused with current observations, and consolidated to retain event frames as the task progresses. At inference, the policy retrieves historical representations and fuses them with the current working representation to predict actions, without invoking the 3D teacher or explicitly reconstructing the scene. After benchmark-specific downstream adaptation, GeoMem-VLA achieves 98.9% average success on LIBERO and 91.10% / 91.24% on RoboTwin's clean/randomized settings. Across five real-world tasks, GeoMem-VLA achieves 78.0% average success. These results suggest that internalizing and remembering 3D scene understanding is a promising approach to robot control.
Figures & tables
Figure 1: Overview of GeoMem-VLA . Top: In the illustrative cover and pick up the apple task, the current observation alone may not reveal the apple’s location once it is occluded by the towel, motivating the retention of spatial context over time. Middle: GeoMem-VLA learns geometry-supervised GeoPlan representations and retains them in causal episode memory for action generation. Bottom: Performance comparisons across manipulation benchmarks and real-world tasks.
Figure 2: Architecture of GeoMem-VLA . Images, language, and learnable 3D marker tokens are encoded by the VLM. The geometry branch is trained with scene-geometry targets and demonstration-derived future end-effector tracks, whereas the action expert is trained to predict robot actions. Geometric and observational features are retained in episodic memory and fused to form the context for action generation. The geometry teacher and reconstruction head are used only during training.
Method
Spatial
Object
Goal
Long
Average
OpenVLA (FT) ( Kim et al., 2024 )
84.7
88.4
79.2
53.7
76.5
TraceVLA ( Zheng et al., 2025 )
84.6
85.2
75.1
54.1
74.8
SpatialVLA ( Qu et al., 2025 )
88.2
89.9
78.6
55.5
78.1
4D-VLA ( Zhang et al., 2026b )
88.9
95.2
90.9
79.1
88.6
DreamVLA ( Zhang et al., 2026d )
97.5
94.0
89.5
89.5
92.6
π0 ( Driess et al., 2026 )
96.8
98.8
95.8
85.2
94.2
Table 1: LIBERO Success Rates (%). References beside method names identify the first-party source for each published result. Averages cover the four displayed suites. Bold and underline mark the highest and second-highest distinct values in each column, including ties.
Method
Clean
Randomized
Average
X-VLA ( Zheng et al., 2026 )
70.00
39.00
54.50
Track4Action ( Wang et al., 2026a )
80.44
81.48
80.96
ABot-M0 ( Yang et al., 2026b )
86.06
85.08
85.57
Qwen-VLA-Instruct ( Wang et al., 2026b )
86.10
87.20
86.65
LingBot-VLA ( Wu et al., 2026a )
88.56
86.68
87.62
StarVLA-OFT ( Community, 2026 )
88.18
88.32
88.25
Table 2: RoboTwin 2.0 success rates (%). Clean and Randomized refer to the 50-task evaluation (called Easy/Hard in some reports). Each Boldface and underlining indicate the highest and second-highest distinct values in each column, including ties.
Configuration
Geo.
Mem.
Spatial
Object
Goal
Long
Average
GeoMem-VLA (full)
✓
✓
99.6
99.8
99.6
96.6
98.9
w/o geometry branch
✗
✓
98.0
99.6
98.8
95.2
97.9
w/o temporal memory
✓
✗
99.4
99.6
99.4
94.2
98.2
StarVLA-GR00T
✗
✗
97.8
98.8
97.4
92.0
96.5
Table 3: Component Ablations on LIBERO. Success rate (%) over 50 trials per task. Geo. and Mem. indicate our GeoPlan branch and causal episode memory, respectively.
Configuration
Geo.
Mem.
Clean
Randomized
Average
GeoMem-VLA (full)
✓
✓
91.10
91.24
91.17
w/o geometry branch
✗
✓
89.68
89.74
89.71
w/o temporal memory
✓
✗
89.84
90.52
90.18
StarVLA-GR00T
✗
✗
88.00
88.50
88.25
StarVLA-OFT
✗
✗
88.18
88.32
88.25
Table 4: Component Ablations on RoboTwin 2.0. Success rate (%) over 50 tasks with 100 episodes per task and setting. Geo. and Mem. indicate our GeoPlan branch and causal episode memory.
Figure 3: Geometry, future motion, and retained memory in RoboTwin. (a,b) Randomized examples show RGB observations, VGGT point-map targets, predicted point maps, and future EEF tracks (solid: prediction; dashed: demonstration). (c) Final memory retention in Randomized stacking: the observational row shows the intersection of both observation banks, while the geometric row shows geometric memory (dots: retained; crosses: not retained; vertical shaded bands: shared timestamps), with corresponding frames enlarged below.
Figure 4: Real-world Manipulation Evaluation of GeoMem-VLA. The figure shows the real-world experiments task settings and success rates across five tasks. Each bar reports the success rate over 10 independent closed-loop rollouts.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
LIBERO
RoboTwin
Astribot
AgileX
Action dimension
7
14
16
7
State dimension
8
14
16
7
Prediction horizon H
8
50
50
50
Training steps
300000
200000
20000
20000
Gradient accumulation
4
8
1
2
Teacher input size
336
336
336
336
Appendix
Table 5: Training configurations. All use Qwen3-VL-4B and a frozen VGGT teacher. The AgileX settings apply to both tasks run through the shared training launcher.
Full
w/o Geo
w/o Mem
Full
w/o Geo
w/o Mem
#
Task
C
R
C
R
C
R
#
Task
C
R
C
R
C
R
1
adjust_bottle
100
100
100
100
100
100
26
place_can_basket
87
93
79
79
74
76
2
beat_block_hammer
99
94
93
93
98
92
27
place_cans_plasticbox
99
100
100
99
98
97
3
blocks_ranking_rgb
99
95
97
96
99
99
28
place_container_plate
100
100
98
98
99
98
4
blocks_ranking_size
87
86
86
85
83
80
29
place_dual_shoes
85
81
84
70
87
88
5
click_alarmclock
92
97
87
93
91
95
30
place_empty_cup
100
100
100
100
100
100
Appendix
Table 6: Per-task results on RoboTwin 2.0. Success rate (%) for all 50 tasks under Clean (C) and Randomized (R) settings, 100 episodes per task. GeoMem-VLA is compared against its two ablated variants: without the geometry branch (w/o Geo) and without temporal memory (w/o Mem).
Task
Setting
Point-map RMSE
EEF RMSE (mm)
Open microwave
Clean
0.076
3.69
Open microwave
Randomized
0.094
3.74
Stack three blocks
Clean
0.070
3.07
Stack three blocks
Randomized
0.064
3.64
Appendix
Table 7: Coverage of the 200K reconstruction diagnostics. All four available cases and all 16 reconstructed frames are included. Point-map errors are unweighted; EEF errors include moving and stationary arms.
Figure 5: Detailed future EEF tracks for the examples in Figure 3 . Dashed curves are demonstration-derived targets; solid curves are predictions from the auxiliary head. Each arm contributes 50 waypoints. Dots mark target starting positions. Both arms within a row use equal spatial scale, with coordinates displayed in centimeters. The right panels show Euclidean error at every future index.
Figure 6: Geometry reconstruction and memory analysis on RoboTwin manipulation tasks. For each example, we visualize: (a) dense geometry targets provided by the frozen VGGT teacher, (b) reconstructed point maps from GeoPlan representations, (c) predicted future end-effector tracks, (d) retrieval attention across perceptual, cognitive, and geometric memories, and (e) retained-event alignment between observation and geometric memory banks.
Figure 7: Successful real-world rollouts on Astribot S1. The figure presents representative execution sequences for three manipulation tasks: pick and stand sticks, separate file pockets, and transfer cup across the rack. GeoMem-VLA completes long-horizon manipulation procedures under randomized object configurations, demonstrating accurate spatial interaction and effective task-state retention.
Figure 8: Failure cases of GeoMem-VLA on Astribot S1 tasks. The figure shows representative unsuccessful rollouts for pick and stand sticks and transfer cup across the rack.
Figure 9: Successful real-world rollouts on AgileX Cobot Magic 2. The figure presents representative execution sequences for two manipulation tasks: cover and find apple under towel and pick cube repeatedly. GeoMem-VLA completes long-horizon manipulation procedures involving hidden-object reasoning and repeated interactions, demonstrating effective spatial understanding and task-state retention.
Figure 10: Failure cases of GeoMem-VLA on AgileX Cobot Magic 2 tasks. The figure shows representative unsuccessful rollouts for cover and find apple under towel and pick cube repeatedly.
Figure 11: Geometry reconstruction and memory analysis on real-world Astribot tasks. For three representative tasks, we visualize: (a) dense teacher geometry estimated from RGB observations, (b) reconstructed point maps from GeoPlan tokens, (c) predicted future end-effector waypoints, (d) memory retrieval similarity across different representation streams, and (e) retained-event alignment between observation and geometric memories.
Figure 12: Geometry reconstruction and memory analysis on real-world AgileX tasks. For two representative tasks, we visualize: (a) dense teacher geometry estimated from RGB observations, (b) reconstructed point maps from GeoPlan tokens, (c) predicted future end-effector waypoints, (d) memory retrieval similarity across different representation streams, and (e) retained-event alignment between observation and geometric memories.
State Key Laboratory of Multimedia Information Processing and National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, Beijing, China · CUHK, Shatin, Hong Kong · AI2Robotics, Beijing, China