From Predictive Geometry to Action: Learning Complementary Predictive Representations for Embodied Manipulation

Xiaoxu Xu; Hao Li; Hongtao Zhang; Yili Tang; Jiafei Cao; Junhao Cai; Haoxiang Ma; Weixin Li

cs.ROOct 20, 2026

From Predictive Geometry to Action: Learning Complementary Predictive Representations for Embodied Manipulation

Authors: Xiaoxu Xu, Hao Li, Hongtao Zhang, Yili Tang, Jiafei Cao, Junhao Cai, Haoxiang Ma, Weixin Li,

Organizations: Beihang University · Shanghai Artificial Intelligence Laboratory

Abstract

Effective manipulation requires a policy to capture not only about the current scene geometry, but also about how that geometry will evolve under interaction. Existing approaches either augment policies with geometric information from current observations or introduce future prediction as auxiliary supervision, leaving open how predictive geometry should be represented for action generation. Therefore, we propose a two-stage framework, which learns complementary predictive representations from multi-view video clips. During pretraining, a frozen 3D foundation model provides dense geometric features that are compressed into predictive geometric representations and optimized to preserve scene structure and its temporal evolution. Action supervision is then applied to geometric transitions to extract a complementary motor representation that emphasizes action-relevant transitions. During downstream post-training, predictive geometry is transferred through geometry queries, while the motor representation guides intermediate features along the action-generation pathway. Future observations are used only as privileged supervision during training and are not required at inference time. Across LIBERO, LIBERO-Plus, SimplerEnv, and real-world bimanual and single-arm tasks, our approach consistently improves downstream policies, including an absolute gain of 10.4 percentage points on the WidowX evaluation. We further demonstrate transfer across multiple VLA architectures and WAMs, with improvements in both future-world and action prediction. The code will be available at https://github.com/xuxiaoxxxx/CPR.

Figures & tables

Figure 1: Comparison of geometry-guidance paradigms for policy learning. (a) Input-centric methods augment policies with spatial information derived from current observations. (b) Prediction-centric methods use future-world or predictive targets as additional supervision. (c) Our approach learns a predictive geometric representation from dense geometric features and derives a complementary motor representation from the evolution of scene geometry. The two representations are transferred through role-specific guidance pathways during training and are not required at inference time.

Figure 2: Overview of complementary predictive representation pretraining. A frozen 3D foundation model extracts dense geometric features from multi-view video observations, which are compressed by the Hierarchical Geometric Tokenizer and temporally modeled by a causal Slot-Temporal Mixer to produce the predictive geometric representation. Geometry reconstruction, masked geometric representation prediction, and geometric transition prediction jointly shape this representation to preserve scene structure and capture its temporal evolution. The resulting geometric transitions are used with action supervision to extract an action-aware motor representation, providing complementary predictive and motor information for policy learning.

Figure 3: Overview of representation-guided policy post-training. The frozen pretrained model provides the predictive geometric representation $\mathbf{G}$ and motor representation $\mathbf{Z}$ as privileged supervision. The two representations are transferred through role-specific pathways: $\mathbf{G}$ supervises dedicated geometry queries, while $\mathbf{Z}$ aligns with intermediate features along the action-generation pathway.

Table 4

Figure 4: Performance comparison on real-robot manipulation tasks.

Figure 5: Transfer of CPR to different downstream VLA models on the WidowX robot.

Methods	Object	Spatial	Goal	Long	Avg.
FastWAM wo/ CPR	100.0	98.2	97.2	95.2	97.6
FastWAM w/ CPR	100.0	99.0	97.4	96.8	98.5

Table 3: Transfer of CPR to FastWAM on the LIBERO.

Figure 6: Probing analysis of representation organization on LIBERO-90. The top compares action-error distributions between the CPR motor and unified representations, while the bottom shows depth-probing results.

Table 9

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Out-of-Distribution success rates on real-world tasks under visual and spatial perturbations.

Methods	Cam.	Rob.	Lang.	Light	Back.	Noise	Layout	Total
OpenVLA-OFT ( Kim et al., 2025 )	56.4	31.9	79.5	88.7	93.3	75.8	74.2	69.6
$\pi_{0}$ ( Black et al., 2024 )	13.8	6.0	58.8	85.0	81.4	79.0	68.9	53.6
WorldVLA ( Cen et al., 2025 )	0.1	27.9	41.6	43.7	17.1	10.9	38.0	25.0
UniVLA ( Bu et al., 2025 )	1.8	46.2	69.6	69.0	81.0	21.2	31.9	43.9
4D-WAM ( Yang et al., 2026a )	45.2	64.3	90.6	94.3	57.7	69.1	79.2	71.0
ELAN4D ( He et al., 2026 )	63.7	70.7	77.8	89.8	91.4	79.9	81.4	78.2

Appendix

Table 6: Comparison of different VLA models on LIBERO-Plus. Success rates (%) are reported under the seven perturbation types. Total denotes the official task-count-weighted mean over all 10,030 perturbed tasks, rather than an unweighted mean of the seven categories.

Figure 8: Visualization of the real-world out-of-distribution perturbations. From left to right: Remove Stopper , with randomly shifted flask positions; Store Tubes , with randomly varied target-box heights; Transfer Liquid , with a randomly moving toy car as a dynamic visual distractor; and Make Sandwich , with additional irrelevant objects added to the workspace. These perturbations introduce changes in object placement, target geometry, dynamic visual context, and scene clutter while preserving the underlying manipulation tasks.

Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Action $\ell_{1}\downarrow$
FastWAM w/o CPR	19.67	0.764	0.229	0.124
FastWAM w/ CPR	22.91	0.803	0.186	0.075

Appendix

Table 7: Zero-shot future-world and action prediction performance of FastWAM on LIBERO-90. PSNR and SSIM are higher-is-better, while LPIPS and action

\ell_{1}

error are lower-is-better.

Figure 10: Qualitative comparison of zero-shot future world prediction on LIBERO-90. Each example contains a main camera view (upper half) and a wrist camera view (lower half), with columns progressing over nine predicted future frames. Within each view, the three rows from top to bottom correspond to ground-truth observations (GT), FastWAM without CPR guidance (w/o CPR), and FastWAM with CPR guidance (w/ CPR).

Figure 11: Distribution of action-probing experiment.

Figure 13: Qualitative visualization of the depth-probing analysis for representation organization. Each example contains a main camera view (upper half) and a wrist camera view (lower half), with columns progressing over nine future frames. Within each view, the rows from top to bottom show the corresponding RGB observations, depth predictions decoded from the unified representation, depth predictions decoded from the predictive geometric representation, and the pseudo-depth references extracted by Depth Anything ( Lin et al., 2025 ) . The predictive geometric representation produces depth maps that more faithfully recover object geometry, relative depth, and overall scene structure than the unified representation.

Methods	Put Spoon	Put Carrot	Stack Cube	Put Eggplant	Avg.
WoG ( Su et al., 2026 )	83.3	62.5	37.5	75.0	64.6
FutureVLA ( Xu et al., 2026 )	79.2	70.8	41.7	79.2	67.7
RepWAM ( Wang et al., 2026a )	87.5	75.0	54.2	58.3	68.8
Ours	95.8	70.8	70.8	50.0	71.9

Appendix

Table 8: Comparison with different predictive representation learning methods under a unified downstream policy framework on the WidowX evaluation.

	$\mathcal{L}_{\text{rec}}$	$\mathcal{L}_{\text{mask}}$	$\mathcal{L}_{\text{trans}}$	Put Spoon	Put Carrot	Stack Cube	Put Eggplant	Avg.
(a)				83.3	58.3	54.2	50.0	61.5
(b)	✓			87.5	66.7	58.3	50.0	65.6
(c)	✓	✓		91.7	70.8	66.7	45.8	68.8
(d)	✓	✓	✓	95.8	70.8	70.8	50.0	71.9

Appendix

Table 9: Ablation of the pretraining objectives for the predictive geometric representation on the WidowX evaluation.

Figure 14: Attention-map comparison between the unguided baseline (middle) and the CPR-guided policy (right). CPR guidance produces more concentrated responses around task-relevant objects and interaction regions.

Motor Input	Put Spoon	Put Carrot	Stack Cube	Put Eggplant	Avg.
Predictive States $\mathbf{G}$	87.5	66.7	66.7	54.2	68.8
Geometric Transitions $\Delta\mathbf{G}$	95.8	70.8	70.8	50.0	71.9

Appendix

Table 10: Effect of the input used to learn the motor representation on the WidowX evaluation.

Alignment Layer	6	12	18	24	30	36
SR (%)	69.8	71.9	67.7	64.6	65.6	61.5

Appendix

Table 11: Effect of the motor representation alignment depth on the WidowX evaluation. The alignment is applied to intermediate features from different layers.

Methods	Put Spoon	Put Carrot	Stack Cube	Put Eggplant	Avg.
DINO	87.5	75.0	54.2	62.5	69.8
VGGT	95.8	70.8	70.8	50.0	71.9

Appendix

Table 12: Effect of the foundation encoder used during predictive representation pretraining on the WidowX evaluation.

$\lambda_{\mathrm{geo}}$	0.1	0.3	0.5	0.7	0.9
SR (%)	69.8	71.9	70.8	68.8	65.6

Appendix

Table 13: Ablation study of the geometry-guidance loss weight

\lambda_{\mathrm{geo}}

on the WidowX evaluation.

Guidance Paradigm	Put Spoon	Put Carrot	Stack Cube	Put Eggplant	Avg.
Input-centric	83.3	66.7	58.3	45.8	63.5
Prediction-centric	87.5	62.5	62.5	50.0	65.6
Ours	95.8	70.8	70.8	50.0	71.9

Appendix

Table 14: Controlled comparison of different geometry-guidance paradigms under the same downstream VLA framework on the WidowX evaluation. The input-centric variant uses current-frame VGGT features as policy input, while the prediction-centric variant predicts future VGGT features as auxiliary guidance.

Figure 16: Qualitative visualization of predictive geometry transferred through the geometry-query pathway. The first panel shows examples from LIBERO, where each example contains a main camera view (upper half) and a wrist-camera view (lower half); within each view, the rows from top to bottom show the RGB observation, the depth prediction decoded from the predicted geometry-query features, and the pseudo-depth reference generated by Depth Anything. The second panel shows real-world manipulation examples with three camera views, corresponding to the main view, the left-arm wrist view, and the right-arm wrist view. For each view, the rows from top to bottom follow the same order: RGB observation, predicted depth, and pseudo-depth reference. The results illustrate that the geometry-query pathway can recover meaningful object geometry and relative spatial structure across both simulated and real-world settings.

Dataset	Ratio
UTAustin Mutex ( Shah et al., 2023 )	2.3
DLR EDAN Shared Control ( Quere et al., 2020 )	0.7
Taco Play ( Rosete-Beas et al., 2023 ; Mees et al., 2022 )	2.3
Austin Sirius Dataset ( Liu et al., 2025 )	1.5
Roboturk ( Mandlekar et al., 2019 )	1.6
Austin Sailor Dataset ( Nasiriany et al., 2022 )	1.7

Appendix

Table 15: Our training data mixture used during the pretraining phase.

Figure 17: Attention mask of FastWAM. $F$ denotes video frames, $G$ indicates predictive geometric representation and $A$ represents actions to be denoised.

Figure 18: Bimanual and single-arm manipulation platforms.

Figure 19: Visualization of four real-robot manipulation tasks.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,685 papers

Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation
Feb 27, 2026Chongyang Xu, Haipeng Li, Shen Cheng +4Bimanual Robotic Manipulation Robotic Manipulation
College of Computer Science, Sichuan University, China · Dexmal · University of Electronic Science and Technology of China, China +1
GeoLAM: Learning Geometry-Grounded Latent Actions from Unlabeled Human Videos
Sep 15, 2026Yifan Xie, Hekun Tian, Jinkun Liu +3Latent Action Learning Geometric Representation Learning
Tsinghua University · Beijing Institute of Technology · Fudan University
Geometric Action Model for Robot Policy Learning
Jun 15, 2026Jisang Han, Seonghu Jeon, Jaewoo Jung +7World Models for Robotics Language-Conditioned Robot Manipulation
KAIST AI · ETH Zurich · ETH AI Center

Feb 27, 2026J/K move · Enter open · S save

Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation

Chongyang Xu, Haipeng Li, Shen Cheng, Jingyu Hu+3

College of Computer Science, Sichuan University, China · Dexmal · University of Electronic Science and Technology of China, China+1

Bimanual Robotic Manipulation Robotic Manipulation

Open paper AskPDF arXiv

Bimanual manipulation requires policies that can reason about 3D geometry, anticipate how it evolves under action, and generate smooth, coordinated motions. However, existing methods typically rely on 2D features with limited spatial awareness, or require explicit point clouds that are difficult to obtain reliably in real-world settings. At the same time, recent 3D geometric foundation models show that accurate and diverse 3D structure can be reconstructed directly from RGB images in a fast and robust manner. We leverage this opportunity and propose a framework that builds bimanual manipulation directly on a pre-trained 3D geometric foundation model. Our policy fuses geometry-aware latents, 2D semantic features, and proprioception into a unified state representation, and uses diffusion model to jointly predict a future action chunk and a future 3D latent that decodes into a dense pointmap. By explicitly predicting how the 3D scene will evolve together with the action sequence, the policy gains strong spatial understanding and predictive capability using only RGB observations. We evaluate our method both in simulation on the RoboTwin benchmark and in real-world robot executions. Our approach consistently outperforms 2D-based and point-cloud-based baselines, achieving state-of-the-art performance in manipulation success, inter-arm coordination, and 3D spatial prediction accuracy. Code is available at https://github.com/Chongyang-99/GAP.git.

Abstract

Robotic Manipulation

Explore similar work

Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation

GeoLAM: Learning Geometry-Grounded Latent Actions from Unlabeled Human Videos

Geometric Action Model for Robot Policy Learning