cs.RONov 16, 2026

Lift, See, Act: Hierarchical Robot Policy Pretraining with 3D Foundation Models

Authors: Yiyuan Ge, Changxing Ding, Ziyu Hao, Zijie Zheng, Xiangmin Xu

Organizations: School of Electronic and Information Engineering, South China University of Technology

Abstract

Robot policy pretraining based on human videos is crucial for improving the policy's generalization ability. One main challenge for this task is the lack of explicit action-relevant representations in such unlabeled data. Recent works tend to estimate 3D hand motion trajectories from videos using 3D foundation models (3D FMs). However, they keep solely the sparse trajectories as supervision, ignoring the rich, fine-grained information produced during the trajectory extraction process. To address this issue, we propose LSA, a hierarchical robot policy pretraining framework that includes three phases: Lift, See, and Act. Specifically, the Lift phase elevates 2D video observations into dense 3D representations by adaptively aligning features in the policy's shallow layers with the intermediate representations extracted from multiple 3D FMs. Building upon these lifted representations, the See phase equips the policy with explicit geometric and interactive awareness through dual guidance. It introduces depth-map reconstruction to help the model comprehend 3D spatial layouts and utilizes hand region cues to explicitly supervise the encoder's attention toward task-relevant interaction hotspots. Empowered by the dense 3D representation learning and precise spatial guidance, our policy achieves robust and accurate robotic manipulation in the Act phase. Extensive experiments on diverse simulation and real-world tasks demonstrate that LSA significantly outperforms current state-of-the-art approaches. The project page is available at: https://yiyuan00.github.io/Lift-See-Act.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions

    Oct 25, 2026Haoyu Zhao, Xingyue Zhao, Zhian Su +11VLA Model TrainingRobot Foundation Models

  2. VidAct: Learning Manipulation from In-the-Wild Videos with Object-Centric 3D Awareness

    Sep 29, 2026Hang Li, Mingxin Zhang, Zihan Wu +9Object-Centric Representation LearningRobotic Manipulation

  3. 3PoinTr: From Human Videos to Robot Policies with 3D Point-Track Plans

    Mar 9, 2026Adam Hung, Bardienus Pieter Duisterhof, Jeffrey IchnowskiHuman-to-Robot Motion ImitationRobot Policy Learning