Lift, See, Act: Hierarchical Robot Policy Pretraining with 3D Foundation Models
Organizations: School of Electronic and Information Engineering, South China University of Technology
Abstract
Robot policy pretraining based on human videos is crucial for improving the policy's generalization ability. One main challenge for this task is the lack of explicit action-relevant representations in such unlabeled data. Recent works tend to estimate 3D hand motion trajectories from videos using 3D foundation models (3D FMs). However, they keep solely the sparse trajectories as supervision, ignoring the rich, fine-grained information produced during the trajectory extraction process. To address this issue, we propose LSA, a hierarchical robot policy pretraining framework that includes three phases: Lift, See, and Act. Specifically, the Lift phase elevates 2D video observations into dense 3D representations by adaptively aligning features in the policy's shallow layers with the intermediate representations extracted from multiple 3D FMs. Building upon these lifted representations, the See phase equips the policy with explicit geometric and interactive awareness through dual guidance. It introduces depth-map reconstruction to help the model comprehend 3D spatial layouts and utilizes hand region cues to explicitly supervise the encoder's attention toward task-relevant interaction hotspots. Empowered by the dense 3D representation learning and precise spatial guidance, our policy achieves robust and accurate robotic manipulation in the Act phase. Extensive experiments on diverse simulation and real-world tasks demonstrate that LSA significantly outperforms current state-of-the-art approaches. The project page is available at: https://yiyuan00.github.io/Lift-See-Act.
Figures & tables
| Method | MetaWorld | DexArt | Adroit | ||||||||||
| HA | DT | AS | HI | PPW | ShP | DI | StP | BU | TO | DR | HA-A | PE | |
| Direct 3D Policy | |||||||||||||
| DP3 ( RSS’24 ) | \text{91.1}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{2.8}}} | \text{84.9}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{2.0}}} | \text{51.1}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{3.3}}} | \text{33.8}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{4.0}}} | \text{32.9}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{3.0}}} | \text{48.4}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{4.7}}} | \text{40.4}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{3.5}}} | \text{39.1}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{4.0}}} | \text{26.7}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{1.7}}} | \text{71.7}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{2.2}}} | \text{63.3}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{1.7}}} | \textbf{100.0}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{0.0}}} | \text{40.6}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{1.0}}} |
| KAN-We-Flow ( ICRA’26 ) | \text{89.3}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{1.4}}} | \text{89.8}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{2.8}}} | \text{61.8}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{4.1}}} | \text{58.7}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{1.4}}} | \text{68.0}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{4.0}}} | \text{70.2}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{2.8}}} | \text{68.9}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{2.0}}} | \text{66.7}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{4.0}}} | \text{31.1}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{2.6}}} | \underline{\text{76.7}}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{1.7}}} | \underline{\text{80.5}}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{2.5}}} | \textbf{100.0}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{0.0}}} | \underline{\text{56.7}}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{3.4}}} |
| 2D Pretrain Policy | |||||||||||||
| CLIP ( ICML’21 ) | \text{95.7}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{1.2}}} | \text{78.3}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{4.6}}} | \text{88.7}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{4.0}}} | \text{61.7}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{5.7}}} | \underline{\text{92.3}}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{1.2}}} | \text{72.3}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{5.7}}} | \text{82.7}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{3.2}}} | \text{70.3}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{3.1}}} | \text{58.7}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{3.5}}} | \text{42.0}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{0.0}}} | \text{36.7}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{4.6}}} | \text{75.0}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{3.4}}} | \text{36.7}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{4.3}}} |
| Method | Pluck Flower | Pick Trash | Place Toy | Wipe PC | Wipe PC U1 | Wipe PC U2 | Avg. |
| DINOv3 [ 45 ] | 20.0 | 30.0 | 20.0 | 30.0 | 10.0 | 20.0 | 21.7 |
| R3M-A [ 37 ] | 10.0 | 30.0 | 30.0 | 40.0 | 0.0 | 10.0 | 20.0 |
| VIP [ 47 ] | 50.0 | 40.0 | 50.0 | 40.0 | 20.0 | 20.0 | 36.7 |
| HRP [ 8 ] | 40.0 | 40.0 | 60.0 | 50.0 | 20.0 | 40.0 | 41.7 |
| Ours | 50.0 | 70.0 | 80.0 | 80.0 | 70.0 | 60.0 | 68.3 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | SR (%) | Params (M) | GFLOPs | FPS | Slowdown |
| Full LSA | 92.2 | 86.1 | 35.1 | 230.6 | 1.0 |
| Direct Depth Pro | 40.7 | 953.0 | 382.3 | 59.6 | 3.9 |
| Direct VGGT | 29.8 | 911.0 | 493.6 | 18.7 | 12.3 |
| Direct Dual-FM | 26.9 | 1,863.7 | 875.9 | 14.5 | 15.9 |
| LSA w/o Lift + Depth Pro | 81.4 | 1,039.4 | 417.5 | 47.0 | 4.9 |
| LSA w/o Lift + VGGT | 61.6 | 997.3 | 528.8 | 17.5 | 13.2 |
| Teacher Combination | SR (%) |
| VGGT + Depth Pro | 92.2 |
| DA3 + Depth Pro | 92.0 |
| VGGT + UniDepthV2 | 92.5 |
| DA3 + UniDepthV2 | 91.8 |
| Layer (Align, Recon) | Ours | Varying Alignment | Varying Reconstruction | Reversed Order | Shared Layer | |||
| (3, 6) | (1, 6) | (5, 6) | (3, 4) | (3, 8) | (6, 3) | (3, 3) | (6, 6) | |
| SR (%) | 92.2 | 89.5 | 88.6 | 88.3 | 84.6 | 89.3 | 85.8 | 82.3 |
| Method | HA | DT | AS | HI | PPW | ShP | DI | StP | Avg. |
| VITRA [ 12 ] | 82.6 | ||||||||
| Moto [ 67 ] | 62.3 | ||||||||
| H-RDT [ 32 ] | 87.6 | ||||||||
| LSA (Ours) | 92.0 |
| Place Toy | Wipe PC | Average Drop | ||||||
| Method | Original | Background | Low-Light | Original | Background | Low-Light | Background | Low-Light |
| HRP | 65.0 | 35.0 ( ) | 30.0 ( ) | 60.0 | 35.0 ( ) | 25.0 ( ) | ||
| LSA | 80.0 | 65.0 ( ) | 60.0 ( ) | 75.0 | 60.0 ( ) | 50.0 ( ) | ||
Robot Learning
Robot learning develops methods for robots to acquire skills and adapt from demonstrations, interaction, data, and physical models.
See all 343 papers