TTT-Parkour: Rapid Test-Time Training for Perceptive Robot Parkour

Shaoting Zhu; Baijun Ye; Jiaxuan Wang; Jiakang Chen; Ziwen Zhuang; Linzhan Mou; Runhan Huang; Hang Zhao

cs.ROFeb 2, 2026

TTT-Parkour: Rapid Test-Time Training for Perceptive Robot Parkour

Authors: Shaoting Zhu, Baijun Ye, Jiaxuan Wang, Jiakang Chen, Ziwen Zhuang, Linzhan Mou, Runhan Huang, Hang Zhao

Organizations: Tsinghua University · Shanghai Qi Zhi Institute · Princeton University

Abstract

Achieving highly dynamic parkour with humanoid robots on complex terrains unseen during the pre-training stage remains a challenge in robotics. Although general locomotion policies demonstrate capabilities across broad terrain distributions, they often struggle with arbitrary and highly challenging environments. To overcome this limitation, we propose a real-to-sim-to-real framework for rapid test-time training (TTT): environment-specific policy fine-tuning in simulation before deployment on a newly encountered terrain. We adopt a two-stage end-to-end learning paradigm: a policy is first pre-trained on diverse procedurally generated terrains, followed by rapid fine-tuning on high-fidelity meshes reconstructed from real-world captures. Specifically, we develop an efficient, feed-forward, and high-fidelity geometry reconstruction pipeline using RGB-D inputs, ensuring both speed and quality during test-time training. We demonstrate that TTT-Parkour empowers humanoid robots to master complex obstacles, including wedges, stakes, boxes, trapezoids, and narrow beams. The entire pipeline, including capture, reconstruction, and test-time training, takes approximately 10 minutes for most tested terrains. Experiments show improved traversal on the tested terrains after fine-tuning, with direct sim-to-real deployment and no further policy updates on the robot.

Figures & tables

Figure 1: Rapid test-time training on terrains unseen during pre-training. By reconstructing the scene and fine-tuning in simulation, our framework enables the robot to master challenging obstacles in approximately 10 minutes for most tested terrains, turning failure (left) into success (right).

Figure 2: TTT-Parkour . Our framework consists of three stages: (1) Pre-training: A general policy is pre-trained on diverse procedurally generated terrains to learn robust locomotion primitives. (2) Test-time Training (TTT): We reconstruct high-fidelity and simulation-ready meshes from real-world captures using feed-forward reconstruction with automatic scale recovery and frame alignment. The policy is then rapidly fine-tuned on these specific terrains in simulation. (3) Sim-to-Real Deployment: The adapted policy is deployed on the target terrain without further policy updates on the robot.

Figure 3: Efficient Geometry Reconstruction. Our pipeline consists of four stages: (1) Real-World Capture. (2) Feed-forward Reconstruction provides initial scene geometry. (3) Scale Recovery corrects metric scale discrepancies by aligning inferred depth with sensor depth. (4) Physically-consistent Frame Alignment registers the terrain to the simulation coordinate system by aligning the $z$ -axis with gravity and the $x$ -axis with the traversal direction using 3D semantic segmentation.

Figure 4: Real-world experiments. The robot successfully traverses extremely challenging terrains, including: (a) Wedges, (b) Stakes, (c) Boxes, (d) Trapezoids, (e) Narrow beams, and (f) Mixed terrain.

Terrain	Pre-train	Scratch-1	Scratch-13	TTT-13	TTT-1
		(25k)	(30k)	(1.5k)	(Converged)
Boxes	98.27±0.21%	0.00±0.00%	0.00±0.00%	98.80±0.46%	100.00±0.00%
Wedges	0.30±0.30%	0.00±0.00%	0.00±0.00%	99.93±0.12%	99.90±0.00%
Nar.1	82.70±0.70%	100.00±0.00%	0.00±0.00%	99.53±0.21%	99.97±0.06%
Nar.2	87.27±1.24%	100.00±0.00%	0.00±0.00%	100.00±0.00%	99.97±0.06%
Nar.3	71.63±4.08%	0.23±0.12%	0.00±0.00%	99.20±0.17%	99.37±0.15%

Table 1: Simulation success rates (%). Mean±std over 3 evaluation seeds.

Panel of Figure 5 — Figure 5: TTT-1 simulation success during fine-tuning on (a) Boxes, Narrow1–3, Trapezoid1, and Mix1; (b) Wedges, Stake1–3, Mix2, and Trapezoid2; and (c) Boston. (d) Real-world success of Pre-train and TTT-1 over 5 trials per terrain.

Figure 6: Geometry comparison. Our pipeline with RGB-D input balances reconstruction quality and efficiency when generating simulation-ready meshes. It maintains realistic geometric fidelity with significantly fewer artifacts compared to LiDAR and iPhone scans, where major artifacts are highlighted by red circles. Top row: Stake1, bottom row: Narrow1.

Methods	RGB-D	LiDAR Scanner	iPhone	Hand-crafted
Pros	Balanced quality and efficiency	Professional, best scale accuracy	Most accessible, fast acquisition	No artifacts
Cons	Scale precision slightly lower than LiDAR	Terrain junction artifacts, tedious post-processing	Unstable reconstruction results, flying point artifacts	High manual effort, over-perfect geometry causes sim-to-real gap
Time	2 min 10 s	20 min	4 min	1 h

Table 2: Comparison of different sources for reconstruction. Time indicates duration from data capture to final mesh.

Terrain	RGB-D	LiDAR	iPhone	Hand-crafted
Stake1	80%	80%	0%	40%
Narrow1	80%	100%	80%	20%
Wedge	100%	60%	20%	20%
Trapezoid1	80%	60%	20%	40%

Table 3: Real-world success rates (%) with different reconstruction sources.

Terrain	RGB-D	LiDAR	iPhone	MapAnything	Pi3
Stake1	0.002	0.016	0.074	0.383	0.863
Narrow1	0.028	0.005	0.056	0.720	1.172
Wedge	0.013	0.015	0.027	0.944	0.423
Trapezoid1	0.018	0.024	0.033	0.988	0.400

Table 4: Absolute relative scale errors (lower is better).

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Space	Signal	Dim	Description
Actor Obs	Base Angular Velocity $\boldsymbol{\omega}_{t}$	3	IMU gyroscope
	Projected Gravity $\mathbf{g}_{t}$	3	Gravity direction in base frame
	Velocity Commands $\mathbf{c}_{t}$	3	$(v_{x},v_{y},\omega_{z})$
	Joint Positions $\mathbf{q}_{t}$	29	29-DoF joint angles
	Joint Velocities $\dot{\mathbf{q}}_{t}$	29	29-DoF joint velocities
	Last Action $\mathbf{a}_{t-1}$	29	Previous target joint positions

Appendix

Table 5: Observation and action space specification.

Parameter	Value
CNN channels	[4]
CNN kernel sizes	[3]
CNN strides	[1]
CNN paddings	[1]
CNN pooling	MaxPool
Encoder projection units	[256, 256]

Appendix

Table 6: Detailed hyperparameters for the network architecture.

Reward Term	Weight	Mathematical Formulation
Task Reward
Linear Vel Tracking ( $v_{xy}$ )	2.0	$\exp(-\\|\mathbf{v}_{xy}^{*}-\mathbf{v}_{xy}\\|^{2}/0.5^{2})$
Angular Vel Tracking ( $\omega_{z}$ )	0.1	$\exp(-(\omega_{z}^{*}-\omega_{z})^{2}/0.5^{2})$
Heading Error	−1.0	$\|\omega_{z}^{*}\|$
Don’t Wait	−0.5	$\mathbb{I}(v_{x}^{*}>0.3)\cdot(\mathbb{I}(v_{x}<0.15)+\mathbb{I}(v_{x}<0)+\mathbb{I}(v_{x}<-0.15))$
Is Alive	3.0	$+1$

Appendix

Table 7: Detailed specification of reward terms, weights, and mathematical formulations.

Terrain	RGB-D	LiDAR	iPhone	MapAnything	Pi3
Stake1	0.050 / 0.078	0.055 / 0.149	0.115/0.194	0.115/0.227	0.195/0.343
Narrow1	0.052 / 0.076	0.047 / 0.071	0.135/0.232	0.115/0.187	0.211/0.379
Wedge	0.029 / 0.036	0.045 / 0.057	0.048/0.067	0.198/0.344	0.236/0.396
Trapezoid1	0.029 / 0.036	0.031 / 0.080	0.151/0.247	0.210/0.353	0.209/0.356

Appendix

Table 8: Reconstruction error: CD / HD95 (m; lower is better).

Terrain	Scratch-1	TTT-13	TTT-1
Boxes	>25k	0	0
Wedges	>25k	250	100
Nar.1	17k	70	20
Nar.2	17k	70	20
Nar.3	>25k	70	40
Trap.1	>25k	230	70

Appendix

Table 9: Iterations to reach 97% success. Boston’s TTT-13 entry is a lower bound from the first 1k iterations.

Terrains / Methods	Last Layer	Residual	Adapter	Full Fine-tuning (Ours)
Narrow1	120	40	40	20
Trapezoid1	60	70	60	70
Mix2	200	>1000	260	120

Appendix

Table 10: Comparison of different test-time training strategies: Iterations to reach a 97% success rate.

Terrain	Pre-training	Real-world test
Boxes	Size 0.30–0.82; height 0.30–0.50; gap $\leq$ 0.35	Height 0.40–0.50; gap 0.31–0.32
Wedges	Size 0.15–0.82; height 0.13–0.35; slope $\sim 30^{\circ}$ /60 ∘	Height $\sim$ 0.31; slope 30 ∘
Narrow	Span 1.43–2.63; width 0.15–0.60; height 0.25–0.45; curved	Length $\leq$ 2.0; width 0.14–0.27; height 0.32–0.48; irregularly curved
Stakes	Diameter 0.16–0.24; height 0.30–0.45; 5–7 piles	Diameter 0.18; height $\sim$ 0.40; irregular layouts
Trapezoids	Top length 0.135–0.275; width 0.81–0.89; height 0.15–0.55	Width up to 1.0; height 0.35–0.55
Mix1/2	Single terrain family per training terrain	Cross-family obstacle compositions

Appendix

Table 11: Pre-training and real-world terrain distributions. Dimensions are in meters.

Figure 7: Detailed dimensions of the real-world terrains.

Figure 8: Detailed dimensions of the real-world terrains.

Figure 9: Detailed dimensions of the real-world terrains.

Figure 10: Cross-terrain simulation success (%). Rows identify the fine-tuning terrain; columns identify the evaluation terrain. The top row is the pre-trained policy. Blue outlines mark evaluation on the fine-tuning terrain.

Robot Locomotion

Robot locomotion studies how robots move efficiently and reliably across varied terrains, environments, and movement modes.

See all 233 papers

Perceptive Humanoid Parkour: Chaining Dynamic Human Skills via Motion Matching
Feb 17, 2026Zhen Wu, Xiaoyu Huang, Lujie Yang +8Humanoid Robot Locomotion Robot Skill Learning
Amazon FAR · UC Berkeley · CMU +1
FCTS: Failure-Guided Curriculum Tree Search Over Generated Terrain for Legged Parkour
Oct 15, 2026Narayanan Palghat Parameswaran, Dong-Ki Kim, Jay Patrikar +4Legged Robotics Curriculum Learning
Carnegie Mellon University · FieldAI
REAL: Robust Extreme Agility via Spatio-Temporal Policy Learning and Physics-Guided Filtering
Mar 18, 2026Jialong Liu, Dehan Shen, Yanbo Wen +2Robot Policy Learning Robot Locomotion
PEAK-Lab, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, 511453, China.

Feb 17, 2026J/K move · Enter open · S save

Perceptive Humanoid Parkour: Chaining Dynamic Human Skills via Motion Matching

Zhen Wu, Xiaoyu Huang, Lujie Yang, Yuanhang Zhang+7

Amazon FAR · UC Berkeley · CMU+1

Humanoid Robot Locomotion Robot Skill Learning

Open paper AskPDF arXiv

While recent advances in humanoid locomotion have achieved stable walking on varied terrains, capturing the agility and adaptivity of highly dynamic human motions remains an open challenge. In particular, agile parkour in complex environments demands not only low-level robustness, but also human-like motion expressiveness, long-horizon skill composition, and perception-driven decision-making. In this paper, we present Perceptive Humanoid Parkour (PHP), a modular framework that enables humanoid robots to autonomously perform long-horizon, vision-based parkour across challenging obstacle courses. Our approach first leverages motion matching, formulated as nearest-neighbor search in a feature space, to compose retargeted atomic human skills into long-horizon kinematic trajectories. This framework enables the flexible composition and smooth transition of complex skill chains while preserving the elegance and fluidity of dynamic human motions. Next, we train motion-tracking reinforcement learning (RL) expert policies for these composed motions, and distill them into a single depth-based, multi-skill student policy, using a combination of DAgger and RL. Crucially, the combination of perception and skill composition enables autonomous, context-aware decision-making: using only onboard depth sensing and a discrete 2D velocity command, the robot selects and executes whether to step over, climb onto, vault or roll off obstacles of varying geometries and heights. We validate our framework with extensive real-world experiments on a Unitree G1 humanoid robot, demonstrating highly dynamic parkour skills such as climbing tall obstacles up to 1.25m (96% robot height), as well as long-horizon multi-obstacle traversal with closed-loop adaptation to real-time obstacle perturbations.

Abstract

Robot Locomotion

Explore similar work

Perceptive Humanoid Parkour: Chaining Dynamic Human Skills via Motion Matching

FCTS: Failure-Guided Curriculum Tree Search Over Generated Terrain for Legged Parkour

REAL: Robust Extreme Agility via Spatio-Temporal Policy Learning and Physics-Guided Filtering