Vision as Proprioception: Estimating Humanoid Robot Pose from Video

Jiaxi Jiang; Matthias Heyrman; René Zurbrügg; Marco Hutter; Christian Holz

cs.ROOct 18, 2026

Vision as Proprioception: Estimating Humanoid Robot Pose from Video

Authors: Jiaxi Jiang, Matthias Heyrman, René Zurbrügg, Marco Hutter, Christian Holz

Organizations: ETH Zürich

Abstract

Estimating a humanoid robot's internal state from video has a wide range of real-world applications but remains unexplored. During interactions with a robot, its onboard state information may be unavailable due to privacy, ownership, or communication constraints. Meanwhile, online and AI-generated robot videos provide rich sources of data but typically lack corresponding joint configurations, limiting their direct use in robot learning. To address these issues, we present HumanoidPose, a new framework that treats vision as proprioception to recover full-body joint configurations in world space from video alone. Specifically, we combine pretrained human-pose features with robot kinematics in a unified model that predicts each embodiment's joint configurations. We retarget human motions to sixteen different humanoid embodiments, enabling scalable learning entirely from synthetic data. To improve real-world generalization, we further leverage unlabeled online videos through self-training to develop a 2D robot-pose detector. In our evaluation, HumanoidPose improves joint-angle and joint-position accuracy over the evaluated baselines and reduces joint-angle error by 18-32% relative to the best-performing baseline on each of the three real humanoid platforms. We further show improved global localization and generalization to unseen embodiments. HumanoidPose enables visual understanding of other robots during interaction and extraction of robot motion from online and generated videos.

Figures & tables

Fig. 2: HumanoidPose overview. (a) Retargeted human motions and randomized rendering provide synthetic robot pose supervision. (b) Pseudo-labels from unlabeled online videos adapt the detector through self-training. (c) Frozen human-pretrained features and kinematic queries drive a shared decoder for joint angles and root pose. Camera motion and forward kinematics place predictions in world coordinates. The pose decoder is trained solely on synthetic data.

Fig. 3: The training corpus. Human motion capture retargeted onto sixteen humanoids spanning $3.0\times$ in height ( $0.60$ – $1.78$ m) and $19$ – $43$ degrees of freedom, rendered in Unreal Engine under randomized scene, illumination and viewpoint. One sample is shown per robot, paired with the configuration used to render it.

Method	Synthetic								Real Robot
	$\mathrm{MPJAE}\,\downarrow_{{}^{\circ}}$				$\mathrm{MPJPE}\,\downarrow_{\mathrm{cm}}$				$\mathrm{MPJAE}\,\downarrow_{{}^{\circ}}$			$\mathrm{MPJPE}\,\downarrow_{\mathrm{cm}}$
	All (16)	G1	PM01	N1	All (16)	G1	PM01	N1	G1	PM01	N1	G1	PM01	N1
Keypoint fitting	21.21	19.86	27.86	20.32	10.84	9.85	15.27	8.74	10.52	6.99	14.70	5.93	6.14	5.63
PromptHMR [ 5 ] + GMR [ 6 ]	18.03	20.37	24.05	18.26	8.78	10.17	12.82	7.45	12.50	16.11	19.57	5.31	6.02	7.46
PromptHMR [ 5 ] + GMR [ 6 ] + SynFT	16.82	19.39	22.83	17.04	7.98	10.03	12.85	6.99	10.46	16.45	15.42	4.88	11.88	6.33
HoRoPose [ 2 ]	15.09	14.21	20.60	16.19	9.61	10.05	12.06	8.91	12.98	15.36	13.72	13.24	11.01	7.24

TABLE I: Pose estimation on synthetic data and real robots. MPJAE ( ∘ ) and base-frame MPJPE (cm). All (16) averages robots equally. One HumanoidPose model serves all robots and both domains. Best in bold and second best underlined.

Fig. 4: Pose recovery across real humanoid platforms. Self-recorded RGB frames (top) and predicted 3D configurations (bottom) for G1, PM01 and N1. The same HumanoidPose model recovers locomotion, arm movements, crouching and supported poses across different robot structures and appearances.

Method	RMSE $\,\downarrow$	Median $\,\downarrow$	P90 $\,\downarrow$
Keypoint fitting	42.2	21.3	56.3
PromptHMR [ 5 ]	66.2	37.4	96.2
PromptHMR [ 5 ] + height scaling	39.0	16.8	51.1
HumanoidPose (Ours)	23.4	11.3	21.7

TABLE II: Global root-translation errors on G1 (cm) after fixed-scale rigid alignment. P90 denotes the

90

th percentile.

Fig. 5: Few-shot adaptation to a held-out embodiment. G1 test error decreases with more synthetic training examples. The horizontal axis uses

\log(1+N)

	Synthetic		Real
Variant	MPJAE $\,\downarrow$	MPJPE $\,\downarrow$	MPJAE $\,\downarrow$	MPJPE $\,\downarrow$
Ours	9.48	4.49	7.09	3.82
w/o joint identity	10.94	5.05	7.15	4.02
w/o direct robot decoding	16.82	7.98	12.12	6.19
w/o kinematic conditioning	12.43	6.67	7.62	4.15
w/o human-pretrained encoder	11.14	5.57	7.95	4.65

TABLE III: Core design comparisons on synthetic data from sixteen robots and real data from three platforms. MPJAE ( ∘ ) and MPJPE (cm).

	Synthetic		G1		PM01		N1
Detector	Det. $\,\uparrow$	PCK $\,\uparrow$	Cov. $\,\uparrow$	MPJAE $\,\downarrow$	Cov. $\,\uparrow$	MPJAE $\,\downarrow$	Cov. $\,\uparrow$	MPJAE $\,\downarrow$
YOLO11	56.8	38.6	98.6	7.82	96.5	5.89	77.8	17.65
YOLO11 + SynFT	100	93.6	81.1	8.65	75.6	5.87	99.3	8.98
YOLO11 + SynFT + ST	100	92.1	98.8	7.11	96.8	5.36	99.4	8.90

TABLE IV: Robot-adapted 2D pose detector stages: synthetic detection rate and PCK (%), and real coverage (%) and MPJAE ( ∘ ) of our fixed pose decoder.

Robotic Perception

Robotic perception enables robots to interpret scenes, objects, motion, and contact from cameras and other sensors to guide action.

See all 321 papers

EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos
Oct 14, 2026Meng Cao, Can Zhao, Jinjie Wei +12Robotic Manipulation Cross-Embodiment Robot Learning
Mohamed bin Zayed University of Artificial Intelligence · Shanghai Jiao Tong University · X Square Robot +2
BeyondRetarget: Learning Executable Humanoid Motions Directly from Monocular Video
Sep 24, 2026Tianyu Xiong, Yi Lu, Jinrui Wang +6Human Motion Generation Cross-Embodiment Robot Learning
School of Electronic Science and Engineering, Nanjing University, Nanjing, China · Jiangsu Mobile Information System Integration Co., Ltd., Nanjing, China · China Mobile Zijin (Jiangsu) Innovation Research Institute Co., Ltd., Nanjing, China +1
Morphology-Consistent Humanoid Interaction through Robot-Centric Video Synthesis
Mar 20, 2026Weisheng Xu, Jian Li, Yi Gu +12Robot Motion Generation Humanoid Robotics
Hong Kong University of Science and Technology (Guangzhou) · Harbin Institute of Technology, Shenzhen · Shenzhen University +1

Oct 14, 2026J/K move · Enter open · S save

EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos

Meng Cao, Can Zhao, Jinjie Wei, Haonan Zhao+11

Mohamed bin Zayed University of Artificial Intelligence · Shanghai Jiao Tong University · X Square Robot+2

Robotic Manipulation Cross-Embodiment Robot Learning Robot Imitation Learning Latent Action Learning World Models for Robotics Joint-Embedding Predictive Architecture

Open paper AskPDF arXiv

Egocentric human videos offer a scalable alternative to costly robot demonstra- tions, yet transferring their manipulation priors to robot control remains chal- lenging. The commonly adopted retargeting-based approaches convert human wrist and finger poses into embodiment-specific action commands through hand- crafted geometric mappings. However, such mappings require meticulous recal- ibration across different embodiments and risk propagating hand-tracking errors into downstream robot control pipelines. To this end, we propose EgoJEPA, a three-stage framework that learns embodiment-agnostic representations from ego- centric human videos without rule-based human-to-robot retargeting: 1) Human- action pre-training learns effect-aware latent actions through JEPA-style visual prediction conditioned on hand-motion differences; 2) Latent-action-based world modeling jointly predicts future visual dynamics and latent actions, which decou- ples human prior pre-training from robot-specific control parameterizations; 3) Embodiment-specific tuning adapts the pretrained model to diverse robots through lightweight action adapters. Experiments on five real-world manipulation tasks across three robot embodiments demonstrate the effectiveness and efficiency of EgoJEPA. It achieves a 74% average success rate under the out-of-domain evalu- ation, compared with 65% for the retargeting baseline, while reducing cumulative cross-embodiment wall-clock cost by 47.2%, from 743.2 to 392.7 hours.

Abstract

Robotic Perception

Explore similar work

EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos

BeyondRetarget: Learning Executable Humanoid Motions Directly from Monocular Video

Morphology-Consistent Humanoid Interaction through Robot-Centric Video Synthesis