EmbodMocap: In-the-Wild 4D Human-Scene Reconstruction for Embodied Agents

Wenjia Wang; Liang Pan; Huaijin Pi; Yuke Lou; Xuqian Ren; Yifan Wu; Zhouyingcheng Liao; Lei Yang

cs.CVFeb 26, 2026

EmbodMocap: In-the-Wild 4D Human-Scene Reconstruction for Embodied Agents

Authors: Wenjia Wang, Liang Pan, Huaijin Pi, Yuke Lou, Xuqian Ren, Yifan Wu, Zhouyingcheng Liao, Lei Yang,

Organizations: The University of Hong Kong · Tampere University · The Chinese University of Hong Kong · Max-Planck Institute for Informatics

Abstract

Human behaviors in the real world naturally encode rich, long-term contextual information that can be leveraged to train embodied agents for perception, understanding, and acting. However, existing capture systems typically rely on costly studio setups and wearable devices, limiting the large-scale collection of scene-conditioned human motion data in the wild. To address this, we propose EmbodMocap, a portable and affordable data collection pipeline using two moving iPhones. Our key idea is to jointly calibrate dual RGB-D sequences to reconstruct both humans and scenes within a unified metric world coordinate frame. The proposed method allows metric-scale and scene-consistent capture in everyday environments without static cameras or markers, bridging human motion and scene geometry seamlessly. Compared with optical capture ground truth, we demonstrate that the dual-view setting exhibits a remarkable ability to mitigate depth ambiguity, achieving superior alignment and reconstruction performance over single iphone or monocular models. Based on the collected data, we empower three embodied AI tasks: monocular human-scene-reconstruction, where we fine-tune on feedforward models that output metric-scale, world-space aligned humans and scenes; physics-based character animation, where we prove our data could be used to scale human-object interaction skills and scene-aware motion tracking; and robot motion control, where we train a humanoid robot via sim-to-real RL to replicate human motions depicted in videos. Experimental results validate the effectiveness of our pipeline and its contributions towards advancing embodied AI research.

Figures & tables

Datasets	Publication	Device					Outcome
Datasets		Mocap Suit	Scanner	Static Cam.	Dyna. Cam.	Total Cost($)	Mesh	Dyna.Anno.	Outdoor
PROX [ 12 ]	ICCV2019	-	Structure Sensor	Kinetic-One	-	2K	✓	✗	✗
RICH [ 19 ]	CVPR 2022	-	Leica RTC360	6-8 $\times$ Cameras	1 $\times$ Camera	20K+	✓	✓	✓
EgoBody [ 76 ]	ECCV2022	-	1 $\times$ IPhone	5 $\times$ Azure Kinect	Hololens2	9K	✓	✓	✗
SLOPER4D [ 6 ]	CVPR2023	Noitom PN+NUC11	Ouster-os1 LiDAR	-	DJI-Action2+TLS	20K	✓	✓	✓
EMDB [ 22 ]	ICCV 2023	EM Sensors	-	-	1 $\times$ IPhone	15K	✗	✓	✓

Table 1: Comparison of 4D Human & Scene datasets based on different features.

Figure 2: EmbodMocap: We propose an affordable dataset capture and processing system. From left to right, the four stages (Stage-I to Stage-IV) illustrate our core logic: leveraging high-quality camera matrices provided by SpectacularAI [ 56 ] and aligning sequence coordinates to the scene’s world frame. For detailed explanations, please refer to Sec. 3 .

$\mathcal{L}_{track}$	$\mathcal{L}_{chamfer}$	$\mathcal{L}_{reproj}$	$\mathcal{L}_{smooth}$	$\mathcal{L}_{kp3d}$	IoU(%) $\uparrow$	Reproj $\downarrow$	Depth $\downarrow$	Jitter $\downarrow$
✗	✓	✓	✓	✓	54.3	44.2	2.372	0.0371
✓	✗	✓	✓	✓	72.5	10.9	0.081	0.0131
✓	✓	✗	✓	✓	72.3	11.1	0.079	0.0130
✓	✓	✓	✗	✓	72.1	10.4	0.087	0.0160
✓	✓	✓	✓	✗	59.3	20.4	0.609	0.0126
✓	✓	✓	✓	✓	73.0	9.3	0.078	0.0128

Table 2: The performance of different optimization settings.

Figure 3: Our dual view vs . single view results in optical studio.

Method	chunk=100		chunk=500		chunk=1000		RTE $\downarrow$
Method	WA-MPJPE $\downarrow$	W-MPJPE $\downarrow$	WA-MPJPE $\downarrow$	W-MPJPE $\downarrow$	WA-MPJPE $\downarrow$	W-MPJPE $\downarrow$
GVHMR	66.56	123.44	124.61	333.34	179.47	593.79	1.85
Single-View V1	124.68	218.22	233.06	489.11	297.83	768.31	2.71
Single-View V2	108.31	211.83	231.41	357.22	338.42	762.80	3.65
Dual View	56.61	72.86	76.90	99.75	119.45	169.11	1.13

Table 3: Comparision among monocular model, single view optimization, with dual view optimization(ours)

Finetuned		EMDB
Pi3	VIMO	WA-MPJPE $\downarrow$	W-MPJPE $\downarrow$	RTE $\downarrow$
✗	✗	83.56	229.04	1.78
✗	✓	82.89	222.93	1.73
✓	✓	82.21	220.65	1.71

Table 4: Comparison of Finetuned Models on EMDB Benchmarks

Figure 4: Quality results of proposed 4D Human & Scene Reconstruction pipeline on EMDB dataset.

Task	Data	Clips	Duration (min)	Rate (%) $\uparrow$	Error (cm) $\downarrow$	APD $\uparrow$
Follow	Optical Mocap	12	1.59	99.9	6.0	20.17 $\pm$ 0.19
	Ours 1X	12	1.48	99.9	6.7	18.42 $\pm$ 0.22
	Ours 2X	24	3.06	99.7	6.8	18.45 $\pm$ 0.17
	Ours Full	148	22.43	99.8	6.2	19.69 $\pm$ 0.32
	Monocular	148	22.43	98.0	7.2	19.85 $\pm$ 0.39
Climb	Optical Mocap	7	0.28	99.9	2.7	22.03 $\pm$ 0.30

Table 5: Comparison of data duration, Success Rate, Contact Error, and APD for different skills among 3 data settings.

Panel of Figure 5 — Figure 5: Qualitative comparison of 4 basic skills and 2 additional skills.

Figure 6: We present qualitative results of scene-aware motion tracking, showing four long-term motion examples in different scenes (a, b, c, and d), including daily indoor and outdoor interactions such as walking, sitting, lying, stair climbing, and touching. Our motion tracking framework not only accurately tracks the reference motion but also ensures physical realism, resolving subtle issues, such as interpenetration and floating artifacts, present in the reference data (see zoomed-in views on the right).

Scene	Clips	Duration (min)	Status	Rate (%)	Eps. Len. (s)
a	14	12.31	Succ.	87.2	9.97 $\pm$ 0.21
a			Fail.	12.8	3.94 $\pm$ 2.10
b	6	3.62	Succ.	96.7	9.99 $\pm$ 0.12
b			Fail.	3.3	4.16 $\pm$ 2.38
c	12	7.87	Succ.	95.9	9.98 $\pm$ 0.17
c			Fail.	4.1	5.43 $\pm$ 2.18

Table 6: Quantitative evaluation of scene-aware motion tracking and dataset statistics across four 3D scenes.

Figure 7: A real-world humanoid robot imitating human motions depicted in videos.

Figure 9: An example in finding the contact marker in software (e.g., Meshlab) and corresponding keyframe index(the frames selected here are just for demo).

Panel of Figure 10 — Figure 10: Statistical information of collected dataset.

Figure 11: Rendered SMPL and depth images of the captured dataset in camera space.

Figure 12: 3D demo of the captured dataset.

3D Human Reconstruction

3D Human Reconstruction recovers human shape, pose, appearance, and motion from images, video, or other sensor data.

See all 105 papers

MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstruction
Feb 17, 2026Qiang Zhang, Jiahao Ma, Peiran Liu +20Terrain-Aware Robot Locomotion Robot Imitation Learning
X-Humanoid · The Hong Kong University of Science and Technology (Guangzhou) · The Australian National University +4
EgoExoMoCap: Distributed Ego-Exo Human Motion Capture
Jul 17, 2026Jiaxi Jiang, Bharat Lal Bhatnagar, Nan Yang +6Markerless Motion Capture 3D Human Pose Estimation
1Meta Reality Labs · *This work was done during an internship at Meta. · 2ETH Zürich
ReScene: Structured Indoor Scene Reconstruction from Multi-View Captures
Jun 26, 2026Haoran Xu, Lechao Zhang, Daoguo Dong +2Multi-View Geometry 3D Reconstruction
School of Computer Science and Technology, East China Normal University · Institute of Trustworthy Embodied AI, Fudan University

Feb 17, 2026J/K move · Enter open · S save

MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstruction

Qiang Zhang, Jiahao Ma, Peiran Liu, Shuai Shi+19

X-Humanoid · The Hong Kong University of Science and Technology (Guangzhou) · The Australian National University+4

Terrain-Aware Robot Locomotion Robot Imitation Learning Motion Retargeting

Open paper AskPDF arXiv

Humanoid motion control has witnessed significant breakthroughs in recent years, with deep reinforcement learning (RL) emerging as a primary catalyst for achieving complex, human-like behaviors. However, the high dimensionality and intricate dynamics of humanoid robots make manual motion design impractical, leading to a heavy reliance on expensive motion capture (MoCap) data. These datasets are not only costly to acquire but also frequently lack the necessary geometric context of the surrounding physical environment. Consequently, existing motion synthesis frameworks often suffer from a decoupling of motion and scene, resulting in physical inconsistencies such as contact slippage or mesh penetration during terrain-aware tasks. In this work, we present MeshMimic, an innovative framework that bridges 3D scene reconstruction and embodied intelligence to enable humanoid robots to learn coupled "motion-terrain" interactions directly from video. By leveraging state-of-the-art 3D vision models, our framework precisely segments and reconstructs both human trajectories and the underlying 3D geometry of terrains and objects. We introduce an optimization algorithm based on kinematic consistency to extract high-quality motion data from noisy visual reconstructions, alongside a contact-invariant retargeting method that transfers human-environment interaction features to the humanoid agent. Experimental results demonstrate that MeshMimic achieves robust, highly dynamic performance across diverse and challenging terrains. Our approach proves that a low-cost pipeline utilizing only consumer-grade monocular sensors can facilitate the training of complex physical interactions, offering a scalable path toward the autonomous evolution of humanoid robots in unstructured environments.

Abstract

3D Human Reconstruction

Explore similar work

MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstruction

EgoExoMoCap: Distributed Ego-Exo Human Motion Capture

ReScene: Structured Indoor Scene Reconstruction from Multi-View Captures