SocialMirror: Reconstructing 3D Human Interaction Behaviors from Monocular Videos with Semantic and Geometric Guidance

Qi Xia; Peishan Cong; Ziyi Wang; Yujing Sun; Qin Sun; Xinge Zhu; Mao Ye; Ruigang Yang

cs.CVApr 15, 2026

SocialMirror: Reconstructing 3D Human Interaction Behaviors from Monocular Videos with Semantic and Geometric Guidance

Authors: Qi Xia, Peishan Cong, Ziyi Wang, Yujing Sun, Qin Sun, Xinge Zhu, Mao Ye, Ruigang Yang,

Organizations: ShanghaiTech University · Nanyang Technological University · Guangzhou Institute of Energy Conversion, CAS · University of Science and Technology of China ·

Abstract

Accurately reconstructing human behavior in close-interaction scenarios is crucial for enabling realistic virtual interactions in augmented reality, precise motion analysis in sports, and natural collaborative behavior in human-robot tasks. Reliable reconstruction in these contexts significantly enhances the realism and effectiveness of AI-driven interactive applications. However, human reconstruction from monocular videos in close-interaction scenarios remains challenging due to severe mutual occlusions, leading local motion ambiguity, disrupted temporal continuity and spatial relationship error. In this paper, we propose SocialMirror, a diffusion-based framework that integrates semantic and geometric cues to effectively address these issues. Specifically, we first leverage high-level interaction descriptions generated by a vision-language model to guide a semantic-guided motion infiller, hallucinating occluded bodies and resolving local pose ambiguities. Next, we propose a sequence-level temporal refiner that enforces smooth, jitter-free motions, while incorporating geometric constraints during sampling to ensure plausible contact and spatial relationships. Evaluations on multiple interaction benchmarks show that SocialMirror achieves state-of-the-art performance in reconstructing interactive human meshes, demonstrating strong generalization across unseen datasets and in-the-wild scenarios. The code will be released upon publication.

Figures & tables

Figure 2: The framework of SocialMirror, which integrates semantic guidance from vision-language annotations and further refine the result with geometric constraints. Trans Block refers to the transformer block.

Figure 3: Qualitative comparison results.

Method	Hi4D								3DPW
Method	$\downarrow$ RE	$\downarrow$ GE	$\downarrow$ Int	$\downarrow$ Smoothness	$\downarrow$ Pen	$\downarrow$ MPJPE	$\downarrow$ PA	$\downarrow$ VPE	$\downarrow$ RE	$\downarrow$ GE	$\downarrow$ Int	$\downarrow$ Smoothness	$\downarrow$ Pen	$\downarrow$ MPJPE	$\downarrow$ PA	$\downarrow$ VPE
Human4D [ 18 ]	-	-	-	-	-	72.1	52.4	88.6	-	-	-	-	-	72.9	49.1	107.0
BEV [ 47 ]	210.5	223.5	131.0	-	1953.6	91.8	59.3	101.2	235.2	291.8	145.6	-	233.8	135.0	81.9	169.7
GroupRec [ 19 ]	113.2	122.3	98.8	-	1858.4	82.4	51.6	88.6	204.6	235.2	110.6	-	100.9	73.3	48.7	109.4
BUDDI [ 36 ]	200.3	216.4	102.6	-	1879.3	96.8	70.6	116.0	228.4	289.4	113.1	-	203.5	83.6	53.6	93.8
CloseInt [ 20 ]	99.0	114.9	81.4	4.6	3947.6	63.1	47.5	76.4	121.1	134.0	75.6	19.9	101.6	59.0	45.3	73.2

Table 1: Comparisons on Hi4D and 3DPW. PA denotes PA-MPJPE and VPE denotes MPVPE.

Semantic-Guided	Temporal Motion Refiner			Geometry	$\downarrow$ RE	$\downarrow$ GE	$\downarrow$ Int	$\downarrow$ Smoothness	$\downarrow$ MPJPE	$\downarrow$ PA	$\downarrow$ VPE
Motion Infiller	-	factorized loss guidance	contact mask	Optimizer	$\downarrow$ RE	$\downarrow$ GE	$\downarrow$ Int	$\downarrow$ Smoothness	$\downarrow$ MPJPE	$\downarrow$ PA	$\downarrow$ VPE
					100.4	119.0	90.5	4.7	62.4	47.5	78.3
✓					91.2	102.7	73.4	4.1	63.5	48.7	80.7
✓	✓				91.5	103.5	74.2	3.9	63.3	48.5	80.2
✓	✓			✓	91.0	102.4	73.3	4.1	63.5	48.7	80.7
	✓	✓		✓	89.3	100.6	69.2	4.0	63.2	48.5	80.3

Table 3: Ablation studies on the impact of semantic and geometric information on Hi4D. Temporal Motion Refiner with a dash (-) indicates Temporal Motion Refiner without factorized loss guidance and contact mask.

IoU	0.0			$(0,0.25]$			$(0.25,0.5]$			$(0.5,0.75]$			$(0.75,1.0]$
	CloseInt	Ours	Improve.	CloseInt	Ours	Improve.	CloseInt	Ours	Improve.	CloseInt	Ours	Improve.	CloseInt	Ours	Improve.
$\downarrow$ RE	85.4	78.3	8.3	93.3	82.3	11.8	104.1	91.4	12.2	104.3	93.8	10.1	108.4	106.7	1.6
$\downarrow$ GE	99.4	89.0	10.5	112.4	96.3	14.3	120.4	101.2	16.0	118.3	105.3	11.0	125.6	117.7	6.3
$\downarrow$ Int	67.8	62.6	7.7	87.2	68.6	21.3	98.2	79.3	19.3	93.3	77.5	16.9	93.3	82.4	11.7
$\downarrow$ MPJPE	40.9	44.7	-9.3	49.2	51.3	-4.3	64.2	65.7	-2.3	74.6	72.6	2.7	84.1	88.1	-4.8
$\downarrow$ PA	31.1	35.2	-13.2	37.6	40.1	-6.7	49.3	51.3	-4.1	55.6	54.9	1.3	58.6	60.6	-3.4

Table 4: Results under various occlusion severities on Hi4D. Darker colors indicate greater improvements. Improve. is improvement rate.

Prompting Example
Given the image sequence of two human interaction, generate 0, 1 or more joint-joint contact pair(s) according to the following background information, rules, and examples. Joint-joint contact pair should exactly reflect the human interaction shown in the image sequence.
[Start of background Information]
Human has JOINTS: [‘pelvis’, ‘left_hip’, ‘right_hip’, ‘left_knee’, ‘right_knee’, ‘left_ankle’, ‘right_ankle’, ‘left_foot’, ‘right_foot’, ‘neck’, ‘left_collar’, ‘right_collar’, ‘head’, ‘left_shoulder’, ‘right_shoulder’, ‘left_elbow’, ‘right_elbow’, ‘left_wrist’, ‘right_wrist’ ].
[End of background Information]
[Start of rules]
1.Each joint-joint pair should be formatted into {JOINT, JOINT, TIME-STEP, TIME-STEP}. JOINT should be replaced by JOINT in the background information. IMPORTANT: The first JOINT belongs to person 1, and the second JOINT belongs to person 2. Each joint-joint pair represents a contact of a joint of person 1 and a joint of person 2. The first TIME-STEP is the start frame number of contact, and the second TIME-STEP is the end frame number of contact.

Table 5: Detailed prompting example for VLM Annotator.

Embedding Layer	$\downarrow$ R-MPJPE	$\downarrow$ G-MPJPE	$\downarrow$ Int	$\downarrow$ MPJPE	$\downarrow$ PA-MPJPE
Linear	102.3	110.0	84.9	81.9	66.8
STGCN	81.7	93.2	62.5	60.8	47.8

Table 6: Ablation studies on the impact of motion embedding layer.

Action	handshake	high-five	kiss	dance	fight
$\downarrow$ R-MPJPE	78.0/65.8 (20.0)	60.5/53.5 (19.2)	81.7/67.7 (20.9)	96.4/87.6 (9.4)	110.3/100.3 (8.2)
$\downarrow$ G-MPJPE	93.2/72.5 (23.8)	84.9/70.9 (19.7)	98.9/79.7 (19.6)	109.2/97.9 (9.4)	131.4/120.1 (6.5)
$\downarrow$ Int	36.9/31.1 (15.6)	26.7/25.1 (5.9)	33.3/23.0 (63.9)	39.9/32.4 (18.8)	46.7/41.1 (11.9)
$\downarrow$ Pen	194.8/71.9 (63.1)	107.4/50.1 (53.3)	15409.3/5570.1 (63.9)	5477.1/2455.0 (55.2)	636.6/226.2 (64.5)

Table 7: Comparison of CloseInt and our method, CloseInt/Ours (Improvement), across different actions on Hi4D.

Method	Hi4D							3DPW
Method	$\downarrow$ RE	$\downarrow$ GE	$\downarrow$ Int	$\downarrow$ Pen	$\downarrow$ MPJPE	$\downarrow$ PA.	$\downarrow$ VPE	$\downarrow$ RE	$\downarrow$ GE	$\downarrow$ Int	$\downarrow$ Pen	$\downarrow$ MPJPE	$\downarrow$ PA.	$\downarrow$ VPE
CloseInt	99.0	114.9	81.4	3947.6	63.1	47.5	76.4	135.7	159.1	95.5	342.7	79.9	52.9	95.1
Ours	83.6	95.2	68.5	2380.5	62.2	47.5	79.3	104.8	162.7	89.9	109.7	65.1	49.0	79.7
CloseInt (Eval. Only)	181.1	232.1	182.7	1973.8	109.1	62.5	132.0	194.4	340.2	128.4	101.6	88.6	63.6	110.7
Ours (Eval. Only)	165.2	184.1	153.0	2380.3	105.2	63.6	129.4	174.7	307.4	125.4	109.7	87.5	63.3	109.8

Table 8: Cross Dataset Evaluation on Hi4D and 3DPW.

Method	$\downarrow$ RE	$\downarrow$ GE	$\downarrow$ Int	$\downarrow$ Pen	$\downarrow$ MPJPE	$\downarrow$ PA.	$\downarrow$ VPE
CloseInt	134.8	297.5	182.5	482.6	70.2	38.6	82.6
Ours	123.8	287.2	176.8	480.3	69.8	39.7	80.8

Table 9: Comparisons on Harmony4D.

Panel of Figure 5 — Figure 5: Visualization results on in-the-wild data.

Figure 6: VLM Annotator failed to describe human interaction.

Figure 7: Challenging case with prolonged, severe occlusions.

3D Human Reconstruction

3D Human Reconstruction recovers human shape, pose, appearance, and motion from images, video, or other sensor data.

See all 105 papers

Recovering Physically Plausible Human-Object Interactions from Monocular Videos
Jun 3, 2026Dingbang Huang, Etienne Vouga, Qixing Huang +13D Reconstruction Monocular 3D Reconstruction
University of Texas at Austin
HORIA: 4D Human and Object Reconstruction from a Monocular Video with Interaction-Aware Re-Rendering
Oct 19, 2026Hyeongjin Nam, Taeryung Lee, Daniel Sungho Jung +1Human-Scene Interaction Reconstruction 4D Reconstruction
Dept. of ECE&ASRI, Seoul National University · IPAI, Seoul National University
RHINO: Reconstructing Human Interactions with Novel Objects from Monocular Videos
May 16, 2026Lixin Xue, Chengwei Zheng, Georgios Paschalidis +43D Reconstruction Monocular 3D Reconstruction
1ETH Zürich · 2The University of Tokyo · University of Amsterdam +1

Jun 3, 2026J/K move · Enter open · S save

Recovering Physically Plausible Human-Object Interactions from Monocular Videos

Dingbang Huang, Etienne Vouga, Qixing Huang, Georgios Pavlakos

University of Texas at Austin

3D Reconstruction Monocular 3D Reconstruction Human-Scene Interaction Reconstruction Human-Object Interaction

Open paper AskPDF arXiv

In this paper, we propose RePHO, a method to reconstruct physically plausible human-object interactions (HOI) from monocular videos. While existing kinematic-based approaches produce visually plausible motion, they often result in physically implausible artifacts such as interpenetration and object floating. To overcome these issues, we introduce a physics-guided reconstruction framework. We begin with a kinematic estimate and then refine it by training a policy with reinforcement learning (RL). This policy is optimized to reproduce the interaction in a physics simulator. Because kinematic estimates are typically noisy, naive RL training can fail. Therefore, we propose an adaptive sampling strategy with a dual self-updating mechanism that can identify the frames with the most informative and reliable kinematic reconstruction. Our process progressively improves reconstruction quality and yields physically consistent HOI sequences. We demonstrate our approach on two standard HOI benchmarks and achieve clear improvements in physical plausibility metrics over state-of-the-art methods. Project Page: https://dingbang777.github.io/RePHO/

Abstract

3D Human Reconstruction

Explore similar work

Recovering Physically Plausible Human-Object Interactions from Monocular Videos

HORIA: 4D Human and Object Reconstruction from a Monocular Video with Interaction-Aware Re-Rendering

RHINO: Reconstructing Human Interactions with Novel Objects from Monocular Videos