Social Humanoid: Socially Responsive Human-Humanoid Interaction

Vongani H. Maluleke; James Ni; Kie Horiuchi; Lea Wilken; Jiahui Lei; Evonne Ng; Roei Herzig; Jitendra Malik

cs.RONov 5, 2026

Social Humanoid: Socially Responsive Human-Humanoid Interaction

Authors: Vongani H. Maluleke, James Ni, Kie Horiuchi, Lea Wilken, Jiahui Lei, Evonne Ng, Roei Herzig, Jitendra Malik,

Organizations: UC Berkeley† · Sony Group Corporation‡ · Meta§

Abstract

Humanoid robots are typically trained as solitary agents, leaving them unable to coordinate physically with human partners who naturally anticipate and adjust to one another. To bridge this gap, we present Social Humanoid, a hierarchical system that generates and executes socially responsive whole-body motion on a 29-DoF Unitree G1 conditioned on a human partner's motion. Its high-level policy performs flexible sequence modeling of human and humanoid motion through a shared tokenized representation, generating reactive humanoid motion and anticipating the partner's motion when they are occluded, while a low-level whole-body tracking controller executes the generated motion on the robot. To train the system, we introduce a two-phase adaptive retargeting procedure that retargets one interaction partner to the robot while morphing the other partner to preserve contact dynamics, reachability, and relative body scale. We apply this procedure across five motion-capture datasets to produce a new Human-Humanoid Interaction (HHI) benchmark of 260 hours of human-humanoid interactions. Our results demonstrate that Social Humanoid produces coordinated, human-aligned behavior across diverse interaction types in simulation and the real world. Project page: https://von31.github.io/social-humanoid/

Figures & tables

Figure 1: Social Humanoid enables humanoids to perceive, anticipate, and respond to people with coordinated whole-body behavior. Examples include a hug (top) and a greeting followed by posing together for a photo (bottom). Given RGB-D video, our method reconstructs the observed human motion, then conditions on it to predict future human motion and a humanoid response. The generated motions are executed by a low-level tracking controller. Video results are available on our webpage: https://von31.github.io/social-humanoid/

Figure 2: Social Humanoid Architecture. From RGB-D observations of a human partner (red), we reconstruct the human pose and localize the body in 3D using depth and estimated gravity. Relative human and humanoid states are encoded through the VQ-VAE encoders $\mathrm{E}_{\mathrm{h}}$ and $\mathrm{E}_{\mathrm{r}}$ . The resulting latent motion tokens $z_{*}$ , relative pose transforms $T^{*}$ , and canonical motion transforms $\Delta T^{*}$ are concatenated and processed by the DFoT policy, then decoded via $\mathrm{D}_{h}$ and $\mathrm{D}_{r}$ to future human behavior (blue) and socially coordinated humanoid motion (purple). SONIC executes the motion in sim or the real-world, and the robot state is updated for closed-loop interaction.

Figure 3: Shared-Codebook VQ-VAE . Human and humanoid motion sequences are independently encoded into latent features and mapped to a shared vector-quantized codebook that learns a common motion vocabulary across embodiments. Embodiment-specific decoders reconstruct the respective motions, enabling aligned, transferable motion representations between humans and humanoids.

Figure 4: Adaptive-HHI Retargeting. Our retargeting pipeline starts from two ground truth humans. In the first pass, we retarget the ego human (orange) to the humanoid using a contact-adaptive weight matrix to encourage body pose in regions close to the physical interaction to be preserved. In a second pass, we co-morph the partner’s pose towards more plausible interaction via a contact-gated morphing strategy driven by the amount of physical contact present in the interaction.

Table 5

		Pose Distortion	Contact Preservation	Foot Skating	Interaction	Downstream RL Policy
Dataset	Method	MPJPE [cm] $\downarrow$	Dur. $\uparrow$	Dur. $\downarrow$	$E_{\text{wrist}}$ [cm] $\downarrow$	Success Rate(%) $\uparrow$
CHI3D	PHC	6.179	0.478	0.107	22.382	68
	OmniRetarget	4.184	0.915	0.009	1.379	100
	Ours	3.893	0.936	0.003	0.501	100
Hi4D	PHC	6.892	0.089	0.087	21.088	79
	OmniRetarget	4.704	0.827	0.006	1.729	100

Table 3: Retargeting comparison across three interaction datasets. Best per column in bold;

\uparrow

\downarrow

indicate higher/lower is better.

Figure 5: Retargeting Qualitative Results. Each panel shows a ground-truth human–human interaction (ego in orange, partner in dark blue) next to its retargeted counterpart. The ego is retargeted to the humanoid (tan), and the partner’s pose is morphed to preserve interaction (light blue). Across diverse interactions our Adaptive-HHI Retargeting method preserves the contacts and relative positioning between the two agents.

Figure 6: Real-world deployment . Frames run left to right in time. Top two rows: separate high-five and hand-wave trials. Bottom: a single continuous take in which the humanoid transitions directly from a handshake into a bend

Scenario	Sim	Real
Handshake	100	100
Hand wave	100	100
High-five	90	90
Hug	100	100
Bend	100	100
Linked arms	100	100

Table 4: Success rate (%) in sim and on the real G1.

Figure 7: Shadow teleoperation . The G1 shadows a human operator performing arm movements (top) and sitting down on a couch (bottom). The operator’s motion is encoded with $E_{h}$ , quantized against the shared codebook $\mathcal{C}$ , and decoded with $D_{r}$ into humanoid motion. Frames run left to right in time.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Clips	Total Frames	Total Time (hrs)
CHI3D [ 15 ]	245	25,588	0.24
Hi4D [ 62 ]	200	23,890	0.22
Harmony4D [ 33 ]	416	153,160	1.42
Inter-X [ 59 ]	22,776	4,049,990	37.50
Embody 3D [ 44 ]	14,692	23,864,668	220.97

Appendix

Table A.1: Summary statistics of our retargeted benchmark. Clip counts, frames, and durations are reported after retargeting each source dataset to the Unitree G1 with Adaptive-HHI.

Method	Hard Kin.	Preserve Interact.	Partner Co-Morph	Adaptive Contact
IMMA [ 46 ]	✓	✗	✗	✗
PHC [ 38 ]	✗	✗	✗	✗
GMR [ 66 ]	✗	✗	✗	✗
VideoMimic [ 3 ]	Soft	✗	✗	✗
OmniRetarget [ 60 ]	✓	✗	✗	✗
Adaptive-HHI Retargeting (Ours)	✓	✓	✓	✓

Appendix

Table A.2: Comparison of retargeting methods. Adaptive-HHI extends OmniRetarget’s interaction-mesh SOCP base with partner co-morphing (joint optimization over the partner SMPL-X) and distance-weighted, wrist-boosted adaptive Laplacian weighting for interaction and human-scale preservation.

	Interaction	Contact Preservation	Foot Skating
Method	$E_{\text{wrist}}$ [cm] $\downarrow$	Dur. $\uparrow$	Dur. $\downarrow$
Baseline (OmniRetarget)	1.379 $\pm$ 0.550	0.915 $\pm$ 0.160	0.009 $\pm$ 0.015
+ contact-adaptive weights ( $w_{v,t}$ )	0.976 $\pm$ 0.534	0.933 $\pm$ 0.131	0.014 $\pm$ 0.018
+ contact-and-reach preservation (Ours)	0.501 $\pm$ 0.290	0.936 $\pm$ 0.119	0.0034 $\pm$ 0.010

Appendix

Table A.3: Adaptive retargeting ablation on CHI3D. Mean

\pm

std over clips. Best per column in bold.

Hyperparameter	Value
Pass I: Laplacian term
Base keypoint weight	10
Wrist / foot keypoint multiplier	$\times 4$ / $\times 0.1$
Laplacian target scale $s$	$h_{\text{robot}}/h_{\text{ego}}$
Pass I: distance and regularization
Wrist / elbow distance weight $\lambda_{j}$	50 / 25

Appendix

Table A.4: Retargeting hyperparameters. Values are shared across all datasets without per-clip tuning.

	FD $\downarrow$				MPJPE [cm] $\downarrow$				Code Agree. $\uparrow$
Method	H	R	H $\to$ R	R $\to$ H	H	R	H $\to$ R	R $\to$ H	$H{=}R$
No alignment	0.0047	0.0026	0.2062	0.1570	3.665	2.809	12.520	12.951	0.000
Continuous only	0.0045	0.0026	0.0034	0.0052	3.665	2.656	3.240	4.239	0.508
Discrete only	0.0054	0.0031	0.0046	0.0054	3.855	2.876	3.063	4.036	0.634
Both (Ours)	0.0047	0.0020	0.0028	0.0044	3.556	2.613	2.793	3.706	0.780

Appendix

Table A.5: Shared-codebook design ablation on HHI. H and R denote human and robot motion; H

\to

R and R

\to

H denote encoding in one embodiment and decoding in the other. FD is Fréchet distance. MPJPE is the root-relative mean per-joint position error of the decoded pose against ground truth (cm; 21 SMPL-X body joints for H, 29 actuated joints of the G1 for R, hands excluded). Code Agree. is the fraction of paired H/R frames assigned the same code. Best per column in bold.

Figure A.1: Cross-embodiment motion translation with a shared-codebook VQ-VAE . Human motion (red) is encoded into a discrete latent space shared between the human and humanoid embodiments, then decoded into humanoid motion (purple). Each example shows three frames of the input human motion alongside the translated humanoid motion in simulation (top row), and the same frames executed on the physical humanoid robot (bottom row).

Module	Component	Input shape	Operation & hyperparameters
Shared-Codebook VQ-VAE	Human & humanoid encoder	$(T,P,D_{\text{in}}{+}D_{\text{c}})$	1D Conv + ResNet, 2 layers, $d_{\text{vq}}{=}512$
	Codebook	$(T^{\prime},P,d_{\text{vq}})$	VQ, $K{=}512$ codes, EMA updates
	Human & humanoid decoder	$(T^{\prime},P,d_{\text{vq}}{+}D_{\text{c}})$	Mirror of encoder, 2 layers
DFoT	Noise emb.	$(T^{\prime},P,d_{\text{e}})$	Sinusoidal, $d_{\text{e}}{=}256$
	Agent emb.	$(T^{\prime},P,d_{\text{e}})$	Learned, $d_{\text{e}}{=}256$
	Input proj.	$(T^{\prime},P,D_{\text{m}}{+}2d_{\text{e}})$	Linear + LayerNorm, 3 layers, ${\rightarrow}\,d{=}512$

Appendix

Table A.6: Network configuration details.

T

is the raw sequence length and

T^{\prime}=T/\omega

the number of tokens after temporal compression (

\omega{=}4

);

P

is the number of interacting agents;

D_{\text{in}}

is the per-agent pose dimension (

D_{h}

D_{r}

);

D_{\text{c}}

is the local conditioning dimension;

d_{\text{e}}

is the embedding dimension; and

D_{\text{m}}=d_{\text{vq}}+4\times 9\times P

is the unified motion token dimension tracking localized cross-agent transformations.

Stage	p50 (ms)	p95 (ms)
perception	104.4	120.4
fetch	7.9	8.4
sync	20.3	21.2
wire	52.7	66.4
JPEG+zstd	23.5	24.4
human	79.4	115.6

Appendix

Table A.7: End-to-End Latency.

Human-Robot Interaction

Human-robot interaction studies how people and robots communicate, coordinate, and safely work or live together.

See all 467 papers

Human2Humanoid: Physics-Aware Cross-Morphology Motion Retargeting for Humanoid Robots
Jun 2, 2026Tianchen Huang, Feiyang Yuan, Junchi Gu +5Cross-Embodiment Robot Learning Motion Retargeting
Institute of Humanoid Robots, Department of Precision Machinery and Precision Instrumentation, University of Science and Technology of China, Hefei, Anhui 230026, China
SocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generation
Sep 27, 2026Chengqun Yang, Tengjie Zhu, Liang Xu +10Human Motion Generation Co-Speech Gesture Generation
Shanghai Jiao Tong University · ZTE Corporation
Social Structure Matters in 3D Human-Human Interaction Generation
Jun 23, 2026Zhongju Wang, Beier Wang, Yatao Bian +6Text-to-Motion Generation Human Motion Generation
University of New South Wales · National University of Singapore · NVIDIA +3

Jun 2, 2026J/K move · Enter open · S save

Human2Humanoid: Physics-Aware Cross-Morphology Motion Retargeting for Humanoid Robots

Tianchen Huang, Feiyang Yuan, Junchi Gu, Shurui Fang+4

Institute of Humanoid Robots, Department of Precision Machinery and Precision Instrumentation, University of Science and Technology of China, Hefei, Anhui 230026, China

Cross-Embodiment Robot Learning Motion Retargeting Humanoid Robotics Human-to-Robot Motion Imitation

Open paper AskPDF arXiv

Retargeting human motion to humanoid robots is critical for teleoperation, imitation learning and human-robot interaction. However, it remains challenging because of substantial morphological discrepancies between humans and robots, including differences in skeletal topology, limb proportions and degrees of freedom, as well as the scarcity of paired motion data. This paper presents Human2Humanoid, an unsupervised motion retargeting framework that transfers human motions to humanoid robot behaviors with high fidelity. To bridge the domain gap under unpaired data, we adopt a CycleGAN-based architecture equipped with a skeleton-aware graph convolutional network to capture topology-dependent motion features. To address cross-domain scale mismatches, we introduce a morphology-invariant end-effector consistency loss that aligns normalized end-effector trajectories to preserve motion semantics across embodiments. To improve physical plausibility and reduce contact artifacts, we impose explicit physics-aware feasibility constraints to encourage reproduction of the contact patterns in the source motion. Experimental results show that the proposed method successfully retargets human motion to the Unitree G1 humanoid robot without paired data, and outperforms existing methods in both downstream controllability and physical feasibility.

Abstract

Human-Robot Interaction

Explore similar work

Human2Humanoid: Physics-Aware Cross-Morphology Motion Retargeting for Humanoid Robots

SocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generation

Social Structure Matters in 3D Human-Human Interaction Generation