Disentangled Hierarchical VAE for 3D Human-Human Interaction Generation

Zichen Geng; Zeeshan Hayder; Bo Miao; Jian Liu; Wei Liu; Ajmal Mian

cs.CVFeb 24, 2026

Disentangled Hierarchical VAE for 3D Human-Human Interaction Generation

Authors: Zichen Geng, Zeeshan Hayder, Bo Miao, Jian Liu, Wei Liu, Ajmal Mian

Organizations: Department of CSSE, The University of Western Australia, Crawley WA 6009, Australia · Data61, CSIRO, Acton ACT 2601, Australia · Australian Institute for Machine Learning, The University of Adelaide, Adelaide SA 5000, Australia · NERC-RVC, Hunan University, Changsha 410012, China

Abstract

Generating realistic 3D Human-Human Interaction (HHI) requires coherent modeling of the physical plausibility of the agents and their interaction semantics. Existing methods compress all motion information into a single latent representation, limiting their ability to capture fine-grained actions and inter-agent interactions. This often leads to semantic misalignment and physically implausible artifacts, such as penetration or missed contact. We propose Disentangled Hierarchical Variational Autoencoder (DHVAE) based latent diffusion for structured and controllable HHI generation. DHVAE explicitly disentangles the global interaction context and individual motion patterns into a decoupled latent structure by employing a CoTransformer module. To mitigate implausible and physically inconsistent contacts in HHI, we incorporate contrastive learning constraints with our DHVAE to promote a more discriminative and physically plausible latent interaction space. For high-fidelity interaction synthesis, DHVAE employs a DDIM-based diffusion denoising process in the hierarchical latent space, enhanced by a skip-connected AdaLN-Transformer denoiser. Extensive evaluations show that DHVAE achieves superior motion fidelity, text alignment, and physical plausibility with greater computational efficiency.

Figures & tables

Figure 1: (a) InterLDM two-in-one , (b) InterMask intermask encode all motion information into a single latent. (c) Our encodes individual motions and interactions into separate disentangled latents.

Figure 2: Architecture of our DHVAE to encode the structured latent representation $\mathbf{z}_{o},\mathbf{z}_{a},\mathbf{z}_{b}$ . The global latent token $\mathbf{z}_{o}$ will learn an interaction plausible space via contrastive learning. The encoded structured representation will be passed into a skip-connected AdaLN Transformer to learn the denoise process.

\mathcal{L}_{\text{triplet}}=\max(0,d(\mathbf{z}_{o},\mathbf{z}_{o}^{+})-d(\mathbf{z}_{o},\mathbf{z}_{o}^{-})+m)

Algorithm 1 Contrastive Learning for Interaction Latent

\mathbf{z}_{o}

InterHuman
Model	R-Prec@1 $\uparrow$	R-Prec@2 $\uparrow$	R-Prec@3 $\uparrow$	FID $\downarrow$	MM Dist $\downarrow$	Diversity $\rightarrow$	MModality $\uparrow$
Ground Truth	$0.452^{\pm.008}$	$0.610^{\pm.009}$	$0.701^{\pm.008}$	$0.273^{\pm.007}$	$3.755^{\pm.008}$	$7.948^{\pm.064}$	-
InterGen intergen	$0.371^{\pm.010}$	$0.515^{\pm.012}$	$0.624^{\pm.010}$	$5.918^{\pm.079}$	$5.108^{\pm.014}$	$7.387^{\pm.029}$	$\textbf{2.141}^{\pm.063}$
MLD* mld	$0.392^{\pm.005}$	$0.533^{\pm.005}$	$0.612^{\pm.004}$	$6.158^{\pm.082}$	$3.817^{\pm.003}$	$7.785^{\pm.048}$	$1.236^{\pm.035}$
MoMat-MoGen momatmogen	${0.449}^{\pm.004}$	$0.591^{\pm.003}$	$0.666^{\pm.004}$	$5.674^{\pm.120}$	${3.790}^{\pm.001}$	$8.021^{\pm.035}$	$1.295^{\pm.023}$
in2IN in2in	${0.455}^{\pm.004}$	${0.611}^{\pm.008}$	${0.687}^{\pm.009}$	$5.177^{\pm.120}$	${3.790}^{\pm.002}$	$7.940^{\pm.047}$	$1.061^{\pm.038}$

Table 1: Comparisons on InterHuman and InterX datasets. The best results are in bold , and the second-best are underlined . Methods with * are implemented by us. All results are run 20 times. For a fair comparison, we set the latent size of MLD to be the same as ours, i.e.

3\times 256

Model	PV $\downarrow$	PFR $\downarrow$	PDR $\downarrow$	Contact $\uparrow$
MLD*	0.503	0.108	0.175	0.427
InterMask	0.873	0.149	0.243	0.349
TIMotion	0.485	0.122	0.104	0.466
Ours w/o triplet	0.446	0.107	0.102	0.445
Ours	$\mathbf{0.390}$	0.064	0.087	0.581

Table 8: Penetration metrics and contact ratio for state-of-the-art models.

Figure 3: Qualitative Comparison with InterMask intermask on InterHuman Dataset, indicating superior text alignment, fidelity, and physical plausibility. The body meshes are arranged sequentially from left to right, with colors progressing from light to dark.

DHVAE				Denoiser			Metrics
Triplet Loss	$\mathbf{z}_{o}$	CoTRM	MLD-VAE	SPE	TokenScale	Skip	rFID $\downarrow$	gFID $\downarrow$	RP@1 $\uparrow$
-	✓	✓	-	✓	✓	✓	0.499	5.063	0.492
✓	-	✓	-	✓	✓	✓	0.669	5.886	0.468
-	-	✓	-	✓	✓	✓	0.667	6.005	0.463
✓	✓	-	-	✓	✓	✓	0.632	5.593	0.476
-	-	-	✓	✓	✓	✓	1.024	5.694	0.457

Table 9: Ablation study of DHVAE and Denoiser components. ✓ indicates the component is used. For a fair comparison with MLD, we set the latent size of MLD-VAE to be

3\times 256

Figure 4: User Study of DHVAE compared to InterMask and TIMotion

Figure 5: Metrics along the change along classifier-free-guidance scale on InterHuman dataset

Figure 6: Metrics along the change along classifier-free-guidance scale on InterX dataset

Dataset	Model	R-Prec@1 $\uparrow$	R-Prec@2 $\uparrow$	R-Prec@3 $\uparrow$	FID $\downarrow$	MM Dist $\downarrow$	Diversity $\rightarrow$
InterHuman	Ground Truth	$0.452^{\pm 0.008}$	$0.610^{\pm 0.007}$	$0.701^{\pm 0.008}$	$0.273^{\pm 0.007}$	$3.755^{\pm 0.008}$	$7.948^{\pm 0.064}$
	InterGen intergen	$0.371^{\pm.010}$	$0.515^{\pm.012}$	$0.624^{\pm.010}$	$5.918^{\pm.079}$	$5.108^{\pm.014}$	$7.387^{\pm.029}$
	MLD * mld	$0.392^{\pm.005}$	$0.533^{\pm.005}$	$0.612^{\pm.004}$	$6.158^{\pm.082}$	$3.817^{\pm.003}$	$7.785^{\pm.048}$
	InterLDM	${0.427}^{\pm.004}$	$0.559^{\pm.050}$	$0.638^{\pm.004}$	$5.619^{\pm.091}$	$1.862^{\pm.007}$	$7.888^{\pm.041}$
	MoMat-MoGen momatmogen	${0.449}^{\pm.004}$	$0.591^{\pm.003}$	$0.666^{\pm.004}$	$5.674^{\pm.120}$	$\underline{3.790}^{\pm.001}$	$8.021^{\pm.035}$
	InterMask intermask	${0.449}^{\pm.004}$	${0.599}^{\pm.005}$	${0.681}^{\pm.004}$	$\underline{5.153}^{\pm.061}$	$\underline{3.790}^{\pm.002}$	$\textbf{7.944}^{\pm.033}$

Table 10: Comparison results on InterHuman including InterLDM . The best results are marked with bold font, and the second-best results are marked with underline . The group with * is implemented by us. To avoid coincidence, each metric is repeated 20 times, and a standard deviation is provided.

Figure 7: Comparison with TIMotion on InterHuman

Panel of Figure 8 — Figure 8: PCA projections of the interaction latent $\mathbf{z}_{o}$ in the hand-shaking scenario. (a) shows encodings of ground-truth motion triplets; (b) visualizes $\mathbf{z}_{o}$ of generated motions from prompts. Both compare DHVAE with and without contrastive learning.

Figure 9: Visualization ablation study for Contrastive learning

Panel of Figure 10 — Figure 10: T-SNE projections of latents on the InterHuman test set.

Setting	PV $\downarrow$	PFR $\downarrow$	PDR $\downarrow$	Contact $\uparrow$	FID $\downarrow$
$\sigma_{c}=0.02,~\sigma_{u}=0.30$	0.392	0.073	0.088	0.557	5.050
$\sigma_{c}=0.03,~\sigma_{u}=0.30$	0.387	0.069	0.082	0.589	5.093
$\sigma_{c}=0.05,~\sigma_{u}=0.30$	0.390	0.064	0.087	0.581	5.015
$\sigma_{c}=0.06,~\sigma_{u}=0.30$	0.368	0.067	0.088	0.592	5.047
$\sigma_{c}=0.08,~\sigma_{u}=0.30$	0.395	0.070	0.085	0.572	5.022
$\sigma_{c}=0.10,~\sigma_{u}=0.30$	0.402	0.068	0.090	0.524	5.035

Table 11: Performance under different

\sigma_{c}

and

\sigma_{u}

settings.

VAE settings	number	rFID $\downarrow$	FID $\downarrow$	R-Prec@1 $\uparrow$	R-Prec@2 $\uparrow$	R-Prec@3 $\uparrow$	MM Dist $\downarrow$
Layer Number	Ground Truth	-	$0.273$	$0.452$	$0.610$	$0.701$	$3.755$
	(4, 1)	0.539	5.290	0.492	0.645	0.717	3.775
	(4, 3)	0.503	5.015	0.496	0.647	0.720	3.772
	(4, 5)	0.522	5.145	0.489	0.643	0.712	3.780
	(4, 7)	0.486	5.312	0.483	0.635	0.707	3.782
	(6, 3)	0.479	5.349	0.499	0.650	0.721	3.770

Table 12: Ablation study on different DHVAE layer settings on InterHuman. The tuple in Layer Number represents the number of individual Transformer Encoder layers and the CoTransformer Encoder Layer, respectively.

VAE settings	number	rFID $\downarrow$	FID $\downarrow$	R-Prec@1 $\uparrow$	R-Prec@2 $\uparrow$	R-Prec@3 $\uparrow$	MM Dist $\downarrow$
Latent Size	l=1	0.503	5.015	0.496	0.647	0.720	3.772
	l=2	0.459	5.116	0.501	0.653	0.725	3.770
	l=3	0.447	5.269	0.496	0.644	0.721	3.771
	l=5	0.433	5.571	0.490	0.640	0.715	3.778
	l=7	0.462	5.670	0.486	0.638	0.709	3.782

Table 13: Ablation study on different latent sizes for DHVAE on InterHuman.

Dataset	Number of Layers	R-Prec@1 $\uparrow$	R-Prec@2 $\uparrow$	R-Prec@3 $\uparrow$	FID $\downarrow$	MM Dist $\downarrow$	Diversity $\rightarrow$
InterHuman	Ground Truth	$0.452$	$0.610$	$0.701$	$0.273$	$3.755$	$7.948$
	l=7	$0.486$	$0.632$	$0.708$	$5.301$	$3.783$	$7.928$
	l=9	$0.491$	$0.642$	$0.718$	$5.107$	$3.772$	$7.970$
	l=11	0.498	0.649	0.721	$5.122$	3.770	$7.988$
	l=13	$\underline{0.496}$	$\underline{0.647}$	$\underline{0.720}$	$\underline{5.015}$	$3.772$	7.952
	l=15	$0.493$	$0.643$	$0.716$	4.983	$3.775$	$\underline{7.940}$

Table 14: Performance with different number of Transformer Encoder Layer

Figure 11: HHI motion sequences generated on InterHuman dataset

Figure 12: HHI motion sequences generated on InterX dataset, including gestures

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

Social Structure Matters in 3D Human-Human Interaction Generation
Jun 23, 2026Zhongju Wang, Beier Wang, Yatao Bian +6Text-to-Motion Generation Human Motion Generation
University of New South Wales · National University of Singapore · NVIDIA +3
GNOCHI: Generative Neural mOdel for Close Human-Human Interactions
Jul 11, 2026Gonzalo Gómez-Nogales, Marc Comino-Trinidad, Andrés Casado-Elvira +1Human Motion Generation Human-Human-Object Interaction Generation
Universidad Rey Juan Carlos, Madrid, Spain.
ViHOI: Human-Object Interaction Synthesis with Visual Priors
Mar 25, 2026Songjin Cai, Linjie Zhong, Ling Guo +1HOI Generation Human Motion Generation
South China University of Technology

Jun 23, 2026J/K move · Enter open · S save

Social Structure Matters in 3D Human-Human Interaction Generation

Zhongju Wang, Beier Wang, Yatao Bian, Pichao Wang+5

University of New South Wales · National University of Singapore · NVIDIA+3

Text-to-Motion Generation Human Motion Generation Human-Human-Object Interaction Generation

Open paper AskPDF arXiv

Although text-to-motion generation has achieved strong progress in synthesizing realistic single-person motions from language, extending it to text-driven 3D human-human interaction (HHI) remains non-trivial, as HHI requires modeling the underlying social structure that governs phase progression, actor roles, and inter-actor coordination. In this paper, we formulate HHI generation as a social structure modeling and grounding problem: the model must first infer how an interaction unfolds and how the two actors coordinate their roles, and then realize this structure as continuous, physically plausible, and partner-aware 3D motion. To study how such structure should be modeled, we first examine the capability boundary of large language models (LLMs) for HHI generation. Our analysis shows that LLMs can think by recovering phase decompositions and partner-aware roles, but cannot directly move, as they fail to generate dynamic, physically plausible, and interaction-aware motion. This motivates our planner-executor paradigm, Think with LLM, Move with Motion Skill. The LLM planner converts implicit interaction semantics into motion-aligned social supervision by decomposing interactions into phases, assigning partner-aware actor roles, and aligning them with motion sequence. The motion executor then grounds the planned social structure into coordinated two-person motion by adapting a pretrained solo motion model with LoRA, previous-phase self-conditioning, and ego-relative partner conditioning. Together, our Solo-to-Social framework bridges social organization and motion realization, producing 3D HHI with improved phase consistency, role alignment, and partner-aware coordination.

Abstract

Human Motion Generation

Explore similar work

Social Structure Matters in 3D Human-Human Interaction Generation

GNOCHI: Generative Neural mOdel for Close Human-Human Interactions

ViHOI: Human-Object Interaction Synthesis with Visual Priors