MAMHOI: Factorizing Scene-Aware Human-Object Interaction through Affordances

Mingyuan Lei; Yoonchang Sung; Tat-Jen Cham

cs.CVOct 8, 2026

MAMHOI: Factorizing Scene-Aware Human-Object Interaction through Affordances

Authors: Mingyuan Lei, Yoonchang Sung, Tat-Jen Cham

Organizations: College of Computing and Data Science Nanyang Technological University, Singapore

Abstract

Generating realistic human-object interactions (HOI) in complex 3D scenes requires two complementary capabilities: reasoning about interaction feasibility in the environment and synthesizing realistic human-object motion. However, supervision for these capabilities is rarely available jointly at scale. Human-scene datasets provide rich information about environment-aware motion, while human-object datasets capture detailed interaction dynamics, yet paired human-object-scene data remain scarce. We present MAMHOI, an affordance-mediated factorization for scene-aware human-object interaction generation. MAMHOI factorizes scene-aware HOI generation through an explicit motion-affordance interface between scene understanding and motion synthesis: a scene-conditioned model first predicts where and how an interaction can be feasibly executed, and an affordance-conditioned HOI model then generates the corresponding human-object motion. This factorization allows scene understanding and interaction dynamics to be learned from complementary sources of supervision without requiring paired human-object-scene data. Experiments in complex indoor environments show that MAMHOI reduces object--scene penetration while better preserving human--object interaction quality, yielding more realistic and physically feasible scene-aware interactions. Project page: https://leimingyuan.github.io/MAMHOI-project-page/

Figures & tables

Figure 1: MAMHOI learns from Scene-Free HOI priors and transfers the ability to complex environments through scene-aware motion affordances. This factorization enables physically plausible interactions across cluttered scenes and diverse objects.

Figure 2: Overview of MAMHOI. (A) The scene-understanding module predicts a task-specific affordance from the scene, planned object path, and instruction, where the instruction guides the task-relevant human placement along the object path. (B) The affordance interface lifts the predicted affordance to 3D and queries local occupancy around the pelvis, object, and goal position of current segment, and encodes them into affordance features through a ViT-based Affordance-voxel encoder. (C) The resulting affordance features condition segment-wise HOI generation through cross-attention, with each generated segment initializing the next. Through this design, scene-aware HOI generation is factorized into a scene-to-affordance factor learned from indoor HSI data and an affordance-to-motion factor learned from scene-free HOI data.

Method	FID $\downarrow$	R-precision $\uparrow$			MMD $\downarrow$	Diversity $\uparrow$
		top-1	top-2	top-3
CHOIS [ 15 ]	2.7892	0.7428	0.8678	0.9062	3.9086	8.3841
InfBaGel [ 31 ]	2.2451	0.8149	0.9159	0.9423	3.4893	9.2597
Ours	1.8361	0.7764	0.9279	0.9543	3.2798	8.6923
Ours - $\mathcal{L}_{\mathrm{rank}}$	0.9403	0.8173	0.8750	0.9375	3.1553	8.7915

Table 1: Interaction-only evaluation on the OMOMO validation split. All methods retain their standard non-scene guidance or trajectory constraints and no scene-specific guidance is applied. InfBaGel is evaluated with its native pelvis-trajectory constraint. Best results are shown in bold and second-best results are underlined.

Method	Human-Scene				Object-Scene			Human-Object
	$\bar{D}^{\mathrm{vtx}}\downarrow$	$\bar{D}^{\mathrm{frm}}\downarrow$	$R^{\mathrm{frm}}\downarrow$	$J_{\mathrm{H}}^{\mathrm{seq}}\downarrow$	$\bar{D}^{\mathrm{vtx}}\downarrow$	$\bar{D}^{\mathrm{frm}}\downarrow$	$D_{\mathrm{path}}^{\mathrm{obj}}\downarrow$	$R_{\mathrm{con}}\uparrow$	$\bar{D}^{\mathrm{vtx}}\downarrow$	$\bar{D}^{\mathrm{frm}}\downarrow$
	$\times 10^{-3}$	$\times 10^{-3}$	$\%$		$\times 10^{-3}$	$\times 10^{-3}$		$\%$	$\times 10^{-3}$	$\times 10^{-3}$
CHOIS	47.337	36.232	29.645	7808.0	0.545	2.267	0.1532	90.97	1.353	2.366
CHOIS + $\mathcal{G}_{\mathrm{sdf}}$	32.936	24.925	18.437	7906.9	0.487	2.041	0.1537	90.44	1.435	2.458
InfBaGel	36.478	26.315	31.848	5258.7	3.136	7.281	–	92.39	1.507	2.424
Ours	28.587	22.009	14.990	4429.3	0.403	1.440	0.1492	94.87	1.416	2.200

Table 2: Comparison of methods on the constructed scene-aware test set. We evaluate human–scene penetration, object–scene penetration, object trajectory adherence, motion smoothness, and human–object interaction quality.

\mathcal{G}_{\mathrm{sdf}}

denotes the same generic test-time scene-SDF guidance applied to CHOIS and MAMHOI. InfBaGel is evaluated with its native Bump-aware Guidance. Best results are shown in bold and second-best results are underlined.

Figure 3: Qualitative comparison on the constructed scene-aware test set. $\mathcal{G}_{\mathrm{sdf}}$ denotes the generic test-time scene-SDF refinement applied to CHOIS and MAMHOI, while InfBaGel is evaluated with its native Bump-aware Guidance. Penetration regions are highlighted from yellow to red.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Human-Scene				Object-Scene			Human-Object
	$\bar{D}^{\mathrm{vtx}}\downarrow$	$\bar{D}^{\mathrm{frm}}\downarrow$	$R^{\mathrm{frm}}\downarrow$	$J_{\mathrm{H}}^{\mathrm{seq}}\downarrow$	$\bar{D}^{\mathrm{vtx}}\downarrow$	$\bar{D}^{\mathrm{frm}}\downarrow$	$D_{\mathrm{path}}^{\mathrm{obj}}\downarrow$	$R_{\mathrm{con}}\uparrow$	$\bar{D}^{\mathrm{vtx}}\downarrow$	$\bar{D}^{\mathrm{frm}}\downarrow$
	$\times 10^{-3}$	$\times 10^{-3}$	$\%$		$\times 10^{-3}$	$\times 10^{-3}$		$\%$	$\times 10^{-3}$	$\times 10^{-3}$
Scene-aligned	41.016	32.254	26.858	7299.8	0.624	2.185	0.1544	91.99	1.443	2.504
Scene-aligned + $\mathcal{G}_{\mathrm{sdf}}$	32.240	25.043	17.404	7686.9	0.551	1.921	0.1535	93.02	1.487	2.536
Ours	28.587	22.009	14.990	4429.3	0.403	1.440	0.1492	94.87	1.416	2.200
Ours + $\mathcal{G}_{\mathrm{sdf}}$	25.884	19.889	13.745	4276.1	0.398	1.498	0.1507	95.69	1.431	2.243

Appendix

Table 3: Ablation of the coordinate frame used for local scene voxels. Both variants use identical interaction-centric voxel regions; the scene-aligned variant retains the global scene orientation, while ours canonicalizes the voxels with respect to the human orientation. Human-aligned canonicalization consistently improves scene-aware motion generation, with test-time SDF guidance providing additional collision reduction. Best and second-best results are shown in bold and underlined, respectively.

Method	Human-Scene				Object-Scene			Human-Object
	$\bar{D}^{\mathrm{vtx}}\downarrow$	$\bar{D}^{\mathrm{frm}}\downarrow$	$R^{\mathrm{frm}}\downarrow$	$J_{\mathrm{H}}^{\mathrm{seq}}\downarrow$	$\bar{D}^{\mathrm{vtx}}\downarrow$	$\bar{D}^{\mathrm{frm}}\downarrow$	$D_{\mathrm{path}}^{\mathrm{obj}}\downarrow$	$R_{\mathrm{con}}\uparrow$	$\bar{D}^{\mathrm{vtx}}\downarrow$	$\bar{D}^{\mathrm{frm}}\downarrow$
	$\times 10^{-3}$	$\times 10^{-3}$	$\%$		$\times 10^{-3}$	$\times 10^{-3}$		$\%$	$\times 10^{-3}$	$\times 10^{-3}$
Ours - $\mathcal{L}_{\mathrm{rank}}$	30.133	22.890	17.342	6907.6	0.517	2.002	0.1464	92.27	1.394	2.361
Ours - $\mathcal{L}_{\mathrm{rank}}$ + $\mathcal{G}_{\mathrm{sdf}}$	27.194	20.717	14.813	6818.2	0.458	1.843	0.1463	92.49	1.401	2.357
Ours	28.587	22.009	14.990	4429.3	0.403	1.440	0.1492	94.87	1.416	2.200
Ours + $\mathcal{G}_{\mathrm{sdf}}$	25.884	19.889	13.745	4276.1	0.398	1.498	0.1507	95.69	1.431	2.243

Appendix

Table 4: Ablation study of the counterfactual ranking loss

\mathcal{L}_{\mathrm{rank}}

, with and without test-time SDF guidance. “Ours” denotes the model trained with

\mathcal{L}_{\mathrm{rank}}

. Best and second-best results are shown in bold and underlined, respectively.

Method	Human-Scene				Object-Scene			Human-Object
	$\bar{D}^{\mathrm{vtx}}\downarrow$	$\bar{D}^{\mathrm{frm}}\downarrow$	$R^{\mathrm{frm}}\downarrow$	$J_{\mathrm{H}}^{\mathrm{seq}}\downarrow$	$\bar{D}^{\mathrm{vtx}}\downarrow$	$\bar{D}^{\mathrm{frm}}\downarrow$	$D_{\mathrm{path}}^{\mathrm{obj}}\downarrow$	$R_{\mathrm{con}}\uparrow$	$\bar{D}^{\mathrm{vtx}}\downarrow$	$\bar{D}^{\mathrm{frm}}\downarrow$
	$\times 10^{-3}$	$\times 10^{-3}$	$\%$		$\times 10^{-3}$	$\times 10^{-3}$		$\%$	$\times 10^{-3}$	$\times 10^{-3}$
CHOIS	68.559	55.442	50.064	14079.9	0.476	4.468	0.1173	56.90	0.806	2.611
HOSIG	45.652	38.502	26.572	18099.7	7.910	18.200	0.4471	98.26	2.457	4.969
Lingo	24.541	19.870	27.566	13077.9	–	–	–	–	–	–
Ours	25.534	19.680	17.282	10252.5	0.003	0.044	0.0964	90.11	2.460	4.694

Appendix

Table 5: Comparison on the TRUMANS-based scene-aware test set. Missing or inapplicable metrics are denoted by “–”. Best and second-best available results are shown in bold and underlined, respectively.

Figure 4: Example interface used in the perceptual study. For each trial, participants view two motion sequences generated under the same scene and task configuration and select the interaction that appears more physically plausible and natural in the scene. Method identities are hidden, while trial order and left–right placement are randomized.

Method A	Method B	A votes	B votes	Total votes	A pref.	B pref.
Ours	CHOIS	195	30	225	86.7%	13.3%
Ours+ $\mathcal{G}_{\mathrm{sdf}}$	CHOIS+ $\mathcal{G}_{\mathrm{sdf}}$	182	34	216	84.3%	15.7%
Ours	CHOIS+ $\mathcal{G}_{\mathrm{sdf}}$	204	17	221	92.3%	7.7%
Ours+ $\mathcal{G}_{\mathrm{sdf}}$	CHOIS	180	42	222	81.1%	18.9%
Total judgments				884

Appendix

Table 6: Aggregated results of the perceptual study. “Method A” and “Method B” are used only to organize the table and do not correspond to fixed left or right positions in the study interface; the displayed left–right placement was randomized independently for every trial.

Explore similar work

MaMi-HOI: Harmonizing Global Kinematics and Local Geometry for Human-Object Interaction Generation
May 7, 2026Hao Wang, Shiqi Wang, Qi LiuHOI Generation Human Motion Generation
School of Future Technology, South China University of Technology, Guangzhou, Guangdong, China.
Learning to Generate Human-Human-Object Interactions from Textual Descriptions
Nov 25, 2025Jeonghyeon Na, Sangwon Baik, Inhee Lee +2HOI Generation Human Motion Generation
Seoul National University
PhotoHOI: Synthesizing 3D Hand-Object Interactions from a Single RGB Photograph
Aug 3, 2026Zhenhao Zhang, Jiajun Zhang, Wei Min +1HOI Generation Dexterous Manipulation
Tsinghua University · Shadow AI

May 7, 2026, cs.ROJ/K move · Enter open · S save

MaMi-HOI: Harmonizing Global Kinematics and Local Geometry for Human-Object Interaction Generation

Hao Wang, Shiqi Wang, Qi Liu

School of Future Technology, South China University of Technology, Guangzhou, Guangdong, China.

HOI Generation Human Motion Generation Diffusion Models Human-Object Interaction

Open paper AskPDF arXiv

Generating realistic 3D Human-Object Interactions (HOI) is a fundamental task for applications ranging from embodied AI to virtual content creation, which requires harmonizing high-level semantic intent with strict low-level physical constraints. Existing methods excel at semantic alignment, however, they struggle to maintain precise object contact. We reveal a key finding termed \textit{Geometric Forgetting}: as diffusion model depth increases, semantic feature tend to overshadow object geometry feature, causing the model to lose its perception to object geometry. To address this, we propose MaMi-HOI, a hierarchical framework reconciling \textbf{Ma}cro-level kinematic fluidity with \textbf{Mi}cro-level spatial precision. First, to counteract geometric forgetting, we introduce the Geometry-Aware Proximity Adapter (GAPA), which explicitly re-injects dense object details to perform residual snapping corrections for precise contact. Nevertheless, such aggressive local enforcement can disrupt global dynamics, leading to robotic stiffness. In response, we introduce the Kinematic Harmony Adapter (KHA), which proactively aligns whole-body posture with spatial objectives, ensuring the skeleton actively accommodates constraints without compromising naturalness. Extensive experiments validate that MaMi-HOI simultaneously achieves natural motion and precise contact. Crucially, it extends generation capabilities to long-term tasks with complex trajectories, effectively bridging the gap between global navigation and high-fidelity manipulation in 3D scenes. Code is available at https://github.com/DON738110198/MaMi-HOI.git

Abstract

Explore similar work

MaMi-HOI: Harmonizing Global Kinematics and Local Geometry for Human-Object Interaction Generation

Learning to Generate Human-Human-Object Interactions from Textual Descriptions

PhotoHOI: Synthesizing 3D Hand-Object Interactions from a Single RGB Photograph