Harnessing Coupled Stream Completion For Human-Object Interaction Modeling

Dawei Guan; Di Yang; Jiangtao Wang

cs.CVSep 26, 2026

Harnessing Coupled Stream Completion For Human-Object Interaction Modeling

Authors: Dawei Guan, Di Yang, Jiangtao Wang

Organizations: School of Artificial Intelligence and Data Science, University of Science and Technology of China

Abstract

Text-conditioned human-object interaction (HOI) generation requires body motion, object trajectories & rotations, and hand articulation to remain coordinated. These components differ in scale and dynamics, but must agree on contact, relative pose, and timing. A shared representation may limit the distinct structure of each stream, while independent generation prevents each stream from responding to changes in the others. Latent supervision alone also does not directly constrain contact after decoding. We propose TRACE, a continuous latent framework that keeps stream states separate and couples their updates. TRACE encodes body, object, and hand motion into separate latents and predicts each stream velocity from the complete current interaction state. Geometric losses on decoded motion further constrain contact and object-relative motion over time. The same model supports completion of any single absent stream from the other two. Frozen flow features also serve as input to a language model for HOI understanding. Experiments on InterAct, OMOMO, and BEHAVE show that joint completion training improves generation and that frozen flow features improve understanding over raw-motion encoding. On InterAct, TRACE achieves the highest contact precision, recall, and F1 among the compared methods.

Figures & tables

Figure 2: Overview of TRACE. Left: HOI Generation. We form different modalities, e.g., body, hand, and object, each compressed into latent vectors by its own encoder. After adding stream and temporal embeddings, a coupled flow transformer predicts their velocities, which are integrated by a flow sampler. Then, different modalities’ decoders reconstruct them into clean motion. Right: HOI Extensions. Leveraging its HOI representation, we enable HOI understanding via language model fine-tuning. Joint training with three completion paths, each targeting one stream, adds completion capability while improving HOI Generation.

Figure 3: Qualitative comparison with baselines. Our method yields more realistic human-object interactions, fewer contact/penetration artifacts, more accurate finger positioning, and better text-motion alignment.

Method	R-Precision $\uparrow$			FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
Method	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{\mathrm{F1}}$
Ground Truth	$0.600^{\scriptscriptstyle\pm 0.004}$	$0.834^{\scriptscriptstyle\pm 0.001}$	$0.909^{\scriptscriptstyle\pm 0.001}$	$0.000^{\scriptscriptstyle\pm 0.000}$	$1.475^{\scriptscriptstyle\pm 0.003}$	$7.781^{\scriptscriptstyle\pm 0.140}$	$0.083^{\scriptscriptstyle\pm 0.204}$	$0.076^{\scriptscriptstyle\pm 0.000}$	$0.208^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$
HOI-Diff	$0.413^{\scriptscriptstyle\pm 0.010}$	$0.624^{\scriptscriptstyle\pm 0.009}$	$0.740^{\scriptscriptstyle\pm 0.011}$	$0.689^{\scriptscriptstyle\pm 0.031}$	$3.029^{\scriptscriptstyle\pm 0.007}$	$7.620^{\scriptscriptstyle\pm 0.085}$	$0.072^{\scriptscriptstyle\pm 0.160}$	$\mathbf{0.103}^{\scriptscriptstyle\pm 0.010}$	$0.084^{\scriptscriptstyle\pm 0.006}$	$0.722^{\scriptscriptstyle\pm 0.002}$	$0.447^{\scriptscriptstyle\pm 0.026}$	$0.501^{\scriptscriptstyle\pm 0.022}$
CHOIS	$\underline{0.439}^{\scriptscriptstyle\pm 0.005}$	$\underline{0.660}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.766}^{\scriptscriptstyle\pm 0.003}$	$0.572^{\scriptscriptstyle\pm 0.008}$	$2.781^{\scriptscriptstyle\pm 0.017}$	$\underline{7.717}^{\scriptscriptstyle\pm 0.115}$	$0.119^{\scriptscriptstyle\pm 0.234}$	$0.131^{\scriptscriptstyle\pm 0.013}$	$0.115^{\scriptscriptstyle\pm 0.002}$	$0.710^{\scriptscriptstyle\pm 0.012}$	$0.520^{\scriptscriptstyle\pm 0.004}$	$0.541^{\scriptscriptstyle\pm 0.007}$
InterDiff	$\mathbf{0.501}^{\scriptscriptstyle\pm 0.009}$	$\mathbf{0.722}^{\scriptscriptstyle\pm 0.003}$	$\mathbf{0.824}^{\scriptscriptstyle\pm 0.007}$	$0.215^{\scriptscriptstyle\pm 0.001}$	$2.461^{\scriptscriptstyle\pm 0.013}$	$\underline{7.717}^{\scriptscriptstyle\pm 0.115}$	$0.092^{\scriptscriptstyle\pm 0.178}$	$0.116^{\scriptscriptstyle\pm 0.011}$	$0.124^{\scriptscriptstyle\pm 0.003}$	$0.715^{\scriptscriptstyle\pm 0.000}$	$0.562^{\scriptscriptstyle\pm 0.013}$	$0.584^{\scriptscriptstyle\pm 0.003}$
Text2HOI	$0.428^{\scriptscriptstyle\pm 0.003}$	$0.637^{\scriptscriptstyle\pm 0.000}$	$0.745^{\scriptscriptstyle\pm 0.000}$	$0.331^{\scriptscriptstyle\pm 0.003}$	$2.665^{\scriptscriptstyle\pm 0.002}$	$7.631^{\scriptscriptstyle\pm 0.000}$	$\underline{0.055}^{\scriptscriptstyle\pm 0.123}$	$\underline{0.105}^{\scriptscriptstyle\pm 0.003}$	$0.102^{\scriptscriptstyle\pm 0.001}$	$0.711^{\scriptscriptstyle\pm 0.001}$	$0.488^{\scriptscriptstyle\pm 0.000}$	$0.532^{\scriptscriptstyle\pm 0.002}$

Table 1: Quantitative comparisons on the InterAct dataset between our method and baseline approaches. We report R-Precision with a batch size of 256.

Figure 4: Qualitative results of stream completion. From left to right, we show the ground-truth interaction and the completion of object, hand, and body streams. Gray denotes observed motion, while pink, magenta, and teal denote the generated object, hands, and body, respectively.

Methods	R-Precision $\uparrow$			MM Dist. $\downarrow$	Bleu@1 $\uparrow$	Bleu@4 $\uparrow$	Rouge $\uparrow$	Cider $\uparrow$	BertScore $\uparrow$
Methods	Top 1	Top 2	Top 3	MM Dist. $\downarrow$	Bleu@1 $\uparrow$	Bleu@4 $\uparrow$	Rouge $\uparrow$	Cider $\uparrow$	BertScore $\uparrow$
Real	0.592	0.757	0.853	1.486	–	–	–	–	–
TM2T	0.206	0.351	0.466	3.342	35.18	24.12	56.17	2.61	52.16
MotionGPT	0.224	0.391	0.504	3.086	39.86	29.63	60.84	3.42	57.83
LaMPM2T	0.243	0.415	0.532	2.894	44.37	34.42	66.14	4.09	63.18
HOIGPT	0.256	0.452	0.570	2.756	50.31	39.22	71.08	4.89	69.05

Table 2: Comparison of HOI understanding on InterAct. Our model is trained for 100 epochs.

Figure 5: Qualitative comparison of stream modeling. We compare unified, independent, and coupled stream modeling. Red boxes highlight object drift and hand-object misalignment in the first two variants, while coupled modeling yields coherent interactions.

Strategy	R-Precision $\uparrow$			FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
Strategy	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{F1}$
GT	$0.600^{\scriptscriptstyle\pm 0.004}$	$0.834^{\scriptscriptstyle\pm 0.001}$	$0.909^{\scriptscriptstyle\pm 0.001}$	$0.000^{\scriptscriptstyle\pm 0.000}$	$1.475^{\scriptscriptstyle\pm 0.003}$	$7.781^{\scriptscriptstyle\pm 0.140}$	$0.083^{\scriptscriptstyle\pm 0.204}$	$0.076^{\scriptscriptstyle\pm 0.000}$	$0.208^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$
Independent	$0.383^{\scriptscriptstyle\pm 0.005}$	$0.584^{\scriptscriptstyle\pm 0.007}$	$0.710^{\scriptscriptstyle\pm 0.005}$	$0.261^{\scriptscriptstyle\pm 0.017}$	$2.447^{\scriptscriptstyle\pm 0.008}$	$8.003^{\scriptscriptstyle\pm 0.056}$	$\underline{0.052}^{\scriptscriptstyle\pm 0.109}$	$\mathbf{0.103}^{\scriptscriptstyle\pm 0.001}$	$0.146^{\scriptscriptstyle\pm 0.002}$	$0.756^{\scriptscriptstyle\pm 0.003}$	$0.579^{\scriptscriptstyle\pm 0.003}$	$0.606^{\scriptscriptstyle\pm 0.003}$
Unified	$\underline{0.400}^{\scriptscriptstyle\pm 0.004}$	$\underline{0.609}^{\scriptscriptstyle\pm 0.006}$	$\underline{0.737}^{\scriptscriptstyle\pm 0.004}$	$\underline{0.213}^{\scriptscriptstyle\pm 0.016}$	$\underline{2.306}^{\scriptscriptstyle\pm 0.007}$	$7.896^{\scriptscriptstyle\pm 0.054}$	$0.059^{\scriptscriptstyle\pm 0.133}$	$0.130^{\scriptscriptstyle\pm 0.001}$	$\underline{0.186}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.818}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.764}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.760}^{\scriptscriptstyle\pm 0.002}$
Coupled	$\mathbf{0.404}^{\scriptscriptstyle\pm 0.003}$	$\mathbf{0.615}^{\scriptscriptstyle\pm 0.007}$	$\mathbf{0.744}^{\scriptscriptstyle\pm 0.005}$	$\mathbf{0.199}^{\scriptscriptstyle\pm 0.015}$	$\mathbf{2.256}^{\scriptscriptstyle\pm 0.006}$	$7.927^{\scriptscriptstyle\pm 0.051}$	$\mathbf{0.047}^{\scriptscriptstyle\pm 0.146}$	$\underline{0.117}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.196}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.830}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.810}^{\scriptscriptstyle\pm 0.002}$	$\mathbf{0.799}^{\scriptscriptstyle\pm 0.004}$

Table 3: Ablation study of stream modeling on InterAct. We report R-Precision with a batch size of 256.

Space	R-Precision $\uparrow$			FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
Space	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{F1}$
GT	$0.600^{\scriptscriptstyle\pm 0.004}$	$0.834^{\scriptscriptstyle\pm 0.001}$	$0.909^{\scriptscriptstyle\pm 0.001}$	$0.000^{\scriptscriptstyle\pm 0.000}$	$1.475^{\scriptscriptstyle\pm 0.003}$	$7.781^{\scriptscriptstyle\pm 0.140}$	$0.083^{\scriptscriptstyle\pm 0.204}$	$0.076^{\scriptscriptstyle\pm 0.000}$	$0.208^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$
Raw	$\underline{0.399}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.607}^{\scriptscriptstyle\pm 0.006}$	$\underline{0.735}^{\scriptscriptstyle\pm 0.004}$	$\underline{0.228}^{\scriptscriptstyle\pm 0.014}$	$\underline{2.314}^{\scriptscriptstyle\pm 0.007}$	$7.918^{\scriptscriptstyle\pm 0.052}$	$\underline{0.055}^{\scriptscriptstyle\pm 0.132}$	$\underline{0.135}^{\scriptscriptstyle\pm 0.001}$	$\underline{0.182}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.809}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.750}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.748}^{\scriptscriptstyle\pm 0.003}$
VQ	$0.384^{\scriptscriptstyle\pm 0.004}$	$0.586^{\scriptscriptstyle\pm 0.006}$	$0.711^{\scriptscriptstyle\pm 0.004}$	$0.269^{\scriptscriptstyle\pm 0.016}$	$2.406^{\scriptscriptstyle\pm 0.007}$	$7.884^{\scriptscriptstyle\pm 0.053}$	$0.063^{\scriptscriptstyle\pm 0.001}$	$0.146^{\scriptscriptstyle\pm 0.184}$	$0.158^{\scriptscriptstyle\pm 0.002}$	$0.771^{\scriptscriptstyle\pm 0.002}$	$0.681^{\scriptscriptstyle\pm 0.003}$	$0.685^{\scriptscriptstyle\pm 0.003}$
VAE	$\mathbf{0.404}^{\scriptscriptstyle\pm 0.003}$	$\mathbf{0.615}^{\scriptscriptstyle\pm 0.007}$	$\mathbf{0.744}^{\scriptscriptstyle\pm 0.005}$	$\mathbf{0.199}^{\scriptscriptstyle\pm 0.015}$	$\mathbf{2.256}^{\scriptscriptstyle\pm 0.006}$	$7.927^{\scriptscriptstyle\pm 0.051}$	$\mathbf{0.047}^{\scriptscriptstyle\pm 0.146}$	$\mathbf{0.117}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.196}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.830}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.810}^{\scriptscriptstyle\pm 0.002}$	$\mathbf{0.799}^{\scriptscriptstyle\pm 0.004}$

Table 4: Ablation study of latent representation on InterAct. We report R-Precision with a batch size of 256.

$\mathcal{L}_{\mathrm{c}}$	$\mathcal{L}_{\mathrm{A/E}}$	R@3 $\uparrow$	FID $\downarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
$\mathcal{L}_{\mathrm{c}}$	$\mathcal{L}_{\mathrm{A/E}}$	R@3 $\uparrow$	FID $\downarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{F1}$
$\times$	$\times$	$0.736^{\scriptscriptstyle\pm 0.004}$	$0.220^{\scriptscriptstyle\pm 0.016}$	$0.053^{\scriptscriptstyle\pm 0.138}$	$0.137^{\scriptscriptstyle\pm 0.002}$	$0.173^{\scriptscriptstyle\pm 0.002}$	$0.796^{\scriptscriptstyle\pm 0.003}$	$0.708^{\scriptscriptstyle\pm 0.004}$	$0.715^{\scriptscriptstyle\pm 0.004}$
$\checkmark$	$\times$	$\underline{0.740}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.208}^{\scriptscriptstyle\pm 0.015}$	$\underline{0.050}^{\scriptscriptstyle\pm 0.104}$	$\underline{0.126}^{\scriptscriptstyle\pm 0.001}$	$\underline{0.187}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.814}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.770}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.765}^{\scriptscriptstyle\pm 0.003}$
$\checkmark$	$\checkmark$	$\mathbf{0.744}^{\scriptscriptstyle\pm 0.005}$	$\mathbf{0.199}^{\scriptscriptstyle\pm 0.015}$	$\mathbf{0.047}^{\scriptscriptstyle\pm 0.146}$	$\mathbf{0.117}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.196}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.830}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.810}^{\scriptscriptstyle\pm 0.002}$	$\mathbf{0.799}^{\scriptscriptstyle\pm 0.004}$

Table 5: Ablation study of geometric constraints on InterAct. We report R-Precision with a batch size of 256.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	R-Precision $\uparrow$			FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
Method	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{\mathrm{F1}}$
Ground Truth	$0.318^{\scriptscriptstyle\pm 0.003}$	$0.560^{\scriptscriptstyle\pm 0.001}$	$0.731^{\scriptscriptstyle\pm 0.001}$	$0.000^{\scriptscriptstyle\pm 0.000}$	$1.499^{\scriptscriptstyle\pm 0.000}$	$7.400^{\scriptscriptstyle\pm 0.047}$	$0.012^{\scriptscriptstyle\pm 0.039}$	$0.067^{\scriptscriptstyle\pm 0.000}$	$0.262^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$
HOI-Diff	$0.221^{\scriptscriptstyle\pm 0.005}$	$0.403^{\scriptscriptstyle\pm 0.001}$	$0.542^{\scriptscriptstyle\pm 0.001}$	$0.987^{\scriptscriptstyle\pm 0.023}$	$2.520^{\scriptscriptstyle\pm 0.022}$	$7.248^{\scriptscriptstyle\pm 0.066}$	$0.021^{\scriptscriptstyle\pm 0.058}$	$0.131^{\scriptscriptstyle\pm 0.005}$	$0.135^{\scriptscriptstyle\pm 0.002}$	$0.862^{\scriptscriptstyle\pm 0.000}$	$0.696^{\scriptscriptstyle\pm 0.000}$	$0.739^{\scriptscriptstyle\pm 0.001}$
CHOIS	$0.291^{\scriptscriptstyle\pm 0.000}$	$0.515^{\scriptscriptstyle\pm 0.009}$	$0.676^{\scriptscriptstyle\pm 0.008}$	$0.132^{\scriptscriptstyle\pm 0.016}$	$1.746^{\scriptscriptstyle\pm 0.002}$	$7.417^{\scriptscriptstyle\pm 0.104}$	$0.019^{\scriptscriptstyle\pm 0.053}$	$0.096^{\scriptscriptstyle\pm 0.004}$	$\underline{0.230}^{\scriptscriptstyle\pm 0.001}$	$\underline{0.937}^{\scriptscriptstyle\pm 0.000}$	$\underline{0.900}^{\scriptscriptstyle\pm 0.004}$	$\mathbf{0.910}^{\scriptscriptstyle\pm 0.002}$
InterDiff	$0.312^{\scriptscriptstyle\pm 0.003}$	$0.548^{\scriptscriptstyle\pm 0.004}$	$0.718^{\scriptscriptstyle\pm 0.004}$	$0.163^{\scriptscriptstyle\pm 0.009}$	$1.587^{\scriptscriptstyle\pm 0.006}$	$7.387^{\scriptscriptstyle\pm 0.045}$	$0.025^{\scriptscriptstyle\pm 0.062}$	$0.100^{\scriptscriptstyle\pm 0.002}$	$0.206^{\scriptscriptstyle\pm 0.000}$	$\mathbf{0.940}^{\scriptscriptstyle\pm 0.016}$	$0.863^{\scriptscriptstyle\pm 0.016}$	$0.890^{\scriptscriptstyle\pm 0.017}$
Text2HOI	$0.286^{\scriptscriptstyle\pm 0.001}$	$0.513^{\scriptscriptstyle\pm 0.001}$	$0.655^{\scriptscriptstyle\pm 0.001}$	$0.168^{\scriptscriptstyle\pm 0.000}$	$1.832^{\scriptscriptstyle\pm 0.006}$	$7.359^{\scriptscriptstyle\pm 0.034}$	$0.020^{\scriptscriptstyle\pm 0.056}$	$0.127^{\scriptscriptstyle\pm 0.002}$	$0.163^{\scriptscriptstyle\pm 0.000}$	$0.919^{\scriptscriptstyle\pm 0.001}$	$0.757^{\scriptscriptstyle\pm 0.002}$	$0.807^{\scriptscriptstyle\pm 0.002}$

Appendix

Table A: Quantitative comparisons on the OMOMO dataset between our method and baseline approaches. We report R-Precision with a batch size of 256.

Method	R-Precision $\uparrow$			FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
Method	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{\mathrm{F1}}$
Ground Truth	$0.559^{\scriptscriptstyle\pm 0.005}$	$0.910^{\scriptscriptstyle\pm 0.000}$	$0.926^{\scriptscriptstyle\pm 0.000}$	$0.000^{\scriptscriptstyle\pm 0.000}$	$1.483^{\scriptscriptstyle\pm 0.018}$	$6.940^{\scriptscriptstyle\pm 0.153}$	$0.181^{\scriptscriptstyle\pm 0.224}$	$0.094^{\scriptscriptstyle\pm 0.001}$	$0.204^{\scriptscriptstyle\pm 0.001}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$
HOI-Diff	$0.167^{\scriptscriptstyle\pm 0.009}$	$0.298^{\scriptscriptstyle\pm 0.012}$	$0.358^{\scriptscriptstyle\pm 0.018}$	$1.463^{\scriptscriptstyle\pm 0.044}$	$4.152^{\scriptscriptstyle\pm 0.038}$	$6.723^{\scriptscriptstyle\pm 0.074}$	$0.179^{\scriptscriptstyle\pm 0.238}$	$0.171^{\scriptscriptstyle\pm 0.007}$	$0.100^{\scriptscriptstyle\pm 0.003}$	$0.646^{\scriptscriptstyle\pm 0.008}$	$0.515^{\scriptscriptstyle\pm 0.010}$	$0.519^{\scriptscriptstyle\pm 0.007}$
CHOIS	$0.219^{\scriptscriptstyle\pm 0.000}$	$0.396^{\scriptscriptstyle\pm 0.003}$	$0.492^{\scriptscriptstyle\pm 0.022}$	$0.705^{\scriptscriptstyle\pm 0.027}$	$3.423^{\scriptscriptstyle\pm 0.045}$	$6.863^{\scriptscriptstyle\pm 0.061}$	$0.191^{\scriptscriptstyle\pm 0.253}$	$0.128^{\scriptscriptstyle\pm 0.011}$	$\underline{0.170}^{\scriptscriptstyle\pm 0.003}$	$0.717^{\scriptscriptstyle\pm 0.018}$	$0.662^{\scriptscriptstyle\pm 0.000}$	$0.650^{\scriptscriptstyle\pm 0.005}$
InterDiff	$0.258^{\scriptscriptstyle\pm 0.027}$	$0.443^{\scriptscriptstyle\pm 0.019}$	$0.531^{\scriptscriptstyle\pm 0.027}$	$0.630^{\scriptscriptstyle\pm 0.033}$	$3.219^{\scriptscriptstyle\pm 0.040}$	$6.862^{\scriptscriptstyle\pm 0.066}$	$0.208^{\scriptscriptstyle\pm 0.264}$	$0.143^{\scriptscriptstyle\pm 0.009}$	$\mathbf{0.173}^{\scriptscriptstyle\pm 0.005}$	$0.723^{\scriptscriptstyle\pm 0.001}$	$\underline{0.682}^{\scriptscriptstyle\pm 0.013}$	$0.676^{\scriptscriptstyle\pm 0.004}$
Text2HOI	$0.135^{\scriptscriptstyle\pm 0.003}$	$0.242^{\scriptscriptstyle\pm 0.000}$	$0.322^{\scriptscriptstyle\pm 0.008}$	$1.291^{\scriptscriptstyle\pm 0.073}$	$3.815^{\scriptscriptstyle\pm 0.026}$	$6.544^{\scriptscriptstyle\pm 0.013}$	$0.177^{\scriptscriptstyle\pm 0.233}$	$0.179^{\scriptscriptstyle\pm 0.003}$	$0.128^{\scriptscriptstyle\pm 0.001}$	$0.703^{\scriptscriptstyle\pm 0.011}$	$0.579^{\scriptscriptstyle\pm 0.007}$	$0.587^{\scriptscriptstyle\pm 0.008}$

Appendix

Table B: Quantitative comparisons on the BEHAVE dataset between our method and baseline approaches. We report R-Precision with a batch size of 256.

Methods	R-Precision $\uparrow$			MM Dist. $\downarrow$	Bleu@1 $\uparrow$	Bleu@4 $\uparrow$	Rouge $\uparrow$	Cider $\uparrow$	BertScore $\uparrow$
Methods	Top 1	Top 2	Top 3	MM Dist. $\downarrow$	Bleu@1 $\uparrow$	Bleu@4 $\uparrow$	Rouge $\uparrow$	Cider $\uparrow$	BertScore $\uparrow$
Real	0.345	0.609	0.778	1.499	–	–	–	–	–
TM2T	0.231	0.421	0.598	2.173	80.62	74.18	90.87	7.46	91.72
MotionGPT	0.254	0.459	0.632	2.046	84.51	79.36	91.92	8.16	92.31
LaMPM2T	0.276	0.490	0.661	1.928	88.74	85.61	92.96	9.37	93.21
HOIGPT	0.291	0.522	0.688	1.824	91.56	85.18	93.58	9.08	93.14

Appendix

Table C: Comparison of HOI understanding on OMOMO. Our model is trained for 100 epochs.

Methods	R-Precision $\uparrow$			MM Dist. $\downarrow$	Bleu@1 $\uparrow$	Bleu@4 $\uparrow$	Rouge $\uparrow$	Cider $\uparrow$	BertScore $\uparrow$
Methods	Top 1	Top 2	Top 3	MM Dist. $\downarrow$	Bleu@1 $\uparrow$	Bleu@4 $\uparrow$	Rouge $\uparrow$	Cider $\uparrow$	BertScore $\uparrow$
Real	0.913	0.944	0.956	1.498	–	–	–	–	–
TM2T	0.246	0.383	0.501	4.143	23.18	9.62	37.16	4.9	22.18
MotionGPT	0.276	0.421	0.544	3.846	26.43	11.57	40.13	5.9	24.16
LaMPM2T	0.306	0.459	0.586	3.596	29.14	13.92	43.09	6.7	26.12
HOIGPT	0.336	0.497	0.632	3.301	32.64	16.52	46.57	8.32	30.61

Appendix

Table D: Comparison of HOI understanding on BEHAVE. Our model is trained for 100 epochs.

Dataset	Setting	Target	Human reconstruction $\downarrow$			Object reconstruction $\downarrow$		Interaction $\uparrow$
Dataset	Setting	Target	B-MPJPE	Root	H-JPE	Center	V2V	$C_{\rm prec}$	$C_{\rm rec}$	$C_{\rm F1}$
InterAct	Text/BPS only	Object	–	–	–	69.90	74.79	0.420	0.172	0.195
		Hands	–	–	726.46	–	–	0.390	0.148	0.164
		Body	170.74	574.55	–	–	–	–	–	–
	Streams only	Object	–	–	–	19.25	34.77	0.503	0.309	0.330
		Hands	–	–	44.91	–	–	0.669	0.613	0.610

Appendix

Table E: Stream completion on InterAct, OMOMO, and BEHAVE. For each dataset, one jointly trained checkpoint is evaluated under three conditioning settings.

Figure A: Qualitative HOI understanding. Adding hand and object evidence produces more complete interaction descriptions. $T$ denotes the reference description.

Strategy	R-Precision $\uparrow$			FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
Strategy	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{F1}$
GT	$0.318^{\scriptscriptstyle\pm 0.003}$	$0.560^{\scriptscriptstyle\pm 0.001}$	$0.731^{\scriptscriptstyle\pm 0.001}$	$0.000^{\scriptscriptstyle\pm 0.000}$	$1.499^{\scriptscriptstyle\pm 0.000}$	$7.400^{\scriptscriptstyle\pm 0.047}$	$0.012^{\scriptscriptstyle\pm 0.039}$	$0.067^{\scriptscriptstyle\pm 0.000}$	$0.262^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$
Independent	$0.325^{\scriptscriptstyle\pm 0.004}$	$0.570^{\scriptscriptstyle\pm 0.007}$	$0.731^{\scriptscriptstyle\pm 0.004}$	$0.120^{\scriptscriptstyle\pm 0.011}$	$1.701^{\scriptscriptstyle\pm 0.020}$	$7.467^{\scriptscriptstyle\pm 0.124}$	$\mathbf{0.015}^{\scriptscriptstyle\pm 0.078}$	$\mathbf{0.077}^{\scriptscriptstyle\pm 0.006}$	$0.174^{\scriptscriptstyle\pm 0.002}$	$0.849^{\scriptscriptstyle\pm 0.011}$	$0.651^{\scriptscriptstyle\pm 0.012}$	$0.696^{\scriptscriptstyle\pm 0.012}$
Unified	$\underline{0.339}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.597}^{\scriptscriptstyle\pm 0.006}$	$\underline{0.759}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.093}^{\scriptscriptstyle\pm 0.010}$	$\underline{1.590}^{\scriptscriptstyle\pm 0.018}$	$\mathbf{7.411}^{\scriptscriptstyle\pm 0.118}$	$0.022^{\scriptscriptstyle\pm 0.032}$	$0.097^{\scriptscriptstyle\pm 0.006}$	$\underline{0.218}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.906}^{\scriptscriptstyle\pm 0.010}$	$\underline{0.842}^{\scriptscriptstyle\pm 0.011}$	$\underline{0.855}^{\scriptscriptstyle\pm 0.011}$
Coupled	$\mathbf{0.342}^{\scriptscriptstyle\pm 0.003}$	$\mathbf{0.601}^{\scriptscriptstyle\pm 0.006}$	$\mathbf{0.764}^{\scriptscriptstyle\pm 0.002}$	$\mathbf{0.087}^{\scriptscriptstyle\pm 0.009}$	$\mathbf{1.557}^{\scriptscriptstyle\pm 0.017}$	$\underline{7.422}^{\scriptscriptstyle\pm 0.122}$	$\underline{0.018}^{\scriptscriptstyle\pm 0.054}$	$\underline{0.089}^{\scriptscriptstyle\pm 0.005}$	$\mathbf{0.227}^{\scriptscriptstyle\pm 0.002}$	$\mathbf{0.917}^{\scriptscriptstyle\pm 0.009}$	$\mathbf{0.889}^{\scriptscriptstyle\pm 0.009}$	$\mathbf{0.894}^{\scriptscriptstyle\pm 0.010}$

Appendix

Table F: Ablation study of stream modeling on OMOMO. We report R-Precision with a batch size of 256.

Strategy	R-Precision $\uparrow$			FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
Strategy	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{F1}$
GT	$0.559^{\scriptscriptstyle\pm 0.005}$	$0.910^{\scriptscriptstyle\pm 0.000}$	$0.926^{\scriptscriptstyle\pm 0.000}$	$0.000^{\scriptscriptstyle\pm 0.000}$	$1.483^{\scriptscriptstyle\pm 0.018}$	$6.940^{\scriptscriptstyle\pm 0.153}$	$0.181^{\scriptscriptstyle\pm 0.224}$	$0.094^{\scriptscriptstyle\pm 0.001}$	$0.204^{\scriptscriptstyle\pm 0.001}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$
Independent	$0.414^{\scriptscriptstyle\pm 0.011}$	$0.550^{\scriptscriptstyle\pm 0.007}$	$0.643^{\scriptscriptstyle\pm 0.011}$	$0.842^{\scriptscriptstyle\pm 0.014}$	$3.392^{\scriptscriptstyle\pm 0.048}$	$7.056^{\scriptscriptstyle\pm 0.051}$	$0.194^{\scriptscriptstyle\pm 0.122}$	$\mathbf{0.142}^{\scriptscriptstyle\pm 0.007}$	$0.104^{\scriptscriptstyle\pm 0.003}$	$0.676^{\scriptscriptstyle\pm 0.004}$	$0.451^{\scriptscriptstyle\pm 0.005}$	$0.486^{\scriptscriptstyle\pm 0.005}$
Unified	$\underline{0.435}^{\scriptscriptstyle\pm 0.010}$	$\underline{0.580}^{\scriptscriptstyle\pm 0.006}$	$\underline{0.678}^{\scriptscriptstyle\pm 0.010}$	$\underline{0.716}^{\scriptscriptstyle\pm 0.012}$	$\underline{3.188}^{\scriptscriptstyle\pm 0.046}$	$\mathbf{6.956}^{\scriptscriptstyle\pm 0.030}$	$\underline{0.177}^{\scriptscriptstyle\pm 0.194}$	$0.174^{\scriptscriptstyle\pm 0.007}$	$\underline{0.136}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.730}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.622}^{\scriptscriptstyle\pm 0.004}$	$\underline{0.631}^{\scriptscriptstyle\pm 0.004}$
Coupled	$\mathbf{0.438}^{\scriptscriptstyle\pm 0.009}$	$\mathbf{0.584}^{\scriptscriptstyle\pm 0.004}$	$\mathbf{0.681}^{\scriptscriptstyle\pm 0.009}$	$\mathbf{0.688}^{\scriptscriptstyle\pm 0.017}$	$\mathbf{3.145}^{\scriptscriptstyle\pm 0.040}$	$\underline{6.978}^{\scriptscriptstyle\pm 0.047}$	$\mathbf{0.169}^{\scriptscriptstyle\pm 0.139}$	$\underline{0.163}^{\scriptscriptstyle\pm 0.006}$	$\mathbf{0.141}^{\scriptscriptstyle\pm 0.003}$	$\mathbf{0.737}^{\scriptscriptstyle\pm 0.002}$	$\mathbf{0.650}^{\scriptscriptstyle\pm 0.009}$	$\mathbf{0.653}^{\scriptscriptstyle\pm 0.003}$

Appendix

Table G: Ablation study of stream modeling on BEHAVE. We report R-Precision with a batch size of 256.

Method	Time / batch (s) $\downarrow$	Time / sequence (s) $\downarrow$	Relative latency ( $\times$ ) $\downarrow$
HOI-Diff	$35.20^{\pm 0.12}$	$0.550$	$10.2$
InterDiff	$42.88^{\pm 0.35}$	$0.670$	$12.5$
LIGHT	$192.76^{\pm 0.57}$	$3.012$	$56.0$
TRACE	$3.44^{\pm 0.01}$	$0.054$	$1.0$

Appendix

Table H: Inference efficiency on InterAct. All methods generate a batch of 64 interactions, each containing 300 frames, on one NVIDIA A800 GPU using BF16.

Explore similar work

Date pendingcs.CV

Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interaction

Modeling 4D human-object interaction (HOI) is a compelling challenge in computer vision and an essential technology powering virtual and mixed-reality applications. While existing works have achieved promising results on specific HOI tasks-such as text-conditioned HOI generation and human motion generation from object motion, they typically rely on task-specific architectures and lack a unified framework capable of handling diverse conditional inputs. Building on this, we propose Uni-HOI, a unified framework that learns the joint distribution among text, human motion, and object motion. By leveraging large language models (LLMs) and two motion-specific vector quantized variational autoencoders (VQ-VAEs), we convert heterogeneous motion data into token sequences compatible with LLM inputs, enabling seamless integration and joint modeling of all three modalities. We introduce a two-stage training strategy: the first stage performs multi-task learning on a large-scale HOI dataset to capture the underlying correlations among the three modalities, while the second stage fine-tunes the model on specific tasks to further enhance performance. Extensive experiments demonstrate that Uni-HOI achieves remarkable performances on multiple HOI-related tasks including text-driven HOI generation, object motion-driven human motion generation (optionally with text) and human motion-driven object motion prediction within a unified framework.

Mengfei Zhang, Jinlu Zhang, Zhigang Tu

Wuhan University Wuhan, China · Peking University Beijing, China

Articulated Objects Human Motion Prediction Unified Framework Multi--Task Learning Text Analysis and Detection

Ask PDF

Jul 2, 2026cs.CV

JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation

Text driven hand object interaction (HOI) generation is gaining attention for immersive applications and robotics, yet producing physically plausible interactions remains challenging. Even when individual motions appear natural, small contact errors can cause conspicuous artifacts such as floating and interpenetration. Prior methods mitigate these issues using explicit contact cues or implicit grasp priors, but typically rely on multi stage pipelines and fail to model temporally evolving contact. We present JointHOI, a single stage diffusion framework that jointly generates 3D hand object motion and dynamic, distance based contact maps from text. By treating contact as an auxiliary inner modality, joint generation enables the model to learn contact motion coupling during training. At inference, contact guided sampling enforces consistency between generated contact maps and motion implied geometry, improving temporal stability and reducing penetration and floating. Experiments on GRAB and ARCTIC demonstrate consistent improvements in text adherence and physical plausibility over prior methods.

Mingyeong Song, Jungbin Cho, Jisoo Kim +5

Ewha Womans University, Seoul, Korea · Yonsei University, Seoul, Korea · Carnegie Mellon University, Pittsburgh PA, USA +1

Hand-Object Interaction Detection Human Motion Generation Contact Diffusion Models

Ask PDF

Sep 29, 2026cs.CV

PAMI: Part Anchored Motion for Text to Human-Object Interaction Generation

Text-conditioned full-body human-object interaction (HOI) generation requires synthesizing human motion and object trajectories that match the input text while remaining precisely coordinated over time. Most methods represent the human and object as separate trajectories and predict the global human-object couplings. Learning this complex, dynamically changing relationship implicitly, however, often yields object drift, missed contact, and penetration. We introduce PAMI, a Part-Anchored Motion framework for Interaction generation. Inspired by the classic Hough Transform, our key idea is to localize object motion by letting body-part anchors vote for it: we express object motion relative to multiple body-part anchors and use PamiVAE to learn an interaction latent space, decoding frame-wise weights that aggregate these part-specific votes. Building on this representation, PAMI generates interactions in a coarse-to-fine hierarchy. PamiGen first generates a coarse human-object interaction from text in this structured latent space, and PamiRefiner then recursively resolves fine-grained contact geometry using a hybrid surface-sensing representation, combining long-range probes that capture overall body-part influence with short-range sensors that resolve detailed contacts near the object surface. Experiments on InterAct show that PAMI generates more faithful interactions and more accurate human-relative object motion than previous methods, achieving 14.5% higher contact recall than the previous state of the art. Extensive ablations validate the contributions of both the part-anchored voting representation and hybrid surface-sensing refinement.

Chuqiao Li, Xianghui Xie, Yong Cao +2

Tübingen AI Center, University of Tübingen, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany

Interaction Data Text Analysis and Detection Latent Space

Ask PDF