Harnessing Coupled Stream Completion For Human-Object Interaction Modeling

Dawei Guan; Di Yang; Jiangtao Wang

cs.CVSep 26, 2026

Harnessing Coupled Stream Completion For Human-Object Interaction Modeling

Authors: Dawei Guan, Di Yang, Jiangtao Wang

Organizations: School of Artificial Intelligence and Data Science, University of Science and Technology of China

Abstract

Text-conditioned human-object interaction (HOI) generation requires body motion, object trajectories & rotations, and hand articulation to remain coordinated. These components differ in scale and dynamics, but must agree on contact, relative pose, and timing. A shared representation may limit the distinct structure of each stream, while independent generation prevents each stream from responding to changes in the others. Latent supervision alone also does not directly constrain contact after decoding. We propose TRACE, a continuous latent framework that keeps stream states separate and couples their updates. TRACE encodes body, object, and hand motion into separate latents and predicts each stream velocity from the complete current interaction state. Geometric losses on decoded motion further constrain contact and object-relative motion over time. The same model supports completion of any single absent stream from the other two. Frozen flow features also serve as input to a language model for HOI understanding. Experiments on InterAct, OMOMO, and BEHAVE show that joint completion training improves generation and that frozen flow features improve understanding over raw-motion encoding. On InterAct, TRACE achieves the highest contact precision, recall, and F1 among the compared methods.

Figures & tables

Figure 2: Overview of TRACE. Left: HOI Generation. We form different modalities, e.g., body, hand, and object, each compressed into latent vectors by its own encoder. After adding stream and temporal embeddings, a coupled flow transformer predicts their velocities, which are integrated by a flow sampler. Then, different modalities’ decoders reconstruct them into clean motion. Right: HOI Extensions. Leveraging its HOI representation, we enable HOI understanding via language model fine-tuning. Joint training with three completion paths, each targeting one stream, adds completion capability while improving HOI Generation.

Figure 3: Qualitative comparison with baselines. Our method yields more realistic human-object interactions, fewer contact/penetration artifacts, more accurate finger positioning, and better text-motion alignment.

Method	R-Precision $\uparrow$			FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
Method	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{\mathrm{F1}}$
Ground Truth	$0.600^{\scriptscriptstyle\pm 0.004}$	$0.834^{\scriptscriptstyle\pm 0.001}$	$0.909^{\scriptscriptstyle\pm 0.001}$	$0.000^{\scriptscriptstyle\pm 0.000}$	$1.475^{\scriptscriptstyle\pm 0.003}$	$7.781^{\scriptscriptstyle\pm 0.140}$	$0.083^{\scriptscriptstyle\pm 0.204}$	$0.076^{\scriptscriptstyle\pm 0.000}$	$0.208^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$
HOI-Diff	$0.413^{\scriptscriptstyle\pm 0.010}$	$0.624^{\scriptscriptstyle\pm 0.009}$	$0.740^{\scriptscriptstyle\pm 0.011}$	$0.689^{\scriptscriptstyle\pm 0.031}$	$3.029^{\scriptscriptstyle\pm 0.007}$	$7.620^{\scriptscriptstyle\pm 0.085}$	$0.072^{\scriptscriptstyle\pm 0.160}$	$\mathbf{0.103}^{\scriptscriptstyle\pm 0.010}$	$0.084^{\scriptscriptstyle\pm 0.006}$	$0.722^{\scriptscriptstyle\pm 0.002}$	$0.447^{\scriptscriptstyle\pm 0.026}$	$0.501^{\scriptscriptstyle\pm 0.022}$
CHOIS	$\underline{0.439}^{\scriptscriptstyle\pm 0.005}$	$\underline{0.660}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.766}^{\scriptscriptstyle\pm 0.003}$	$0.572^{\scriptscriptstyle\pm 0.008}$	$2.781^{\scriptscriptstyle\pm 0.017}$	$\underline{7.717}^{\scriptscriptstyle\pm 0.115}$	$0.119^{\scriptscriptstyle\pm 0.234}$	$0.131^{\scriptscriptstyle\pm 0.013}$	$0.115^{\scriptscriptstyle\pm 0.002}$	$0.710^{\scriptscriptstyle\pm 0.012}$	$0.520^{\scriptscriptstyle\pm 0.004}$	$0.541^{\scriptscriptstyle\pm 0.007}$
InterDiff	$\mathbf{0.501}^{\scriptscriptstyle\pm 0.009}$	$\mathbf{0.722}^{\scriptscriptstyle\pm 0.003}$	$\mathbf{0.824}^{\scriptscriptstyle\pm 0.007}$	$0.215^{\scriptscriptstyle\pm 0.001}$	$2.461^{\scriptscriptstyle\pm 0.013}$	$\underline{7.717}^{\scriptscriptstyle\pm 0.115}$	$0.092^{\scriptscriptstyle\pm 0.178}$	$0.116^{\scriptscriptstyle\pm 0.011}$	$0.124^{\scriptscriptstyle\pm 0.003}$	$0.715^{\scriptscriptstyle\pm 0.000}$	$0.562^{\scriptscriptstyle\pm 0.013}$	$0.584^{\scriptscriptstyle\pm 0.003}$
Text2HOI	$0.428^{\scriptscriptstyle\pm 0.003}$	$0.637^{\scriptscriptstyle\pm 0.000}$	$0.745^{\scriptscriptstyle\pm 0.000}$	$0.331^{\scriptscriptstyle\pm 0.003}$	$2.665^{\scriptscriptstyle\pm 0.002}$	$7.631^{\scriptscriptstyle\pm 0.000}$	$\underline{0.055}^{\scriptscriptstyle\pm 0.123}$	$\underline{0.105}^{\scriptscriptstyle\pm 0.003}$	$0.102^{\scriptscriptstyle\pm 0.001}$	$0.711^{\scriptscriptstyle\pm 0.001}$	$0.488^{\scriptscriptstyle\pm 0.000}$	$0.532^{\scriptscriptstyle\pm 0.002}$

Table 1: Quantitative comparisons on the InterAct dataset between our method and baseline approaches. We report R-Precision with a batch size of 256.

Figure 4: Qualitative results of stream completion. From left to right, we show the ground-truth interaction and the completion of object, hand, and body streams. Gray denotes observed motion, while pink, magenta, and teal denote the generated object, hands, and body, respectively.

Methods	R-Precision $\uparrow$			MM Dist. $\downarrow$	Bleu@1 $\uparrow$	Bleu@4 $\uparrow$	Rouge $\uparrow$	Cider $\uparrow$	BertScore $\uparrow$
Methods	Top 1	Top 2	Top 3	MM Dist. $\downarrow$	Bleu@1 $\uparrow$	Bleu@4 $\uparrow$	Rouge $\uparrow$	Cider $\uparrow$	BertScore $\uparrow$
Real	0.592	0.757	0.853	1.486	–	–	–	–	–
TM2T	0.206	0.351	0.466	3.342	35.18	24.12	56.17	2.61	52.16
MotionGPT	0.224	0.391	0.504	3.086	39.86	29.63	60.84	3.42	57.83
LaMPM2T	0.243	0.415	0.532	2.894	44.37	34.42	66.14	4.09	63.18
HOIGPT	0.256	0.452	0.570	2.756	50.31	39.22	71.08	4.89	69.05

Table 2: Comparison of HOI understanding on InterAct. Our model is trained for 100 epochs.

Figure 5: Qualitative comparison of stream modeling. We compare unified, independent, and coupled stream modeling. Red boxes highlight object drift and hand-object misalignment in the first two variants, while coupled modeling yields coherent interactions.

Strategy	R-Precision $\uparrow$			FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
Strategy	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{F1}$
GT	$0.600^{\scriptscriptstyle\pm 0.004}$	$0.834^{\scriptscriptstyle\pm 0.001}$	$0.909^{\scriptscriptstyle\pm 0.001}$	$0.000^{\scriptscriptstyle\pm 0.000}$	$1.475^{\scriptscriptstyle\pm 0.003}$	$7.781^{\scriptscriptstyle\pm 0.140}$	$0.083^{\scriptscriptstyle\pm 0.204}$	$0.076^{\scriptscriptstyle\pm 0.000}$	$0.208^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$
Independent	$0.383^{\scriptscriptstyle\pm 0.005}$	$0.584^{\scriptscriptstyle\pm 0.007}$	$0.710^{\scriptscriptstyle\pm 0.005}$	$0.261^{\scriptscriptstyle\pm 0.017}$	$2.447^{\scriptscriptstyle\pm 0.008}$	$8.003^{\scriptscriptstyle\pm 0.056}$	$\underline{0.052}^{\scriptscriptstyle\pm 0.109}$	$\mathbf{0.103}^{\scriptscriptstyle\pm 0.001}$	$0.146^{\scriptscriptstyle\pm 0.002}$	$0.756^{\scriptscriptstyle\pm 0.003}$	$0.579^{\scriptscriptstyle\pm 0.003}$	$0.606^{\scriptscriptstyle\pm 0.003}$
Unified	$\underline{0.400}^{\scriptscriptstyle\pm 0.004}$	$\underline{0.609}^{\scriptscriptstyle\pm 0.006}$	$\underline{0.737}^{\scriptscriptstyle\pm 0.004}$	$\underline{0.213}^{\scriptscriptstyle\pm 0.016}$	$\underline{2.306}^{\scriptscriptstyle\pm 0.007}$	$7.896^{\scriptscriptstyle\pm 0.054}$	$0.059^{\scriptscriptstyle\pm 0.133}$	$0.130^{\scriptscriptstyle\pm 0.001}$	$\underline{0.186}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.818}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.764}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.760}^{\scriptscriptstyle\pm 0.002}$
Coupled	$\mathbf{0.404}^{\scriptscriptstyle\pm 0.003}$	$\mathbf{0.615}^{\scriptscriptstyle\pm 0.007}$	$\mathbf{0.744}^{\scriptscriptstyle\pm 0.005}$	$\mathbf{0.199}^{\scriptscriptstyle\pm 0.015}$	$\mathbf{2.256}^{\scriptscriptstyle\pm 0.006}$	$7.927^{\scriptscriptstyle\pm 0.051}$	$\mathbf{0.047}^{\scriptscriptstyle\pm 0.146}$	$\underline{0.117}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.196}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.830}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.810}^{\scriptscriptstyle\pm 0.002}$	$\mathbf{0.799}^{\scriptscriptstyle\pm 0.004}$

Table 3: Ablation study of stream modeling on InterAct. We report R-Precision with a batch size of 256.

Space	R-Precision $\uparrow$			FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
Space	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{F1}$
GT	$0.600^{\scriptscriptstyle\pm 0.004}$	$0.834^{\scriptscriptstyle\pm 0.001}$	$0.909^{\scriptscriptstyle\pm 0.001}$	$0.000^{\scriptscriptstyle\pm 0.000}$	$1.475^{\scriptscriptstyle\pm 0.003}$	$7.781^{\scriptscriptstyle\pm 0.140}$	$0.083^{\scriptscriptstyle\pm 0.204}$	$0.076^{\scriptscriptstyle\pm 0.000}$	$0.208^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$
Raw	$\underline{0.399}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.607}^{\scriptscriptstyle\pm 0.006}$	$\underline{0.735}^{\scriptscriptstyle\pm 0.004}$	$\underline{0.228}^{\scriptscriptstyle\pm 0.014}$	$\underline{2.314}^{\scriptscriptstyle\pm 0.007}$	$7.918^{\scriptscriptstyle\pm 0.052}$	$\underline{0.055}^{\scriptscriptstyle\pm 0.132}$	$\underline{0.135}^{\scriptscriptstyle\pm 0.001}$	$\underline{0.182}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.809}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.750}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.748}^{\scriptscriptstyle\pm 0.003}$
VQ	$0.384^{\scriptscriptstyle\pm 0.004}$	$0.586^{\scriptscriptstyle\pm 0.006}$	$0.711^{\scriptscriptstyle\pm 0.004}$	$0.269^{\scriptscriptstyle\pm 0.016}$	$2.406^{\scriptscriptstyle\pm 0.007}$	$7.884^{\scriptscriptstyle\pm 0.053}$	$0.063^{\scriptscriptstyle\pm 0.001}$	$0.146^{\scriptscriptstyle\pm 0.184}$	$0.158^{\scriptscriptstyle\pm 0.002}$	$0.771^{\scriptscriptstyle\pm 0.002}$	$0.681^{\scriptscriptstyle\pm 0.003}$	$0.685^{\scriptscriptstyle\pm 0.003}$
VAE	$\mathbf{0.404}^{\scriptscriptstyle\pm 0.003}$	$\mathbf{0.615}^{\scriptscriptstyle\pm 0.007}$	$\mathbf{0.744}^{\scriptscriptstyle\pm 0.005}$	$\mathbf{0.199}^{\scriptscriptstyle\pm 0.015}$	$\mathbf{2.256}^{\scriptscriptstyle\pm 0.006}$	$7.927^{\scriptscriptstyle\pm 0.051}$	$\mathbf{0.047}^{\scriptscriptstyle\pm 0.146}$	$\mathbf{0.117}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.196}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.830}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.810}^{\scriptscriptstyle\pm 0.002}$	$\mathbf{0.799}^{\scriptscriptstyle\pm 0.004}$

Table 4: Ablation study of latent representation on InterAct. We report R-Precision with a batch size of 256.

$\mathcal{L}_{\mathrm{c}}$	$\mathcal{L}_{\mathrm{A/E}}$	R@3 $\uparrow$	FID $\downarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
$\mathcal{L}_{\mathrm{c}}$	$\mathcal{L}_{\mathrm{A/E}}$	R@3 $\uparrow$	FID $\downarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{F1}$
$\times$	$\times$	$0.736^{\scriptscriptstyle\pm 0.004}$	$0.220^{\scriptscriptstyle\pm 0.016}$	$0.053^{\scriptscriptstyle\pm 0.138}$	$0.137^{\scriptscriptstyle\pm 0.002}$	$0.173^{\scriptscriptstyle\pm 0.002}$	$0.796^{\scriptscriptstyle\pm 0.003}$	$0.708^{\scriptscriptstyle\pm 0.004}$	$0.715^{\scriptscriptstyle\pm 0.004}$
$\checkmark$	$\times$	$\underline{0.740}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.208}^{\scriptscriptstyle\pm 0.015}$	$\underline{0.050}^{\scriptscriptstyle\pm 0.104}$	$\underline{0.126}^{\scriptscriptstyle\pm 0.001}$	$\underline{0.187}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.814}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.770}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.765}^{\scriptscriptstyle\pm 0.003}$
$\checkmark$	$\checkmark$	$\mathbf{0.744}^{\scriptscriptstyle\pm 0.005}$	$\mathbf{0.199}^{\scriptscriptstyle\pm 0.015}$	$\mathbf{0.047}^{\scriptscriptstyle\pm 0.146}$	$\mathbf{0.117}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.196}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.830}^{\scriptscriptstyle\pm 0.001}$	$\mathbf{0.810}^{\scriptscriptstyle\pm 0.002}$	$\mathbf{0.799}^{\scriptscriptstyle\pm 0.004}$

Table 5: Ablation study of geometric constraints on InterAct. We report R-Precision with a batch size of 256.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	R-Precision $\uparrow$			FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
Method	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{\mathrm{F1}}$
Ground Truth	$0.318^{\scriptscriptstyle\pm 0.003}$	$0.560^{\scriptscriptstyle\pm 0.001}$	$0.731^{\scriptscriptstyle\pm 0.001}$	$0.000^{\scriptscriptstyle\pm 0.000}$	$1.499^{\scriptscriptstyle\pm 0.000}$	$7.400^{\scriptscriptstyle\pm 0.047}$	$0.012^{\scriptscriptstyle\pm 0.039}$	$0.067^{\scriptscriptstyle\pm 0.000}$	$0.262^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$
HOI-Diff	$0.221^{\scriptscriptstyle\pm 0.005}$	$0.403^{\scriptscriptstyle\pm 0.001}$	$0.542^{\scriptscriptstyle\pm 0.001}$	$0.987^{\scriptscriptstyle\pm 0.023}$	$2.520^{\scriptscriptstyle\pm 0.022}$	$7.248^{\scriptscriptstyle\pm 0.066}$	$0.021^{\scriptscriptstyle\pm 0.058}$	$0.131^{\scriptscriptstyle\pm 0.005}$	$0.135^{\scriptscriptstyle\pm 0.002}$	$0.862^{\scriptscriptstyle\pm 0.000}$	$0.696^{\scriptscriptstyle\pm 0.000}$	$0.739^{\scriptscriptstyle\pm 0.001}$
CHOIS	$0.291^{\scriptscriptstyle\pm 0.000}$	$0.515^{\scriptscriptstyle\pm 0.009}$	$0.676^{\scriptscriptstyle\pm 0.008}$	$0.132^{\scriptscriptstyle\pm 0.016}$	$1.746^{\scriptscriptstyle\pm 0.002}$	$7.417^{\scriptscriptstyle\pm 0.104}$	$0.019^{\scriptscriptstyle\pm 0.053}$	$0.096^{\scriptscriptstyle\pm 0.004}$	$\underline{0.230}^{\scriptscriptstyle\pm 0.001}$	$\underline{0.937}^{\scriptscriptstyle\pm 0.000}$	$\underline{0.900}^{\scriptscriptstyle\pm 0.004}$	$\mathbf{0.910}^{\scriptscriptstyle\pm 0.002}$
InterDiff	$0.312^{\scriptscriptstyle\pm 0.003}$	$0.548^{\scriptscriptstyle\pm 0.004}$	$0.718^{\scriptscriptstyle\pm 0.004}$	$0.163^{\scriptscriptstyle\pm 0.009}$	$1.587^{\scriptscriptstyle\pm 0.006}$	$7.387^{\scriptscriptstyle\pm 0.045}$	$0.025^{\scriptscriptstyle\pm 0.062}$	$0.100^{\scriptscriptstyle\pm 0.002}$	$0.206^{\scriptscriptstyle\pm 0.000}$	$\mathbf{0.940}^{\scriptscriptstyle\pm 0.016}$	$0.863^{\scriptscriptstyle\pm 0.016}$	$0.890^{\scriptscriptstyle\pm 0.017}$
Text2HOI	$0.286^{\scriptscriptstyle\pm 0.001}$	$0.513^{\scriptscriptstyle\pm 0.001}$	$0.655^{\scriptscriptstyle\pm 0.001}$	$0.168^{\scriptscriptstyle\pm 0.000}$	$1.832^{\scriptscriptstyle\pm 0.006}$	$7.359^{\scriptscriptstyle\pm 0.034}$	$0.020^{\scriptscriptstyle\pm 0.056}$	$0.127^{\scriptscriptstyle\pm 0.002}$	$0.163^{\scriptscriptstyle\pm 0.000}$	$0.919^{\scriptscriptstyle\pm 0.001}$	$0.757^{\scriptscriptstyle\pm 0.002}$	$0.807^{\scriptscriptstyle\pm 0.002}$

Appendix

Table A: Quantitative comparisons on the OMOMO dataset between our method and baseline approaches. We report R-Precision with a batch size of 256.

Method	R-Precision $\uparrow$			FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
Method	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{\mathrm{F1}}$
Ground Truth	$0.559^{\scriptscriptstyle\pm 0.005}$	$0.910^{\scriptscriptstyle\pm 0.000}$	$0.926^{\scriptscriptstyle\pm 0.000}$	$0.000^{\scriptscriptstyle\pm 0.000}$	$1.483^{\scriptscriptstyle\pm 0.018}$	$6.940^{\scriptscriptstyle\pm 0.153}$	$0.181^{\scriptscriptstyle\pm 0.224}$	$0.094^{\scriptscriptstyle\pm 0.001}$	$0.204^{\scriptscriptstyle\pm 0.001}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$
HOI-Diff	$0.167^{\scriptscriptstyle\pm 0.009}$	$0.298^{\scriptscriptstyle\pm 0.012}$	$0.358^{\scriptscriptstyle\pm 0.018}$	$1.463^{\scriptscriptstyle\pm 0.044}$	$4.152^{\scriptscriptstyle\pm 0.038}$	$6.723^{\scriptscriptstyle\pm 0.074}$	$0.179^{\scriptscriptstyle\pm 0.238}$	$0.171^{\scriptscriptstyle\pm 0.007}$	$0.100^{\scriptscriptstyle\pm 0.003}$	$0.646^{\scriptscriptstyle\pm 0.008}$	$0.515^{\scriptscriptstyle\pm 0.010}$	$0.519^{\scriptscriptstyle\pm 0.007}$
CHOIS	$0.219^{\scriptscriptstyle\pm 0.000}$	$0.396^{\scriptscriptstyle\pm 0.003}$	$0.492^{\scriptscriptstyle\pm 0.022}$	$0.705^{\scriptscriptstyle\pm 0.027}$	$3.423^{\scriptscriptstyle\pm 0.045}$	$6.863^{\scriptscriptstyle\pm 0.061}$	$0.191^{\scriptscriptstyle\pm 0.253}$	$0.128^{\scriptscriptstyle\pm 0.011}$	$\underline{0.170}^{\scriptscriptstyle\pm 0.003}$	$0.717^{\scriptscriptstyle\pm 0.018}$	$0.662^{\scriptscriptstyle\pm 0.000}$	$0.650^{\scriptscriptstyle\pm 0.005}$
InterDiff	$0.258^{\scriptscriptstyle\pm 0.027}$	$0.443^{\scriptscriptstyle\pm 0.019}$	$0.531^{\scriptscriptstyle\pm 0.027}$	$0.630^{\scriptscriptstyle\pm 0.033}$	$3.219^{\scriptscriptstyle\pm 0.040}$	$6.862^{\scriptscriptstyle\pm 0.066}$	$0.208^{\scriptscriptstyle\pm 0.264}$	$0.143^{\scriptscriptstyle\pm 0.009}$	$\mathbf{0.173}^{\scriptscriptstyle\pm 0.005}$	$0.723^{\scriptscriptstyle\pm 0.001}$	$\underline{0.682}^{\scriptscriptstyle\pm 0.013}$	$0.676^{\scriptscriptstyle\pm 0.004}$
Text2HOI	$0.135^{\scriptscriptstyle\pm 0.003}$	$0.242^{\scriptscriptstyle\pm 0.000}$	$0.322^{\scriptscriptstyle\pm 0.008}$	$1.291^{\scriptscriptstyle\pm 0.073}$	$3.815^{\scriptscriptstyle\pm 0.026}$	$6.544^{\scriptscriptstyle\pm 0.013}$	$0.177^{\scriptscriptstyle\pm 0.233}$	$0.179^{\scriptscriptstyle\pm 0.003}$	$0.128^{\scriptscriptstyle\pm 0.001}$	$0.703^{\scriptscriptstyle\pm 0.011}$	$0.579^{\scriptscriptstyle\pm 0.007}$	$0.587^{\scriptscriptstyle\pm 0.008}$

Appendix

Table B: Quantitative comparisons on the BEHAVE dataset between our method and baseline approaches. We report R-Precision with a batch size of 256.

Methods	R-Precision $\uparrow$			MM Dist. $\downarrow$	Bleu@1 $\uparrow$	Bleu@4 $\uparrow$	Rouge $\uparrow$	Cider $\uparrow$	BertScore $\uparrow$
Methods	Top 1	Top 2	Top 3	MM Dist. $\downarrow$	Bleu@1 $\uparrow$	Bleu@4 $\uparrow$	Rouge $\uparrow$	Cider $\uparrow$	BertScore $\uparrow$
Real	0.345	0.609	0.778	1.499	–	–	–	–	–
TM2T	0.231	0.421	0.598	2.173	80.62	74.18	90.87	7.46	91.72
MotionGPT	0.254	0.459	0.632	2.046	84.51	79.36	91.92	8.16	92.31
LaMPM2T	0.276	0.490	0.661	1.928	88.74	85.61	92.96	9.37	93.21
HOIGPT	0.291	0.522	0.688	1.824	91.56	85.18	93.58	9.08	93.14

Appendix

Table C: Comparison of HOI understanding on OMOMO. Our model is trained for 100 epochs.

Methods	R-Precision $\uparrow$			MM Dist. $\downarrow$	Bleu@1 $\uparrow$	Bleu@4 $\uparrow$	Rouge $\uparrow$	Cider $\uparrow$	BertScore $\uparrow$
Methods	Top 1	Top 2	Top 3	MM Dist. $\downarrow$	Bleu@1 $\uparrow$	Bleu@4 $\uparrow$	Rouge $\uparrow$	Cider $\uparrow$	BertScore $\uparrow$
Real	0.913	0.944	0.956	1.498	–	–	–	–	–
TM2T	0.246	0.383	0.501	4.143	23.18	9.62	37.16	4.9	22.18
MotionGPT	0.276	0.421	0.544	3.846	26.43	11.57	40.13	5.9	24.16
LaMPM2T	0.306	0.459	0.586	3.596	29.14	13.92	43.09	6.7	26.12
HOIGPT	0.336	0.497	0.632	3.301	32.64	16.52	46.57	8.32	30.61

Appendix

Table D: Comparison of HOI understanding on BEHAVE. Our model is trained for 100 epochs.

Dataset	Setting	Target	Human reconstruction $\downarrow$			Object reconstruction $\downarrow$		Interaction $\uparrow$
Dataset	Setting	Target	B-MPJPE	Root	H-JPE	Center	V2V	$C_{\rm prec}$	$C_{\rm rec}$	$C_{\rm F1}$
InterAct	Text/BPS only	Object	–	–	–	69.90	74.79	0.420	0.172	0.195
		Hands	–	–	726.46	–	–	0.390	0.148	0.164
		Body	170.74	574.55	–	–	–	–	–	–
	Streams only	Object	–	–	–	19.25	34.77	0.503	0.309	0.330
		Hands	–	–	44.91	–	–	0.669	0.613	0.610

Appendix

Table E: Stream completion on InterAct, OMOMO, and BEHAVE. For each dataset, one jointly trained checkpoint is evaluated under three conditioning settings.

Figure A: Qualitative HOI understanding. Adding hand and object evidence produces more complete interaction descriptions. $T$ denotes the reference description.

Strategy	R-Precision $\uparrow$			FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
Strategy	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{F1}$
GT	$0.318^{\scriptscriptstyle\pm 0.003}$	$0.560^{\scriptscriptstyle\pm 0.001}$	$0.731^{\scriptscriptstyle\pm 0.001}$	$0.000^{\scriptscriptstyle\pm 0.000}$	$1.499^{\scriptscriptstyle\pm 0.000}$	$7.400^{\scriptscriptstyle\pm 0.047}$	$0.012^{\scriptscriptstyle\pm 0.039}$	$0.067^{\scriptscriptstyle\pm 0.000}$	$0.262^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$
Independent	$0.325^{\scriptscriptstyle\pm 0.004}$	$0.570^{\scriptscriptstyle\pm 0.007}$	$0.731^{\scriptscriptstyle\pm 0.004}$	$0.120^{\scriptscriptstyle\pm 0.011}$	$1.701^{\scriptscriptstyle\pm 0.020}$	$7.467^{\scriptscriptstyle\pm 0.124}$	$\mathbf{0.015}^{\scriptscriptstyle\pm 0.078}$	$\mathbf{0.077}^{\scriptscriptstyle\pm 0.006}$	$0.174^{\scriptscriptstyle\pm 0.002}$	$0.849^{\scriptscriptstyle\pm 0.011}$	$0.651^{\scriptscriptstyle\pm 0.012}$	$0.696^{\scriptscriptstyle\pm 0.012}$
Unified	$\underline{0.339}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.597}^{\scriptscriptstyle\pm 0.006}$	$\underline{0.759}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.093}^{\scriptscriptstyle\pm 0.010}$	$\underline{1.590}^{\scriptscriptstyle\pm 0.018}$	$\mathbf{7.411}^{\scriptscriptstyle\pm 0.118}$	$0.022^{\scriptscriptstyle\pm 0.032}$	$0.097^{\scriptscriptstyle\pm 0.006}$	$\underline{0.218}^{\scriptscriptstyle\pm 0.002}$	$\underline{0.906}^{\scriptscriptstyle\pm 0.010}$	$\underline{0.842}^{\scriptscriptstyle\pm 0.011}$	$\underline{0.855}^{\scriptscriptstyle\pm 0.011}$
Coupled	$\mathbf{0.342}^{\scriptscriptstyle\pm 0.003}$	$\mathbf{0.601}^{\scriptscriptstyle\pm 0.006}$	$\mathbf{0.764}^{\scriptscriptstyle\pm 0.002}$	$\mathbf{0.087}^{\scriptscriptstyle\pm 0.009}$	$\mathbf{1.557}^{\scriptscriptstyle\pm 0.017}$	$\underline{7.422}^{\scriptscriptstyle\pm 0.122}$	$\underline{0.018}^{\scriptscriptstyle\pm 0.054}$	$\underline{0.089}^{\scriptscriptstyle\pm 0.005}$	$\mathbf{0.227}^{\scriptscriptstyle\pm 0.002}$	$\mathbf{0.917}^{\scriptscriptstyle\pm 0.009}$	$\mathbf{0.889}^{\scriptscriptstyle\pm 0.009}$	$\mathbf{0.894}^{\scriptscriptstyle\pm 0.010}$

Appendix

Table F: Ablation study of stream modeling on OMOMO. We report R-Precision with a batch size of 256.

Strategy	R-Precision $\uparrow$			FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	Interaction $\uparrow$
Strategy	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist. $\downarrow$	Diversity $\rightarrow$	FSR $\downarrow$	Pene. $\downarrow$	Contact $\rightarrow$	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{F1}$
GT	$0.559^{\scriptscriptstyle\pm 0.005}$	$0.910^{\scriptscriptstyle\pm 0.000}$	$0.926^{\scriptscriptstyle\pm 0.000}$	$0.000^{\scriptscriptstyle\pm 0.000}$	$1.483^{\scriptscriptstyle\pm 0.018}$	$6.940^{\scriptscriptstyle\pm 0.153}$	$0.181^{\scriptscriptstyle\pm 0.224}$	$0.094^{\scriptscriptstyle\pm 0.001}$	$0.204^{\scriptscriptstyle\pm 0.001}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$	$1.000^{\scriptscriptstyle\pm 0.000}$
Independent	$0.414^{\scriptscriptstyle\pm 0.011}$	$0.550^{\scriptscriptstyle\pm 0.007}$	$0.643^{\scriptscriptstyle\pm 0.011}$	$0.842^{\scriptscriptstyle\pm 0.014}$	$3.392^{\scriptscriptstyle\pm 0.048}$	$7.056^{\scriptscriptstyle\pm 0.051}$	$0.194^{\scriptscriptstyle\pm 0.122}$	$\mathbf{0.142}^{\scriptscriptstyle\pm 0.007}$	$0.104^{\scriptscriptstyle\pm 0.003}$	$0.676^{\scriptscriptstyle\pm 0.004}$	$0.451^{\scriptscriptstyle\pm 0.005}$	$0.486^{\scriptscriptstyle\pm 0.005}$
Unified	$\underline{0.435}^{\scriptscriptstyle\pm 0.010}$	$\underline{0.580}^{\scriptscriptstyle\pm 0.006}$	$\underline{0.678}^{\scriptscriptstyle\pm 0.010}$	$\underline{0.716}^{\scriptscriptstyle\pm 0.012}$	$\underline{3.188}^{\scriptscriptstyle\pm 0.046}$	$\mathbf{6.956}^{\scriptscriptstyle\pm 0.030}$	$\underline{0.177}^{\scriptscriptstyle\pm 0.194}$	$0.174^{\scriptscriptstyle\pm 0.007}$	$\underline{0.136}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.730}^{\scriptscriptstyle\pm 0.003}$	$\underline{0.622}^{\scriptscriptstyle\pm 0.004}$	$\underline{0.631}^{\scriptscriptstyle\pm 0.004}$
Coupled	$\mathbf{0.438}^{\scriptscriptstyle\pm 0.009}$	$\mathbf{0.584}^{\scriptscriptstyle\pm 0.004}$	$\mathbf{0.681}^{\scriptscriptstyle\pm 0.009}$	$\mathbf{0.688}^{\scriptscriptstyle\pm 0.017}$	$\mathbf{3.145}^{\scriptscriptstyle\pm 0.040}$	$\underline{6.978}^{\scriptscriptstyle\pm 0.047}$	$\mathbf{0.169}^{\scriptscriptstyle\pm 0.139}$	$\underline{0.163}^{\scriptscriptstyle\pm 0.006}$	$\mathbf{0.141}^{\scriptscriptstyle\pm 0.003}$	$\mathbf{0.737}^{\scriptscriptstyle\pm 0.002}$	$\mathbf{0.650}^{\scriptscriptstyle\pm 0.009}$	$\mathbf{0.653}^{\scriptscriptstyle\pm 0.003}$

Appendix

Table G: Ablation study of stream modeling on BEHAVE. We report R-Precision with a batch size of 256.

Method	Time / batch (s) $\downarrow$	Time / sequence (s) $\downarrow$	Relative latency ( $\times$ ) $\downarrow$
HOI-Diff	$35.20^{\pm 0.12}$	$0.550$	$10.2$
InterDiff	$42.88^{\pm 0.35}$	$0.670$	$12.5$
LIGHT	$192.76^{\pm 0.57}$	$3.012$	$56.0$
TRACE	$3.44^{\pm 0.01}$	$0.054$	$1.0$

Appendix

Table H: Inference efficiency on InterAct. All methods generate a batch of 64 interactions, each containing 300 frames, on one NVIDIA A800 GPU using BF16.

Explore similar work

Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interaction
Date pendingMengfei Zhang, Jinlu Zhang, Zhigang TuArticulated Objects Human Motion Prediction
Wuhan University Wuhan, China · Peking University Beijing, China
JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation
Jul 2, 2026Mingyeong Song, Jungbin Cho, Jisoo Kim +5Hand-Object Interaction Detection Human Motion Generation
Ewha Womans University, Seoul, Korea · Yonsei University, Seoul, Korea · Carnegie Mellon University, Pittsburgh PA, USA +1
PAMI: Part Anchored Motion for Text to Human-Object Interaction Generation
Sep 29, 2026Chuqiao Li, Xianghui Xie, Yong Cao +2Interaction Data Text Analysis and Detection
Tübingen AI Center, University of Tübingen, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany

Date pending, cs.CVJ/K move · Enter open · S save

Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interaction

Mengfei Zhang, Jinlu Zhang, Zhigang Tu

Wuhan University Wuhan, China · Peking University Beijing, China

Articulated Objects Human Motion Prediction Unified Framework Multi--Task Learning Text Analysis and Detection

Open paper AskPDF arXiv

Modeling 4D human-object interaction (HOI) is a compelling challenge in computer vision and an essential technology powering virtual and mixed-reality applications. While existing works have achieved promising results on specific HOI tasks-such as text-conditioned HOI generation and human motion generation from object motion, they typically rely on task-specific architectures and lack a unified framework capable of handling diverse conditional inputs. Building on this, we propose Uni-HOI, a unified framework that learns the joint distribution among text, human motion, and object motion. By leveraging large language models (LLMs) and two motion-specific vector quantized variational autoencoders (VQ-VAEs), we convert heterogeneous motion data into token sequences compatible with LLM inputs, enabling seamless integration and joint modeling of all three modalities. We introduce a two-stage training strategy: the first stage performs multi-task learning on a large-scale HOI dataset to capture the underlying correlations among the three modalities, while the second stage fine-tunes the model on specific tasks to further enhance performance. Extensive experiments demonstrate that Uni-HOI achieves remarkable performances on multiple HOI-related tasks including text-driven HOI generation, object motion-driven human motion generation (optionally with text) and human motion-driven object motion prediction within a unified framework.

Abstract

Explore similar work

Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interaction

JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation

PAMI: Part Anchored Motion for Text to Human-Object Interaction Generation