Human-object motion synthesis requires coordinated human participation as objects move toward specified goals. However, reaching object targets does not ensure that contact is established and maintained, while adjustments that improve contact may interfere with task completion. We introduce CoRelate, a diffusion framework that coordinates full-body human-object motion through explicit interaction relations and task feedback for rigid and articulated objects. We represent intended participation through hand-part relations that specify the participating hand, its part-relative reference position, and participation interval. These relations serve both as generation conditions and as references for measuring interaction realization. At each denoising step, a temporal refiner trained jointly with the generator combines relation deviations, task residuals, and motion features from the current prediction to jointly update human and object motion across the sequence. After generation, temporal motion correction addresses remaining contact failures by optimizing local motion segments while preserving object trajectories and protected human motion. Experiments demonstrate improved object-goal satisfaction, contact F1, and complete interaction success over the evaluated baselines. Ablation studies support the benefits of relation conditioning and the complementary roles of task and relation feedback.
Figures & tables
Figure 2: Overview of CoRelate. (A) From the task conditions and a task-only proposal X , we construct relations R : a hand, a wrist reference xrel in the part frame that moves with the part, and establish, hold, and release phases. (B) R conditions the generator Dθ and is the reference for relation feedback z (blue paths). At each denoising step, Fψ combines z , task residuals etask , and features f to update human and object motion, with R and goals g fixed. (C) Temporal correction then edits local hand motion with the object trajectory fixed followed by finger refinement.
Object
Contact
Motion quality
GT fidelity
Overall
Method
T e ↓
T xy ↓
C prec ↑
C rec ↑
C F1 ↑
MCC ↑
P hand ↓
FS ↓
FID ↓
R prec ↑
Div. →
MPJPE ↓
T obj ↓
O obj ↓
CIS ↑
MDM †
7.07
3.03
0.832
0.732
0.728
88.4
0.45
4.53
8.652
0.895
1.375
13.49
12.56
0.959
19.7
Pred-OMOMO
7.55
3.48
0.825
0.696
0.710
83.2
0.39
4.25
3.621
0.858
1.353
14.38
13.17
0.964
9.9
CHOIS
7.57
3.49
0.848
0.714
0.732
83.8
0.44
4.04
6.208
0.890
1.350
13.50
13.17
0.965
10.4
HOI-Dyn
6.59
3.02
0.842
0.644
0.678
80.8
0.38
4.84
6.568
0.870
1.364
14.88
13.03
0.924
12.5
MaMi-HOI
4.30
1.76
0.868
0.771
0.777
89.5
0.42
4.13
1.297
0.922
1.347
11.14
9.09
0.753
36.8
Table 1: Comparison on FullBodyManipulation dataset. CoRelate achieves lower object-target errors and higher contact F1 and complete interaction success than the evaluated baselines. † denotes task-adapted methods.
Task
Contact
Motion quality
Overall
Method
completion ↑
C acc tem ↑
C acc body ↑
C F1 ↑
MCC ↑
Pene ↓
FS ↓
Jerk obj ↓
FID ↓
R prec ↑
Div. →
CIS ↑
CoDA †
22.0
0.502
0.918
0.407
56.7
0.872
0.180
0.488
0.227
0.560
1.197
13.2
SK-HOI †
33.0
0.425
0.920
0.425
35.0
0.743
0.565
0.589
0.385
0.340
1.159
16.2
CoRelate
92.0
0.652
0.910
0.775
71.7
0.721
0.225
0.546
0.117
0.840
1.271
36.0
Table 2: Comparison on ParaHome dataset. CoRelate achieves higher task completion, contact F1, and complete interaction success than the evaluated baselines. † denotes task-adapted methods.
Figure 3: Qualitative comparisons. We mark the final distance to the goal and the frames where the object moves without a hand. CoRelate maintains hand contact as objects move toward their goals.
Object
Contact
Motion quality
GT fidelity
Overall
Configuration
T e ↓
T xy ↓
C F1 ↑
MCC ↑
P hand ↓
FS ↓
FID ↓
R prec ↑
MPJPE ↓
T obj ↓
O obj ↓
CIS ↑
Base
3.17
1.83
0.761
89.4
0.44
4.15
1.248
0.939
12.01
8.78
0.748
41.2
+ Relation conditioning
2.93
1.74
0.821
94.3
0.41
3.97
0.902
0.934
11.51
8.91
0.754
46.4
+ Joint refiner w/o feedback
2.97
1.74
0.821
93.4
0.43
4.02
0.926
0.927
11.47
8.85
0.751
47.8
+ Task feedback
0.93
0.75
0.825
94.2
0.41
4.13
1.116
0.932
11.50
8.58
0.753
66.4
+ Relation feedback (Full)
0.85
0.49
0.854
95.3
0.42
4.06
0.921
0.945
11.32
8.22
0.760
71.2
Table 3: Ablation study. Each design choice contributes and task and relation feedback show complementary gains when combined.
Figure 4: Qualitative ablation visualization. We progressively add task and relation feedback to the no-feedback baseline and highlight incorrect interactions.
Text-conditioned full-body human-object interaction (HOI) generation requires synthesizing human motion and object trajectories that match the input text while remaining precisely coordinated over time. Most methods represent the human and object as separate trajectories and predict the global human-object couplings. Learning this complex, dynamically changing relationship implicitly, however, often yields object drift, missed contact, and penetration. We introduce PAMI, a Part-Anchored Motion framework for Interaction generation. Inspired by the classic Hough Transform, our key idea is to localize object motion by letting body-part anchors vote for it: we express object motion relative to multiple body-part anchors and use PamiVAE to learn an interaction latent space, decoding frame-wise weights that aggregate these part-specific votes. Building on this representation, PAMI generates interactions in a coarse-to-fine hierarchy. PamiGen first generates a coarse human-object interaction from text in this structured latent space, and PamiRefiner then recursively resolves fine-grained contact geometry using a hybrid surface-sensing representation, combining long-range probes that capture overall body-part influence with short-range sensors that resolve detailed contacts near the object surface. Experiments on InterAct show that PAMI generates more faithful interactions and more accurate human-relative object motion than previous methods, achieving 14.5% higher contact recall than the previous state of the art. Extensive ablations validate the contributions of both the part-anchored voting representation and hybrid surface-sensing refinement.
Chuqiao Li, Xianghui Xie, Yong Cao +2
Tübingen AI Center, University of Tübingen, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany
Synthesizing human--object interaction (HOI) videos has broad practical value in e-commerce, digital advertising, and virtual marketing. However, current diffusion models, despite their photorealistic rendering capability, still frequently fail on (i) the structural stability of sensitive regions such as hands and faces and (ii) physically plausible contact (e.g., avoiding hand--object interpenetration). We present CoInteract, an end-to-end framework for HOI video synthesis conditioned on a person reference image, a product reference image, text prompts, and speech audio. CoInteract introduces two complementary designs embedded into a Diffusion Transformer (DiT) backbone. First, we propose a Human-Aware Mixture-of-Experts (MoE) that routes tokens to lightweight, region-specialized experts via spatially supervised routing, improving fine-grained structural fidelity with minimal parameter overhead. Second, we propose Spatially-Structured Co-Generation, a dual-stream training paradigm that jointly models an RGB appearance stream and an auxiliary HOI structure stream to inject interaction geometry priors. During training, the HOI stream attends to RGB tokens and its supervision regularizes shared backbone weights; at inference, the HOI branch is removed for zero-overhead RGB generation. Experimental results demonstrate that CoInteract significantly outperforms existing methods in structural stability, logical consistency, and interaction realism.
Synthesizing realistic full-body human interactions with articulated objects is a fundamental challenge for embodied AI and graphics, with applications in robotics training and virtual agents. Existing models remain limited: some focus on simple activities with static objects, while others restrict attention to hand-only manipulation. This leaves open the problem of generating coordinated full-body motion that approaches, manipulates, and moves articulated objects in a realistic and generalizable way. The key difficulty lies in reasoning jointly about locomotion, fine-grained contact, and object articulation. Models must capture subtle hand-object correspondences that transfer across object geometries, while also producing seamless transitions from navigation to manipulation. At the same time, the scarcity of large-scale paired motion-scene data makes it difficult to generalize across diverse object positions and shapes. We introduce a text-conditioned diffusion model that addresses these challenges through three core ideas: an object-centric representation that unifies hand-object contact with object surfaces, a mixed-domain training strategy that balances locomotion and interaction, and a contact-based augmentation scheme that expands training diversity. Through experiments, our method demonstrated strong generalization to unseen object configurations, surpassing current state-of-the-art methods.
Xiaohan Zhang, Sebastian Starke, Alexander Winkler +3
Tübingen AI Center, University of Tübingen · Max Planck Institute for Informatics, Saarland Informatics Campus · Meta Reality Labs Research