Human-Object Interaction (HOI) modelling captures how humans act upon and relate to objects, typically expressed as <person, action, object> triplets. Existing approaches split into two disjoint families: HOI generation synthesises scenes from structured triplets and layout, but fails to integrate mixed conditions like HOI and object-only entities; and HOI editing modifies interactions via text, yet struggles to decouple pose from physical contact and scale to multiple interactions. We introduce OneHOI, a unified diffusion transformer framework that consolidates HOI generation and editing into a single conditional denoising process driven by shared structured interaction representations. At its core, the Relational Diffusion Transformer (R-DiT) models verb-mediated relations through role- and instance-aware HOI tokens, layout-based spatial Action Grounding, a Structured HOI Attention to enforce interaction topology, and HOI RoPE to disentangle multi-HOI scenes. Trained jointly with modality dropout on our HOI-Edit-44K, along with HOI and object-centric datasets, OneHOI supports layout-guided, layout-free, arbitrary-mask, and mixed-condition control, achieving state-of-the-art results across both HOI generation and editing. Code is available at https://jiuntian.github.io/OneHOI/.
Figures & tables
Figure 2: Unified HOI generation and editing. OneHOI enables a single-model multi-step workflow. It begins with (i) Mixed-Condition Generation , synthesising a complex scene from layout-guided HOIs with arbitrary shape. Then, it performs (ii) Layout-free HOI Editing , ( e.g . , change him to plant the flag), followed by (iii) Layout-guided HOI Editing ( e.g . , add another astronaut and driving a rover) and (iv) Attribute Editing ( e.g . , change to Mars). More examples in Fig. 14 of the Appendix.
Figure 3: (a) OneHOI unifies HOI editing and generation tasks on a DiT backbone. The pipeline features an HOI Encoder to inject role and instance cues, and Structured HOI Attention to enforce verb-mediated topology and spatial grounding. (b, c) To separate instances, in contrast to the Original RoPE (b), HOI RoPE (c) provides unique positional indices for each interaction.
Figure 4: Action-token→image attention heatmaps from the baseline. The “Between” region proposed in InteractDiffusion [ 14 ] misses where the action actually attends, while our “Union” region (subject ∪ object) better matches the attention footprint.
Figure 5: (a) HOI attention mask. Colours match Fig. 3(a) legend, grey hatched indicates blocked attention. Direct Sn↔On is blocked to enforce verb-mediated topology. Sn,On,An attend to image I1 only within Rns,Rno,Rna , respectively, as shown in (b).
Method
HOI Editing
Image Quality
Editability-Identity
HOI Editability
PickScore
HPS
ImageReward
Null-Text Inversion [ 10 , 28 ]
0.443
0.390
20.81
0.2483
−0.3329
MasaCtrl [ 2 ]
0.371
0.260
20.14
0.2212
−0.7136
HOIEdit [ 43 ]
0.349
0.240
19.51
0.2129
−1.0289
InstructPix2Pix [ 1 ]
0.380
0.269
20.28
0.2178
−0.7717
TurboEdit [ 8 ]
0.434
0.326
20.36
0.2437
−0.3821
Table 1: Quantitative comparison for layout-free HOI editing on IEBench benchmark. Our method significantly outperforms others across all metrics for editing and image quality. Best results are in bold , second best are underlined . Final row shows the closed-source baseline.
Task
Method
Layout-guided HOI Editing
Image Quality
Editability-Identity
HOI Editability
Spatial
PickScore
HPS
ImageReward
Single HOI Editing
InteractEdit + InteractDiffusion
0.559
0.520
0.749
20.53
0.2418
−0.3072
Ours
0.638
0.570
0.822
21.04
0.2678
0.2897
Multi HOI Editing
Ours ∗
0.435
0.329
0.675
21.22
0.2742
0.1954
Table 2: Quantitative results for our novel layout-guided HOI editing tasks. We report strong performance for both single- and multi-HOI editing, establishing the first baseline for these new capabilities.
Method
Controllability
Image Quality
Spatial
HOI
PickScore
HPS
ImageReward
GLIGEN [ 22 ]
0.5150
0.3344
20.46
0.2322
−0.4103
InstanceDiffusion [ 37 ]
0.5228
0.3476
20.06
0.2312
−0.2532
MIGC++ [ 47 , 46 ]
0.5331
0.3616
20.16
0.2208
−0.6492
Eligen [ 45 ]
0.4371
0.3061
2 1.28
0 .2496
0.3921
InteractDiffusion [ 14 ]
0 .5768
0 .4505
20.37
0.2283
−0.3194
Table 3: Quantitative comparison for HOI generation task. Our method outperforms leading layout-conditioned and HOI-aware models on both controllability and image quality metrics.
Figure 6: Qualitative comparison for layout-free HOI editing. Our method successfully renders the new interaction while preserving identity. In contrast, baseline methods often produce artifacts, fail to change the pose, or lose the subject’s identity.
Figure 7: Qualitative comparison for HOI generation. While object-level methods correctly place entities, they fail to synthesise specified interactions. Ours renders semantically and geometrically consistent interactions, demonstrating a deeper relational understanding.
Figure 8: Layout-guided editing examples. Our model supports single-HOI (top) and multi-HOI edits (bottom), limiting changes to target layouts while preserving scene consistency.
Figure 9: Versatile control in HOI generation. Our model supports conditioning on both arbitrary-shape masks (top) and a mix of HOI and object-only inputs within a single scene (bottom), demonstrating its compositional capabilities.
Figure 10: Progressively adding components improves the interaction’s plausibility, only the full model (4) successfully rendering the complex, two-handed action of both “holding” and “petting.”
Figure 11: Evaluation Interface. Web-based survey used for data collection. Participants performed side-by-side comparisons of two models based on a source image and target edit instruction.
Figure 12: Results of the Human Preference Study. Aggregated preference percentages for HOI Correctness and Physical Plausibility, Identity Preservation, and Overall Quality. The top bar in each category compares OneHOI (Ours) against QwenImageEdit, while the bottom bar compares it against InteractEdit.
Figure 13: Attention footprint of Flux.1. “Union” better matches the attention footprint compared to “Between”.
Task Scenario
Metric
Task-specific
Unified (Ours)
Generation
Spatial ↑ / HOI Acc. ↑
0.422 / 0.177
0.443 / 0.224
Layout-Free Edit
Editability-Identity ↑ / HOI Editability ↑
0.574 / 0.464
0.611 / 0.562
Multi-HOI Edit
Editability-Identity ↑ / HOI Editability ↑
0.391 / 0.287
0.435 / 0.329
Appendix
Table 5: Ablation on Unification.
Figure 14: Versatile workflow for unified HOI generation and editing using OneHOI. OneHOI enables a seamless, multi-step workflow within a single model, showcasing diverse conditional control. Starting with: Top Row: Urban Park Scene. (1) Mixed-Condition Generation synthesises a complex scene from layout-guided HOIs ( i.e . , walking dog ) and arbitrary shape-guided independent objects ( i.e . , lamp post , leash ), alongside another HOI ( i.e . , person sitting on bench ). (2) Multi-HOI Editing simultaneously updates two distinct interactions ( i.e . , holding dog , person lying on bench ). (3) Single-HOI Editing modifies one interaction ( i.e . , holding ball ). (4) Attribute Editing changes an object’s colour ( i.e . , black → blue ). Bottom Row: Ocean Survival Scene. (1) Mixed-Condition Generation creates a challenging open-water scenario from a person standing on a boat and arbitrary shape-guided floating debris. (2) Layout-guided HOI Editing precisely changes the person’s action ( i.e . , paddling the boat ). (3) HOI Editing (Add) introduces a new interaction ( i.e . , white Bengal tiger roaring and lying on the boat ). (4) Attribute Editing (Scene) transforms the entire environment ( i.e . , day → stormy , calm → turbulent ocean).
Figure 15: Examples from the HOI-Edit-44K dataset.
Figure 16: Treemap visualising the distribution of the interacting object categories in the HOI-Edit-44K dataset. The size of each block corresponds to the category’s frequency.
Figure 17: Treemap visualising the distribution of target action categories in the HOI-Edit-44K dataset.
Figure 22
Figure 22: Additional qualitative results for HOI generation. These examples further highlight the limitations of baselines, which often fail to render the specific action even when the objects are placed correctly. In the first row (standing on a chair) , all baseline methods incorrectly generate a child sitting on a chair, while our model is the only one that correctly synthesises the ‘standing on’ pose. Similarly, for holding a spoon (row 2) , baselines produce general eating scenes, with Eligen and InteractDiff showing a fork instead. Our model, in contrast, correctly renders the person holding a spoon. This challenge is more pronounced in complex multi-HOI prompts. For row 3 (flipping, jumping, and riding a skateboard) , baselines fail to capture the ‘flipping’ or ‘jumping’ motions, rendering a simple ‘riding’ pose at best. In row 4 (drinking with bottle while holding it) , most methods fail to combine both ‘holding’ or ‘drinking’. In contrast, our model generates coherent images that plausibly reflects all specified interactions, demonstrating superior compositional understanding.
Figure 23: Additional qualitative comparisons for layout-free HOI edits. Row 1 (jump → sit on skateboard) : Baselines show incorrect poses (Qwen, Flux.1), unnatural actions (InteractEdit), or severe artifacts (HOIEdit), while ours renders “sit on” interaction. Row 2 (kick → hold ball) : Most baselines fail to alter the pose, while ours renders the “hold” action. Row 3 (hold → jump snowboard) : Most methods failed to render “jump”. Although InteractEdit renders jump, it fails to preserve the snowboard’s identity. Ours renders the jump while maintaining the identity of both the person and the snowboard. Row 4 (walk → feed dog) : Only ours renders a coherent “feeding” interaction while preserving the identities of both subjects, demonstrating its superior capability in handling complex relational changes.
Figure 24: Examples from the MultiHOIEdit benchmark.
The way humans interact with each other, including interpersonal distances, spatial configuration, and motion, varies significantly across different situations. To enable machines to understand such complex, context-dependent behaviors, it is essential to model multiple people in relation to the surrounding scene context. In this paper, we present a novel research problem to model the correlations between two people engaged in a shared interaction involving an object. We refer to this formulation as Human-Human-Object Interactions (HHOIs). To overcome the lack of dedicated datasets for HHOIs, we present a newly captured HHOIs dataset and a method to synthesize HHOI data by leveraging image generative models. As an intermediary, we obtain individual human-object interaction (HOIs) and human-human interaction (HHIs) from the HHOIs, and with these data, we train an text-to-HOI and text-to-HHI model using score-based diffusion model. Finally, we present a unified generative framework that integrates the two individual model, capable of synthesizing complete HHOIs in a single advanced sampling process. Our method extends HHOI generation to multi-human settings, enabling interactions involving more than two individuals. Experimental results show that our method generates realistic HHOIs conditioned on textual descriptions, outperforming previous approaches that focus only on single-human HOIs. Furthermore, we introduce multi-human motion generation involving objects as an application of our framework.
Synthesizing human--object interaction (HOI) videos has broad practical value in e-commerce, digital advertising, and virtual marketing. However, current diffusion models, despite their photorealistic rendering capability, still frequently fail on (i) the structural stability of sensitive regions such as hands and faces and (ii) physically plausible contact (e.g., avoiding hand--object interpenetration). We present CoInteract, an end-to-end framework for HOI video synthesis conditioned on a person reference image, a product reference image, text prompts, and speech audio. CoInteract introduces two complementary designs embedded into a Diffusion Transformer (DiT) backbone. First, we propose a Human-Aware Mixture-of-Experts (MoE) that routes tokens to lightweight, region-specialized experts via spatially supervised routing, improving fine-grained structural fidelity with minimal parameter overhead. Second, we propose Spatially-Structured Co-Generation, a dual-stream training paradigm that jointly models an RGB appearance stream and an auxiliary HOI structure stream to inject interaction geometry priors. During training, the HOI stream attends to RGB tokens and its supervision regularizes shared backbone weights; at inference, the HOI branch is removed for zero-overhead RGB generation. Experimental results demonstrate that CoInteract significantly outperforms existing methods in structural stability, logical consistency, and interaction realism.
Recent advances in video diffusion models have spurred interest in human-object interaction (HOI) video generation, which demands fine-grained control over interaction logic beyond single-subject animation. However, existing HOI methods rely heavily on explicit motion control, limiting scalability and generalization across diverse objects and interactions. In this study, we propose AgentHOI, a text-driven HOI video generation following a thinking-before-generation framework that bridges the gap between high-level textual intent and physical execution through multi-agent reasoning over perception, interaction, and motion planning. Building upon the generated interaction plans, we further strengthen text-driven motion understanding. We introduce an implicit text-motion alignment strategy that distills text-to-motion priors into the video diffusion model, enabling robust HOI synthesis without explicit motion inputs at inference. Experiments show that AgentHOI significantly improves interaction naturalness, object appearance preservation, and adherence to complex textual instructions across challenging object-centric scenarios such as wearing and riding. The code is available at https://github.com/bone-11/agenthoi.
Ziyao Huang, Shunkai Li, Juan Cao +7
University of Chinese Academy of Sciences · Tencent HunYuan