cs.CVApr 15, 2026
SaveOneHOI: Unifying Human-Object Interaction Generation and Editing
Organizations: Nanyang Technological University · Sun Yat-sen University · VinUniversity · Universiti Malaya
Abstract
Human-Object Interaction (HOI) modelling captures how humans act upon and relate to objects, typically expressed as <person, action, object> triplets. Existing approaches split into two disjoint families: HOI generation synthesises scenes from structured triplets and layout, but fails to integrate mixed conditions like HOI and object-only entities; and HOI editing modifies interactions via text, yet struggles to decouple pose from physical contact and scale to multiple interactions. We introduce OneHOI, a unified diffusion transformer framework that consolidates HOI generation and editing into a single conditional denoising process driven by shared structured interaction representations. At its core, the Relational Diffusion Transformer (R-DiT) models verb-mediated relations through role- and instance-aware HOI tokens, layout-based spatial Action Grounding, a Structured HOI Attention to enforce interaction topology, and HOI RoPE to disentangle multi-HOI scenes. Trained jointly with modality dropout on our HOI-Edit-44K, along with HOI and object-centric datasets, OneHOI supports layout-guided, layout-free, arbitrary-mask, and mixed-condition control, achieving state-of-the-art results across both HOI generation and editing. Code is available at https://jiuntian.github.io/OneHOI/.
Figures & tables
Figure 2: Unified HOI generation and editing. OneHOI enables a single-model multi-step workflow. It begins with (i) Mixed-Condition Generation , synthesising a complex scene from layout-guided HOIs with arbitrary shape. Then, it performs (ii) Layout-free HOI Editing , ( e.g . , change him to plant the flag), followed by (iii) Layout-guided HOI Editing ( e.g . , add another astronaut and driving a rover) and (iv) Attribute Editing ( e.g . , change to Mars). More examples in Fig. 14 of the Appendix.
Figure 3: (a) OneHOI unifies HOI editing and generation tasks on a DiT backbone. The pipeline features an HOI Encoder to inject role and instance cues, and Structured HOI Attention to enforce verb-mediated topology and spatial grounding. (b, c) To separate instances, in contrast to the Original RoPE (b), HOI RoPE (c) provides unique positional indices for each interaction.
Figure 4: Action-token→image attention heatmaps from the baseline. The “Between” region proposed in InteractDiffusion [ 14 ] misses where the action actually attends, while our “Union” region (subject object) better matches the attention footprint.
Figure 5: (a) HOI attention mask. Colours match Fig. 3(a) legend, grey hatched indicates blocked attention. Direct is blocked to enforce verb-mediated topology. attend to image only within , respectively, as shown in (b).
| Method | HOI Editing | Image Quality | |||
| Editability-Identity | HOI Editability | PickScore | HPS | ImageReward | |
| Null-Text Inversion [ 10 , 28 ] | 0.443 | 0.390 | 20.81 | 0.2483 | |
| MasaCtrl [ 2 ] | 0.371 | 0.260 | 20.14 | 0.2212 | |
| HOIEdit [ 43 ] | 0.349 | 0.240 | 19.51 | 0.2129 | |
| InstructPix2Pix [ 1 ] | 0.380 | 0.269 | 20.28 | 0.2178 | |
| TurboEdit [ 8 ] | 0.434 | 0.326 | 20.36 | 0.2437 | |
Table 1: Quantitative comparison for layout-free HOI editing on IEBench benchmark. Our method significantly outperforms others across all metrics for editing and image quality. Best results are in bold , second best are underlined . Final row shows the closed-source baseline.
| Task | Method | Layout-guided HOI Editing | Image Quality | ||||
|---|---|---|---|---|---|---|---|
| Editability-Identity | HOI Editability | Spatial | PickScore | HPS | ImageReward | ||
| Single HOI Editing | InteractEdit + InteractDiffusion | 0.559 | 0.520 | 0.749 | 20.53 | 0.2418 | |
| Ours | 0.638 | 0.570 | 0.822 | 21.04 | 0.2678 | 0.2897 | |
| Multi HOI Editing | Ours ∗ | 0.435 | 0.329 | 0.675 | 21.22 | 0.2742 | |
Table 2: Quantitative results for our novel layout-guided HOI editing tasks. We report strong performance for both single- and multi-HOI editing, establishing the first baseline for these new capabilities.
| Method | Controllability | Image Quality | |||
| Spatial | HOI | PickScore | HPS | ImageReward | |
| GLIGEN [ 22 ] | 0.5150 | 0.3344 | 20.46 | 0.2322 | |
| InstanceDiffusion [ 37 ] | 0.5228 | 0.3476 | 20.06 | 0.2312 | |
| MIGC++ [ 47 , 46 ] | 0.5331 | 0.3616 | 20.16 | 0.2208 | |
| Eligen [ 45 ] | 0.4371 | 0.3061 | 2 1.28 | 0 .2496 | 0.3921 |
| InteractDiffusion [ 14 ] | 0 .5768 | 0 .4505 | 20.37 | 0.2283 | |
Table 3: Quantitative comparison for HOI generation task. Our method outperforms leading layout-conditioned and HOI-aware models on both controllability and image quality metrics.
Figure 6: Qualitative comparison for layout-free HOI editing. Our method successfully renders the new interaction while preserving identity. In contrast, baseline methods often produce artifacts, fail to change the pose, or lose the subject’s identity.
Figure 7: Qualitative comparison for HOI generation. While object-level methods correctly place entities, they fail to synthesise specified interactions. Ours renders semantically and geometrically consistent interactions, demonstrating a deeper relational understanding.
Figure 8: Layout-guided editing examples. Our model supports single-HOI (top) and multi-HOI edits (bottom), limiting changes to target layouts while preserving scene consistency.
Figure 9: Versatile control in HOI generation. Our model supports conditioning on both arbitrary-shape masks (top) and a mix of HOI and object-only inputs within a single scene (bottom), demonstrating its compositional capabilities.
Figure 10: Progressively adding components improves the interaction’s plausibility, only the full model (4) successfully rendering the complex, two-handed action of both “holding” and “petting.”
| Components | HOI Generation | Multi-HOI Edit | ||||||
|---|---|---|---|---|---|---|---|---|
| AG | Enc | Attn | HRoPE | HOI Acc. | IR | EI | IR | |
| BL | 0.3061 | 0.3921 | - | - | ||||
| (1) | ✓ | 0.4138 | 0.3156 | 0.423 | 0.1118 | |||
| (2) | ✓ | ✓ | 0.4254 | 0.4602 | 0.422 | 0.1306 | ||
| (3) | ✓ | ✓ | ✓ | 0.4504 | 0.4861 | 0.433 | 0.1944 | |
| (4) | ✓ | ✓ | ✓ | ✓ | 0.4528 | 0.5224 | 0.435 | 0.2046 |
Table 4: Ablation study on core components. AG: Action Grounding, Enc: HOI Encoder, Attn: HOI Attention, HRoPE: HOI RoPE, EI: Editability-Identity, IR: ImageReward.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Evaluation Interface. Web-based survey used for data collection. Participants performed side-by-side comparisons of two models based on a source image and target edit instruction.
Figure 12: Results of the Human Preference Study. Aggregated preference percentages for HOI Correctness and Physical Plausibility, Identity Preservation, and Overall Quality. The top bar in each category compares OneHOI (Ours) against QwenImageEdit, while the bottom bar compares it against InteractEdit.
Figure 13: Attention footprint of Flux.1. “Union” better matches the attention footprint compared to “Between”.
| Task Scenario | Metric | Task-specific | Unified (Ours) |
|---|---|---|---|
| Generation | Spatial / HOI Acc. | 0.422 / 0.177 | 0.443 / 0.224 |
| Layout-Free Edit | Editability-Identity / HOI Editability | 0.574 / 0.464 | 0.611 / 0.562 |
| Multi-HOI Edit | Editability-Identity / HOI Editability | 0.391 / 0.287 | 0.435 / 0.329 |
Table 5: Ablation on Unification.
Figure 14: Versatile workflow for unified HOI generation and editing using OneHOI. OneHOI enables a seamless, multi-step workflow within a single model, showcasing diverse conditional control. Starting with: Top Row: Urban Park Scene. (1) Mixed-Condition Generation synthesises a complex scene from layout-guided HOIs ( i.e . , walking dog ) and arbitrary shape-guided independent objects ( i.e . , lamp post , leash ), alongside another HOI ( i.e . , person sitting on bench ). (2) Multi-HOI Editing simultaneously updates two distinct interactions ( i.e . , holding dog , person lying on bench ). (3) Single-HOI Editing modifies one interaction ( i.e . , holding ball ). (4) Attribute Editing changes an object’s colour ( i.e . , black blue ). Bottom Row: Ocean Survival Scene. (1) Mixed-Condition Generation creates a challenging open-water scenario from a person standing on a boat and arbitrary shape-guided floating debris. (2) Layout-guided HOI Editing precisely changes the person’s action ( i.e . , paddling the boat ). (3) HOI Editing (Add) introduces a new interaction ( i.e . , white Bengal tiger roaring and lying on the boat ). (4) Attribute Editing (Scene) transforms the entire environment ( i.e . , day stormy , calm turbulent ocean).
Figure 15: Examples from the HOI-Edit-44K dataset.
Figure 16: Treemap visualising the distribution of the interacting object categories in the HOI-Edit-44K dataset. The size of each block corresponds to the category’s frequency.
Figure 17: Treemap visualising the distribution of target action categories in the HOI-Edit-44K dataset.
Figure 22: Additional qualitative results for HOI generation. These examples further highlight the limitations of baselines, which often fail to render the specific action even when the objects are placed correctly. In the first row (standing on a chair) , all baseline methods incorrectly generate a child sitting on a chair, while our model is the only one that correctly synthesises the ‘standing on’ pose. Similarly, for holding a spoon (row 2) , baselines produce general eating scenes, with Eligen and InteractDiff showing a fork instead. Our model, in contrast, correctly renders the person holding a spoon. This challenge is more pronounced in complex multi-HOI prompts. For row 3 (flipping, jumping, and riding a skateboard) , baselines fail to capture the ‘flipping’ or ‘jumping’ motions, rendering a simple ‘riding’ pose at best. In row 4 (drinking with bottle while holding it) , most methods fail to combine both ‘holding’ or ‘drinking’. In contrast, our model generates coherent images that plausibly reflects all specified interactions, demonstrating superior compositional understanding.
Figure 23: Additional qualitative comparisons for layout-free HOI edits. Row 1 (jump → sit on skateboard) : Baselines show incorrect poses (Qwen, Flux.1), unnatural actions (InteractEdit), or severe artifacts (HOIEdit), while ours renders “sit on” interaction. Row 2 (kick → hold ball) : Most baselines fail to alter the pose, while ours renders the “hold” action. Row 3 (hold → jump snowboard) : Most methods failed to render “jump”. Although InteractEdit renders jump, it fails to preserve the snowboard’s identity. Ours renders the jump while maintaining the identity of both the person and the snowboard. Row 4 (walk → feed dog) : Only ours renders a coherent “feeding” interaction while preserving the identities of both subjects, demonstrating its superior capability in handling complex relational changes.
Figure 24: Examples from the MultiHOIEdit benchmark.
More in this topic
HOI Generation
Human-object interaction generation creates 3D motion, grasps, and videos of people manipulating or coordinating with objects.
See all 83 papers