COSMI: COmpositional Synthesis of Multi-object Interactions
Authors: Daniel Eskandar, Ilya A. Petrov, Gerard Pons-Moll
Organizations: University of Tübingen, Germany · Tübingen AI Center, Germany · Zuse School ELIZA, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany
Generative models of human-object interaction are bounded by the data that exists: everyday activities involve several objects, but most captured datasets record one at a time, as multi-object capture is combinatorially expensive. Our observation is that interactions are local, so single-object captures already contain the parts of multi-object activities. We compose them: contact-consistent clips of single interactions, mirrored to balance the hands, transfer between bodies, and a language model and geometric checks admit only the pairings that are plausible, semantically and physically. Therefore, the dataset grows combinatorially with the clips rather than recording time. The COSMI dataset holds 222k sequences and 275 hours with up to five objects, nearly thirty times the largest multi-object capture, and can be extended by adding datasets or even hand-object recordings. On this data we train the COSMI method, a text-to-interaction diffusion transformer that follows how the data is built: weight-shared object slots generate a variable number of objects, predicted relative to the body parts that move them. On a benchmark with an unseen object and unseen interaction combinations, models trained on the dataset generalize to the unseen combinations. COSMI outperforms baselines in text alignment and contact accuracy, where its margin is largest on the unseen object. Code, models, and the dataset pipeline will be released on the project page: https://ptrvilya.github.io/cosmi.
Figures & tables
Figure 1: COSMI. Left: sequences of the COSMI dataset, composed from single-object captures and rendered with clothed scans. Right: composition of one sequence from a sitting clip, a right-hand action, a left-hand action obtained by mirroring, and a backpack, rendered with a clothed scan.
Figure 2: Composition of one sequence. Atomic clips are extracted by interaction rules, annotated by a language model, and mirrored. A sitting target receives two compatible hand actions, their limbs and objects are transferred onto its body, and the caption is assembled accordingly.
Figure 3: COSMI method. One transformer denoises the human tokens H , one per frame, and the object tokens On , one per four frames, at diffusion step t . Conditioned on the CLIP text tokens W , the initial poses , and the name and BPS geometry of every object, it generates the human motion and each object motion relative to the joint that drives it.
Figure 4: Hand-object recordings as source clips. A HOT3D recording of the hands and the object (left) becomes a full-body clip once SMPL-X is fitted to the headset and the MANO hands (middle), and the composition pairs it with a sitting clip like any captured source (right).
Figure 5: Qualitative comparison. Generations for a test prompt with the unseen black chair and two hand actions in sequence. COSMI keeps the person seated, and generates plausible contacts.
Method
black chair
walk + apple
sit + flip phone
camera + mug
foot + drink
stand + bottle
T-Acc ↑
B-Acc ↑
T-Acc ↑
B-Acc ↑
T-Acc ↑
B-Acc ↑
T-Acc ↑
B-Acc ↑
T-Acc ↑
B-Acc ↑
T-Acc ↑
B-Acc ↑
MDM
0.823
0.917
0.971
0.946
0.903
0.930
0.955
0.953
0.926
0.958
0.977
0.929
PriorMDM
0.895
0.916
0.983
0.950
0.941
0.937
0.965
0.956
0.942
0.954
0.969
0.939
COSMI (ours)
0.975
0.960
0.995
0.974
0.981
0.959
0.984
0.984
0.953
0.981
0.993
0.967
Table 4: Unseen object and combinations. Contact accuracy on the test sequences with the unseen object, the black chair, and with the five unseen combinations.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Qualitative comparison with the baselines. Generations for test prompts of the three-object tier (top) and of the all-objects tier (bottom), which includes more than 3 objects and therefore no HIMO. COSMI keeps every object in the hand that drives it, whereas the baselines let objects drift away from the hands or float beside the body.
Figure 7: Failure cases. Left: the hinge of the flip phone has limited movement range. Right: the unseen black chair is placed off the body, which intersects its armrest.
Figure 8: Hand-object recordings as source clips. A HOT3D recording (second column) provides the hands and the object. A body fitted to the head-mounted device and the MANO hands (third column) turns it into a source clip, which the rules and the composition process like any other, here onto sitting and large-object targets (first column), giving the composed sequences (fourth column).
Figure 9: Clothed scans animated with composed sequences. 4D-DRESS scans (left), registered as SMPL-X+D, are driven by the joint transforms of composed sequences, shown on the body model in the top row. Objects follow through their joint-relative transforms, and the finger motion is preserved on the hands.
Figure 10: Atomic clips per interaction rule. Clips extracted by each of the thirteen rules of Table 6 , the walking sequences of AMASS included. Target rules (T) engage the whole body, source rules (S) one or both hands, and augmentation rules (A) worn objects. Holds and worn objects are single frames.
Figure 11: Annotation prompts. The two per-clip prompts of the annotation with the answer of Qwen3, for a clip of the InterCap briefcase whose given label hold is corrected to move .
Figure 12: Mirroring. Global reflection of the object breaks the contact with the mirrored hand, and local reflection keeps the contact but mirrors the mesh. Our local mirroring poses the original mesh like its reflection with a proper rotation and preserves both.
Figure 13: Composition graph. An arrow leads from every source rule to the rules it can be composed onto.
Figure 14: Composition of sequences. Left: the target clip, which provides the body motion, and the source clips, whose limbs are colored, blue for the left hand, red for the right hand, and purple for both hands. Right: the composed sequence, in which the source limbs replace their counterparts and every object follows its limb through Eq. 1 . Rows two and four are sequential compositions.
Figure 15: Sequences of the COSMI dataset. Composed sequences with their captions: simultaneous and sequential actions on walking, sitting, and standing bodies, with worn and articulated objects.
w/o G
w/ G
Per window (ms)
total
157
1,140
denoising, 100 steps
138
1,122
decoding to meshes
5
5
Per sequence (s)
119 frames, 1 window
0.15
1.08
Appendix
Table 7: Runtime. Generation time on one RTX 5090 without and with the contact-consistent guidance (G), at batch size one unless stated.
The way humans interact with each other, including interpersonal distances, spatial configuration, and motion, varies significantly across different situations. To enable machines to understand such complex, context-dependent behaviors, it is essential to model multiple people in relation to the surrounding scene context. In this paper, we present a novel research problem to model the correlations between two people engaged in a shared interaction involving an object. We refer to this formulation as Human-Human-Object Interactions (HHOIs). To overcome the lack of dedicated datasets for HHOIs, we present a newly captured HHOIs dataset and a method to synthesize HHOI data by leveraging image generative models. As an intermediary, we obtain individual human-object interaction (HOIs) and human-human interaction (HHIs) from the HHOIs, and with these data, we train an text-to-HOI and text-to-HHI model using score-based diffusion model. Finally, we present a unified generative framework that integrates the two individual model, capable of synthesizing complete HHOIs in a single advanced sampling process. Our method extends HHOI generation to multi-human settings, enabling interactions involving more than two individuals. Experimental results show that our method generates realistic HHOIs conditioned on textual descriptions, outperforming previous approaches that focus only on single-human HOIs. Furthermore, we introduce multi-human motion generation involving objects as an application of our framework.
Synthesizing realistic full-body human interactions with articulated objects is a fundamental challenge for embodied AI and graphics, with applications in robotics training and virtual agents. Existing models remain limited: some focus on simple activities with static objects, while others restrict attention to hand-only manipulation. This leaves open the problem of generating coordinated full-body motion that approaches, manipulates, and moves articulated objects in a realistic and generalizable way. The key difficulty lies in reasoning jointly about locomotion, fine-grained contact, and object articulation. Models must capture subtle hand-object correspondences that transfer across object geometries, while also producing seamless transitions from navigation to manipulation. At the same time, the scarcity of large-scale paired motion-scene data makes it difficult to generalize across diverse object positions and shapes. We introduce a text-conditioned diffusion model that addresses these challenges through three core ideas: an object-centric representation that unifies hand-object contact with object surfaces, a mixed-domain training strategy that balances locomotion and interaction, and a contact-based augmentation scheme that expands training diversity. Through experiments, our method demonstrated strong generalization to unseen object configurations, surpassing current state-of-the-art methods.
Xiaohan Zhang, Sebastian Starke, Alexander Winkler +3
Tübingen AI Center, University of Tübingen · Max Planck Institute for Informatics, Saarland Informatics Campus · Meta Reality Labs Research
Recent advances in 4D Human-Object Interaction (HOI) generation have enabled increasingly realistic motion synthesis, particularly for single-object manipulation. Yet current research overlooks an inherent property of human behavior: people naturally coordinate both hands and manipulate multiple objects simultaneously. To address this gap, we present Dex2HOI, a unified diffusion model for single- and two-object HOI synthesis from text. At its core, Dex2HOI employs a Dual-Stream Diffusion approach, where each object is processed in a dedicated interaction stream and coordinated through bidirectional cross-attention. To synthesize the final motion, we introduce a Motion Fusion Network integrated with novel hand-relative object representations and contact-aware conditioning applied across the whole sequence. By sampling the diffusion process autoregressively over prefix-conditioned windows, Dex2HOI generates arbitrarily long sequences at real-time speed omitting redundant test-time optimization, achieving up to x540 inference speed-up over prior state-of-the-art methods. Extensive evaluation on both single- and two-object benchmarks demonstrates state-of-the-art quantitative results, marking a step beyond conventional single-object HOI generation and toward expressive multi-object manipulation. Project: https://cpratikaki.github.io/dex2hoi/
Chrysa Pratikaki, Pablo Ruiz-Ponce, Jiankang Deng +2
Imperial College London, UK · University of Alicante, Spain