COSMI: COmpositional Synthesis of Multi-object Interactions

Daniel Eskandar; Ilya A. Petrov; Gerard Pons-Moll

cs.CVOct 2, 2026

COSMI: COmpositional Synthesis of Multi-object Interactions

Authors: Daniel Eskandar, Ilya A. Petrov, Gerard Pons-Moll

Organizations: University of Tübingen, Germany · Tübingen AI Center, Germany · Zuse School ELIZA, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany

Abstract

Generative models of human-object interaction are bounded by the data that exists: everyday activities involve several objects, but most captured datasets record one at a time, as multi-object capture is combinatorially expensive. Our observation is that interactions are local, so single-object captures already contain the parts of multi-object activities. We compose them: contact-consistent clips of single interactions, mirrored to balance the hands, transfer between bodies, and a language model and geometric checks admit only the pairings that are plausible, semantically and physically. Therefore, the dataset grows combinatorially with the clips rather than recording time. The COSMI dataset holds 222k sequences and 275 hours with up to five objects, nearly thirty times the largest multi-object capture, and can be extended by adding datasets or even hand-object recordings. On this data we train the COSMI method, a text-to-interaction diffusion transformer that follows how the data is built: weight-shared object slots generate a variable number of objects, predicted relative to the body parts that move them. On a benchmark with an unseen object and unseen interaction combinations, models trained on the dataset generalize to the unseen combinations. COSMI outperforms baselines in text alignment and contact accuracy, where its margin is largest on the unseen object. Code, models, and the dataset pipeline will be released on the project page: https://ptrvilya.github.io/cosmi.

Figures & tables

Figure 1: COSMI. Left: sequences of the COSMI dataset, composed from single-object captures and rendered with clothed scans. Right: composition of one sequence from a sitting clip, a right-hand action, a left-hand action obtained by mirroring, and a backpack, rendered with a clothed scan.

Figure 2: Composition of one sequence. Atomic clips are extracted by interaction rules, annotated by a language model, and mirrored. A sitting target receives two compatible hand actions, their limbs and objects are transferred onto its body, and the caption is assembled accordingly.

Figure 3: COSMI method. One transformer denoises the human tokens $\mathcal{H}$ , one per frame, and the object tokens $\mathcal{O}_{n}$ , one per four frames, at diffusion step $t$ . Conditioned on the CLIP text tokens $\mathcal{W}$ , the initial poses , and the name and BPS geometry of every object, it generates the human motion and each object motion relative to the joint that drives it.

Figure 4: Hand-object recordings as source clips. A HOT3D recording of the hands and the object (left) becomes a full-body clip once SMPL-X is fitted to the headset and the MANO hands (middle), and the composition pairs it with a sitting clip like any captured source (right).

Figure 5: Qualitative comparison. Generations for a test prompt with the unseen black chair and two hand actions in sequence. COSMI keeps the person seated, and generates plausible contacts.

Method	black chair		walk + apple		sit + flip phone		camera + mug		foot + drink		stand + bottle
Method	T-Acc $\uparrow$	B-Acc $\uparrow$	T-Acc $\uparrow$	B-Acc $\uparrow$	T-Acc $\uparrow$	B-Acc $\uparrow$	T-Acc $\uparrow$	B-Acc $\uparrow$	T-Acc $\uparrow$	B-Acc $\uparrow$	T-Acc $\uparrow$	B-Acc $\uparrow$
MDM	0.823	0.917	0.971	0.946	0.903	0.930	0.955	0.953	0.926	0.958	0.977	0.929
PriorMDM	0.895	0.916	0.983	0.950	0.941	0.937	0.965	0.956	0.942	0.954	0.969	0.939
COSMI (ours)	0.975	0.960	0.995	0.974	0.981	0.959	0.984	0.984	0.953	0.981	0.993	0.967

Table 4: Unseen object and combinations. Contact accuracy on the test sequences with the unseen object, the black chair, and with the five unseen combinations.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Qualitative comparison with the baselines. Generations for test prompts of the three-object tier (top) and of the all-objects tier (bottom), which includes more than 3 objects and therefore no HIMO. COSMI keeps every object in the hand that drives it, whereas the baselines let objects drift away from the hands or float beside the body.

Figure 7: Failure cases. Left: the hinge of the flip phone has limited movement range. Right: the unseen black chair is placed off the body, which intersects its armrest.

Figure 8: Hand-object recordings as source clips. A HOT3D recording (second column) provides the hands and the object. A body fitted to the head-mounted device and the MANO hands (third column) turns it into a source clip, which the rules and the composition process like any other, here onto sitting and large-object targets (first column), giving the composed sequences (fourth column).

Figure 9: Clothed scans animated with composed sequences. 4D-DRESS scans (left), registered as SMPL-X+D, are driven by the joint transforms of composed sequences, shown on the body model in the top row. Objects follow through their joint-relative transforms, and the finger motion is preserved on the hands.

Figure 10: Atomic clips per interaction rule. Clips extracted by each of the thirteen rules of Table 6 , the walking sequences of AMASS included. Target rules (T) engage the whole body, source rules (S) one or both hands, and augmentation rules (A) worn objects. Holds and worn objects are single frames.

Panel of Figure 11 — Figure 11: Annotation prompts. The two per-clip prompts of the annotation with the answer of Qwen3, for a clip of the InterCap briefcase whose given label hold is corrected to move .

Figure 12: Mirroring. Global reflection of the object breaks the contact with the mirrored hand, and local reflection keeps the contact but mirrors the mesh. Our local mirroring poses the original mesh like its reflection with a proper rotation and preserves both.

Figure 13: Composition graph. An arrow leads from every source rule to the rules it can be composed onto.

Figure 14: Composition of sequences. Left: the target clip, which provides the body motion, and the source clips, whose limbs are colored, blue for the left hand, red for the right hand, and purple for both hands. Right: the composed sequence, in which the source limbs replace their counterparts and every object follows its limb through Eq. 1 . Rows two and four are sequential compositions.

Figure 15: Sequences of the COSMI dataset. Composed sequences with their captions: simultaneous and sequential actions on walking, sitting, and standing bodies, with worn and articulated objects.

	w/o G	w/ G
Per window (ms)
total	157	1,140
denoising, 100 steps	138	1,122
decoding to meshes	5	5
Per sequence (s)
119 frames, 1 window	0.15	1.08

Appendix

Table 7: Runtime. Generation time on one RTX 5090 without and with the contact-consistent guidance (G), at batch size one unless stated.

HOI Generation

Human-object interaction generation creates 3D motion, grasps, and videos of people manipulating or coordinating with objects.

See all 84 papers

Learning to Generate Human-Human-Object Interactions from Textual Descriptions
Nov 25, 2025Jeonghyeon Na, Sangwon Baik, Inhee Lee +2HOI Generation Human Motion Generation
Seoul National University
GIRAF: Towards Generalizable Human Interactions with Articulated Objects
Jul 8, 2026Xiaohan Zhang, Sebastian Starke, Alexander Winkler +3HOI Generation Text-to-Motion Generation
Tübingen AI Center, University of Tübingen · Max Planck Institute for Informatics, Saarland Informatics Campus · Meta Reality Labs Research
Dex2HOI: Dexterous Bimanual Two-Object Interaction Generation
May 28, 2026Chrysa Pratikaki, Pablo Ruiz-Ponce, Jiankang Deng +2HOI Generation Human Motion Generation
Imperial College London, UK · University of Alicante, Spain

Nov 25, 2025J/K move · Enter open · S save

Learning to Generate Human-Human-Object Interactions from Textual Descriptions

Jeonghyeon Na, Sangwon Baik, Inhee Lee, Junyoung Lee+1

Seoul National University

HOI Generation Human Motion Generation Human-Human-Object Interaction Generation Human-Object Interaction

Open paper AskPDF arXiv

The way humans interact with each other, including interpersonal distances, spatial configuration, and motion, varies significantly across different situations. To enable machines to understand such complex, context-dependent behaviors, it is essential to model multiple people in relation to the surrounding scene context. In this paper, we present a novel research problem to model the correlations between two people engaged in a shared interaction involving an object. We refer to this formulation as Human-Human-Object Interactions (HHOIs). To overcome the lack of dedicated datasets for HHOIs, we present a newly captured HHOIs dataset and a method to synthesize HHOI data by leveraging image generative models. As an intermediary, we obtain individual human-object interaction (HOIs) and human-human interaction (HHIs) from the HHOIs, and with these data, we train an text-to-HOI and text-to-HHI model using score-based diffusion model. Finally, we present a unified generative framework that integrates the two individual model, capable of synthesizing complete HHOIs in a single advanced sampling process. Our method extends HHOI generation to multi-human settings, enabling interactions involving more than two individuals. Experimental results show that our method generates realistic HHOIs conditioned on textual descriptions, outperforming previous approaches that focus only on single-human HOIs. Furthermore, we introduce multi-human motion generation involving objects as an application of our framework.

Abstract

HOI Generation

Explore similar work

Learning to Generate Human-Human-Object Interactions from Textual Descriptions

GIRAF: Towards Generalizable Human Interactions with Articulated Objects

Dex2HOI: Dexterous Bimanual Two-Object Interaction Generation