Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis

Liang Xu; Chengqun Yang; Zili Lin; Xintao Lv; Yichao Yan; Xin Jin; Zhibo Chen; Xiaokang Yang

cs.CVAug 20, 2026

Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis

Authors: Liang Xu, Chengqun Yang, Zili Lin, Xintao Lv, Yichao Yan, Xin Jin, Zhibo Chen, Xiaokang Yang,

Organizations: Shanghai Jiao Tong University · Eastern Institute of Technology, Ningbo · University of Science and Technology of China

Abstract

The capability to perceive and synthesize human-human interactions is fundamental to developing intelligent digital human systems. However, existing datasets and modeling approaches are fundamentally constrained by low-fidelity kinematics, the omission of dexterous hand gestures and a severe lack of rich multimodal annotations. Furthermore, fragmented interaction representations and inconsistent evaluation protocols also impede fair and rigorous benchmarking. To systematically address these bottlenecks, we present Inter-X++, a comprehensive and large-scale benchmark designed to empower versatile HHI analysis. Captured via a novel hybrid motion capture system, Inter-X++ provides 11,388 high-fidelity interaction sequences and over 8.1M frames, featuring precise whole-body movements and detailed finger articulations. Meanwhile, we enrich the data foundation with multifaceted annotations, including hierarchical fine-grained textual descriptions, interaction categories, causal interaction orders, the relationship and personality of the subjects, as well as vertex-level contact maps and physically regularized constraints. Leveraging these elaborate annotations, we formulate a unified testing ground comprising four categories of downstream tasks that symmetrically span both generative and perceptive paradigms. To eliminate benchmarking ambiguities, we systematically standardize the interaction representations and evaluation protocols. Finally, we go beyond dataset construction to propose OpenHHI, a single and unified HHI representation and modeling framework that jointly optimizes interaction reconstruction and semantic understanding. Extensive experiments reveal that OpenHHI achieves state-of-the-art performance on both generation and perception tasks. This definitively proves that our unified representation successfully bridges interaction understanding and generation simultaneously.

Figures & tables

Fig. 1: Overview of Inter-X++. We present Inter-X++ as a comprehensive benchmark for multimodal human-human interaction analysis structured across three dimensions of interaction data, multimodal annotations and unified representation. For interaction data, Inter-X++ contains high-fidelity MoCap results with vertex-level contact annotations while incorporating physical constraints. For multimodal annotations, we enrich Inter-X++ with hierarchical text descriptions, interaction categories, causal interaction order, relationship and personality annotations. We also propose a unified HHI representation named OpenHHI that is universally applicable to both generative and perceptive downstream tasks.

Dataset	Year	Sequences	Frames	Texts	Scheme	Modality	Hands	Asyn.	Rel.& Pst.	Physical	Contact
UMPM [ 8 ]	2011	36	400K	✗	MoCap	Skel.	✗	✗	✗	✗	✗
SBU Kinect [ 9 ]	2012	300	7.5K	✗	RGB+D	Skel.	✗	✗	✗	✗	✗
You2Me [ 22 ]	2020	42	77K	✗	RGB+D	Skel.	✗	✗	✗	✗	✗
NTU120 [ 23 ]	2019	8,276	462K	✗	RGB+D	Skel.	✗	✗	✗	✗	✗
Chi3D [ 10 ]	2020	373	63K	✗	MoCap	SMPL-X	✓	✗	✗	✗	✗
ExPI [ 24 ]	2022	115	30K	✗	mRGB	Skel.	✗	✗	✗	✗	✗

TABLE I: Dataset Comparisons. A comprehensive comparison between Inter-X++ and existing human-human interaction datasets. Sequences : Total number of interaction clips; Frames : Total number of 3D motion frames; Texts : The number of textual annotations; Scheme : Motion data acquisition strategy; Modality : Kinematic representation (“Skel.” denotes skeleton). Additionally, Hands , Asyn. , Rel.&Pst. , Physical , and Contact indicate the inclusion of articulated hand gestures, asymmetry annotations, interpersonal relationships and personalities, physical plausibility, and explicit contact annotations, respectively. ∗ denotes the integration of upgraded hierarchical language descriptions.

Fig. 2: The Inter-X++ whole-body hybrid MoCap system. (a). Spatial integration and alignment of the optical MoCap suit and the IMU-based gloves is achieved via a rigid triangular bracket of reflective markers; (b). Detailed view of the reflective marker configuration; (c). Temporal synchronization of the human body and hand kinematics within the unified whole-body motion capture system.

Fig. 3: Motion characteristics of Inter-X++. Our proposed Inter-X++ dataset is distinguished by the 1) Precise contact for close HHIs; 2) Dexterous hand gestures; 3) Diverse action and reaction patterns.

Method	R Precision $\uparrow$			FID $\downarrow$	MM Dist $\downarrow$	Diversity $\rightarrow$	MModality $\uparrow$
Method	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist $\downarrow$	Diversity $\rightarrow$	MModality $\uparrow$
Real	$0.448^{\pm 0.003}$	$0.639^{\pm 0.003}$	$0.743^{\pm 0.003}$	$0.001^{\pm 0.0001}$	$3.652^{\pm 0.009}$	$9.662^{\pm 0.078}$	-
Axis-angle	${0.302}^{\pm{0.004}}$	${0.464}^{\pm{0.004}}$	${0.570}^{\pm{0.004}}$	${3.702}^{\pm{0.0855}}$	${4.807}^{\pm{0.0160}}$	${8.452}^{\pm{0.057}}$	${2.559}^{\pm{0.078}}$
Quaternion	${0.349}^{\pm{0.005}}$	${0.533}^{\pm{0.005}}$	$\underline{0.652^{\pm 0.005}}$	$\mathbf{0.748^{\pm 0.0323}}$	$\underline{4.293^{\pm 0.0131}}$	$\mathbf{9.797^{\pm 0.064}}$	$\mathbf{2.891^{\pm 0.091}}$
Rotation matrix	${0.105}^{\pm{0.001}}$	${0.193}^{\pm{0.003}}$	${0.267}^{\pm{0.002}}$	${91.074}^{\pm{0.1451}}$	${10.190}^{\pm{0.0120}}$	${3.385}^{\pm{0.053}}$	${1.370}^{\pm{0.037}}$
6D rotation	$\mathbf{0.389^{\pm 0.004}}$	$\mathbf{0.577^{\pm 0.005}}$	$\mathbf{0.689^{\pm 0.004}}$	$\underline{1.195^{\pm 0.0320}}$	$\mathbf{3.963^{\pm 0.0172}}$	$\underline{9.906^{\pm 0.075}}$	${2.695}^{\pm{0.090}}$

TABLE II: Experimental comparison of the human-human interaction representations on Inter-X++. We evaluate five interaction representations under a unified experimental protocol.

\pm

indicates 95% confidence interval and

\rightarrow

means the closer the better. Bold indicates best results and underline indicates second best results.

Method	Succ(%) $\uparrow$	$E_{\text{g-mpjpe}}$ $\downarrow$	$E_{\text{mpjpe}}$ $\downarrow$	$\text{E}_{\text{acc}}$ $\downarrow$	$\text{E}_{\text{vel}}$ $\downarrow$
PHC-Base [ 96 ]	76.79	70.26	70.19	14.51	13.34
PHC* (Epoch 6,000)	80.17	61.68	61.97	11.92	11.87
PHC* (Epoch 7,500)	80.21	65.29	63.64	12.33	12.27
PHC* (Epoch 9,000)	81.30	58.58	61.95	11.72	11.75
PHC* (Epoch 10,500)	81.20	59.73	62.76	12.11	12.06

TABLE III: Quantitative results of interaction imitation on the Inter-X++ dataset. PHC* means that we fine-tune the PHC-Base [ 96 ] model on the Inter-X++ dataset with different epochs.

Method	R Precision $\uparrow$			FID $\downarrow$	MM Dist $\downarrow$	Diversity $\rightarrow$	MModality $\uparrow$
Method	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist $\downarrow$	Diversity $\rightarrow$	MModality $\uparrow$
Real	$0.448^{\pm 0.003}$	$0.639^{\pm 0.003}$	$0.743^{\pm 0.003}$	$0.001^{\pm 0.0001}$	$3.652^{\pm 0.009}$	$9.662^{\pm 0.078}$	-
TEMOS [ 43 ]	$0.092^{\pm 0.003}$	$0.171^{\pm 0.003}$	$0.238^{\pm 0.002}$	$29.258^{\pm 0.0694}$	$6.867^{\pm 0.013}$	$4.738^{\pm 0.078}$	$0.672^{\pm 0.041}$
T2M [ 34 ]	$0.184^{\pm 0.010}$	$0.298^{\pm 0.006}$	$0.396^{\pm 0.005}$	$5.481^{\pm 0.3820}$	$9.576^{\pm 0.006}$	$5.771^{\pm 0.151}$	$2.761^{\pm 0.042}$
MDM [ 42 ]	$0.203^{\pm 0.009}$	$0.329^{\pm 0.007}$	$0.426^{\pm 0.005}$	$23.701^{\pm 0.0569}$	$9.548^{\pm 0.014}$	$5.856^{\pm 0.077}$	$\underline{3.490^{\pm 0.061}}$
MDM(GRU) [ 42 ]	$0.179^{\pm 0.006}$	$0.299^{\pm 0.005}$	$0.387^{\pm 0.007}$	$32.617^{\pm 0.1221}$	$9.557^{\pm 0.019}$	$7.003^{\pm 0.134}$	$3.430^{\pm 0.035}$

TABLE IV: Experimental results of text-conditioned HHI generation on the Inter-X++ dataset, where

\pm

indicates 95% confidence interval and

\rightarrow

means the closer the better. * denotes our reproduced results based on the official codebase.

Fig. 4: Visualization results of text-conditioned human-human interaction generation on the Inter-X++ dataset. Text is provided as the input condition, yielding interaction sequences as the generated output.

Method	FID $\downarrow$	Acc. $\uparrow$	Div. $\rightarrow$	Multimod. $\rightarrow$
Real	$0.281^{\pm 0.002}$	$0.990^{\pm 0.0000}$	$12.890^{\pm 0.028}$	$22.391^{\pm 0.195}$
Action2Motion [ 40 ]	$20.295^{\pm 12.081}$	$0.766^{\pm 0.0003}$	$11.581^{\pm 0.024}$	$15.345^{\pm 0.245}$
ACTOR [ 41 ]	$9.392^{\pm 0.816}$	$0.855^{\pm 0.0003}$	$11.594^{\pm 0.029}$	$15.327^{\pm 0.195}$
MDM [ 42 ]	$12.426^{\pm 2.584}$	$0.896^{\pm 0.0004}$	$13.492^{\pm 0.033}$	$\mathbf{22.042^{\pm 0.153}}$
MDM(GRU) [ 42 ]	$35.003^{\pm 7.876}$	$0.716^{\pm 0.0006}$	$\underline{12.579^{\pm 0.038}}$	$16.456^{\pm 0.100}$
Actformer [ 6 ]	$\underline{8.067^{\pm 0.653}}$	$\underline{0.945^{\pm 0.0007}}$	$12.512^{\pm 0.05}$	$16.187^{\pm 0.189}$

TABLE V: Experimental results of action-conditioned interaction generation on the Inter-X++ dataset.

Method	FID $\downarrow$	Acc. $\uparrow$	Div. $\rightarrow$	Multimod. $\rightarrow$
Real	$0.260^{\pm 0.0021}$	$0.988^{\pm 0.0000}$	$12.115^{\pm 0.031}$	$21.498^{\pm 0.131}$
MDM [ 42 ]	$6.747^{\pm 0.3153}$	$0.903^{\pm 0.0001}$	$12.264^{\pm 0.051}$	$19.681^{\pm 0.234}$
MDM(GRU) [ 42 ]	$19.968^{\pm 1.1700}$	$0.752^{\pm 0.0003}$	$12.351^{\pm 0.049}$	$18.056^{\pm 0.156}$
RAIG [ 20 ]	$6.372^{\pm 0.2154}$	$0.908^{\pm 0.0001}$	$12.330^{\pm 0.060}$	$\underline{20.071^{\pm 0.299}}$
AGRoL [ 105 ]	$\underline{4.386^{\pm 0.2186}}$	$\underline{0.925^{\pm 0.0001}}$	$\underline{12.204^{\pm 0.042}}$	$\mathbf{20.199^{\pm 0.226}}$
OpenHHI (Ours)	$\mathbf{3.172^{\pm 0.1645}}$	$\mathbf{0.947^{\pm 0.0001}}$	$\mathbf{12.132^{\pm 0.039}}$	$18.873^{\pm 0.185}$

TABLE VI: Experimental results of human reaction generation based on action labels on the Inter-X++ dataset.

Method	Top-1 (%)	Top-5 (%)
ST-GCN [ 64 ]	64.62	90.16
2s-AGCN [ 65 ]	75.22	93.73
HD-GCN [ 68 ]	77.40	94.73
CTR-GCN [ 67 ]	82.19	96.72
MS-G3D [ 66 ]	83.30	97.09
OpenHHI (Ours)	84.88	97.13

TABLE VII: Experimental results of skeleton-based human-human interaction recognition on the Inter-X++ dataset.

Method	Bleu@1 $\uparrow$	Bleu@4 $\uparrow$	Rouge $\uparrow$	Cider $\uparrow$	BertScore $\uparrow$
RAEs [ 106 ]	28.6	9.7	34.1	25.9	10.2
Seq2Seq [ 107 ]	53.8	18.5	45.2	61.9	27.1
SeqGAN [ 108 ]	45.4	14.1	36.8	52.3	21.4
TM2T [ 17 ]	56.8	21.6	48.2	75.5	32.7
OpenHHI (Ours)	70.7	35.4	48.9	49.1	35.1

TABLE VIII: Experimental results of human-human interaction captioning on the Inter-X++ dataset.

Fig. 5: Qualitative results of human-human interaction captioning on Inter-X++. Each example presents a sampled interaction sequence together with its ground-truth description and the caption generated by OpenHHI. The predictions coherently describe the coordinated behaviors of both participants, including their roles, body movements, and interaction details.

Method	Accuracy (%)
ST-GCN [ 64 ]	62.3
2s-AGCN [ 65 ]	68.2
HD-GCN [ 68 ]	70.6
CTR-GCN [ 67 ]	74.5
MS-G3D [ 66 ]	76.8
OpenHHI (Ours)	79.1

TABLE IX: Experimental results of causal interaction order inference on the Inter-X++ dataset.

Method	FID $\downarrow$	Acc. $\uparrow$	Div. $\rightarrow$	Multimod. $\rightarrow$
Real	$0.281^{\pm 0.002}$	$0.990^{\pm 0.0000}$	$12.890^{\pm 0.028}$	$22.391^{\pm 0.195}$
Action2Motion [ 40 ]	$21.182^{\pm 13.319}$	$0.737^{\pm 0.0005}$	$11.492^{\pm 0.032}$	$14.934^{\pm 0.258}$
ACTOR [ 41 ]	$9.796^{\pm 0.862}$	$0.867^{\pm 0.0003}$	$11.862^{\pm 0.039}$	$15.174^{\pm 0.245}$
MDM [ 42 ]	$11.762^{\pm 1.854}$	$0.912^{\pm 0.0002}$	$\underline{13.025^{\pm 0.028}}$	$\mathbf{21.742^{\pm 0.106}}$
MDM(GRU) [ 42 ]	$31.688^{\pm 4.492}$	$0.753^{\pm 0.0006}$	$12.259^{\pm 0.039}$	$16.271^{\pm 0.206}$
Actformer [ 6 ]	$\underline{8.544^{\pm 0.684}}$	$\underline{0.932^{\pm 0.0006}}$	$12.116^{\pm 0.062}$	$16.122^{\pm 0.183}$

TABLE X: Experimental results of action-conditioned stylized human interaction generation on the Inter-X++ dataset.

Method	Openness	Conscientiousness	Extraversion	Agreeableness	Neuroticism
ST-GCN [ 64 ]	21.16	25.38	34.91	23.67	13.02
2s-AGCN [ 65 ]	23.46	31.27	38.72	24.88	13.57
HD-GCN [ 68 ]	25.92	33.19	41.33	26.83	14.29
CTR-GCN [ 67 ]	27.78	35.41	43.52	29.43	15.63
MS-G3D [ 66 ]	28.36	37.88	46.23	29.07	16.35
OpenHHI (Ours)	30.74	40.12	48.67	31.05	18.21

TABLE XI: The

R^{2}

results (%) of the personality assessment on the Inter-X++ dataset.

$\rho$	R Precision $\uparrow$			FID $\downarrow$	MM Dist $\downarrow$	Diversity $\rightarrow$	MModality $\uparrow$
$\rho$	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist $\downarrow$	Diversity $\rightarrow$	MModality $\uparrow$
Real	$0.448^{\pm 0.003}$	$0.639^{\pm 0.003}$	$0.743^{\pm 0.003}$	$0.001^{\pm 0.0001}$	$3.652^{\pm 0.009}$	$9.662^{\pm 0.078}$	-
InterMask* [ 52 ]	$0.389^{\pm 0.004}$	$0.577^{\pm 0.005}$	$0.689^{\pm 0.004}$	$1.195^{\pm 0.0320}$	$3.963^{\pm 0.0172}$	$9.906^{\pm 0.075}$	$\mathbf{2.695^{\pm 0.090}}$
$0.1$	$0.422^{\pm 0.005}$	$0.613^{\pm 0.004}$	$0.722^{\pm 0.004}$	$\mathbf{0.366^{\pm 0.0183}}$	$3.660^{\pm 0.018}$	$9.616^{\pm 0.085}$	$\underline{1.901^{\pm 0.057}}$
$0.5$	$\mathbf{0.437^{\pm 0.005}}$	$\mathbf{0.627^{\pm 0.004}}$	$\underline{0.732^{\pm 0.005}}$	$0.573^{\pm 0.0209}$	$\mathbf{3.599^{\pm 0.017}}$	$\mathbf{9.685^{\pm 0.085}}$	$1.796^{\pm 0.071}$
$1.0$	$\underline{0.429^{\pm 0.005}}$	$\underline{0.624^{\pm 0.005}}$	$\mathbf{0.733^{\pm 0.005}}$	$\underline{0.480^{\pm 0.0196}}$	$\underline{3.602^{\pm 0.021}}$	$9.748^{\pm 0.084}$	$1.784^{\pm 0.050}$

TABLE XII: Ablation of the relative weighting between the reconstruction and interaction-understanding objectives in OpenHHI for Level-3 text-conditioned HHI generation. We fix

\omega_{\mathrm{rec}}=1

and vary

\rho=\omega_{\mathrm{und}}/\omega_{\mathrm{rec}}

. InterMask* denotes our reproduced baseline using the official implementation. Bold and underlined values indicate the best and second-best results, respectively.

Texts	Method	R Precision $\uparrow$			FID $\downarrow$	MM Dist $\downarrow$	Diversity $\rightarrow$	MModality $\uparrow$
Texts	Method	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist $\downarrow$	Diversity $\rightarrow$	MModality $\uparrow$
Level-1	Real	$0.394^{\pm 0.003}$	$0.584^{\pm 0.004}$	$0.694^{\pm 0.003}$	$0.001^{\pm 0.0001}$	$3.719^{\pm 0.011}$	$9.013^{\pm 0.072}$	-
Level-1	OpenHHI	$\mathbf{0.376^{\pm 0.004}}$	$\mathbf{0.549^{\pm 0.004}}$	$\mathbf{0.647^{\pm 0.003}}$	$\mathbf{2.654^{\pm 0.0516}}$	$\mathbf{4.187^{\pm 0.018}}$	$\mathbf{8.060^{\pm 0.070}}$	$2.086^{\pm 0.074}$
Level-2	Real	$0.398^{\pm 0.004}$	$0.591^{\pm 0.004}$	$0.699^{\pm 0.003}$	$0.001^{\pm 0.0001}$	$3.704^{\pm 0.011}$	$8.819^{\pm 0.076}$	-
Level-2	OpenHHI	$\mathbf{0.384^{\pm 0.004}}$	$\mathbf{0.566^{\pm 0.004}}$	$\mathbf{0.669^{\pm 0.004}}$	$\mathbf{2.774^{\pm 0.0625}}$	$\mathbf{4.053^{\pm 0.015}}$	$\mathbf{8.401^{\pm 0.076}}$	$2.036^{\pm 0.057}$
Level-3	Real	$0.448^{\pm 0.003}$	$0.639^{\pm 0.003}$	$0.743^{\pm 0.003}$	$0.001^{\pm 0.0001}$	$3.652^{\pm 0.009}$	$9.662^{\pm 0.078}$	-

TABLE XIII: Quantitative results of OpenHHI for text-conditioned HHI generation using hierarchical descriptions of different granularities. “Real” reports the statistics of ground-truth motions paired with the corresponding text level.

Fig. 6: Qualitative comparison of HHI generation conditioned on hierarchical descriptions from Inter-X++. Each column presents motions generated for the same interaction at three levels of textual granularity. Coarser prompts permit broader motion variations, whereas finer descriptions provide more precise control.

A01: Hug	A02: Handshake	A03: Wave	A04: Grab
A05: Hit	A06: Kick	A07: Posing	A08: Push
A09: Pull	A10: Sit on leg	A11: Slap	A12: Pat on back
A13: Point finger at	A14: Walk towards	A15: Knock over	A16: Step on foot
A17: High-five	A18: Chase	A19: Whisper in ear	A20: Support with hand
A21: Finger-guessing	A22: Dance	A23: Link arms	A24: Shoulder to shoulder
A25: Bend	A26: Carry on back	A27: Massage shoulder	A28: Massage leg

TABLE XIV: The 40 common daily human-human interaction categories included in the Inter-X++ dataset.

Fig. 7: Examples of the hierarchical textual annotations in Inter-X++. For each interaction, Level-1 gives a concise action summary, Level-2 describes the overall interaction in greater detail, and Level-3 additionally specifies participant-specific movements, body parts, and spatial relations.

Fig. 8: Additional qualitative results of text-conditioned human-human interaction generation on Inter-X++. Each example shows a generated interaction sequence together with its input description.

Fig. 9: Representative failure cases of text-conditioned human-human interaction generation. The top row highlights frames with inaccurate or physically implausible contacts using red boxes, while the bottom row illustrates inaccurate grounding of spatial and body-part constraints, which are emphasized in red in the input descriptions.

Fig. 10: Additional qualitative results of human-human interaction captioning on Inter-X++. Each example presents a sampled interaction sequence together with its ground-truth description and the caption generated by OpenHHI.

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 284 papers

Nov 25, 2025

Learning to Generate Human-Human-Object Interactions from Textual Descriptions

The way humans interact with each other, including interpersonal distances, spatial configuration, and motion, varies significantly across different situations. To enable machines to understand such complex, context-dependent behaviors, it is essential to model multiple people in relation to the surrounding scene context. In this paper, we present a novel research problem to model the correlations between two people engaged in a shared interaction involving an object. We refer to this formulation as Human-Human-Object Interactions (HHOIs). To overcome the lack of dedicated datasets for HHOIs, we present a newly captured HHOIs dataset and a method to synthesize HHOI data by leveraging image generative models. As an intermediary, we obtain individual human-object interaction (HOIs) and human-human interaction (HHIs) from the HHOIs, and with these data, we train an text-to-HOI and text-to-HHI model using score-based diffusion model. Finally, we present a unified generative framework that integrates the two individual model, capable of synthesizing complete HHOIs in a single advanced sampling process. Our method extends HHOI generation to multi-human settings, enabling interactions involving more than two individuals. Experimental results show that our method generates realistic HHOIs conditioned on textual descriptions, outperforming previous approaches that focus only on single-human HOIs. Furthermore, we introduce multi-human motion generation involving objects as an application of our framework.

Jeonghyeon Na, Sangwon Baik, Inhee Lee +2

Seoul National University

HOI Generation Human Motion Generation Human-Human-Object Interaction Generation Human-Object Interaction

Ask PDF

Mar 30, 2026

HandX: Scaling Bimanual Motion and Interaction Generation

Synthesizing human motion has advanced rapidly, yet realistic hand motion and bimanual interaction remain underexplored. Whole-body models often miss the fine-grained cues that drive dexterous behavior, finger articulation, contact timing, and inter-hand coordination, and existing resources lack high-fidelity bimanual sequences that capture nuanced finger dynamics and collaboration. To fill this gap, we present HandX, a unified foundation spanning data, annotation, and evaluation. We consolidate and filter existing datasets for quality, and collect a new motion-capture dataset targeting underrepresented bimanual interactions with detailed finger dynamics. For scalable annotation, we introduce a decoupled strategy that extracts representative motion features, e.g., contact events and finger flexion, and then leverages reasoning from large language models to produce fine-grained, semantically rich descriptions aligned with these features. Building on the resulting data and annotations, we benchmark diffusion and autoregressive models with versatile conditioning modes. Experiments demonstrate high-quality dexterous motion generation, supported by our newly proposed hand-focused metrics. We further observe clear scaling trends: larger models trained on larger, higher-quality datasets produce more semantically coherent bimanual motion. Our dataset is released to support future research.

Zimu Zhang, Yucheng Zhang, Xiyan Xu +8

University of Illinois Urbana-Champaign · Specs Inc. · Snap Inc.

Human Motion Generation Diffusion Models for Motion Generation

Ask PDF

Aug 17, 2026

HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object-Interaction

Humanoid intelligence requires learning over an extremely diverse space of whole-body motions and physically grounded interactions. However, existing embodied datasets remain fundamentally limited: internet-scale video data lack precise physical states and interaction grounding, while laboratory motion datasets provide high fidelity but only narrow behavioral coverage. This mismatch creates a critical bottleneck for scalable humanoid policy learning. We present HiPHI, a 600+ hour scale high-fidelity whole-body human motion dataset designed to systematically maximize coverage of the human motion and interaction manifold. HiPHI is theoretically guided by FrameNet, a linguistic framework organizing human primitives. Created using an optical motion capture pipeline, HiPHI provides sub-millimeter spatial marker tracking accuracy for full-body human motion and mesh-level object trajectories. We further introduce a benchmark suite evaluating motion-space diversity, interaction grounding, object consistency, and physical AI applications. Our analyses demonstrate that HiPHI significantly expands motion coverage compared to existing motion datasets while maintaining high-fidelity interaction quality, and establishes a scalable data foundation for training, evaluating, and generalizing humanoid policies in real-world embodied tasks, where similar extensions are also applicable to motion prior models in computer graphics. Project page: https://noitom-robotics.github.io/hiphi/

Jiahao Ji, Ji Ma, Runhan Zhang +12

Noitom Robotics · National University of Singapore · The University of Hong Kong +2

Humanoid Robotics Robot Learning Human-Object Interaction

Ask PDF

Abstract

Human Motion Generation

Explore similar work

Learning to Generate Human-Human-Object Interactions from Textual Descriptions

HandX: Scaling Bimanual Motion and Interaction Generation

HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object-Interaction