Synthesizing human motion has advanced rapidly, yet realistic hand motion and bimanual interaction remain underexplored. Whole-body models often miss the fine-grained cues that drive dexterous behavior, finger articulation, contact timing, and inter-hand coordination, and existing resources lack high-fidelity bimanual sequences that capture nuanced finger dynamics and collaboration. To fill this gap, we present HandX, a unified foundation spanning data, annotation, and evaluation. We consolidate and filter existing datasets for quality, and collect a new motion-capture dataset targeting underrepresented bimanual interactions with detailed finger dynamics. For scalable annotation, we introduce a decoupled strategy that extracts representative motion features, e.g., contact events and finger flexion, and then leverages reasoning from large language models to produce fine-grained, semantically rich descriptions aligned with these features. Building on the resulting data and annotations, we benchmark diffusion and autoregressive models with versatile conditioning modes. Experiments demonstrate high-quality dexterous motion generation, supported by our newly proposed hand-focused metrics. We further observe clear scaling trends: larger models trained on larger, higher-quality datasets produce more semantically coherent bimanual motion. Our dataset is released to support future research.
Figures & tables
Dataset
Duration (h)
Frames (M)
Text Granularity
Text (K)
Motion-X [ 28 ]
144.2
15.6
coarse
8.1
InterAct [ 64 ]
30.7
3.3
coarse
48.6
Hand motion datasets
BOTH2Hands [ 76 ]
8.31
1.8
coarse
23.5
HandDiffuse [ 29 ]
2.0
0.25
–
–
InterHand2.6M [ 38 ]
24.0
2.6
–
–
Table 1: Comparison of major hand motion datasets. Left: Dataset scale. Values are reported as HQ (Raw) where “HQ” denotes high-quality filtered data (Sec. A.4 ) and “Raw” (when available) indicates the original data. “Coarse” denotes short descriptions without articulation detail, while “action” denotes only categorical labels. HandX provides fine-grained, multi-level language descriptions. Right: Statistics of bimanual motion quality. Metrics are defined in Sec. A.5 . HandX provides contact-rich bimanual motions.
Figure 2: Two benchmark models. ( a ) Diffusion model. Text embeddings for the left hand, right hand, and bimanual interaction are separately cross-attended with noisy motion embeddings, and then fused through residual connections to predict denoised motion embeddings. ( b ) Autoregressive model, consisting of Finite Scalar Quantization (FSQ) and a text-prefix autoregressive model. Unlike the diffusion model, it concatenates the left-hand, right-hand, and bimanual text descriptions with separator tokens to form a text prefix, and formulates bimanual motion generation as a token prediction task over motion tokenized by FSQ.
Figure 3: Qualitative results of our unified framework, showing ( a ) high-fidelity text-to-motion generation with fine-grained articulation and contact, and ( b ) bimanual motion synthesis given versatile spatiotemporal conditions. Gray hands denote the input condition, green hands denote the generation, and orange hands denote the extended long-horizon generation.
Dataset Ratio
Decoder Layers
R-Precision ↑
FID ↓
Diversity →
Matching Dist ↓
Intra-hand Interaction ↑
Top 1
Top 2
Top 3
Cprec
Crec
CF1
Ground Truth
0.854±0.094
0.925±0.059
0.948±0.043
0.000±0.000
6.887±0.078
4.360±0.373
0.984±0.000
0.984±0.000
0.984±0.000
0.05
4
0.142±0.059
0.228±0.068
0.296±0.071
2.574 ±0.109
7.406 ±0.069
6.581 ±0.158
0.628 ±0.005
0.447 ±0.006
0.523 ±0.006
0.05
8
0.223 ±0.071
0.334 ±0.077
0.417 ±0.079
1.548 ±0.048
7.161 ±0.054
6.100 ±0.159
0.659 ±0.004
0.576 ±0.006
0.615 ±0.005
0.05
12
0.343 ±0.094
0.485 ±0.092
0.573 ±0.086
1.837 ±0.033
5.935 ±0.016
5.331±0.165
0.701 ±0.002
0.553 ±0.003
0.618 ±0.002
0.2
4
0.145 ±0.062
0.233 ±0.073
0.301 ±0.080
2.181 ±0.051
6.903 ±0.064
6.412 ±0.238
0.703 ±0.008
0.379 ±0.008
0.493 ±0.008
Table 2: Ablation study on model size and data size. For R-precision, we adopt a batch size of 32. We observe clear scaling trends for our primary metrics, e.g . , R-Precision improves consistently as we scale both data and model sizes, while Intra-hand CF1 shows a strong positive trend, culminating in the best performance with 12 layers of decoder and all training data.
Model Size(M)
Codebook Size
R-Precision ↑
FID ↓
Diversity →
Matching Dist ↓
Intra-hand Interaction ↑
Top 1
Top 2
Top 3
Cprec
Crec
CF1
Ground Truth
0.854
0.925
0.948
0.000
6.887
4.360
0.984
0.984
0.984
4.63
512
0.366
0.495
0.569
8.377
5.504
5.440
0.935
0.357
0.514
26.33
512
0.277
0.401
0.495
5.071
5.872
5.556
0.850
0.422
0.561
29.63
512
0.210
0.327
0.402
4.683
6.031
5.828
0.795
0.419
0.545
38.95
512
0.285
0.398
0.480
5.131
5.985
5.591
0.841
0.408
0.546
Table 3: Ablation study on the codebook size of FSQ and the model size of autoregressive models. For R-precision, we adopt a batch size of 32. Both the FSQ and autoregressive models are trained on the full training dataset. The primary metrics, e.g . , FID, achieve the best performance when both model capacity and codebook size are scaled up. In contrast, scaling only one while keeping the other fixed can degrade performance. For example, R-precision is highest when the autoregressive model size is comparable to the codebook size.
Dataset Ratio
Number of Decoder Layers
R-Precision ↑
FID ↓
Diversity →
Matching Dist ↓
Intra-hand Interaction ↑
Top 1
Top 2
Top 3
Cprec
Crec
CF1
Ground Truth
0.854 ±0.094
0.925 ±0.059
0.948 ±0.043
0.000 ±0.000
6.887 ±0.078
4.360 ±0.373
0.984 ±0.000
0.984 ±0.000
0.984 ±0.000
0.05
4
0.142 ±0.059
0.228 ±0.068
0.296 ±0.071
2.574 ±0.109
7.406 ±0.069
6.581 ±0.158
0.628 ±0.005
0.447 ±0.006
0.523 ±0.006
0.05
8
0.223 ±0.071
0.334 ±0.077
0.417 ±0.079
1.548 ±0.048
7.161 ±0.054
6.100 ±0.159
0.659 ±0.004
0.576 ±0.006
0.615 ±0.005
0.05
12
0.343 ±0.094
0.485 ±0.092
0.573 ±0.086
1.837 ±0.033
5.935 ±0.016
5.331 ±0.165
0.701 ±0.002
0.553 ±0.003
0.618 ±0.002
0.2
8
0.262 ±0.077
0.388 ±0.081
0.466 ±0.084
3.053 ±0.099
7.581 ±0.037
6.242 ±0.164
0.739 ±0.003
0.460 ±0.009
0.567 ±0.006
Table 4: Ablation study of number of decoder layers of diffusion model and size of dataset. For R-precision, we adopt a batch size of 32.By comparing models trained on different ratio of our dataset and different decoder layer numbers, we observe clear scaling trends for our primary metrics. R-Precision improves consistently as we scale both data and model size, while Intra-hand CF1 shows a strong positive trend, culminating in the best performance at the largest scale, with 12 layers of decoder and all training data.
Model Size(M)
Codebook Size
R-Precision ↑
FID ↓
Diversity →
Matching Dist ↓
Intra-hand Interaction ↑
Top 1
Top 2
Top 3
Cprec
Crec
CF1
Ground Truth
0.854
0.925
0.948
0.000
6.887
4.360
0.984
0.984
0.984
29.63
512
0.210
0.327
0.402
4.683
6.031
5.828
0.795
0.419
0.545
29.63
1024
0.236
0.361
0.438
3.459
6.132
5.764
0.810
0.402
0.534
92.27
2048
0.182
0.288
0.354
2.949
6.156
5.882
0.694
0.493
0.574
92.27
4096
0.134
0.221
0.283
3.050
6.025
5.953
0.754
0.376
0.497
Table 5: Ablation study on the codebook size of FSQ and the model size of autoregressive models. For R-precision, we adopt a batch size of 32. Note that following previous works [ 37 , 13 ] , we adopt a deterministic (greedy) decoding strategy during inference, selecting at each step the token with the highest predicted probability. And the FSQ and autoregressive models are both trained on the full training dataset.
Figure 4: Scaling trend of computational scale. We observe a clear log-linear relationship between R-precision and FLOPS, with a high correlation coefficient of 0.96 . R-Precision is evaluated with a batch size of 16.
Figure 5: Qualitative comparison of diffusion models trained with different data scales . The model trained on the full dataset generates more expressive motion with better text alignment.
Figure 6: Qualitative comparison of diffusion models with different model scales . Larger models produce motion that is better aligned with the text and exhibits improved bimanual contact.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A : Our motion capture configuration: ( a ) 36-camera OptiTrack studio and ( b ) placement of 25 markers on each hand.
Figure B : Distribution of filtered clips. This figure shows the intensity distribution of the filtered clips.
Feature
Range
State
Finger Flexing ( ∘ )
[−180,−20)
Hyper extend
[−20,30)
Fully extend
[30,60)
Partially bent
[60,180)
Fully bent
Finger Spacing ( ∘ )
[0,20)
Closed
[20,180)
Open
Appendix
Table A : Quantitative state intervals for different hand motion features used in event segmentation.
Figure C : Structured representation of extracted motion features. The JSON format captures temporal segmentation with frame indices and semantic state descriptions for transition events and constant events.
Figure D : Our prompt design to facilitate LLMs to summarize kinematic features. This is for bimanual motion captioning.
Configuration
FLOPs (G)
R-Precision ↑
FID ↓
L
d
H
FFN
Top 1
Top 2
Top 3
2
256
4
1,024
3.83
0.117±0.072
0.210±0.088
0.295±0.091
5.342±0.073
4
256
4
1,024
4.22
0.160±0.085
0.264±0.098
0.355±0.107
2.705±0.057
4
384
6
1,536
5.22
0.185±0.087
0.298±0.059
0.389±0.092
2.053±0.033
8
384
6
1,536
6.98
0.262±0.101
0.407±0.112
0.501±0.115
1.959±0.066
8
512
8
2,048
9.74
0.318±0.117
0.463±0.124
0.566±0.124
1.830±0.033
Appendix
Table B : Ablation study on diffusion model for analyzing scaling curves. R-Precision is evaluated with a batch size of 16.
Figure E : User study on data quality . Our HandX exhibits high-quality motion and annotations. For annotation quality evaluation, we compare our method with a baseline that utilizes Gemini 3 Pro [ 51 ] to directly caption rendered motion videos.
Model
Layers
Latent dim
FF size
Trainable params
Diffusion
4
256
1,024
4.63M
8
512
1,024
26.33M
12
512
1,024
38.95M
16
768
3,072
260.97M
Autoregressive
8
512
512
29.63M
12
768
768
92.27M
Appendix
Table C : Detailed configurations of the diffusion and autoregressive models. Parameter counts exclude the frozen text encoder. Results from Figure 4 are extracted from this table.
Dataset Ratio
Layers
Inter-hand Cprec ↑
0.05
4
0.7310
0.05
8
0.7381
0.05
12
0.7111
0.2
8
0.7066
0.2
12
0.7971
1.0
4
0.7551
Appendix
Table D : Ablation study on inter-hand interaction. We report the inter-hand contact precision ( Cprec ) on diffusion model across different dataset scales and model depths.
Bimanual articulated-object interaction generation requires a model to capture the evolution of object articulation, coordination between the two hands, and fine-grained hand--object contact. Existing methods typically encode object and hand motion as a unified high-dimensional sequence, making it difficult to explicitly accommodate the different scales of manipulation progress, relative bimanual motion, and local hand pose. We propose DuetHOI, a structured framework for bimanual articulated-object interaction generation. DuetHOI first uses ContactVAE to predict contact intent on the object surface from language and object geometry. Conditioned on this spatial intent, ArtPlanner provides a trajectory-level articulation reference, and DualFormer generates the global interaction using object states, object-centered hand positions, and compact hand-pose tokens learned by ManiVAE. After global generation, ProxiRefine fixes the object trajectory and residually corrects the two hands using current hand--object proximity, improving local surface alignment. Experiments across articulated and rigid objects and single- and bimanual settings show that DuetHOI outperforms three adapted baselines on most contact and hand--object consistency metrics, with particularly strong results in bimanual articulated interaction generation.
Zhi Wang, Yuyan Liu, Liu liu +2
Hefei University of Technology · Shanghai Jiao Tong University · Anhui University
Recent advances in 4D Human-Object Interaction (HOI) generation have enabled increasingly realistic motion synthesis, particularly for single-object manipulation. Yet current research overlooks an inherent property of human behavior: people naturally coordinate both hands and manipulate multiple objects simultaneously. To address this gap, we present Dex2HOI, a unified diffusion model for single- and two-object HOI synthesis from text. At its core, Dex2HOI employs a Dual-Stream Diffusion approach, where each object is processed in a dedicated interaction stream and coordinated through bidirectional cross-attention. To synthesize the final motion, we introduce a Motion Fusion Network integrated with novel hand-relative object representations and contact-aware conditioning applied across the whole sequence. By sampling the diffusion process autoregressively over prefix-conditioned windows, Dex2HOI generates arbitrarily long sequences at real-time speed omitting redundant test-time optimization, achieving up to x540 inference speed-up over prior state-of-the-art methods. Extensive evaluation on both single- and two-object benchmarks demonstrates state-of-the-art quantitative results, marking a step beyond conventional single-object HOI generation and toward expressive multi-object manipulation. Project: https://cpratikaki.github.io/dex2hoi/
Chrysa Pratikaki, Pablo Ruiz-Ponce, Jiankang Deng +2
Imperial College London, UK · University of Alicante, Spain
The capability to perceive and synthesize human-human interactions is fundamental to developing intelligent digital human systems. However, existing datasets and modeling approaches are fundamentally constrained by low-fidelity kinematics, the omission of dexterous hand gestures and a severe lack of rich multimodal annotations. Furthermore, fragmented interaction representations and inconsistent evaluation protocols also impede fair and rigorous benchmarking. To systematically address these bottlenecks, we present Inter-X++, a comprehensive and large-scale benchmark designed to empower versatile HHI analysis. Captured via a novel hybrid motion capture system, Inter-X++ provides 11,388 high-fidelity interaction sequences and over 8.1M frames, featuring precise whole-body movements and detailed finger articulations. Meanwhile, we enrich the data foundation with multifaceted annotations, including hierarchical fine-grained textual descriptions, interaction categories, causal interaction orders, the relationship and personality of the subjects, as well as vertex-level contact maps and physically regularized constraints. Leveraging these elaborate annotations, we formulate a unified testing ground comprising four categories of downstream tasks that symmetrically span both generative and perceptive paradigms. To eliminate benchmarking ambiguities, we systematically standardize the interaction representations and evaluation protocols. Finally, we go beyond dataset construction to propose OpenHHI, a single and unified HHI representation and modeling framework that jointly optimizes interaction reconstruction and semantic understanding. Extensive experiments reveal that OpenHHI achieves state-of-the-art performance on both generation and perception tasks. This definitively proves that our unified representation successfully bridges interaction understanding and generation simultaneously.
Liang Xu, Chengqun Yang, Zili Lin +6
Shanghai Jiao Tong University · Eastern Institute of Technology, Ningbo · University of Science and Technology of China