HandX: Scaling Bimanual Motion and Interaction Generation

Zimu Zhang; Yucheng Zhang; Xiyan Xu; Ziyin Wang; Sirui Xu; Kai Zhou; Bing Zhou; Chuan Guo

cs.CVMar 30, 2026

HandX: Scaling Bimanual Motion and Interaction Generation

Authors: Zimu Zhang, Yucheng Zhang, Xiyan Xu, Ziyin Wang, Sirui Xu, Kai Zhou, Bing Zhou, Chuan Guo,

Organizations: University of Illinois Urbana-Champaign · Specs Inc. · Snap Inc.

Abstract

Synthesizing human motion has advanced rapidly, yet realistic hand motion and bimanual interaction remain underexplored. Whole-body models often miss the fine-grained cues that drive dexterous behavior, finger articulation, contact timing, and inter-hand coordination, and existing resources lack high-fidelity bimanual sequences that capture nuanced finger dynamics and collaboration. To fill this gap, we present HandX, a unified foundation spanning data, annotation, and evaluation. We consolidate and filter existing datasets for quality, and collect a new motion-capture dataset targeting underrepresented bimanual interactions with detailed finger dynamics. For scalable annotation, we introduce a decoupled strategy that extracts representative motion features, e.g., contact events and finger flexion, and then leverages reasoning from large language models to produce fine-grained, semantically rich descriptions aligned with these features. Building on the resulting data and annotations, we benchmark diffusion and autoregressive models with versatile conditioning modes. Experiments demonstrate high-quality dexterous motion generation, supported by our newly proposed hand-focused metrics. We further observe clear scaling trends: larger models trained on larger, higher-quality datasets produce more semantically coherent bimanual motion. Our dataset is released to support future research.

Figures & tables

Dataset	Duration (h)	Frames (M)	Text Granularity	Text (K)
Motion-X [ 28 ]	144.2	15.6	coarse	8.1
InterAct [ 64 ]	30.7	3.3	coarse	48.6
Hand motion datasets
BOTH2Hands [ 76 ]	8.31	1.8	coarse	23.5
HandDiffuse [ 29 ]	2.0	0.25	–	–
InterHand2.6M [ 38 ]	24.0	2.6	–	–

Table 1: Comparison of major hand motion datasets. Left: Dataset scale. Values are reported as HQ (Raw) where “HQ” denotes high-quality filtered data (Sec. A.4 ) and “Raw” (when available) indicates the original data. “Coarse” denotes short descriptions without articulation detail, while “action” denotes only categorical labels. HandX provides fine-grained, multi-level language descriptions. Right: Statistics of bimanual motion quality. Metrics are defined in Sec. A.5 . HandX provides contact-rich bimanual motions.

Figure 2: Two benchmark models. ( a ) Diffusion model. Text embeddings for the left hand, right hand, and bimanual interaction are separately cross-attended with noisy motion embeddings, and then fused through residual connections to predict denoised motion embeddings. ( b ) Autoregressive model, consisting of Finite Scalar Quantization (FSQ) and a text-prefix autoregressive model. Unlike the diffusion model, it concatenates the left-hand, right-hand, and bimanual text descriptions with separator tokens to form a text prefix, and formulates bimanual motion generation as a token prediction task over motion tokenized by FSQ.

Figure 3: Qualitative results of our unified framework, showing ( a ) high-fidelity text-to-motion generation with fine-grained articulation and contact, and ( b ) bimanual motion synthesis given versatile spatiotemporal conditions. Gray hands denote the input condition, green hands denote the generation, and orange hands denote the extended long-horizon generation.

Dataset Ratio	Decoder Layers	R-Precision ↑			FID ↓	Diversity →	Matching Dist ↓	Intra-hand Interaction ↑
Dataset Ratio	Decoder Layers	Top 1	Top 2	Top 3	FID ↓	Diversity →	Matching Dist ↓	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{\mathrm{F1}}$
Ground Truth		$0.854^{\pm 0.094}$	$0.925^{\pm 0.059}$	$0.948^{\pm 0.043}$	$0.000^{\pm 0.000}$	$6.887^{\pm 0.078}$	$4.360^{\pm 0.373}$	$0.984^{\pm 0.000}$	$0.984^{\pm 0.000}$	$0.984^{\pm 0.000}$
0.05	4	$0.142^{\pm 0.059}$	$0.228^{\pm 0.068}$	$0.296^{\pm 0.071}$	2.574 ±0.109	7.406 ±0.069	6.581 ±0.158	0.628 ±0.005	0.447 ±0.006	0.523 ±0.006
0.05	8	0.223 ±0.071	0.334 ±0.077	0.417 ±0.079	1.548 ±0.048	7.161 ±0.054	6.100 ±0.159	0.659 ±0.004	0.576 ±0.006	0.615 ±0.005
0.05	12	0.343 ±0.094	0.485 ±0.092	0.573 ±0.086	1.837 ±0.033	5.935 ±0.016	${\boldsymbol{5.331}}^{\pm{\boldsymbol{0.165}}}$	0.701 ±0.002	0.553 ±0.003	0.618 ±0.002
0.2	4	0.145 ±0.062	0.233 ±0.073	0.301 ±0.080	2.181 ±0.051	6.903 ±0.064	6.412 ±0.238	0.703 ±0.008	0.379 ±0.008	0.493 ±0.008

Table 2: Ablation study on model size and data size. For R-precision, we adopt a batch size of 32. We observe clear scaling trends for our primary metrics, e.g . , R-Precision improves consistently as we scale both data and model sizes, while Intra-hand

C_{\mathrm{F1}}

shows a strong positive trend, culminating in the best performance with 12 layers of decoder and all training data.

Model Size(M)	Codebook Size	R-Precision ↑			FID ↓	Diversity →	Matching Dist ↓	Intra-hand Interaction ↑
Model Size(M)	Codebook Size	Top 1	Top 2	Top 3	FID ↓	Diversity →	Matching Dist ↓	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{\mathrm{F1}}$
Ground Truth		0.854	0.925	0.948	0.000	6.887	4.360	0.984	0.984	0.984
4.63	512	0.366	0.495	0.569	8.377	5.504	5.440	0.935	0.357	0.514
26.33	512	0.277	0.401	0.495	5.071	5.872	5.556	0.850	0.422	0.561
29.63	512	0.210	0.327	0.402	4.683	6.031	5.828	0.795	0.419	0.545
38.95	512	0.285	0.398	0.480	5.131	5.985	5.591	0.841	0.408	0.546

Table 3: Ablation study on the codebook size of FSQ and the model size of autoregressive models. For R-precision, we adopt a batch size of 32. Both the FSQ and autoregressive models are trained on the full training dataset. The primary metrics, e.g . , FID, achieve the best performance when both model capacity and codebook size are scaled up. In contrast, scaling only one while keeping the other fixed can degrade performance. For example, R-precision is highest when the autoregressive model size is comparable to the codebook size.

Dataset Ratio	Number of Decoder Layers	R-Precision ↑			FID ↓	Diversity →	Matching Dist ↓	Intra-hand Interaction ↑
Dataset Ratio	Number of Decoder Layers	Top 1	Top 2	Top 3	FID ↓	Diversity →	Matching Dist ↓	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{\mathrm{F1}}$
Ground Truth		0.854 ±0.094	0.925 ±0.059	0.948 ±0.043	0.000 ±0.000	6.887 ±0.078	4.360 ±0.373	0.984 ±0.000	0.984 ±0.000	0.984 ±0.000
0.05	4	0.142 ±0.059	0.228 ±0.068	0.296 ±0.071	2.574 ±0.109	7.406 ±0.069	6.581 ±0.158	0.628 ±0.005	0.447 ±0.006	0.523 ±0.006
0.05	8	0.223 ±0.071	0.334 ±0.077	0.417 ±0.079	1.548 ±0.048	7.161 ±0.054	6.100 ±0.159	0.659 ±0.004	0.576 ±0.006	0.615 ±0.005
0.05	12	0.343 ±0.094	0.485 ±0.092	0.573 ±0.086	1.837 ±0.033	5.935 ±0.016	5.331 ${}^{\pm\textbf{0.165}}$	0.701 ±0.002	0.553 ±0.003	0.618 ±0.002
0.2	8	0.262 ±0.077	0.388 ±0.081	0.466 ±0.084	3.053 ±0.099	7.581 ±0.037	6.242 ±0.164	0.739 ${}^{\pm\textbf{0.003}}$	0.460 ±0.009	0.567 ±0.006

Table 4: Ablation study of number of decoder layers of diffusion model and size of dataset. For R-precision, we adopt a batch size of 32.By comparing models trained on different ratio of our dataset and different decoder layer numbers, we observe clear scaling trends for our primary metrics. R-Precision improves consistently as we scale both data and model size, while Intra-hand

C_{\mathrm{F1}}

shows a strong positive trend, culminating in the best performance at the largest scale, with 12 layers of decoder and all training data.

Model Size(M)	Codebook Size	R-Precision ↑			FID ↓	Diversity →	Matching Dist ↓	Intra-hand Interaction ↑
Model Size(M)	Codebook Size	Top 1	Top 2	Top 3	FID ↓	Diversity →	Matching Dist ↓	$C_{\mathrm{prec}}$	$C_{\mathrm{rec}}$	$C_{\mathrm{F1}}$
Ground Truth		0.854	0.925	0.948	0.000	6.887	4.360	0.984	0.984	0.984
29.63	512	0.210	0.327	0.402	4.683	6.031	5.828	0.795	0.419	0.545
29.63	1024	0.236	0.361	0.438	3.459	6.132	5.764	0.810	0.402	0.534
92.27	2048	0.182	0.288	0.354	2.949	6.156	5.882	0.694	0.493	0.574
92.27	4096	0.134	0.221	0.283	3.050	6.025	5.953	0.754	0.376	0.497

Table 5: Ablation study on the codebook size of FSQ and the model size of autoregressive models. For R-precision, we adopt a batch size of 32. Note that following previous works [ 37 , 13 ] , we adopt a deterministic (greedy) decoding strategy during inference, selecting at each step the token with the highest predicted probability. And the FSQ and autoregressive models are both trained on the full training dataset.

Figure 4: Scaling trend of computational scale. We observe a clear log-linear relationship between R-precision and FLOPS, with a high correlation coefficient of

0.96

. R-Precision is evaluated with a batch size of 16.

Panel of Figure 5 — Figure 5: Qualitative comparison of diffusion models trained with different data scales . The model trained on the full dataset generates more expressive motion with better text alignment.

Panel of Figure 6 — Figure 6: Qualitative comparison of diffusion models with different model scales . Larger models produce motion that is better aligned with the text and exhibits improved bimanual contact.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A : Our motion capture configuration: ( a ) 36-camera OptiTrack studio and ( b ) placement of 25 markers on each hand.

Figure B : Distribution of filtered clips. This figure shows the intensity distribution of the filtered clips.

Feature	Range	State
Finger Flexing ( ∘ )	$[-180,-20)$	Hyper extend
	$[-20,30)$	Fully extend
	$[30,60)$	Partially bent
	$[60,180)$	Fully bent
Finger Spacing ( ∘ )	$[0,20)$	Closed
Finger Spacing ( ∘ )	$[20,180)$	Open

Appendix

Table A : Quantitative state intervals for different hand motion features used in event segmentation.

Figure C : Structured representation of extracted motion features. The JSON format captures temporal segmentation with frame indices and semantic state descriptions for transition events and constant events.

Figure D : Our prompt design to facilitate LLMs to summarize kinematic features. This is for bimanual motion captioning.

Configuration				FLOPs (G)	R-Precision ↑			FID ↓
L	d	H	FFN	FLOPs (G)	Top 1	Top 2	Top 3	FID ↓
2	256	4	1,024	3.83	$0.117^{\pm 0.072}$	$0.210^{\pm 0.088}$	$0.295^{\pm 0.091}$	$5.342^{\pm 0.073}$
4	256	4	1,024	4.22	$0.160^{\pm 0.085}$	$0.264^{\pm 0.098}$	$0.355^{\pm 0.107}$	$2.705^{\pm 0.057}$
4	384	6	1,536	5.22	$0.185^{\pm 0.087}$	$0.298^{\pm 0.059}$	$0.389^{\pm 0.092}$	$2.053^{\pm 0.033}$
8	384	6	1,536	6.98	$0.262^{\pm 0.101}$	$0.407^{\pm 0.112}$	$0.501^{\pm 0.115}$	$1.959^{\pm 0.066}$
8	512	8	2,048	9.74	$0.318^{\pm 0.117}$	$0.463^{\pm 0.124}$	$0.566^{\pm 0.124}$	$1.830^{\pm 0.033}$

Appendix

Table B : Ablation study on diffusion model for analyzing scaling curves. R-Precision is evaluated with a batch size of 16.

Panel of Figure 17 — Figure E : User study on data quality . Our HandX exhibits high-quality motion and annotations. For annotation quality evaluation, we compare our method with a baseline that utilizes Gemini 3 Pro [ 51 ] to directly caption rendered motion videos.

Model	Layers	Latent dim	FF size	Trainable params
Diffusion	4	256	1,024	4.63M
	8	512	1,024	26.33M
	12	512	1,024	38.95M
	16	768	3,072	260.97M
Autoregressive	8	512	512	29.63M
	12	768	768	92.27M

Appendix

Table C : Detailed configurations of the diffusion and autoregressive models. Parameter counts exclude the frozen text encoder. Results from Figure 4 are extracted from this table.

Dataset Ratio	Layers	Inter-hand $C_{\mathrm{prec}}$ ↑
0.05	4	0.7310
0.05	8	0.7381
0.05	12	0.7111
0.2	8	0.7066
0.2	12	0.7971
1.0	4	0.7551

Appendix

Table D : Ablation study on inter-hand interaction. We report the inter-hand contact precision (

C_{\mathrm{prec}}

) on diffusion model across different dataset scales and model depths.

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 284 papers

DuetHOI: Language-Guided Bimanual Hand--Object Motion Generation with Articulation Planning and Contact Refinement
Mar 9, 2026Zhi Wang, Yuyan Liu, Liu liu +2HOI Generation
Hefei University of Technology · Shanghai Jiao Tong University · Anhui University
Dex2HOI: Dexterous Bimanual Two-Object Interaction Generation
May 28, 2026Chrysa Pratikaki, Pablo Ruiz-Ponce, Jiankang Deng +2HOI Generation Human Motion Generation
Imperial College London, UK · University of Alicante, Spain
Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis
Aug 20, 2026Liang Xu, Chengqun Yang, Zili Lin +6Human Motion Generation
Shanghai Jiao Tong University · Eastern Institute of Technology, Ningbo · University of Science and Technology of China

Mar 9, 2026J/K move · Enter open · S save

DuetHOI: Language-Guided Bimanual Hand--Object Motion Generation with Articulation Planning and Contact Refinement

Zhi Wang, Yuyan Liu, Liu liu, Ruonan Liu+1

Hefei University of Technology · Shanghai Jiao Tong University · Anhui University

HOI Generation

Open paper AskPDF arXiv

Bimanual articulated-object interaction generation requires a model to capture the evolution of object articulation, coordination between the two hands, and fine-grained hand--object contact. Existing methods typically encode object and hand motion as a unified high-dimensional sequence, making it difficult to explicitly accommodate the different scales of manipulation progress, relative bimanual motion, and local hand pose. We propose DuetHOI, a structured framework for bimanual articulated-object interaction generation. DuetHOI first uses ContactVAE to predict contact intent on the object surface from language and object geometry. Conditioned on this spatial intent, ArtPlanner provides a trajectory-level articulation reference, and DualFormer generates the global interaction using object states, object-centered hand positions, and compact hand-pose tokens learned by ManiVAE. After global generation, ProxiRefine fixes the object trajectory and residually corrects the two hands using current hand--object proximity, improving local surface alignment. Experiments across articulated and rigid objects and single- and bimanual settings show that DuetHOI outperforms three adapted baselines on most contact and hand--object consistency metrics, with particularly strong results in bimanual articulated interaction generation.

Abstract

Human Motion Generation

Explore similar work

DuetHOI: Language-Guided Bimanual Hand--Object Motion Generation with Articulation Planning and Contact Refinement

Dex2HOI: Dexterous Bimanual Two-Object Interaction Generation

Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis