SignMimic: Robust High-Quality Sign Language Motion Generation via Human-Shape-Oblivious Pose Transfer Guidance

Zhewen He; Junyi Yu; Haomian Huang; Zhenhua Li; Yi Fang

cs.CVSep 12, 2026

SignMimic: Robust High-Quality Sign Language Motion Generation via Human-Shape-Oblivious Pose Transfer Guidance

Authors: Zhewen He, Junyi Yu, Haomian Huang, Zhenhua Li, Yi Fang

Organizations: New York University Abu Dhabi · ChatSign Technology

Abstract

We study the challenge of sign language video mimicking: given a driving video and a single reference frame, synthesize a video where the target signer reproduces the source motion while preserving identity and linguistic form. Prior pipelines entangle rigid motion, non-rigid deformation, and view-dependent completion in a monolithic generator, causing handshape drift and spatio-temporal instability. We present SignMimic, which (i) applies a TNet-based model to study SE(3) rigid canonicalization to stabilize global pose, (ii) performs non-rigid adaptation in a canonical space to preserve fine-grained articulators (hands/face) and coarticulation via NIF2D, and (iii) uses Pose-MAE-style completion before conditional video diffusion. This factorization injects geometric and linguistic priors, yielding shape and spatio-temporal consistency. On several large-scale datasets (ASL 50K, How2Sign, CSL News), SignMimic achieves state-of-the-art-level performance on video quality, identity similarity, and frame continuity while also achieving minimal loss when performing back translation (SLT) on generated videos. Ablations confirm the role of rigid canonicalization, non-rigid adaptation, and completion. Code is available at https://anonymous.4open.science/r/UniSignMimicTurbo-6088; model checkpoints and video examples will be released.

Figures & tables

Figure 1 : An overview of the SignMimic pipeline. Our framework first extracts a pose sequence from a source video via DWPose. This pose is then adapted to a reference identity by our core retargeting modules (TNet & NIF2D) and refined by a Pose MAE to handle occlusions. Finally, a pre-trained generator (MimicMotion) synthesizes the output video. Blue (snowflake icon) indicates frozen, pre-trained models, while orange (flame icon) indicates trainable modules.

Loss	Applies to	Definition
Masked reconstruction $\mathcal{L}_{\text{recon}}$	All parts; masked & high-conf joints	$\displaystyle\mathcal{L}_{\text{recon}}=\frac{1}{\|\Omega\|}\sum_{(t,j)\in\Omega}\rho_{\delta}\!\big(\hat{X}_{t,j}-X_{t,j}\big)$
Velocity $\mathcal{L}_{\text{vel}}$	All parts; masked joints	$\displaystyle\mathcal{L}_{\text{vel}}=\frac{1}{\|\Omega_{v}\|}\sum_{(t,j)\in\Omega_{v}}\!\big\\|\Delta\hat{X}_{t,j}-\Delta X_{t,j}\big\\|_{2}^{2}$
Acceleration $\mathcal{L}_{\text{acc}}$	All parts; masked joints	$\displaystyle\mathcal{L}_{\text{acc}}=\frac{1}{\|\Omega_{a}\|}\sum_{(t,j)\in\Omega_{a}}\!\big\\|\Delta^{2}\hat{X}_{t,j}-\Delta^{2}X_{t,j}\big\\|_{2}^{2}$
Total variation $\mathcal{L}_{\text{tv}}$	All parts; masked joints (predictions only)	$\displaystyle\mathcal{L}_{\text{tv}}=\frac{1}{\|\Omega_{v}\|}\sum_{(t,j)\in\Omega_{v}}\!\big\\|\Delta\hat{X}_{t,j}\big\\|_{1}$
Body bone length $\mathcal{L}_{\text{bone}}$	Body stream; valid bones by subset	$\displaystyle\mathcal{L}_{\text{bone}}=\frac{1}{\sum_{t}\|\mathcal{B}^{\text{valid}}_{t}\|}\!\sum_{t}\!\sum_{(i,k)\in\mathcal{B}^{\text{valid}}_{t}}\Big(\,\\|\hat{X}_{t,i}-\hat{X}_{t,k}\\|_{2}-\\|X_{t,i}-X_{t,k}\\|_{2}\Big)^{2}$

Table 1: Pose MAE Losses

X,\hat{X}\in\mathbb{R}^{T^{\prime}\times V\times 2}

;

\Delta X_{t,j}=X_{t,j}-X_{t-1,j}

;

\Delta^{2}

is the second difference.

\Omega

is the set of masked & high-confidence joints;

\Omega_{v}{=}\{(t\!\geq\!2,j)\!\in\!\Omega\}

;

\Omega_{a}{=}\{(t\!\geq\!3,j)\!\in\!\Omega\}

\mathcal{B}^{\text{valid}}_{t}

are body limb pairs with both endpoints present per subset .

DataSet	Method	FID_VID ( $\downarrow$ )	FVD ( $\downarrow$ )	ID_COS ( $\uparrow$ )	PSNR_TS ( $\uparrow$ )	SSIM_TS ( $\uparrow$ )	$\Delta$ BLEU ( $\uparrow$ )
ASL 50K	SignMimic (Ours)	55.8082	471.3722	0.4451	28.9048	0.9634	-0.89
ASL 50K	SignMimic (w/o MAE)	60.5222	837.8924	0.4744	29.8137	0.9662
ASL 50K	SignMimic (w/o TNet)	97.2169	1222.6924	0.1451	24.6959	0.9214
ASL 50K	SignMimic (w/o NIF2D)	62.5249	678.2377	0.1759	25.8771	0.9299
ASL 50K	Mimicmotion (Benchmark)	111.3754	1309.0642	0.1523	23.7492	0.9038	-2.67
How2Sign	SignMimic (Ours)	64.3899	1139.6051	0.3892	25.8220	0.9582

Table 2: Quantitative results. We evaluate SignMimic against the previous SOTA model MimicMotion on four large-scale datasets: ASL 50K, How2Sign, CSL News and Phoenix-2014. Ablations on TNet, NIF2D, and PoseMAE confirm our contributions. Arrows indicate better directions, data with bold font indicates the best result.

Figure 2 : Qualitative results . The top row displays original video frames from three datasets: How2Sign, CSL News, and Phoenix-2014. The bottom row shows the corresponding mimicked results. Notably, all mimicked images were generated using only a single reference image. (Raw images from ASL 50K Dataset is not available for publication use due to copyright issues)

Figure 3 : Qualitative results of PoseMAE . This figures visualizing the ability of our PoseMAE . 1) Part-wise attention: (a) - (c) shows completion of body and hands, while (d) - (f) shows completion of hands and faces. 2) Different masking strategies, (a),(c),(e) shows spatial masking, while (b), (d), (f) shows temporal masking.

Experiment on Spatial Mask
Mask Parameters			Average Loss Components
Spatial	Temporal	Conf	Accel.	Bone	Recon.	Total	TV	Velocity
15	5	3	6.09e-4	2.09e-4	1.78e-4	8.87e-4	6.17e-2	2.11e-4
5	5	3	$3.62\text{\times}{10}^{-4}$	$1.80\text{\times}{10}^{-4}$	$1.46\text{\times}{10}^{-4}$	$5.75\text{\times}{10}^{-4}$	$3.71\text{\times}{10}^{-2}$	$1.24\text{\times}{10}^{-4}$
25	5	3	$7.55\text{\times}{10}^{-4}$	$2.13\text{\times}{10}^{-4}$	$1.79\text{\times}{10}^{-4}$	$1.08\text{\times}{10}^{-3}$	$7.91\text{\times}{10}^{-2}$	$2.62\text{\times}{10}^{-4}$
Experiment on Temporal Mask

Table 3 : Quantitive study of different masking settings of PoseMAE This table analyzes the impact of the spatial mask ratio, temporal mask ratio, and confidence mask threshold settings on model performance, measured by various loss components. Our baseline configuration is (Spatial=15, Temporal=5, Conf=3), highlighted in bold. Each subsequent section ablates a single parameter while holding the others constant to isolate its effect. All loss values are presented in scientific notation.

Dataset	FPS	Resolution	Role
ASL 50K	30	$640{\times}360$	pretrain & eval
How2Sign (FF)	25	$1280{\times}720$	eval & SLT
CSL News	30	$200{\times}170$	eval
Phoenix-2014	NA	$210{\times}260$	eval

Table 4: Datasets and basic preprocessing. All videos are decoded at a fixed frame rate (FPS) and resized so that the signer occupies the central region with sufficient background context. “FF” denotes the front-facing split.

Module	Input	Output
TNet (per part)	$P_{t}^{\text{part}}$	$2{\times}2$ affine $A_{p}$
Hier. Pose Grafting	$\hat{P}_{t},\hat{P}_{\text{ref}}$	$P_{t}^{\text{canon}}$
Identity encoder $E_{\text{id}}$	$\hat{P}_{\text{ref}}$	$z_{\text{ref}}\in\mathbb{R}^{d_{z}}$
NIF2D $f_{\theta}$	$p^{\text{canon}}_{i},z_{\text{ref}}$	$\Delta p_{i}$
Pose MAE encoder	pose tokens (per part)	latent sequence
Pose MAE decoder	mask tokens + encoder feats	reconstructed pose

Table 5: High-level architecture of each module. Depth / hidden sizes are representative and can be adjusted to the exact configuration used in the implementation (e.g.,

d_{z}

L_{e}

L_{d}

	TNet	NIF2D	Pose MAE
Optimizer	AdamW	AdamW	AdamW
LR	$1{\times}10^{-4}$	$2{\times}10^{-4}$	$1{\times}10^{-4}$
Batch	128 seq	128 clips	128 clips
Epochs / iters	100 epochs	100 epochs	80 epochs
Params (M)	0.1M	0.3M	22M

Table 6: Training hyperparameters and parameter counts for SignMimic modules. “Seq” denotes pose sequences; “clips” denotes driving-clip–level batches. Learning rates, batch sizes and parameter counts can be updated to match the actual implementation.

Method	FID ${}_{\text{VID}}$ $\downarrow$	FVD $\downarrow$	ID COS $\uparrow$	PSNR-TS $\uparrow$	SSIM-TS $\uparrow$
SignMimic (ours)	55.81	471.37	0.4451	28.90	0.9634
MimicMotion (2025)	111.38	1309.06	0.1523	23.75	0.9038
Animate Anyone (2024)	135.95	1499.15	0.2681	27.49	0.9345
Magic Pose (2024)	163.91	1315.78	0.1460	25.24	0.8976

Table 7: Additional benchmark comparison on ASL 50K. Lower is better for FID

{}_{\text{VID}}

and FVD; higher is better for ID COS, PSNR-TS, and SSIM-TS. The best result is highlighted in bond font.

Dataset	BLEU-1	BLEU-2	BLEU-3	BLEU-4	rBLEU	chrF2++	rchrF2++
How2Sign	57.8	26.8	18.3	9.9	8.4	40.6	20.3

Table 8: SLT evaluation results (corpus-level). rBLEU and rchrF2++ denote the reduced versions of BLEU and chrF2++.

Model	FID $\downarrow$	ID COS $\uparrow$	PSNR-TS $\uparrow$	SSIM-TS $\uparrow$
interp_0	52.7979	0.3409	28.5928	0.9643
interp_3	52.5100	0.3425	28.7500	0.9651
interp_6	52.5600	0.3454	28.8600	0.9655
interp_10	51.7726	0.3478	29.0488	0.9665

Table 9: Interpolation experiment on ASL 50K. Increasing interpolation strength from interp_0 to interp_10 yields small but consistent improvements in FID, identity preservation, and temporal reconstruction quality.

Explore similar work

SignRefine: Adapting Foundational Video Models for Sign Language Generation
Sep 8, 2026Anton Pelykh, Edward Fish, Ozge Mercanoglu Sincan +1Sign Language Translation Online Signature Verification
Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, UK
SignRR: Retrieve and Refine Real Motion for Sign Language Production
Aug 28, 2026Fidel Omar Tito Cruz, Angie Sanchez Marquina, Summy Farfan +1Sign Language Translation Object Articulation
University of Central Florida Florida, USA · Universidad Nacional Mayor de San Marcos Lima, Peru · Universidad Catolica San Pablo Arequipa, Peru +1
Physics-Informed Diffusion for Biomechanically Plausible 3D Sign Language Generation
Jul 16, 2026Emanuele Colonna, Moises Diaz, Gennaro Vessio +2Contrastive Alignment Motion
Department of Computer Science, University of Bari Aldo Moro, Bari, Italy · Institute for Technological Development and Innovation in Communications, University of Las Palmas de Gran Canaria, Las Palmas de Gran Canaria, Spain

Sep 8, 2026, cs.CVJ/K move · Enter open · S save

SignRefine: Adapting Foundational Video Models for Sign Language Generation

Anton Pelykh, Edward Fish, Ozge Mercanoglu Sincan, Richard Bowden

Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, UK

Sign Language Translation Online Signature Verification Video Foundation Models Object Articulation Refinevla

Open paper AskPDF arXiv

Sign language video generation demands precise hand and facial articulation, yet modern video diffusion models, trained predominantly on spoken-language video, produce artifacts that render signing unintelligible. We propose SignRefine, a sign language video generation model that produces comprehensible signing from 2D keypoint conditioning alone, generalizing across appearances and visual conditions. Our approach builds on a pretrained video diffusion transformer and introduces local adapters with spatial grounding to selectively refine hand and face regions, steering the strong base model's prior toward accurate articulation. To enable this work and support broader sign language research, we present NVSign, a large-scale dataset of video content natively produced in sign language, offering diverse signer appearances, environments, and natural conversational settings. Trained on this data, our model shows up to 30% improvement in hand pose precision metrics over the strongest baseline and is preferred by sign language users for visual quality and comprehensibility in more than 80% of comparisons.

Abstract

Explore similar work

SignRefine: Adapting Foundational Video Models for Sign Language Generation

SignRR: Retrieve and Refine Real Motion for Sign Language Production

Physics-Informed Diffusion for Biomechanically Plausible 3D Sign Language Generation