cs.CVNov 5, 2026

MV-Sign: Multi-View 3D Motion Integration and Enhanced Text-to-Motion Conditioning for Sign Language Production

Authors: Singyu Muk, Kaustubh Kundu, Kunyuan Xie, Weijie Wang, Kalin Stefanov

Organizations: Independent Researcher, Australia · Monash University, Australia · University of Trento, Italy · Japan Advanced Institute of Science and Technology, Japan

Abstract

Sign language production (SLP) aims to generate appropriate sign language motion from spoken language text. However, current 3D SLP methods suffer from noisy supervision and weak text conditioning. Specifically, these methods typically rely on pseudo-3D motion data fitted from single-view frontal videos, which introduces severe self-occlusion and loses fine-grained hand details, ultimately degrading downstream production. To overcome these limitations, we propose a comprehensive framework that jointly integrates multi-view 3D motion data and optimizes the text-conditioning pathway. First, we introduce How2Sign-MV3D, a high-fidelity dataset constructed via a novel cross-view curation pipeline that fuses multi-view estimates to resolve occlusions and filter non-signing segments. Second, we design a robust SLP model incorporating a sign-aware spatio-temporal VQ-VAE (ST-VQVAE) for joint body-hand tokenization, a pseudo-gloss adapter, and keyframe-aware retrieval to enhance text-to-motion alignment. Evaluated on the How2Sign benchmark, our framework reduces body and hand DTW-PA-JPE by 32.8% and 20.9% respectively compared to the state-of-the-art. Finally, we release two new high-quality 3D datasets, ASLCitizen-3D and MM-WAuslan-MV3D, alongside SGNify-MV as a multi-view video benchmark to support broader community research.

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learning

    Mar 11, 2026Jianhe Low, Alexandre Symeonidis-Herzig, Maksym Ivashechkin +2Human Motion GenerationSign Language Generation

  2. M3T: Discrete Multi-Modal Motion Tokens for Sign Language Production

    Mar 24, 2026Alexandre Symeonidis-Herzig, Jianhe Low, Ozge Mercanoglu Sincan +1Human Motion GenerationMultimodal Generation