RoboSTAR: Next-Scale Autoregressive Sign Language Translation for Humanoid Robots

Yujia Zeng; Chensheng Peng; Yuxin Chen; Alex Shao; Nathan Jew; Masayoshi Tomizuka

cs.CVSep 26, 2026

RoboSTAR: Next-Scale Autoregressive Sign Language Translation for Humanoid Robots

Authors: Yujia Zeng, Chensheng Peng, Yuxin Chen, Alex Shao, Nathan Jew, Masayoshi Tomizuka

Organizations: University of California, Berkeley

Abstract

Sign-language interpretation in public communication relies on qualified professional interpreters and can be difficult to scale, motivating robotic signing as a complementary accessibility interface. We present RoBoSTAR, a text-conditioned sign language production (SLP) framework for generating human-centric sign motion that can be retargeted for robotic execution, with speech supported optionally through an external ASR front end. Conventional autoregressive approaches flatten motion into a single full-resolution token sequence, forcing long-range and local dependencies to be modeled at a uniform temporal granularity. RoBoSTAR instead combines part-wise Finite Scalar Quantization with next-scale autoregression, generating motion over progressively finer temporal resolutions while predicting synchronized body and hand tokens in parallel within each step. This coarse-to-fine formulation provides compact long-range context before progressively refining motion details, while self-conditioning and context corruption improve robustness to cross-scale prediction errors. The generated motion is subsequently retargeted for physical humanoid execution. Extensive qualitative and quantitative evaluations are conducted to demonstrate the effectiveness of RoBoSTAR.

Figures & tables

Figure 1: From language to embodied robotic signing. Given either speech or text, RoBoSTAR generates a continuous sign sequence and retargets it to a humanoid robot for physical execution.

Figure 2: Overview of RoBoSTAR . Speech is first transcribed into text, while written input directly conditions the generator with retrieved signs. Part-wise FSQ tokenization represents sign motion at multiple temporal resolutions, which are generated sequentially through next-scale autoregression. The decoded human motion is subsequently retargeted to the robot in the real world.

Figure 3: Part-wise FSQ pyramid. Finest-scale tokens are obtained from each motion stream, while coarser targets are constructed by temporal pooling and projection onto the FSQ grid. The hierarchy supervises next-scale generation, and only the finest scale is decoded for reconstruction.

Figure 4: Qualitative comparison on How2Sign. Representative frames from ground-truth (GT), SOKE, and RoBoSTAR (Ours) sequences. RoBoSTAR better preserves the reference upper-body motion, hand articulation, and body–hand coordination.

Method	DTW-PA-JPE $\downarrow$		DTW-JPE $\downarrow$		B-T BLEU $\uparrow$				Motion Dynamics
Method	Body	Hand	Body	Hand	B-1	B-2	B-3	B-4	Dynamic $\leftrightarrow 0.8$	Magnitude $\leftrightarrow 1$	Freeze-tail $\downarrow$
GT	–	–	–	–	22.45	8.59	3.95	1.96	0.804	1.000	0.036
PT	10.85	2.66	10.84	10.97	13.44	4.69	1.99	0.91	0.245	0.484	0.232
NAR	4.21	1.79	4.14	5.70	6.81	2.10	0.88	0.42	0.000	0.004	1.000
S-MotionGPT	11.84	3.92	11.91	13.30	12.01	4.33	1.99	1.03	0.005	0.016	0.294
T2S-GPT	4.35	1.82	4.27	5.78	7.13	2.79	1.28	0.60	0.000	0.000	1.000

Table 1: Quantitative comparison on the common How2Sign test subset. GT is reported only as a reference and is excluded from ranking. B-

n

denotes back-translation BLEU-

n

. Best and second-best results among the generation methods are shown in bold and underlined.

Figure 5: Real-robot execution of RoBoSTAR -generated sign motion. Representative frames show generated human-centric motions transferred to the physical robot through morphology-aware retargeting and low-level tracking.

Variant	DTW-PA-JPE $\downarrow$		DTW-JPE $\downarrow$
Variant	Body	Hand	Body	Hand
Single-scale AR	4.66	2.56	4.43	6.81
Next-scale	3.96	1.52	3.87	5.34

Table 2: Effect of next-scale generation.

Quantizer	Reconstruction		Generation
Quantizer	Norm. MAE	Norm. RMSE	DTW-PA Body	DTW-PA Hand	DTW Body	DTW Hand
FSQ	0.5221	0.7053	3.9643	1.5238	3.8672	5.3448
VQ-VAE	0.4098	0.5569	4.7488	2.1826	4.6157	6.3220

Table 3: Reconstruction–generation trade-off between FSQ and VQ-VAE at a matched compression ratio, comparing tokenizer reconstruction fidelity and downstream motion generation.

Configuration	Training		DTW-PA-JPE $\downarrow$		DTW-JPE $\downarrow$		B-T BLEU $\uparrow$				Motion Dynamics
Configuration	SC	GCC	Body	Hand	Body	Hand	B-1	B-2	B-3	B-4	Dynamic $\leftrightarrow 0.8$	Magnitude $\leftrightarrow 1$	Freeze-tail $\downarrow$
Neither	✗	✗	4.19	1.85	4.06	5.90	8.85	3.02	1.37	0.63	0.13	0.23	0.16
GCC only	✗	✓	4.25	1.96	4.10	6.04	8.17	2.74	1.23	0.55	0.15	0.25	0.23
SC only	✓	✗	3.92	1.54	3.82	5.07	17.93	6.34	2.67	1.14	0.34	0.47	0.31
Ours (SC+GCC)	✓	✓	3.96	1.52	3.87	5.34	20.08	7.12	3.04	1.36	0.43	0.52	0.19

Table 4: Ablation study of self-conditioning (SC) and generic context corruption (GCC). The SC+GCC configuration is adopted as our final model. Best and second-best results are shown in bold and underlined.

Figure 6: User study. Mean ratings; error bars denote SEM.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Motion	Elbow (cm)	Wrist (cm)	Wrist ori. ( ∘ )	Joint MAE ( ∘ )
Retargeting	GT	$2.0\pm 0.4$	$4.6\pm 1.4$	$22.9\pm 10.9$	–
Retargeting	Ours	$2.0\pm 0.5$	$5.6\pm 1.9$	$31.3\pm 17.7$	–
Execution (sim.)	GT	$1.6\pm 0.5$	$3.6\pm 0.9$	$10.3\pm 3.2$	$2.1\pm 0.7$
Execution (sim.)	Ours	$0.9\pm 0.3$	$2.1\pm 0.6$	$5.9\pm 1.6$	$1.3\pm 0.4$

Appendix

Table 5: Quantitative physical-embodiment evaluation. Retargeting measures morphology-transfer error, while execution measures closed-loop tracking in physics simulation. Results are mean

\pm

standard deviation across sequences.

Table 12

Finite Scalar Quantization

See all 11 papers

SignRR: Retrieve and Refine Real Motion for Sign Language Production
Aug 28, 2026Fidel Omar Tito Cruz, Angie Sanchez Marquina, Summy Farfan +1Human Motion Generation Motion Representation Learning
University of Central Florida Florida, USA · Universidad Nacional Mayor de San Marcos Lima, Peru · Universidad Catolica San Pablo Arequipa, Peru +1
Conditional Collapse in Sign Language Production: A Diagnostic and a Scaling Argument
Jun 1, 2026Rui Hong, Jana KošeckáSign Language Translation Human Motion Generation
George Mason University
From Sign Language Generation to Humanoid Execution: Vision-Language Guided Retargeting with Collision Mitigation
Jul 20, 2026Nabeela Khan, Bowen Wu, Runwu Shi +5Motion Retargeting Humanoid Robotics
Department of Systems and Control Engineering, Institute of Science Tokyo, Tokyo, Japan · RIKEN Guardian Robot Project, Kyoto, Japan · ATR Hiroshi Ishiguro Laboratories, Kyoto, Japan +1

Aug 28, 2026J/K move · Enter open · S save

SignRR: Retrieve and Refine Real Motion for Sign Language Production

Fidel Omar Tito Cruz, Angie Sanchez Marquina, Summy Farfan, Gissella Bejarano

University of Central Florida Florida, USA · Universidad Nacional Mayor de San Marcos Lima, Peru · Universidad Catolica San Pablo Arequipa, Peru+1

Human Motion Generation Motion Representation Learning Sign Language Generation

Open paper AskPDF arXiv

Sign language production (SLP) aims to generate continuous signing motion from spoken language, often through gloss-to-pose generation. Prior work mainly follows two paradigms. Generative models synthesize motion from a learned prior or from noise, without reference to an observed signing instance, making rare hand configurations and signer-specific articulation difficult to preserve. Retrieval-based methods reuse real, well-articulated motion segments, but concatenating segments from different signers and co-articulation contexts can introduce rhythm and style inconsistencies across the full sequence, not only at segment boundaries. These limitations suggest a complementary solution: use retrieval to provide realistic articulation, and use learned refinement to impose the global coherence that retrieval alone lacks. We therefore propose retrieve-and-refine, a paradigm that starts from real retrieved motion and refines it into a globally coherent signing sequence rather than generating motion from scratch. Our framework, SignRR, initializes motion from a dictionary of real sign segments and refines the full sequence with a part-aware Residual VQ-VAE, where residual quantization preserves fine hand articulation and temporal length differences are handled in the latent space. Experiments on PHOENIX14T and CSL-Daily show that SignRR achieves state-of-the-art back-translation performance while maintaining competitive pose quality.

Abstract

Finite Scalar Quantization

Explore similar work

SignRR: Retrieve and Refine Real Motion for Sign Language Production

Conditional Collapse in Sign Language Production: A Diagnostic and a Scaling Argument

From Sign Language Generation to Humanoid Execution: Vision-Language Guided Retargeting with Collision Mitigation