Paper ID: 2405.10718 • Published May 17, 2024
SignLLM: Sign Language Production Large Language Models
Sen Fang, Lei Wang, Ce Zheng, Chunyu Sui, Mingyu Zhao, Yapeng Tian, Chen Chen
TL;DR
Get AI-generated summaries with premium
Get AI-generated summaries with premium
In this paper, we propose SignLLM, a multilingual Sign Language Production
(SLP) large language model, which includes two novel multilingual SLP modes
MLSF and Prompt2LangGloss that allow sign language gestures generation from
query texts input and question-style prompts input respectively. Both modes can
use a new RL loss based on reinforcement learning and a new RL module named
Priority Learning Channel. These RL components can accelerate the training by
enhancing the model's capability to sample high-quality data. For SignLLM's
training, we introduce Prompt2Sign, a comprehensive multilingual sign language
dataset, which builds from public data, including American Sign Language (ASL)
and seven others. This dataset standardizes information by extracting pose
information from sign language videos into a unified compressed format. We
extensively evaluate SignLLM, demonstrating that our model achieves
state-of-the-art performance on SLP tasks across eight sign languages.