Text-guided human body animation has advanced rapidly, yet facial animation lags due to the scarcity of well-annotated, text-paired facial corpora. To close this gap, we leverage foundation generative models to synthesize a large, balanced corpus of facial behavior. We design prompts suite covering emotions and head motions, generate about 80 hours of facial videos with multiple generators, and fit per-frame 3D facial parameters, yielding large-scale (prompt and parameter) pairs for training. Building on this dataset, we probe language models for bidirectional competence over facial motion via two complementary tasks: (1) Motion2Language: given a sequence of 3D facial parameters, the model produces natural-language descriptions capturing content, style, and dynamics; and (2) Language2Motion: given a prompt, the model synthesizes the corresponding sequence of 3D facial parameters via quantized motion tokens for downstream animation. Extensive experiments show that in this setting language models can both interpret and synthesize facial motion with strong generalization. To best of our knowledge, this is the first work to cast facial-parameter modeling as a language problem, establishing a unified path for text-conditioned facial animation and motion understanding.
Figures & tables
Figure 2: The analysis of the Open3DFaceVid dataset. We summarize the control categories induced by prompts and their corresponding video counts, broken down by underlying T2V backbones. We further visualize the vocabulary with word clouds, separately for emotion-related terms and for full-text prompts, to highlight the diversity and saliency of affective descriptors.
Figure 3: Dataset overview. Top two rows: starting from a fixed text prompt, we vary the random seed and emphasize different prompt keywords to modulate facial identity and video attributes, showcasing subjects across different genders. Bottom three rows: we recover FLAME facial parameters and pair the resulting trajectories with the corresponding prompt, forming Text–3DMM dataset.
Figure 4: Geometry-aware facial tokenization learning. We quantize facial expression codes into a discrete codebook and enforce reconstruction in mesh space. The input facial expression codes are mapped to code indices, decoded back to FLAME meshes (bottom), and supervised with an L1 loss on vertex positions
Figure 5: Motion2Language. Geometry sequences are encoded into discrete facial tokens by the geometry encoder and fed, together with text tokens from the user prompt, into a LLM. Conditioned only on these geometry tokens, the agent generates natural-language descriptions of expression/head motion, enabling interactive question to answering about 3D facial behavior.
Figure 6: Language2Motion. The user provides a natural-language description of the desired facial behavior (top left). The text tokenizer converts the prompt into word-level tokens, while a paired 3D facial sequence is encoded into discrete geometry tokens by the geometry encoder. The autoregressive transformer predicts future geometry tokens conditioned on the text prefix.
Methods
CorE↑
CorM↑
CorI↑
USERE↑
USERM↑
[GPT-4]
[Human]
HumanOmni [ 50 ]
1.84
1.17
1.09
2.04
1.00
Gemini2.5 VLM [ 56 ]
2.45
2.91
3.51
2.88
3.41
Ours
4.02
3.35
3.63
4.29
3.79
Table 1: Quantitative evaluation of Motion2Language. Mean 1–5 correctness scores from GPT-4 and human raters for emotion ( CorE , USERE ), motion ( CorM , USERM ) and intensity ( CorI ). We bold the best. Our geometry-token model consistently outperforms HumanOmni and Gemini-2.5 VLM across the metrics.
Methods
CorE↑
CorM↑
CorI↑
USERE↑
USERM↑
[GPT-4]
[Human]
HumanOmni [ 69 ]
3.66
1.59
1.42
3.82
1.27
Gemini2.5 VLM [ 51 ]
4.21
3.17
3.92
4.17
3.82
Ours
4.02
3.35
3.63
4.29
3.79
Table 2: Quantitative evaluation of Motion2Language with natural images input. We employ the same correctness metric as in Table 1 , while replacing geometry-image inputs with natural facial images to better adapt the evaluation setting to VLMs.
Methods
L2↓
FD ↓
Tok ↑
CorE↑
USER ↑
[Parameters]
[Language]
T2M-X [ 27 ]
0.471
47.59
0.671
2.21
3.40
T2M-GPT [ 65 ]
0.226
37.04
0.895
3.57
3.91
Ours
0.219
31.75
0.920
4.13
3.95
Table 3: Quantitative evaluation of Language2Motion. Comparison with T2M-X and T2M-GPT on parameter-space measures ( L2 /FD/Tok) and language-based metrics ( CorE /USER). L2 /FD assess expression/pose fidelity, Tok is token-level accuracy.
Figure 7: Qualitative comparison for Motion2Language. For representative clips, we show the input frames/geometry. The Gemini/HumanOmni operate on rendered geometry, while our model relies on 3D facial tokens. Our approach produces more accurate descriptions of emotion/motion, closely matching GT annotations.
Figure 8: Qualitative comparison for Language2Motion. Given prompts, we visualize generated 3D facial motion from different method. Our model produces expressions and head poses that more faithfully follow the described affect
Figure 9: Emotion coverage across datasets. Top-row: emotion-label distributions for (1) Open3DFaceVid, (2) MEAD [ 56 ] , and (3) YouTube-derived clips, showing the number and balance of emotion categories (187/8/37 labels, respectively). Bottom-row: 2D projections of the corresponding facial (expression+pose) video embeddings, where Open3DFaceVid exhibits rich, well-separated clusters, while MEAD and YouTube provide coarser and less diverse affective coverage. Each point in the t-SNE [ 36 ] plots corresponds to the facial-expression features of a single video clip. Due to color variety limitations, we are unable to display all categories.
Train-Corpus
CorE↑
CorM↑
CorI↑
USERE↑
USERM↑
[GPT-4]
[Human]
MEAD [ 56 ]
3.03
1.29
3.59
2.92
1.18
YouTube [Gemini]
2.74
3.92
3.76
3.14
3.65
Open3DFaceVid
3.97
3.44
3.85
4.05
3.75
Table 4: Ablation on training corpus for Motion2Language. Identical architectures are trained on MEAD, YouTube (Gemini-annotated) and Open3DFaceVid. The metrics defined in Sec. 5.3 .
Train-Corpus
L2↓
FD ↓
Tok ↑
CorE↑
USER ↑
[Parameters]
[Language]
MEAD [ 56 ]
0.217
32.14
0.892
3.56
2.88
YouTube [Gemini]
0.259
33.47
0.910
3.17
3.59
Open3DFaceVid
0.226
32.48
0.915
3.90
3.92
Table 5: Ablation on training corpus for Language2Motion. Identical models are trained on MEAD, YouTube (Gemini-annotated), and Open3DFaceVid. Training on Open3DFaceVid yields the best trade-off between geometric fidelity and text–motion alignment.
Figure 10: The qualitative ablation on training corpus for Language2Motion. We visualize facial motion generated by models trained on ablation datasets. The MEAD- and YouTube-trained models miss the “pout” behavior, whereas the our model produces clear lip protrusion (red arrows), matching the description.
Figure 11: Qualitative ablation on corpus for Motion2Language. Given user query, we show responses from models trained on MEAD, YouTube and Open3DFaceVid. The MEAD- and YouTube-trained variants produce generic or partially correct descriptions, while the Open3DFaceVid model most accurately captures the subtle worry-to-surprise transition and head motion.
Figure 12: The scaling behavior of Motion2Language and Language2Motion. Left: token accuracy over training iterations for Motion2Language with Qwen3 backbones of different sizes (.6B-32B). Right: token accuracy for Language2Motion with LLaMA-based backbones (0.7B-3B).
Figure 13: Language-controlled expressive ablation. We modify one keyword in prompt and let the Language2Motion model generate the corresponding facial motion. The two happy prompts produce different intensity level, while the angry prompt yields clearly distinct mouth shapes, demonstrating fine-grained text control over emotional style.
Omni-modal large language models (OLLMs) aim to unify multimodal understanding and generation, yet extending them to jointly produce speech and 3D facial animation remains largely underexplored. A key challenge is the mismatch between the discrete semantic reasoning of LLMs and the dense temporal dynamics required for 3D facial motion. We propose Expressive Omni (Ex-Omni), a framework that augments OLLMs with speech-accompanied 3D facial animation. Ex-Omni decouples semantic reasoning from temporal generation through a speech-unit generator with blendshape co-supervision and a non-autoregressive blendshape decoder, where speech units provide temporal scaffolding and hidden speech representations carry facially relevant cues. We further introduce a token-as-query gated fusion (TQGF) interface for controlled semantic injection, as well as InstructS2SF-1200K, a 1.2M-sample weakly supervised dataset for speech-accompanied facial animation. Extensive experiments show that Ex-Omni retains competitive speech QA capability while natively generating coordinated text, speech, and 3D facial animation, and approaches the Audio2Face-3D teacher cascade in synchronization and human preference.
Haoyu Zhang, Zhipeng Li, Yiwen Guo +1
The Chinese University of Hong Kong, Shenzhen · LIGHTSPEED · Independent Researcher
Speech-driven facial animation requires accurate correspondence between acoustic signals and facial motion, especially for articulation-related mouth movements. However, directly mapping speech audio to facial coefficients often overlooks the linguistic and phonetic structure underlying speech production. In this paper, we propose AudioFace, a language-assisted framework for speech-driven blendshape generation that treats mouth-related facial coefficient prediction as a structured generation problem guided by linguistic and articulatory information. Instead of relying solely on acoustic features, our method leverages the prior knowledge of multimodal large language models and introduces transcript- and phoneme-level cues to bridge speech signals with interpretable facial actions. Extensive experiments show that AudioFace achieves superior performance across multiple evaluation metrics, validating the effectiveness of language-assisted and multimodal-prior-guided speech-driven facial animation.
Kai Zheng, Zejian Kang, Rui Mao +4
Westlake University · Zhejiang University · Tiangong University +2
Precise control of 3D facial expressions from text is crucial for virtual avatars, animation, and human-computer interaction, yet existing text-to-3D methods jointly generate identity, expression, and texture, making fine-grained expression control difficult. We instead formulate text-driven expression synthesis as a regression problem in the disentangled parameter space of a 3D Morphable Model (3DMM). This setting, however, requires paired data linking detailed language to precise expression parameters, which are missing from existing resources. To fill this gap, we introduce Txt2Emote, a benchmark of diverse 3D facial expressions with fine-grained textual annotations obtained from GPT-4o and a high-fidelity face tracker, providing both explicit descriptions detailing facial features and implicit descriptions referencing the situational context behind the expression. Leveraging this dataset, we present EmoteGPT, a text-to-3D expression framework based on a Multimodal Large Language Model (MLLM) with a dedicated <Expr> token to semantically ground expression representations, which are then decoded into 3DMM parameters. We further improve EmoteGPT by augmenting training with large-scale image-to-3DMM data, enabling it to surpass state-of-the-art text-to-3D face synthesis methods on emotion recognition metrics and in perceived expressiveness. Integrated into avatar pipelines, our method enables photorealistic and stylized 3D avatars, as well as expressive 3D-consistent 2D face synthesis from textual input.
Haoran Wang, Mohit Mendiratta, Christian Theobalt +1
Max Planck Institute for Informatics, Saarland Informatics Campus · CISPA Helmholtz Center for Information Security