TDMM-LM: Bridging Facial Understanding and Animation via Language Models

Luchuan Song; Pinxin Liu; Haiyang Liu; Zhenchao Jin; Yolo Yunlong Tang; Zichong Xu; Susan Liang; Jing Bi

cs.CVMar 14, 2026

TDMM-LM: Bridging Facial Understanding and Animation via Language Models

Authors: Luchuan Song, Pinxin Liu, Haiyang Liu, Zhenchao Jin, Yolo Yunlong Tang, Zichong Xu, Susan Liang, Jing Bi,

Organizations: University of Rochester · University of Tokyo · University of Michigan · Voxel51

Abstract

Text-guided human body animation has advanced rapidly, yet facial animation lags due to the scarcity of well-annotated, text-paired facial corpora. To close this gap, we leverage foundation generative models to synthesize a large, balanced corpus of facial behavior. We design prompts suite covering emotions and head motions, generate about 80 hours of facial videos with multiple generators, and fit per-frame 3D facial parameters, yielding large-scale (prompt and parameter) pairs for training. Building on this dataset, we probe language models for bidirectional competence over facial motion via two complementary tasks: (1) Motion2Language: given a sequence of 3D facial parameters, the model produces natural-language descriptions capturing content, style, and dynamics; and (2) Language2Motion: given a prompt, the model synthesizes the corresponding sequence of 3D facial parameters via quantized motion tokens for downstream animation. Extensive experiments show that in this setting language models can both interpret and synthesize facial motion with strong generalization. To best of our knowledge, this is the first work to cast facial-parameter modeling as a language problem, establishing a unified path for text-conditioned facial animation and motion understanding.

Figures & tables

Figure 2: The analysis of the Open3DFaceVid dataset. We summarize the control categories induced by prompts and their corresponding video counts, broken down by underlying T2V backbones. We further visualize the vocabulary with word clouds, separately for emotion-related terms and for full-text prompts, to highlight the diversity and saliency of affective descriptors.

Figure 3: Dataset overview. Top two rows: starting from a fixed text prompt, we vary the random seed and emphasize different prompt keywords to modulate facial identity and video attributes, showcasing subjects across different genders. Bottom three rows: we recover FLAME facial parameters and pair the resulting trajectories with the corresponding prompt, forming Text–3DMM dataset.

Figure 4: Geometry-aware facial tokenization learning. We quantize facial expression codes into a discrete codebook and enforce reconstruction in mesh space. The input facial expression codes are mapped to code indices, decoded back to FLAME meshes (bottom), and supervised with an $\mathcal{L}_{1}$ loss on vertex positions

Figure 5: Motion2Language. Geometry sequences are encoded into discrete facial tokens by the geometry encoder and fed, together with text tokens from the user prompt, into a LLM. Conditioned only on these geometry tokens, the agent generates natural-language descriptions of expression/head motion, enabling interactive question to answering about 3D facial behavior.

Figure 6: Language2Motion. The user provides a natural-language description of the desired facial behavior (top left). The text tokenizer converts the prompt into word-level tokens, while a paired 3D facial sequence is encoded into discrete geometry tokens by the geometry encoder. The autoregressive transformer predicts future geometry tokens conditioned on the text prefix.

Methods	$\text{Cor}_{E}$ $\uparrow$	$\text{Cor}_{M}$ $\uparrow$	$\text{Cor}_{I}$ $\uparrow$	$\text{USER}_{E}$ $\uparrow$	$\text{USER}_{M}$ $\uparrow$
Methods	[GPT-4]			[Human]
HumanOmni [ 50 ]	1.84	1.17	1.09	2.04	1.00
Gemini2.5 VLM [ 56 ]	2.45	2.91	3.51	2.88	3.41
Ours	4.02	3.35	3.63	4.29	3.79

Table 1: Quantitative evaluation of Motion2Language. Mean 1–5 correctness scores from GPT-4 and human raters for emotion (

\text{Cor}_{E}

\text{USER}_{E}

), motion (

\text{Cor}_{M}

\text{USER}_{M}

) and intensity (

\text{Cor}_{I}

). We bold the best. Our geometry-token model consistently outperforms HumanOmni and Gemini-2.5 VLM across the metrics.

Methods	$\text{Cor}_{E}$ $\uparrow$	$\text{Cor}_{M}$ $\uparrow$	$\text{Cor}_{I}$ $\uparrow$	$\text{USER}_{E}$ $\uparrow$	$\text{USER}_{M}$ $\uparrow$
Methods	[GPT-4]			[Human]
HumanOmni [ 69 ]	3.66	1.59	1.42	3.82	1.27
Gemini2.5 VLM [ 51 ]	4.21	3.17	3.92	4.17	3.82
Ours	4.02	3.35	3.63	4.29	3.79

Table 2: Quantitative evaluation of Motion2Language with natural images input. We employ the same correctness metric as in Table 1 , while replacing geometry-image inputs with natural facial images to better adapt the evaluation setting to VLMs.

Methods	$L_{2}$ $\downarrow$	FD $\downarrow$	Tok $\uparrow$	$\text{Cor}_{E}$ $\uparrow$	USER $\uparrow$
Methods	[Parameters]			[Language]
T2M-X [ 27 ]	0.471	47.59	0.671	2.21	3.40
T2M-GPT [ 65 ]	0.226	37.04	0.895	3.57	3.91
Ours	0.219	31.75	0.920	4.13	3.95

Table 3: Quantitative evaluation of Language2Motion. Comparison with T2M-X and T2M-GPT on parameter-space measures (

L_{2}

/FD/Tok) and language-based metrics (

\text{Cor}_{E}

/USER).

L_{2}

/FD assess expression/pose fidelity, Tok is token-level accuracy.

Figure 7: Qualitative comparison for Motion2Language. For representative clips, we show the input frames/geometry. The Gemini/HumanOmni operate on rendered geometry, while our model relies on 3D facial tokens. Our approach produces more accurate descriptions of emotion/motion, closely matching GT annotations.

Figure 8: Qualitative comparison for Language2Motion. Given prompts, we visualize generated 3D facial motion from different method. Our model produces expressions and head poses that more faithfully follow the described affect

Figure 9: Emotion coverage across datasets. Top-row: emotion-label distributions for (1) Open3DFaceVid, (2) MEAD [ 56 ] , and (3) YouTube-derived clips, showing the number and balance of emotion categories (187/8/37 labels, respectively). Bottom-row: 2D projections of the corresponding facial (expression+pose) video embeddings, where Open3DFaceVid exhibits rich, well-separated clusters, while MEAD and YouTube provide coarser and less diverse affective coverage. Each point in the t-SNE [ 36 ] plots corresponds to the facial-expression features of a single video clip. Due to color variety limitations, we are unable to display all categories.

Train-Corpus	$\text{Cor}_{E}$ $\uparrow$	$\text{Cor}_{M}$ $\uparrow$	$\text{Cor}_{I}$ $\uparrow$	$\text{USER}_{E}$ $\uparrow$	$\text{USER}_{M}$ $\uparrow$
Train-Corpus	[GPT-4]			[Human]
MEAD [ 56 ]	3.03	1.29	3.59	2.92	1.18
YouTube [Gemini]	2.74	3.92	3.76	3.14	3.65
Open3DFaceVid	3.97	3.44	3.85	4.05	3.75

Table 4: Ablation on training corpus for Motion2Language. Identical architectures are trained on MEAD, YouTube (Gemini-annotated) and Open3DFaceVid. The metrics defined in Sec. 5.3 .

Train-Corpus	$L_{2}$ $\downarrow$	FD $\downarrow$	Tok $\uparrow$	$\text{Cor}_{E}$ $\uparrow$	USER $\uparrow$
Train-Corpus	[Parameters]			[Language]
MEAD [ 56 ]	0.217	32.14	0.892	3.56	2.88
YouTube [Gemini]	0.259	33.47	0.910	3.17	3.59
Open3DFaceVid	0.226	32.48	0.915	3.90	3.92

Table 5: Ablation on training corpus for Language2Motion. Identical models are trained on MEAD, YouTube (Gemini-annotated), and Open3DFaceVid. Training on Open3DFaceVid yields the best trade-off between geometric fidelity and text–motion alignment.

Figure 10: The qualitative ablation on training corpus for Language2Motion. We visualize facial motion generated by models trained on ablation datasets. The MEAD- and YouTube-trained models miss the “pout” behavior, whereas the our model produces clear lip protrusion (red arrows), matching the description.

Figure 11: Qualitative ablation on corpus for Motion2Language. Given user query, we show responses from models trained on MEAD, YouTube and Open3DFaceVid. The MEAD- and YouTube-trained variants produce generic or partially correct descriptions, while the Open3DFaceVid model most accurately captures the subtle worry-to-surprise transition and head motion.

Figure 12: The scaling behavior of Motion2Language and Language2Motion. Left: token accuracy over training iterations for Motion2Language with Qwen3 backbones of different sizes (.6B-32B). Right: token accuracy for Language2Motion with LLaMA-based backbones (0.7B-3B).

Figure 13: Language-controlled expressive ablation. We modify one keyword in prompt and let the Language2Motion model generate the corresponding facial motion. The two happy prompts produce different intensity level, while the angry prompt yields clearly distinct mouth shapes, demonstrating fine-grained text control over emotional style.

Human Avatar Animation

Human avatar animation builds digital people that can be reconstructed, driven by motion or speech, and rendered consistently across poses and views.

See all 91 papers

Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models
Feb 6, 2026Haoyu Zhang, Zhipeng Li, Yiwen Guo +1Multimodal Large Language Models Audio-Driven Facial Animation
The Chinese University of Hong Kong, Shenzhen · LIGHTSPEED · Independent Researcher
AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models
May 8, 2026Kai Zheng, Zejian Kang, Rui Mao +4Audio-Driven Facial Animation
Westlake University · Zhejiang University · Tiangong University +2
EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions
Jul 2, 2026Haoran Wang, Mohit Mendiratta, Christian Theobalt +1Text-to-3D Generation Facial Expression Synthesis
Max Planck Institute for Informatics, Saarland Informatics Campus · CISPA Helmholtz Center for Information Security

Feb 6, 2026J/K move · Enter open · S save

Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models

Haoyu Zhang, Zhipeng Li, Yiwen Guo, Tianshu Yu

The Chinese University of Hong Kong, Shenzhen · LIGHTSPEED · Independent Researcher

Multimodal Large Language Models Audio-Driven Facial Animation Multimodal Generation Facial Expression Synthesis

Open paper AskPDF arXiv

Omni-modal large language models (OLLMs) aim to unify multimodal understanding and generation, yet extending them to jointly produce speech and 3D facial animation remains largely underexplored. A key challenge is the mismatch between the discrete semantic reasoning of LLMs and the dense temporal dynamics required for 3D facial motion. We propose Expressive Omni (Ex-Omni), a framework that augments OLLMs with speech-accompanied 3D facial animation. Ex-Omni decouples semantic reasoning from temporal generation through a speech-unit generator with blendshape co-supervision and a non-autoregressive blendshape decoder, where speech units provide temporal scaffolding and hidden speech representations carry facially relevant cues. We further introduce a token-as-query gated fusion (TQGF) interface for controlled semantic injection, as well as InstructS2SF-1200K, a 1.2M-sample weakly supervised dataset for speech-accompanied facial animation. Extensive experiments show that Ex-Omni retains competitive speech QA capability while natively generating coordinated text, speech, and 3D facial animation, and approaches the Audio2Face-3D teacher cascade in synchronization and human preference.

Abstract

Human Avatar Animation

Explore similar work

Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models

AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models

EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions