Describe Your Facial Expressions by Linking Image Encoders and Large Language Models

Yujian Yuan; Jiabei Zeng; Shiguang Shan

cs.CVNov 5, 2026

Describe Your Facial Expressions by Linking Image Encoders and Large Language Models

Authors: Yujian Yuan, Jiabei Zeng, Shiguang Shan

Organizations: Institute of Computing Technology, Chinese Academy of Sciences Beijing, China · University of Chinese Academy of Sciences Beijing, China

Abstract

This paper presents a novel task of describing human facial expressions of a facial image in natural language, which captures the nuances of facial actions and emotional states beyond traditional emotion categories or facial action units (AUs). To achieve the facial expression captioning model, we propose a three-stage training framework that trains a vision-to-language model using synthesized image-text pairs and the BLIP-2 pre-training techniques. To overcome the challenge of missing training image-text pairs for facial expression captioning, we propose a strategy that involves synthesizing and combining captions using GPT-3.5 and existing annotations on either emotion categories or AUs. Experiments demonstrate the effectiveness of our method in generating captions that describe details of facial actions and emotions, as well as the inferential relationship between them, even when those emotions are not present in the training data. It is also demonstrated that the vision-to-language task enhances the performance of the intermediate visual features on both AU detection and emotion classification tasks.

Figures & tables

Figure 1: Three-stage training framework for facial expression captioning. (Left) Synthesize the training image-text pairs. (Middle) Train AU-BLIP and Emot-BLIP to respectively describe a single aspect of facial expression. (Right) Train Exp-BLIP utilizing synthesized data with fused captions to describe both facial actions and emotions.

Figure 2: Inputs and outputs examples of the rule-based text generators for AUs and emotions(Left) and those of the GPT-3.5 based generator with one-shot prompt(Right) for AUs.

Figure 3: Two-step training procedure of BLIP-2 [ Li et al.(2023b)Li, Li, Savarese, and Hoi ] . The 1 ${}^{\text{st}}$ step ( Left ): BLIP-2 enforces the queries to extract the visual representation most relevant to the text. The 2 ${}^{\text{nd}}$ step( Right ): BLIP-2 bootstraps from frozen LLMs to conduct vision-to-language generative training.

Figure 4: Inputs and outputs examples of the GPT-3.5 based generator with zero-shot prompt for fusing AU and emotion captions. The image described by the fused caption is on the right.

	Originally annotated with AUs					Originally annotated with emotions
	BP4D	DISFA	GFT	RAF-AU	EmotioNet	AffectNet	RAF-DB	FaceME
train( $\#$ image/ $\#$ sub)	$16627^{*}$ /28	$14814^{*}$ /24	$17719^{*}$ /78	3733/-	19046/-	287618/-	3162/-	10052/-
test( $\#$ image/ $\#$ sub)	45805/13	14535/3	$4034^{*}$ /18	868/-	2117/-	4000/-	792/-	-

Table 1: Statistics of training and test data.(#sub:number of subjects; *:sampled set)

Models	RAF-AU			GFT
Models	Meteor $\uparrow$	Rouge $\uparrow$	Ada-sim( $\%$ ) $\uparrow$	Meteor $\uparrow$	Rouge $\uparrow$	Ada-sim( $\%$ ) $\uparrow$
BLIP-2(ViT-G,6.7B)	0.031	0.112	76.32	0.033	0.118	77.58
BLIP-2(ViT-G,6.7B) ${}^{\text{COCO}}$	0.034	0.129	77.31	0.034	0.131	78.47
AU-BLIP(ViT-B,2.7B)	0.236	0.457	90.13	0.278	0.509	93.11
AU-BLIP(ViT-G,2.7B)	0.254	0.478	90.23	0.279	0.487	93.45
AU-BLIP(ViT-G,6.7B)	0.263	0.505	90.55	0.295	0.538	93.97

Table 2: Metrics of AU captioning with different models (ViT-B/G denotes the used image encoder, 2.7B/6.7B denotes the used language model OPT-2.7B/6.7B)

Figure 5: Examples of the generated captions by different models with ViT-G/OPT-6.7B

Models	BP4D (F1 score $\times 100$ )													AffectNet ( Acc. %)
Models	AU1	AU2	AU4	AU6	AU7	AU10	AU12	AU14	AU15	AU17	AU23	AU24	Avg	Emotion
FaRL [ Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen ]	48.4	45.9	47.5	78.9	69.8	82.8	86.0	58.3	42.0	55.9	36.1	38.0	57.5	38.8
Me-GraphAU ∗ [ Luo et al.(2022)Luo, Song, Xie, Shen, and Gunes ]	52.7	44.3	60.9	79.9	80.1	85.3	89.2	69.4	55.4	64.4	49.8	55.1	65.5	-
EfficientNet-B2 ∗ [ Savchenko et al.(2022)Savchenko, Savchenko, and Makarov ]	-	-	-	-	-	-	-	-	-	-	-	-	-	63.0
ViT-G	53.2	34.2	58.7	81.1	71.1	85.7	90.0	56.1	31.2	58.3	43.4	51.1	59.5	44.8
ViT-G ${}^{\text{fine-tune}}$	32.8	18.4	38.1	77.2	68.6	84.0	85.1	60.1	32.3	61.6	35.7	32.9	52.2	50.3

Table 3: Performance of visual representation on BP4D and AffectNet. ( ∗ : original values)

Models	AU1	AU2	AU4	AU5	AU6	AU9	AU10	AU12	AU16	AU17	AU25	AU26	AU27	Avg
FaRL [ Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen ]	57.8	56.3	75.2	56.2	32.8	50.2	61.5	68.9	49.7	38.8	84.4	55.0	60.6	57.5
AU-CNN ∗ [ Yan et al.(2020)Yan, Li, Que, Pei, and Deng ]	60.5	65.6	73.4	69.7	58.2	67.4	68.4	69.6	59.4	25.6	92.1	64.7	82.7	65.9
ViT-G	60.0	57.6	77.2	62.3	35.8	59.7	70.6	73.6	53.6	45.5	88.7	56.3	70.1	62.4
ViT-G ${}^{\text{fine-tune}}$	27.6	34.5	59.1	52.5	23.4	38.5	48.9	51.0	41.9	25.0	86.1	46.5	62.6	46.0
ViT-G(Emot)	61.0	55.3	78.4	62.2	37.5	66.8	70.8	71.5	53.8	48.6	88.8	55.9	69.5	63.1
QFormer(Emot)	57.8	53.6	78.8	62.8	36.1	62.9	67.2	68.4	52.7	42.4	86.6	56.1	62.0	60.6

Table 4: Performance of visual representation on RAF-AU. ( ∗ : original values)

Models	Correctness $\uparrow$	The highest completeness(ratio) $\uparrow$	The lowest completeness(ratio) $\downarrow$
Mix-BLIP(ViT-B,2.7B)	7.61	3.43%	91.17%
Cat-BLIP(ViT-B,2.7B)	7.27	28.85%	7.91%
Exp-BLIP(ViT-B,2.7B)	7.61	67.72%	0.92%

Table 5: Correctness and completeness for different combining strategies

Figure 6: Examples of zero-shot ability of Exp-BLIP(ViT-G,OPT-6.7B)

AU	Noun form	Verb form	AU	Noun form	Verb form
AU1	inner brow raiser	raise inner eyebrow	AU17	chin raiser	raise chin
AU2	outer brow raiser	raise outer eyebrow	AU18	lip pucker	pucker lips
AU4	brow lowerer	lower brow	AU19	tongue show	stick tongue out
AU5	upper lid raiser	raise upper lid	AU20	lip stretch	stretch lips
AU6	cheek raiser	raise cheek	AU21	neck tightener	tighten neck
AU7	lid tightener	tighten lids	AU22	lip funneler	show lip funneler

Table S1: Brief AU descriptions of noun form and verb form.

Models	BP4D (F1 score $\times 100$ )													AffectNet ( Acc. %)
Models	AU1	AU2	AU4	AU6	AU7	AU10	AU12	AU14	AU15	AU17	AU23	AU24	Avg	Emotion
ViT-B	46.3	38.2	50.8	79.3	73.4	85.4	87.9	69.3	24.0	53.0	0.9	24.2	52.7	37.8
ViT-B ${}^{\text{fine-tune}}$	55.1	65.3	64.8	53.3	59.4	83.9	88.4	5.9	43.5	22.2	96.0	87.3	60.4	51.4
ViT-B(Emot)	47.0	37.5	52.9	79.0	73.0	85.8	88.6	68.0	24.3	54.3	7.7	21.9	53.3	38.2
QFormer(Emot)	52.1	44.7	66.7	80.8	74.8	85.2	88.6	50.4	36.9	59.2	14.3	28.5	56.9	54.3
ViT-B(AU)	47.3	39.6	59.6	78.9	73.3	85.7	88.6	68.2	23.3	52.7	2.1	15.8	52.9	38.8

Table S2: Performance on BP4D and AffectNet. (ViT-B/OPT-2.7B)

Models	AU1	AU2	AU4	AU5	AU6	AU7	AU9	AU10	AU12	AU15	AU16	AU17	AU22	AU24	AU25	AU26	AU27	Avg
ViT-B	48.4	49.9	67.9	23.8	32.4	19.7	25.1	47.5	69.1	15.5	24.5	18.4	2.0	16.9	86.7	32.3	58.9	37.6
ViT-B ${}^{\text{fine-tune}}$	66.3	53.9	79.5	67.1	36.7	32.5	79.6	72.9	66.4	46.8	57.7	68.0	19.6	26.5	94.7	67.7	81.8	59.9
ViT-B(Emot)	48.1	50.5	68.9	27.2	34.8	23.0	28.0	50.2	71.1	16.4	27.6	19.8	2.2	18.7	87.4	29.2	65.3	39.3
QFormer(Emot)	54.2	53.9	72.8	58.3	35.7	26.7	53.4	62.8	62.4	26.1	49.5	49.9	5.3	11.7	92.5	42.1	76.6	49.1
ViT-B(AU)	50.3	52.6	71.0	30.8	29.6	22.2	35.5	52.4	70.8	16.1	29.8	28.1	4.4	19.2	87.6	36.2	67.1	41.4
QFormer(AU)	66.6	66.1	78.0	63.3	31.5	25.0	69.7	54.5	61.5	19.6	39.6	62.5	3.6	9.7	94.7	55.3	69.6	51.2

Table S3: Performance on RAF-AU. (ViT-B/OPT-2.7B)

Models	BP4D (F1 score $\times 100$ )													AffectNet ( Acc. %)
Models	AU1	AU2	AU4	AU6	AU7	AU10	AU12	AU14	AU15	AU17	AU23	AU24	Avg	Emotion
ViT-G	34.2	16.7	48.6	80.1	70.0	85.9	89.6	59.6	15.7	51.8	19.1	17.1	49.0	44.8
ViT-G ${}^{\text{fine-tune}}$	44.5	30.6	57.4	79.4	69.2	86.0	88.8	62.1	39.2	59.9	41.7	30.9	57.5	50.3
ViT-G(Emot)	35.5	16.2	53.9	78.1	73.8	87.5	89.7	70.5	35.0	56.7	33.0	21.1	54.3	50.9
QFormer(Emot)	5.9	3.8	43.6	71.7	74.4	82.9	87.7	30.0	4.3	9.8	0.1	12.4	35.6	52.7
ViT-G(AU)	56.3	49.7	66.2	79.5	75.3	85.9	89.0	59.9	46.0	70.4	41.2	40.0	63.3	50.1

Table S4: Performance on BP4D and AffectNet. (ViT-G/OPT-6.7B)

Models	AU1	AU2	AU4	AU5	AU6	AU7	AU9	AU10	AU12	AU15	AU16	AU17	AU22	AU24	AU25	AU26	AU27	Avg
ViT-G	48.9	50.0	71.0	65.4	26.1	12.1	64.5	53.1	60.7	9.8	50.0	48.9	10.4	9.7	90.0	28.4	75.1	45.5
ViT-G ${}^{\text{fine-tune}}$	67.4	64.5	78.7	64.5	34.6	29.5	67.2	68.7	67.2	45.8	61.2	52.4	25.2	23.7	94.2	65.6	81.5	58.3
ViT-G(Emot)	49.8	29.9	74.7	66.5	14.9	9.9	59.3	59.4	58.7	28.9	51.8	47.6	5.3	14.0	88.7	41.0	74.2	45.6
QFormer(Emot)	30.4	9.6	73.1	58.9	8.4	16.5	47.1	47.0	49.9	0.0	34.0	5.0	0.0	0.0	86.9	16.1	73.4	32.7
ViT-G(AU)	72.4	65.4	80.7	75.9	47.8	27.3	76.2	60.3	63.2	35.4	53.1	55.4	30.4	15.9	93.6	58.1	77.3	58.1
QFormer(AU)	74.6	72.1	81.5	71.2	46.5	40.5	75.3	64.0	66.8	45.7	46.7	69.0	28.3	32.4	94.3	61.5	78.3	61.7

Table S5: Performance on RAF-AU. (ViT-G/OPT-6.7B)

Models	BP4D (F1 score $\times 100$ )													AffectNet ( Acc. %)
Models	AU1	AU2	AU4	AU6	AU7	AU10	AU12	AU14	AU15	AU17	AU23	AU24	Avg	Emotion
ViT-B ${}^{\text{fine-tune}}$	55.1	65.3	64.8	53.3	59.4	83.9	88.4	5.9	43.5	22.2	96.0	87.3	60.4	51.4
ViT-B(Emot) ${}^{\text{fine-tune}}$	55.2	44.0	60.0	78.4	73.2	85.6	89.4	62.0	44.9	61.5	40.2	44.1	61.5	52.9
QFormer(Emot) ${}^{\text{fine-tune}}$	55.1	47.5	62.4	77.8	71.0	86.1	89.5	60.2	47.2	61.6	43.9	47.4	62.5	54.7
ViT-B(AU) ${}^{\text{fine-tune}}$	55.6	44.4	62.5	77.8	72.6	85.5	89.3	58.3	45.5	61.6	39.2	47.8	61.7	51.9
QFormer(AU) ${}^{\text{fine-tune}}$	54.5	48.8	62.4	80.1	75.2	85.8	90.0	66.9	49.6	64.5	46.5	45.0	64.1	53.6

Table S6: Performance of fine-tuning on BP4D and AffectNet. (ViT-B/OPT-2.7B)

Models	AU1	AU2	AU4	AU5	AU6	AU7	AU9	AU10	AU12	AU15	AU16	AU17	AU22	AU24	AU25	AU26	AU27	Avg
ViT-B ${}^{\text{fine-tune}}$	66.3	53.9	79.5	67.1	36.7	32.5	79.6	72.9	66.4	46.8	57.7	68.0	19.6	26.5	94.7	67.7	81.8	59.9
ViT-B(Emot) ${}^{\text{fine-tune}}$	68.5	67.3	81.6	62.5	28.2	27.6	76.6	69.9	66.3	49.1	51.1	64.9	9.6	26.8	95.4	56.2	78.4	57.6
QFormer(Emot) ${}^{\text{fine-tune}}$	75.0	71.4	83.6	74.0	46.6	31.2	81.8	71.9	74.0	60.3	65.7	72.9	19.8	23.5	95.8	72.9	86.5	65.1
ViT-B(AU) ${}^{\text{fine-tune}}$	69.3	66.7	82.3	67.5	28.3	32.2	75.5	71.3	64.6	49.1	46.8	62.4	18.2	20.9	95.3	54.6	77.5	57.8
QFormer(AU) ${}^{\text{fine-tune}}$	73.9	69.1	83.9	74.5	48.9	36.8	81.3	77.2	70.8	63.5	66.4	74.9	26.2	39.0	95.7	74.7	84.8	67.2
ViT-B(Mix) ${}^{\text{fine-tune}}$	70.4	66.9	82.9	68.4	23.6	28.6	77.2	71.3	65.0	48.1	45.8	63.5	17.6	34.2	95.0	49.8	75.2	57.9

Table S7: Performance of fine-tuning on RAF-AU. (ViT-B/OPT-2.7B)

Models	AU1	AU2	AU4	AU5	AU6	AU7	AU9	AU12	AU17	AU23	AU24	AU25	Avg	Avg(F1)
FaRL [ Zheng et al.(2022)Zheng, Yang, Zhang, Bao, Chen, Huang, Yuan, Chen, Zeng, and Wen ]	69.0	71.0	76.6	70.6	61.6	68.8	69.2	75.0	70.3	66.7	81.1	69.2	70.8	55.7
DGCN ∗ [ Jia et al.(2023)Jia, Xu, Zhou, Wang, and Li ]	79.8	87.1	68.5	85.5	91.9	93.4	81.1	79.8	86.5	98.4	96.6	65.9	84.5	-
ViT-G	72.7	76.2	80.5	76.6	69.5	70.6	77.6	81.2	74.4	76.4	71.2	89.3	76.3	53.5
ViT-G ${}^{\text{fine-tune}}$	83.5	89.3	80.9	87.3	89.7	82.3	75.7	84.6	73.5	85.5	36.1	58.3	77.2	45.2
ViT-G(Emot)	73.4	72.5	82.3	77.6	71.0	65.2	81.3	80.6	76.6	75.0	68.9	87.0	76.0	53.0
QFormer(Emot)	70.6	74.8	80.8	80.1	73.0	73.0	81.5	77.1	73.5	84.9	72.6	85.1	77.2	52.2

Table S8: Performance of visual representation on RAFAU. (

Acc.\%

; ∗ : original values.)

Figure S1: Examples of the generated captions by different models with ViT-G/OPT-6.7B

Figure S2: Bad cases of the generated captions for emotion-annotated images by different models with ViT-G/OPT-6.7B

Model	Layers	Hidden size D	MLP size	Heads	Params
ViT-Base(ViT-B)	12	768	3072	12	86M
ViT-Giant(ViT-G)	39	1408	6144	16	1843M

Table S9: Details of image encoder model (Vision Transformer) variants.

LLM	$OPT_{2.7B}$	$OPT_{6.7B}$
FFN dim	10240	16384
Hidden size	2560	4096
Word embedding project dim	2560	4096
Attention heads	32
Layers	32
Dropout	0.1

Table S10: Hyperparameters for Large Language Model OPT2.7/OPT6.7 architecture.

Model	$BERT_{Base}$
Hidden size	768
Intermediate size	3072
Attention heads	12
Layers	12
max position embeddings	512

Table S11: Hyperparameters for BERT-Base architecture.

LLM	$OPT_{2.7B}$	$OPT_{6.7B}$
Fine-tuning epochs	20
Warmup steps	1000
Learning rate	1e-5
Batch size	32
AdamW $\beta$	(0.9,0.999)
Weight decay	0.05

Table S12: Hyperparameters for fine-tuning BLIP-2 with ViT-B/ViT-G on image captioning.

Image Captioning

Image captioning generates natural-language descriptions of visual content, with research focused on accuracy, coverage, grounding, and evaluation.

See all 97 papers

EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions
Jul 2, 2026Haoran Wang, Mohit Mendiratta, Christian Theobalt +1Text-to-3D Generation Facial Expression Synthesis
Max Planck Institute for Informatics, Saarland Informatics Campus · CISPA Helmholtz Center for Information Security
Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis
Jul 27, 2026Yifan Hu, Shuwei He, Rui Liu +1TTS Synthesis Audio-Driven Facial Animation
Inner Mongolia University Hohhot, China · SRIBD, School of Artificial Intelligence The Chinese University of Hong Kong Shenzhen, China
TAG: Thinking with Action Unit Grounding for Facial Expression Recognition
Feb 21, 2026Haobo Lin, Tianyi Bai, Jiajun Zhang +5Facial Expression Recognition VLM Reasoning
PKU · JLU · Shanghai AI Lab +2

Jul 2, 2026J/K move · Enter open · S save

EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions

Haoran Wang, Mohit Mendiratta, Christian Theobalt, Adam Kortylewski

Max Planck Institute for Informatics, Saarland Informatics Campus · CISPA Helmholtz Center for Information Security

Text-to-3D Generation Facial Expression Synthesis

Open paper AskPDF arXiv

Precise control of 3D facial expressions from text is crucial for virtual avatars, animation, and human-computer interaction, yet existing text-to-3D methods jointly generate identity, expression, and texture, making fine-grained expression control difficult. We instead formulate text-driven expression synthesis as a regression problem in the disentangled parameter space of a 3D Morphable Model (3DMM). This setting, however, requires paired data linking detailed language to precise expression parameters, which are missing from existing resources. To fill this gap, we introduce Txt2Emote, a benchmark of diverse 3D facial expressions with fine-grained textual annotations obtained from GPT-4o and a high-fidelity face tracker, providing both explicit descriptions detailing facial features and implicit descriptions referencing the situational context behind the expression. Leveraging this dataset, we present EmoteGPT, a text-to-3D expression framework based on a Multimodal Large Language Model (MLLM) with a dedicated <Expr> token to semantically ground expression representations, which are then decoded into 3DMM parameters. We further improve EmoteGPT by augmenting training with large-scale image-to-3DMM data, enabling it to surpass state-of-the-art text-to-3D face synthesis methods on emotion recognition metrics and in perceived expressiveness. Integrated into avatar pipelines, our method enables photorealistic and stylized 3D avatars, as well as expressive 3D-consistent 2D face synthesis from textual input.

Abstract

Image Captioning

Explore similar work

EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions

Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis

TAG: Thinking with Action Unit Grounding for Facial Expression Recognition