TAG: Thinking with Action Unit Grounding for Facial Expression Recognition

Haobo Lin; Tianyi Bai; Jiajun Zhang; Xuanhao Chang; Sheng Lu; Fangming Gu; Zengjie Hu; Wentao Zhang

cs.CVFeb 21, 2026

TAG: Thinking with Action Unit Grounding for Facial Expression Recognition

Authors: Haobo Lin, Tianyi Bai, Jiajun Zhang, Xuanhao Chang, Sheng Lu, Fangming Gu, Zengjie Hu, Wentao Zhang

Organizations: PKU · JLU · Shanghai AI Lab · HKUST ·

Abstract

Facial Expression Recognition (FER) is a fine-grained visual understanding task where reliable predictions require reasoning over localized and meaningful facial cues. Recent vision--language models (VLMs) enable natural language explanations for FER, but their reasoning is often ungrounded, producing fluent yet unverifiable rationales that are weakly tied to visual evidence and prone to hallucination, leading to poor robustness across different datasets. We propose TAG (Thinking with Action Unit Grounding), a vision--language framework that explicitly constrains multimodal reasoning to be supported by facial Action Units (AUs). TAG requires intermediate reasoning steps to be grounded in AU-related facial regions, yielding predictions accompanied by verifiable visual evidence. The model is trained via supervised fine-tuning on AU-grounded reasoning traces followed by reinforcement learning with an AU-aware reward that aligns predicted regions with external AU detectors. Evaluated on RAF-DB, FERPlus, and AffectNet, TAG consistently outperforms strong open-source and closed-source VLM baselines while simultaneously improving visual faithfulness. Ablation and preference studies further show that AU-grounded rewards stabilize reasoning and mitigate hallucination, demonstrating the importance of structured grounded intermediate representations for trustworthy multimodal reasoning in FER. The code will be available at https://github.com/would1920/FER_TAG .

Figures & tables

Figure 1: Two-stage training framework of TAG. TAG is first trained by supervised fine-tuning with AU-grounded reasoning traces, and then refined by GRPO using format, answer, and AU grounding rewards.

Figure 2: Construction pipeline of TAG-310k. Face images are filtered, annotated with activated AUs, and used to generate and select high-quality AU-grounded reasoning trajectories.

Open-source VLMs, Zero Shot
Model	RAF-DB	FERPlus	AffectNet	Average
LLaVA-Next-Llama3-8B [ 34 ]	59.09	73.36	37.17	56.54
Qwen2.5-VL-7B [ 35 ]	56.75	66.36	33.89	52.33
InternVL3.5-8B [ 36 ]	73.47	69.41	38.57	60.48
LLaVA-Next-34B [ 34 ]	78.36	69.38	42.83	63.52
InternVL3-38B [ 36 ]	79.95	68.75	46.63	65.11

Table 1: Main results (accuracy %) on RAF-DB, FERPlus, and AffectNet. Bold indicates the best performance in each column, underlined indicates the second-best. † Methods with this marker use per-dataset tuning (one model per dataset). TAG-7B (SFT only) uses a single unified model for all datasets.

Figure 3: Qualitative case studies of TAG’s AU-grounded reasoning. Green boxes indicate external AU detector outputs with confidence scores, while red boxes represent TAG’s predicted regions.

SFT	$R_{\mathrm{ans}},R_{\mathrm{fmt}}$	$R_{\mathrm{AU}}$	Acc	IoU
			56.75	–
$\checkmark$			89.02	46.73
$\checkmark$	$\checkmark$		90.22	43.46
$\checkmark$	$\checkmark$	$\checkmark$	92.80	60.24

Table 2: Ablation on TAG components with cumulative checkmarked modules and RAF-DB accuracy / AU IoU. The final row shows per-dataset tuning results.

Method	RAF-DB	FERPlus	AffectNet
Baseline (w/o grounding)	56.75	66.36	33.89
SFT w/o <bbox>	86.99	78.99	48.09
SFT (full)	89.02	81.01	50.00
$\Delta$ (w/o <bbox> vs full)	-2.03	-2.02	-1.91

Table 3: Ablation on grounding capability in SFT stage. We remove all <bbox> content while keeping other structure identical to evaluate grounding benefits.

Reward Design	RAF-DB		FERPlus
	Acc	IoU	Acc	IoU
IoU-based	92.80	61.78	91.50	53.27
F1-based	92.31	56.46	91.21	50.84

Table 4: Comparison of AU reward designs: IoU-based (validating bounding box coordinates against AU regions derived from an independent detector, OpenFace) vs F1-based (validating extracted AU labels).

Training Dataset	RAF-DB	FERPlus	AffectNet	Avg.
RL on RAF-DB	92.80	82.30	51.50	75.53
$\Delta$ (vs SFT, RAF-DB)	+3.78	+1.29	+1.50	+1.19
RL on FERPlus	89.50	91.50	50.51	77.17
$\Delta$ (vs SFT, FERPlus)	+0.48	+10.49	+0.51	+2.83

Table 5: Cross-dataset evaluation: RL training on RAF-DB or FERPlus, evaluation on all three datasets.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Human and LLM evaluation results.

Dataset	Classes used	# Test images
RAF-DB	7 (w/o contempt)	3,068
FERPlus	7 (w/o contempt)	3,517
AffectNet	7 (w/o contempt)	3,500

Appendix

Table 6: Evaluation splits under the unified 7-class protocol. We follow the official evaluation splits of each dataset and exclude the contempt category when constructing the test sets.

Figure 5: Training dynamics during two stages.

Panel of Figure 6 — Figure 6: TAG-310K appendix statistics: dataset composition and AU-related distributions.

Figure 7: Distribution of reasoning trace lengths (in tokens).

Panel of Figure 8 — Figure 8: Inference-time grounding analysis across RAF-DB (top row), FERPlus (middle row), and AffectNet (bottom row). Left two columns (a) and (b) show cases where TAG’s predictions align with external AU detectors, demonstrating consistency. Right two columns (c) and (d) show cases where TAG attends to regions beyond external detector coverage, demonstrating emergent grounding capability. Green boxes indicate external AU detector outputs with confidence scores, while red boxes represent TAG’s predicted regions.

Facial Expression Recognition

Facial Expression Recognition studies how computers infer emotion and facial actions from images and video.

See all 79 papers

ActFER: Agentic Facial Expression Recognition via Active Tool-Augmented Visual Reasoning
Apr 10, 2026Shifeng Liu, Zhengye Zhang, Sirui Zhao +7Facial Expression Recognition Affective Computing
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China, Hefei, China · Tsinghua University, Beijing, China · University of Science and Technology of China, Hefei, China +2
LaCoVL-FER: Landmark-Guided Contrastive Learning Network with Vision-Language Enhancement for Facial Expression Recognition
May 19, 2026Jiaxin Wang, Muwei Jian, Hui Yu +2VLM Adaptation Facial Expression Recognition
School of Airspace Science and Engineering, Shandong University, Weihai, China · School of Computer Science and Technology, Shandong University of Finance and Economics, Jinan, China · School of Psychology and Neuroscience, University of Glasgow, Glasgow, U.K. +1
SRE-FER: Regional residual evidence learning for mitigating local evidence dilution in fine-grained facial expression recognition
Aug 9, 2026Jiaye Song, Ruochen Zhang, Yuliang Wang +1Residual Learning Facial Expression Recognition
Tiangong University, No. 399 Binshuixi Road, Xiqing District, Tianjin 300387, China

Apr 10, 2026J/K move · Enter open · S save

ActFER: Agentic Facial Expression Recognition via Active Tool-Augmented Visual Reasoning

Shifeng Liu, Zhengye Zhang, Sirui Zhao, Xinglong Mao+6

State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China, Hefei, China · Tsinghua University, Beijing, China · University of Science and Technology of China, Hefei, China+2

Facial Expression Recognition Affective Computing Active Feature Acquisition Facial Action Unit Detection Agentic RL

Open paper AskPDF arXiv

Recent advances in Multimodal Large Language Models (MLLMs) have created new opportunities for facial expression recognition (FER), moving it beyond pure label prediction toward reasoning-based affect understanding. However, existing MLLM-based FER methods still follow a passive paradigm: they rely on externally prepared facial inputs and perform single-pass reasoning over fixed visual evidence, without the capability for active facial perception. To address this limitation, we propose ActFER, an agentic framework that reformulates FER as active visual evidence acquisition followed by multimodal reasoning. Specifically, ActFER dynamically invokes tools for face detection and alignment, selectively zooms into informative local regions, and reasons over facial Action Units (AUs) and emotions through a visual Chain-of-Thought. To realize such behavior, we further develop Utility-Calibrated GRPO (UC-GRPO), a reinforcement learning algorithm tailored to agentic FER. UC-GRPO uses AU-grounded multi-level verifiable rewards to densify supervision, query-conditional contrastive utility estimation to enable sample-aware dynamic credit assignment for local inspection, and emotion-aware EMA calibration to reduce noisy utility estimates while capturing emotion-wise inspection tendencies. This algorithm enables ActFER to learn both when local inspection is beneficial and how to reason over the acquired evidence. Comprehensive experiments show that ActFER trained with UC-GRPO consistently outperforms passive MLLM-based FER baselines and substantially improves AU prediction accuracy.

Abstract

Facial Expression Recognition

Explore similar work

ActFER: Agentic Facial Expression Recognition via Active Tool-Augmented Visual Reasoning

LaCoVL-FER: Landmark-Guided Contrastive Learning Network with Vision-Language Enhancement for Facial Expression Recognition

SRE-FER: Regional residual evidence learning for mitigating local evidence dilution in fine-grained facial expression recognition