ActFER: Agentic Facial Expression Recognition via Active Tool-Augmented Visual Reasoning

Shifeng Liu; Zhengye Zhang; Sirui Zhao; Xinglong Mao; Zhehan Kan; Zhixiang Wei; Shiwei Wu; Chaoyou Fu

cs.CVApr 10, 2026

ActFER: Agentic Facial Expression Recognition via Active Tool-Augmented Visual Reasoning

Authors: Shifeng Liu, Zhengye Zhang, Sirui Zhao, Xinglong Mao, Zhehan Kan, Zhixiang Wei, Shiwei Wu, Chaoyou Fu,

Organizations: State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China, Hefei, China · Tsinghua University, Beijing, China · University of Science and Technology of China, Hefei, China · Tsinghua University, Shenzhen, China ·

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have created new opportunities for facial expression recognition (FER), moving it beyond pure label prediction toward reasoning-based affect understanding. However, existing MLLM-based FER methods still follow a passive paradigm: they rely on externally prepared facial inputs and perform single-pass reasoning over fixed visual evidence, without the capability for active facial perception. To address this limitation, we propose ActFER, an agentic framework that reformulates FER as active visual evidence acquisition followed by multimodal reasoning. Specifically, ActFER dynamically invokes tools for face detection and alignment, selectively zooms into informative local regions, and reasons over facial Action Units (AUs) and emotions through a visual Chain-of-Thought. To realize such behavior, we further develop Utility-Calibrated GRPO (UC-GRPO), a reinforcement learning algorithm tailored to agentic FER. UC-GRPO uses AU-grounded multi-level verifiable rewards to densify supervision, query-conditional contrastive utility estimation to enable sample-aware dynamic credit assignment for local inspection, and emotion-aware EMA calibration to reduce noisy utility estimates while capturing emotion-wise inspection tendencies. This algorithm enables ActFER to learn both when local inspection is beneficial and how to reason over the acquired evidence. Comprehensive experiments show that ActFER trained with UC-GRPO consistently outperforms passive MLLM-based FER baselines and substantially improves AU prediction accuracy.

Figures & tables

Figure 1. Comparison of FER paradigms. Previous methods rely on passive reasoning over fixed inputs, while ActFER performs tool-augmented agentic inspection. A comparison figure contrasting three paradigms for facial expression recognition. On the left, traditional FER models consume preprocessed face images and output emotion labels. In the middle, MLLM-based FER models take preprocessed faces and perform text-based chain-of-thought reasoning before predicting emotions, reflecting external preprocessing, fixed visual evidence, and single-pass passive reasoning. On the right, ActFER starts from raw images, invokes a tool library for face detection, alignment, and adaptive ROI zooming with an explicit decision of whether and where to zoom, performs multimodal CoT reasoning with AU analysis, and then outputs the final emotion answer, reflecting active internal preprocessing, adaptive local inspection, and tool-augmented agentic reasoning.

Figure 2. Overall architecture of ActFER. The agent combines tool-augmented visual reasoning, perceptual tools, FACS-grounded hierarchical inference, and two-stage SFT+UC-GRPO optimization. Overall ActFER agent showing the input image, perceptual tool layer, FACS-grounded reasoning process, and two-stage training pipeline.

Figure 3. Statistics of the curated training data. A single-column combined figure summarizing the curated training data. It includes the emotion distributions of the SFT and RL subsets and a bar chart comparing AU frequencies between the two splits. The RL subset is more balanced across emotions, especially for low-resource categories, while preserving similar AU coverage.

Model	Benchmark F1				Overall		Per-emotion F1
Model	RAF-DB	FERPlus	AffectNet	SFEW2.0	Acc	F1	Neutral	Happiness	Sad	Surprise	Fear	Disgust	Anger	Contempt
General-purpose MLLMs
InternVL3.5-4B* ( Wang et al., 2025b )	60.69	37.87	44.51	33.96	50.40	43.67	55.56	81.89	53.47	55.06	17.44	35.59	46.68	3.70
Qwen3VL-4B* ( Bai et al., 2025a )	56.13	47.37	36.12	38.57	55.27	44.03	60.12	81.93	54.01	56.06	22.97	23.47	50.00	3.65
Qwen2.5VL-7B ( Bai et al., 2025b )	50.26	46.25	26.97	36.22	53.78	35.34	54.60	67.29	53.91	52.70	3.28	22.26	46.97	0.37
InternVL3-8B ( Zhu et al., 2025 )	52.69	43.34	39.33	37.27	60.59	44.54	65.08	85.56	61.57	40.02	41.35	38.31	54.41	14.55

Table 1. Main emotion-recognition results. The left block reports benchmark-level F1 (%), overall Acc (%) and F1 (%) on FERBench; the right block reports per-emotion F1 (%). * InternVL3.5-4B ( Wang et al., 2025b ) , Qwen3VL-4B ( Bai et al., 2025a ) , EmoLA ( Li et al., 2025b ) , and ExpLLM ( Lan et al., 2025 ) are reproduced from official checkpoints, with evaluation strictly following the FERBench protocol. Results of other open-source baselines are taken from UniFER ( Zhang et al., 2025a ) . Best results are boldfaced ; second best are underlined .

Figure 4. Emotion-wise comparison between ActFER-SFT and ActFER on FERBench. Bars denote the proportion of invoking zoom, and lines denote the per-emotion F1. A two-column comparison plot over eight emotions. For each emotion, two bars show the zoom ratio of ActFER-SFT and Full ActFER, and two lines show their corresponding F1 scores. Full ActFER increases zoom usage across all emotions, with the strongest increase and largest F1 improvement on contempt.

Method	1	2	4	6	9	12	25	26	Avg.
Qwen2.5VL-7B ( Bai et al., 2025b )	20.8	19.2	23.0	17.9	17.1	26.3	30.9	21.6	22.1
Qwen3VL-4B* ( Bai et al., 2025a )	24.7	27.1	76.7	3.1	0.0	73.7	80.7	18.6	38.1
InternVL3.5-4B* ( Wang et al., 2025b )	36.9	40.7	71.8	2.1	35.3	67.6	67.2	4.6	40.8
FEALLM <7B> ( Hu et al., 2025 )	36.9	29.7	70.4	30.2	–	54.5	51.9	24.7	42.6
ActFER-SFT <4B>	30.2	30.8	70.2	51.4	44.1	67.7	82.0	12.5	48.6
ActFER <4B>	37.1	38.2	77.3	58.2	52.3	78.0	92.4	32.0	58.2

Table 2. Comparison of zero-shot AU F1 (%) on DISFA under the standard 8-AU protocol. The table lists the F1 score for each individual AU together with the average F1 (Avg F1) over all 8 AUs. * InternVL3.5-4B ( Wang et al., 2025b ) and Qwen3VL-4B ( Bai et al., 2025a ) are reproduced on DISFA using their official checkpoints.

Variant	$\mathcal{R}_{acc}$	$\mathcal{R}_{tool}$			FERBench			DISFA
Variant	AU-grounded dense reward	Contrastive utility	Emotion-EMA	Symmetric fallback	Acc	F1	Zoom	Avg.F1	Zoom
ActFER-SFT	–	–	–	–	65.70	53.37	32.6	48.6	8.4
Emotion-only RL	✗	✗	–	✓	69.14	61.49	0.0	47.8	0.0
AU-grounded RL	✓	✗	–	✓	68.80	62.72	0.0	50.1	0.0
Zoom-biased RL	✓	✗	–	✗	71.86	62.22	100.0	52.0	100.0
w/o Emotion-EMA	✓	✓	✗	✓	70.01	65.34	90.1	54.2	100.0

Table 3. UC-GRPO ablation of ActFER. The configuration columns indicate whether the AU-grounded dense reward

\mathcal{R}_{acc}

is enabled and which components are included in the tool-side reward

\mathcal{R}_{tool}

. The FERBench block reports emotion Accuracy/F1 and the zoom ratio (i.e., the percentage of samples that invoke zoom); the DISFA block reports average AU F1 and the zoom ratio. For Zoom-biased RL , the entire utility reward

\mathcal{R}_{util}

is disabled, including both the adaptive branch and the symmetric fallback, and is replaced by a constant zoom-preference bonus to bias the policy toward using zoom.

Figure 5. Training-time emotion accuracy for three utility-calibration variants. Full ActFER attains the best late-stage accuracy plateau, while removing EMA or utility calibration results in policy collapse during the later stage of training. A single line chart comparing training-time emotion accuracy for Full ActFER, a variant without emotion-wise EMA, and a variant without utility calibration. Full ActFER reaches the highest late-stage plateau, the w/o EMA variant oscillates more, and the w/o utility variant converges lower.

Panel of Figure 6 — Figure 6. Qualitative analyses of ActFER. (a) The model aligns the face, invokes zoom_in on task-relevant regions, analyzes local AU evidence, and predicts anger. (b) On subtle contempt and sadness examples, adaptive zoom reveals weak local cues that the same model misses without zoom. Two qualitative analyses. Panel a shows an end-to-end ActFER trajectory for an anger example: the system aligns a face, zooms into four facial regions, identifies AU4, AU5, AU15, and AU25, and predicts anger. Panel b compares adaptive zoom with no zoom. For contempt, zoom reveals a slight squint and asymmetric upward mouth-corner pull and corrects a happiness prediction; for sadness, zoom reveals the downward mouth-corner pattern of AU15 and corrects a neutral prediction.

Facial Expression Recognition

Facial Expression Recognition studies how computers infer emotion and facial actions from images and video.

See all 79 papers

TAG: Thinking with Action Unit Grounding for Facial Expression Recognition
Feb 21, 2026Haobo Lin, Tianyi Bai, Jiajun Zhang +5Facial Expression Recognition VLM Reasoning
PKU · JLU · Shanghai AI Lab +2
LaCoVL-FER: Landmark-Guided Contrastive Learning Network with Vision-Language Enhancement for Facial Expression Recognition
May 19, 2026Jiaxin Wang, Muwei Jian, Hui Yu +2VLM Adaptation Facial Expression Recognition
School of Airspace Science and Engineering, Shandong University, Weihai, China · School of Computer Science and Technology, Shandong University of Finance and Economics, Jinan, China · School of Psychology and Neuroscience, University of Glasgow, Glasgow, U.K. +1
SRE-FER: Regional residual evidence learning for mitigating local evidence dilution in fine-grained facial expression recognition
Aug 9, 2026Jiaye Song, Ruochen Zhang, Yuliang Wang +1Residual Learning Facial Expression Recognition
Tiangong University, No. 399 Binshuixi Road, Xiqing District, Tianjin 300387, China

Feb 21, 2026J/K move · Enter open · S save

TAG: Thinking with Action Unit Grounding for Facial Expression Recognition

Haobo Lin, Tianyi Bai, Jiajun Zhang, Xuanhao Chang+4

PKU · JLU · Shanghai AI Lab+2

Facial Expression Recognition VLM Reasoning Faithfulness in VLMs

Open paper AskPDF arXiv

Facial Expression Recognition (FER) is a fine-grained visual understanding task where reliable predictions require reasoning over localized and meaningful facial cues. Recent vision--language models (VLMs) enable natural language explanations for FER, but their reasoning is often ungrounded, producing fluent yet unverifiable rationales that are weakly tied to visual evidence and prone to hallucination, leading to poor robustness across different datasets. We propose TAG (Thinking with Action Unit Grounding), a vision--language framework that explicitly constrains multimodal reasoning to be supported by facial Action Units (AUs). TAG requires intermediate reasoning steps to be grounded in AU-related facial regions, yielding predictions accompanied by verifiable visual evidence. The model is trained via supervised fine-tuning on AU-grounded reasoning traces followed by reinforcement learning with an AU-aware reward that aligns predicted regions with external AU detectors. Evaluated on RAF-DB, FERPlus, and AffectNet, TAG consistently outperforms strong open-source and closed-source VLM baselines while simultaneously improving visual faithfulness. Ablation and preference studies further show that AU-grounded rewards stabilize reasoning and mitigate hallucination, demonstrating the importance of structured grounded intermediate representations for trustworthy multimodal reasoning in FER. The code will be available at https://github.com/would1920/FER_TAG .

Abstract

Facial Expression Recognition

Explore similar work

TAG: Thinking with Action Unit Grounding for Facial Expression Recognition

LaCoVL-FER: Landmark-Guided Contrastive Learning Network with Vision-Language Enhancement for Facial Expression Recognition

SRE-FER: Regional residual evidence learning for mitigating local evidence dilution in fine-grained facial expression recognition