When, Where, and Why: Hierarchical Causal Explainability for Video Face Understanding with MLLMs

Anubhav Kataria; Shreya Ghosh; Jingwen Ye; Abhinav Dhall

cs.CVOct 13, 2026

When, Where, and Why: Hierarchical Causal Explainability for Video Face Understanding with MLLMs

Authors: Anubhav Kataria, Shreya Ghosh, Jingwen Ye, Abhinav Dhall

Organizations: Monash University, Melbourne, Australia · Indian Institute of Technology Kharagpur, Kharagpur, India

Abstract

Understanding facial behaviour in video requires answering three fundamental questions: when a key facial expression occurs, where the model attends on the face at that moment, and why that evidence is necessary rather than merely correlated. Today's multimodal large language models (MLLMs) can label and describe video, but they rarely provide faithful explanations, especially for small facial movements that change over time. We present VideoFace-X-LLM, a framework for explainable facial video understanding comprising two key components. First, we introduce ExplainFace, a weakly supervised video-text corpus for facial behaviour understanding, automatically constructed from CelebV-HQ videos and validated through human evaluation and a two-model audit (at most 2.3% contradictions), demonstrating consistent gains across two backbone architectures. Second, H-TCC (Hierarchical Temporal Spatial Causal Concepts), which finds the key temporal moments (When), the facial regions (Where), and verifies its causal necessity (Why) through targeted occlusion. Fine-tuning on ExplainFace raises appearance Macro F1 from 19.48% to 32.87% (+13.39), with gains of +4.38 on actions, +6.77 on temporal emotion, and +11.84 F1 on deepfake detection. We also measure the explanations pointing lift, face/edge lift, stability, an Adebayo sanity check, and a necessity test rather than only showing examples. Our models, code, and corpus are available on request for research use.

Figures & tables

Figure 1: Overview of VideoFace-X-LLM. Left: the ExplainFace corpus is built by prompting Gemini for multi-aspect video descriptions, then auditing them for quality and bias. Top: the MLLM is fine-tuned on this corpus. Right: H-TCC explains each prediction by finding when the key moment occurs, where the model looks on the face, and why that evidence is needed.

Description aspect	Task it mainly supervises
Visual Elements	Appearance attributes
Actions & Interaction	Action recognition
Emotional Tone & Expressions	Emotion recognition, Action Units
Temporal Flow	Temporal emotion, deepfake (motion cues)
Audio Components	Multimodal context (speech, affect)
Technical Aspects	Manipulation / deepfake artifacts

Table 1: How each ExplainFace description aspect supervises a downstream task.

Aspect	SBERT $\uparrow$	BLEU-1 $\uparrow$	ROUGE-L $\uparrow$
Theme & Narrative	0.46	0.06	0.13
Visual Elements	0.43	0.05	0.14
Actions & Movements	0.46	0.06	0.14
Audio Components	0.45	0.08	0.17
Emotional Tone	0.44	0.04	0.11
Technical Aspects	0.54	0.06	0.16

Table 2: User Study 1: human vs. Gemini descriptions (

531

videos). SBERT measures meaning; BLEU and ROUGE measure word overlap. High SBERT with low BLEU/ROUGE means both describe the same content in different words.

Level	What varies	Samples	Human Acc.
Easy	Random videos	368	99.18%
Medium	Similar content	275	94.55%
Hard	Same-clip segments	274	91.24%
Overall		917	95.42%

Table 3: User Study 2: human video retrieval accuracy. A participant reads a description and picks the matching video from four options. Chance accuracy is

25\%

	Semantic $\uparrow$		Lexical $\downarrow$		NLI (%)
Auditor	BERT	Cosine	BLEU	ROUGE-L	Ent. $\uparrow$	Neu.	Con. $\downarrow$
(Pair)	Score	Sim.
Gemini 31,128	0.93	0.73	0.21	0.48	53.8	43.9	2.30
Qwen3 33,008	0.93	0.74	0.20	0.46	36.9	61.5	1.60

Table 4: Semantic similarity between flagged (interpretive) statements and their suggested observational alternatives. Semantic: meaning; Lexical: word overlap; NLI: Natural Language Inference; Ent: Entailment, Neu.: Neutral, Con.: Contradiction.

Figure 2: Overview of H-TCC. Phase 1 detects the peak moment ( When ); Phase 2 localises where the model attends on the face ( Where ); Phase 3 tests whether that evidence is needed ( Why ). See Sec. 5 .

Figure 3: Qualitative H-TCC results. Phase 1 shows the temporal saliency at different time windows. Phase 2 shows where the model looks for the attribute black hair . Phase 3 shows the necessity test.

Model	Appearance		Action		Emotion				Deepfake		Action Units
					Video-Level		Temporal
	mA $\uparrow$	F1 ${}_{\text{mac}}$ $\uparrow$	mA $\uparrow$	F1 ${}_{\text{mac}}$ $\uparrow$	Acc $\uparrow$	F1 ${}_{\text{mac}}$ $\uparrow$	Acc $\uparrow$	F1 ${}_{\text{mac}}$ $\uparrow$	Acc $\uparrow$	F1 ${}_{\text{mac}}$ $\uparrow$	mA $\uparrow$	Frame F1	Clip F1
InternVideo2.5 (Base)	88.10	19.48	93.69	16.44	82.06	21.96	84.04	20.04	51.01	33.96	88.3	21.1	21.6
Ours (ViT-Only)	86.16	26.29	94.50	17.88	55.61	22.00	71.47	23.83	50.00	35.23	71.0	16.7	16.8
Ours (ViT+LLM)	86.36	32.87	94.80	20.82	60.57	25.68	74.84	26.81	56.32	45.80	79.2	31.6	32.2
Qwen3-VL	89.19	36.85	95.32	23.22	81.09	26.11	85.88	13.06	56.07	47.88	88.9	37.7	38.2

Table 5: Results on six tasks. Appearance/Action/Emotion on CelebV-HQ (identity-independent test split, 6,923 clips), Deepfake on FaceForensics++, Action Units on DISFA. Temporal emotion is frame-by-frame at 10 Hz. mA = Mean Accuracy, F1

{}_{\text{mac}}

= Macro F1 (primary metric, robust to imbalance). "Qwen3-VL + EF" = Qwen3-VL fine-tuned on ExplainFace. Bold = best.

Metric	Emotion	Appearance	Action
Multi-stride stability ( $\uparrow$ )	0.46	0.39	0.41
Adebayo endpoint ( $\rightarrow 0$ )	$-0.12$	$-0.22$	$-0.13$
Spatial pointing lift ( $\uparrow$ , chance $=1$ )	1.42 $\times$	1.40 $\times$	1.40 $\times$
Face lift ( $\uparrow$ , uniform $=1$ )	1.23 $\times$	1.23 $\times$	1.24 $\times$
Edge lift ( $\downarrow$ , want $<1$ )	0.80 $\times$	0.80 $\times$	0.80 $\times$

Table 6: Faithfulness of H-TCC explanations. Higher is better for lifts, stability and causal scores; edge lift should be

<1

; the Adebayo endpoint should fall to about

0

or below.

VLM Interpretability

Vision-language model interpretability explains how image and language representations shape predictions, evidence use, and multimodal reasoning.

See all 259 papers

MLLM-based Textual Explanations for Face Comparison
Mar 17, 2026Redwan Sony, Anil K Jain, Arun RossFaithfulness of Language Model Explanations Face Recognition
Department of Computer Science and Engineering, Michigan State University, East Lansing, MI, USA
Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events
Jun 1, 2026Xiaolin Liu, Yilun Zhu, Xiangyu Zhao +9VLM Evaluation Video QA
Shandong University · Southeast University · Shanghai Jiao Tong University +1
EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment
Aug 21, 2026Guray Ozgur, Mustafa Efe Tamyapar, Naser Damer +1Face Recognition Concept-Based Explanations
Fraunhofer IGD, Fraunhoferstr. 5, 64283 Darmstadt, Germany · TU Darmstadt, Karolinenpl. 5, 64289 Darmstadt, Germany

Mar 17, 2026J/K move · Enter open · S save

MLLM-based Textual Explanations for Face Comparison

Redwan Sony, Anil K Jain, Arun Ross

Department of Computer Science and Engineering, Michigan State University, East Lansing, MI, USA

Faithfulness of Language Model Explanations Face Recognition Multimodal Large Language Models Hallucination Detection in VLMs

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have recently been proposed as a means to generate natural-language explanations for face recognition decisions. While such explanations facilitate human interpretability, their reliability on unconstrained face images remains underexplored. In this work, we systematically analyze MLLM-generated explanations for the unconstrained face verification task on the challenging IJB-S dataset, with a particular focus on extreme pose variation and surveillance imagery. Our results show that even when MLLMs produce correct verification decisions, the accompanying explanations frequently rely on non-verifiable or hallucinated facial attributes that are not supported by visual evidence. We further study the effect of incorporating information from traditional face recognition systems, viz., scores and decisions, alongside the input images. Although such information improves categorical verification performance, it does not consistently lead to faithful explanations. To evaluate the explanations beyond decision accuracy, we introduce a likelihood-ratio-based framework that measures the evidential strength of textual explanations. Our findings highlight fundamental limitations of current MLLMs for explainable face recognition and underscore the need for a principled evaluation of reliable and trustworthy explanations in biometric applications. Code is available at https://github.com/redwankarimsony/LR-MLLMFR-Explainability.

Abstract

VLM Interpretability

Explore similar work

MLLM-based Textual Explanations for Face Comparison

Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events

EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment