Benchmarking the Explanatory Quality of Open-Weight Vision-Language Models in Face Recognition

Laurent Colbois; Sébastien Marcel

cs.CVSep 18, 2026

Benchmarking the Explanatory Quality of Open-Weight Vision-Language Models in Face Recognition

Authors: Laurent Colbois, Sébastien Marcel

Organizations: Idiap Research Institute, Switzerland · Université de Lausanne, Switzerland

Abstract

Vision-Language Models (VLMs) have recently been proposed as promising tools for face recognition, as they can produce natural language explanations alongside similarity scores. This capability is considered appealing for face comparisons in forensic contexts, which require decisions to be transparent and auditable. However, existing evaluations of VLMs for that use case focus mostly on recognition accuracy, while the validity of generated explanations remains unquantified. In this work, we introduce a benchmarking framework for VLM-based face recognition that treats explanation quality as a core evaluation axis. We propose two criteria that explanations should satisfy: relevance, i.e., reliance on identity-stable facial features; and faithfulness, i.e., alignment with the visible image content without hallucinated features. We jointly develop a methodology enabling the quantification of relevance and faithfulness of evaluated models, based on constraining model outputs to a structured explanation format that supports automated querying and auditing. Using this framework, we benchmark several families of open-weight VLMs, jointly evaluating face verification accuracy and explanation quality. Our results highlight remaining shortcomings of produced explanations, and emphasize the need for such explanation quality metrics to get a complete picture of model performance. The proposed benchmark and open-source evaluation harness provide a foundation for proper benchmarking and future fine-tuning of explainable face recognition systems.

Figures & tables

Figure 1: We prompt VLMs with pairs of face images and a question requiring them to assess whether these images represent the same person or not. Moreover, the model is forced to structure its answer following a predefined JSON schema, thus enabling facilitated queryability of the generated explanations for automated scoring.

System Prompt

You are a face identity verification expert.

User Prompt

Analyze the two provided face images (A and B). Compare them feature-by-feature to decide whether they show the same person or different people. Output a single JSON object with exactly these keys: • feature_comparisons : list of facial feature comparisons; for each feature include: – feature : the facial feature name in snakecase format – description_a : description of the feature in image A. – description_b : description of the feature in image B. – comparison : concise comparison of this feature across A and B. • explanation : a brief global justification summarizing how the local comparisons lead to the final decision. • similarity_score : a number between 0.0 (definitely different people) and 1.0 (definitely the same person). Do not nominally mention the names of the individuals in the images; focus only on facial characteristics. Respond strictly in JSON format with the specified keys.

Figure 2: The prompts used for the structured setting

Relevance metric: unstable features
Category	Example keywords
Accessories (27)	glasses, mask, hat, scarf, jewelry, …
Attire (17)	shirt, jacket, collar, sleeve, outfit, …
Environment (16)	background, lighting, shadow, blur, image, …
Expression (15)	expression, smile, gaze, frown, gesture, …
Grooming (16)	makeup, hairstyle, beard, mustache, blemish, …

Table 1: Lexicons used for Relevance and Faithfulness metrics. Counts in parentheses indicate the total number of keywords per category. Note on occlusion acknowledgments: if a listed feature appears in the trigger lexicon but its description simply acknowledges occlusion (by containing these terms), we do not count it as a hallucination. The full lexicon is presented as supplementary material.

	Dataset	LFW			ARFace			Soteria			CelebA
	Metric	EER	ACC	FTA	EER	ACC	FTA	EER	ACC	FTA	EER	ACC	FTA
ArcFace	-	0.3	99.9	0.0	0.1	100.0	0.0	0.3	99.7	0.1	3.0	98.1	0.2
Gemma3	4B	9.5	90.4	0.1	11.3	88.9	0.1	9.0	91.5	0.2	13.5	86.5	0.2
	12B	7.6	92.6	0.0	17.1	85.1	0.0	10.1	90.7	0.1	12.1	88.3	0.2
	27B	5.2	94.7	0.0	7.1	93.0	0.0	4.6	95.5	0.1	10.4	89.6	0.2
Qwen2.5-VL	3B	44.3	55.5	2.6	39.8	60.3	1.8	38.3	63.0	2.2	44.2	55.7	2.6

Table 2: EER , Accuracy (ACC) and FTA , in % in the structured output setting. The best performing VLM (in EER, respectively accuracy) is bolded .

	Dataset	LFW			ARFace			Soteria			CelebA
	Metric	EER	ACC	FTA	EER	ACC	FTA	EER	ACC	FTA	EER	ACC	FTA
Gemma3	4B	+2.6	-2.8	+0.1	+1.5	-1.4	+0.1	+2.9	-2.8	+0.0	+1.4	-1.4	+0.1
	12B	-2.4	+0.5	-0.0	+0.2	+5.6	-0.0	-4.0	+2.7	+0.0	-1.5	+0.4	-0.0
	27B	+1.1	-1.2	-0.0	+0.6	+0.1	-0.0	+0.2	-0.1	+0.0	+1.2	-1.2	-0.0
Qwen2.5-VL	3B	+14.1	-16.2	+2.5	+8.5	-10.7	+1.8	+8.0	-7.5	+2.0	+12.9	-13.5	+2.4
	7B	+19.0	-18.9	+0.3	+17.1	-17.8	+0.2	+15.2	-15.2	+0.2	+14.3	-15.9	+0.1

Table 3: Difference in EER , Accuracy and FTA , in percentage points, in the structured output setting compared to the unstructured output setting. A positive value means that the corresponding metric is higher in the structured output setting than in the unstructured one.

Panel of Figure 3 — Figure 3: Scaling trends for relevance and faithfulness metrics. We report $(1-\mathrm{FTA})\times$ score for relevance and faithfulness, reflecting end-to-end usable explanation quality by accounting for cases where no valid structured explanation is produced (FTA).

Feature	Description A	Description B
hair	short, grey hair	no detectable hair
hairstyle	Long, straight hair framing the face.	Short, dark hair styled back.
glasses	no glasses	black expansive glasses with reflective lenses
beard	Full, graying beard and mustache.	Full, graying beard and mustache.
expression	Neutral expression.	Slightly intense expression.
eyewear	thick rim glasses with noticeable reflection	thin rim glasses with noticeable reflection

Table 4: Examples of unstable features in face descriptions. We showcase a mixture of examples from all models and all datasets.

Panel of Figure 4 — Figure 4: Examples of hallucinations in facial descriptions. The caption is the content of the description field produced by the VLM for that specific image.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Relevance metric: unstable features
Category	Keywords
Accessories	glasses, spectacles, eyewear, goggles, sunglasses, mask, respirator, shield, visor, hat, cap, helmet, beanie, hood, headwear, bandana, headband, scarf, tie, jewelry, earring, piercing, stud, hoop, necklace, clip, bindi
Attire	cloth, shirt, jacket, coat, attire, garment, outfit, dress, suit, uniform, collar, sleeve, zipper, neckline, shoulder, chest, glove
Environment	background, foreground, backdrop, scene, context, lighting, shadow, blur, focus, quality, noise, pixel, image, photo, picture, crop
Expression	expression, emotion, mood, feeling, gaze, look, smile, smiling, laugh, frown, grimace, mouth open, teeth visible, wink, gesture
Grooming	makeup, cosmetic, mascara, liner, lipstick, hairstyle, haircut, hairdo, dye, beard, mustache, shaved, trim, acne, blemish, pimple

Appendix

Table 5: Lexicons used for Relevance and Faithfulness metrics. Note on the ”occlusion acknowledgments” list: if a listed feature is part of the trigger lexicon, but the description of that feature simply acknowledges occlusion (by containing these terms), we do not count it as a hallucination.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 957 papers

Vision Language Model Fusion for Explainable Face Recognition
Aug 25, 2026Ana Estrada-Real, Lydia Alapatt, Christoph Busch +1Face Recognition VLM Interpretability
da/sec Biometrics and Security Research Group Hochschule Darmstadt, Germany
EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment
Aug 21, 2026Guray Ozgur, Mustafa Efe Tamyapar, Naser Damer +1Face Recognition Concept-Based Explanations
Fraunhofer IGD, Fraunhoferstr. 5, 64283 Darmstadt, Germany · TU Darmstadt, Karolinenpl. 5, 64289 Darmstadt, Germany
MLLM-based Textual Explanations for Face Comparison
Mar 17, 2026Redwan Sony, Anil K Jain, Arun RossFaithfulness of Language Model Explanations Face Recognition
Department of Computer Science and Engineering, Michigan State University, East Lansing, MI, USA

Aug 25, 2026J/K move · Enter open · S save

Vision Language Model Fusion for Explainable Face Recognition

Ana Estrada-Real, Lydia Alapatt, Christoph Busch, Christian Rathgeb

da/sec Biometrics and Security Research Group Hochschule Darmstadt, Germany

Face Recognition VLM Interpretability VLM Evaluation

Open paper AskPDF arXiv

Responsible deployment of face verification systems requires more than accurate decisions: systems should also provide interpretable and auditable evidence that enables users to understand, assess, and challenge their decisions. Vision-language models (VLMs) provide a promising foundation for explainable face recognition by combining visual analysis with natural-language reasoning. However, relying on a single model may further limit the decision accuracy as well as provided explanations. This work therefore investigates whether multiple VLMs can be combined to improve recognition accuracy, and to enrich the explanations associated with those decisions. This work evaluates four VLMs as standalone face verification systems and subsequently proposes a fusion framework, where two source models provide similarity scores and textual justifications and a third VLM acts as a decider model. Four different fusion scenarios are considered, progressively providing the decider model with scores, justifications, face images, and combinations of these modalities. Overall, the findings suggest that the value of multi-VLM fusion extends beyond recognition performance. VLMs can provide complementary justifications and perspectives that enable richer explanations of face recognition decisions, supporting greater transparency, auditability, and error analysis. This is relevant to the development of responsible explainable face verification systems, where users and operators should be able to understand not only the final decision but also the evidence and potential sources underlying it. The proposed multimodal VLM, which combines decision scores, explanations, and face images, achieves higher recognition accuracy than state-of-the-art VLMs and domain-specific face recognition models, while also providing fused explanations that are expected to be more robust than those generated by individual VLMs.

Abstract

VLM Evaluation

Explore similar work

Vision Language Model Fusion for Explainable Face Recognition

EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment

MLLM-based Textual Explanations for Face Comparison