Vision Language Model Fusion for Explainable Face Recognition

Ana Estrada-Real; Lydia Alapatt; Christoph Busch; Christian Rathgeb

cs.CVAug 25, 2026

Vision Language Model Fusion for Explainable Face Recognition

Authors: Ana Estrada-Real, Lydia Alapatt, Christoph Busch, Christian Rathgeb

Organizations: da/sec Biometrics and Security Research Group Hochschule Darmstadt, Germany

Abstract

Responsible deployment of face verification systems requires more than accurate decisions: systems should also provide interpretable and auditable evidence that enables users to understand, assess, and challenge their decisions. Vision-language models (VLMs) provide a promising foundation for explainable face recognition by combining visual analysis with natural-language reasoning. However, relying on a single model may further limit the decision accuracy as well as provided explanations. This work therefore investigates whether multiple VLMs can be combined to improve recognition accuracy, and to enrich the explanations associated with those decisions. This work evaluates four VLMs as standalone face verification systems and subsequently proposes a fusion framework, where two source models provide similarity scores and textual justifications and a third VLM acts as a decider model. Four different fusion scenarios are considered, progressively providing the decider model with scores, justifications, face images, and combinations of these modalities. Overall, the findings suggest that the value of multi-VLM fusion extends beyond recognition performance. VLMs can provide complementary justifications and perspectives that enable richer explanations of face recognition decisions, supporting greater transparency, auditability, and error analysis. This is relevant to the development of responsible explainable face verification systems, where users and operators should be able to understand not only the final decision but also the evidence and potential sources underlying it. The proposed multimodal VLM, which combines decision scores, explanations, and face images, achieves higher recognition accuracy than state-of-the-art VLMs and domain-specific face recognition models, while also providing fused explanations that are expected to be more robust than those generated by individual VLMs.

Figures & tables

Year	Reference	Top VLM	Datasets tested	Performance
2024	DeAndres-Tame et al. [ 10 ]	ChatGPT1x1	LFW	93.50% Acc.
2025	Shahreza and Marcel [ 41 ]	GeminiPro1.5	Various (8 Mixed)	70.00% Acc.
2025	Sony et al. [ 43 ]	OpenCLIP-H-14	LFW	81.73% [email protected]%FMR
2026	Narayan et al. [ 34 ]	GeminiPro 1.5	Various (26 Mixed)	82.25% Acc.
2026	Shahreza and Marcel [ 42 ]	Qwen2-VL-7B-Instruct	LFW	93.28% Acc.

TABLE I: Best performing VLMs evaluated for FR in recent benchmarks.

Model	Parameters	Max. context	Size (GB)
Gemma4-31B-it [ 1 ]	31B	256k	62.6
InternVL3-8B [ 50 ]	8B	40k	15.9
Ovis-2.6-30B-A3B [ 27 ]	30B	64K	62.8
Qwen3-VL-8B-Instruct [ 3 ]	8B	256K	17.5

TABLE II: Open source VLMs used for this work.

Fig. 1: Diagram representing the models’ standalone FR test. Each model would get the 200,000 image pairs and be asked to output a similarity score between $[0,1]$ and a justification for each of them.

Fig. 2: Diagram of the models’ fusion. Each model would produce a similarity score and justification for each image pair. This information will then be used part of the fusion. There are four scenarios: scenario A where the decider model will get the similarity scores of two source models, scenario B where the decider model will get the similarity scores and justification of two source models. For scenarios C and D there heterogeneous data-sources are used, scenario C where the decider model has access to the original face image pairs and the similarity scores of two source models, and scenario D where the decider model will also have access to the original face image pairs, in addition to the similarity scores and justifications.

Model	AUC	EER	EER Threshold	d-prime	[email protected]%FMR	[email protected]%FMR	[email protected]%FMR	[email protected]%FMR
AdaFace [ 25 ]	0.9718	0.0562	0.0901	4.8732	0.0605	0.1915	0.0613	0.2343
LVFace [ 48 ]	0.9704	0.0578	0.0836	4.4983	0.0613	0.1673	0.0621	0.2047
MagFace [ 32 ]	0.9741	0.0557	0.1343	4.8431	0.0612	0.2652	0.0620	0.3201

TABLE III: Baseline FR results.

Fig. 3: DET curves showing the performance of the three baseline models.

Model	AUC	EER	EER Threshold	d-prime	[email protected]%FMR	[email protected]%FMR	[email protected]%FMR	[email protected]%FMR
MagFace	0.9741	0.0557	0.1343	4.8431	0.0612	0.2652	0.0620	0.3201
Gemma	0.9985	0.0130	0.5488	6.5239	0.0688	0.8745	0.2138	0.9230
Intern	0.9935	0.0329	0.6899	4.6494	0.3413	0.8653	0.6358	0.8868
Ovis	0.9932	0.0299	0.8228	4.5316	0.3685	0.9138	0.6522	0.9579
Qwen	0.9966	0.0152	0.7662	7.6937	0.1550	0.9508	0.4265	0.9602

TABLE IV: VLMs Standalone FR evaluation.

Panel of Figure 8 — Fig. 4: Distograms of the scores distribution for (a) Gemma (b) Intern (c) Ovis and (d) Qwen.

Fig. 5: DET curves from the standalone VLMs.

Model	AUC	EER	EER Threshold	d-prime	[email protected]%FMR	[email protected]%FMR	[email protected]%FMR	[email protected]%FMR
MagFace	0.9741	0.0557	0.1343	4.8431	0.0612	0.2652	0.0620	0.3201
Gemma	0.9985	0.0130	0.5488	6.5239	0.0688	0.8745	0.2138	0.9230
Gemma $\leftarrow$ Intern + Ovis	0.9962	0.0262	0.7289	4.8478	0.3224	0.8703	0.6486	0.9129
Gemma $\leftarrow$ Qwen + Intern	0.9980	0.0173	0.6665	6.6291	0.1687	0.8738	0.5471	0.9138
Gemma $\leftarrow$ Qwen + Ovis	0.9982	0.0164	0.7183	6.4134	0.1671	0.9020	0.5384	0.9527
Intern $\leftarrow$ Ovis + Gemma	0.9988	0.0144	0.6653	5.7538	0.0893	0.8616	0.3793	0.9090

TABLE V: Scenario A FR evaluation metrics for the 12 fusion combinations.

Panel of Figure 11 — Fig. 6: Scenario A DET curves presented are grouped by decider model: (a) Gemma (b) Intern (c) Ovis and (d) Qwen.

Model	AUC	EER	EER Threshold	d-prime	[email protected]%FMR	[email protected]%FMR	[email protected]%FMR	[email protected]%FMR
MagFace	0.9741	0.0557	0.1343	4.8431	0.0612	0.2652	0.0620	0.3201
Gemma	0.9985	0.0130	0.5488	6.5239	0.0688	0.8745	0.2138	0.9230
Gemma $\leftarrow$ Intern + Ovis	0.9962	0.0261	0.7260	4.8457	0.3239	0.8705	0.6485	0.9129
Gemma $\leftarrow$ Qwen + Intern	0.9977	0.0173	0.6676	6.6205	0.1652	0.8709	0.5458	0.9138
Gemma $\leftarrow$ Qwen + Ovis	0.9978	0.0164	0.7194	6.4181	0.1670	0.9020	0.5383	0.9527
Intern $\leftarrow$ Ovis + Gemma	0.9988	0.0143	0.6806	5.6971	0.0900	0.8620	0.3784	0.9090

TABLE VI: Scenario B FR evaluation metrics for the 12 fusion combinations.

Panel of Figure 13 — Fig. 7: Scenario B DET curves presented are grouped by decider model: (a) Gemma (b) Intern (c) Ovis (d) Qwen.

Model	AUC	EER	EER Threshold	d-prime	[email protected]%FMR	[email protected]%FMR	[email protected]%FMR	[email protected]%FMR
MagFace	0.9741	0.0557	0.1343	4.8431	0.0612	0.2652	0.0620	0.3201
Gemma	0.9985	0.0130	0.5488	6.5239	0.0688	0.8745	0.2138	0.9230
Gemma $\leftarrow$ Intern + Ovis	0.9987	0.0129	0.6759	6.6103	0.0878	0.8782	0.2906	0.9333
Gemma $\leftarrow$ Qwen + Intern	0.9988	0.0108	0.6627	7.5194	0.0649	0.8753	0.2759	0.9330
Gemma $\leftarrow$ Qwen + Ovis	0.9989	0.0112	0.6685	7.4923	0.0626	0.8941	0.3393	0.9362
Intern $\leftarrow$ Ovis + Gemma	0.9969	0.0239	0.7308	5.5130	0.2319	0.9028	0.5054	0.9451

TABLE VII: Scenario C FR evaluation metrics for the 12 fusion combinations.

Panel of Figure 15 — Fig. 8: Scenario C DET curves presented are grouped by decider model: (a) Gemma (b) Intern (c) Ovis (d) Qwen.

Model	AUC	EER	EER Threshold	d-prime	[email protected]%FMR	[email protected]%FMR	[email protected]%FMR	[email protected]%FMR
MagFace	0.9741	0.0557	0.1343	4.8431	0.0612	0.2652	0.0620	0.3201
Gemma	0.9985	0.0130	0.5488	6.5239	0.0688	0.8745	0.2138	0.9230
Gemma $\leftarrow$ Intern + Ovis	0.9985	0.0147	0.7130	5.8847	0.1047	0.8808	0.3595	0.9301
Gemma $\leftarrow$ Qwen + Intern	0.9987	0.0106	0.7276	7.4262	0.0720	0.8871	0.3052	0.9339
Gemma $\leftarrow$ Qwen + Ovis	0.9988	0.0113	0.7549	7.3283	0.0793	0.8973	0.3508	0.9282
Intern $\leftarrow$ Ovis + Gemma	0.9979	0.0184	0.7474	5.2555	0.1977	0.9033	0.4756	0.9500

TABLE VIII: Scenario D FR evaluation metrics for the 12 fusion combinations.

Panel of Figure 17 — Fig. 9: Scenario D DET curves presented are grouped by decider model: (a) Gemma (b) Intern (c) Ovis (d) Qwen.

Panel of Figure 18 — Fig. 10: Edge cases selected for analysing the quality of the explanations.

Panel of Figure 19 — Fig. 11: Edge cases selected for analysing the quality of the explanations, (a) corresponds to scenario C where Ovis had access to only the scores produced by Intern and Gemma, and (b) corresponds to scenario D where Gemma has access to the scores and justifications from Quen and Intern.

Face Recognition

Face recognition matches or identifies people from facial images, videos, and biometric representations across changing conditions.

See all 106 papers

Sep 18, 2026

Benchmarking the Explanatory Quality of Open-Weight Vision-Language Models in Face Recognition

Vision-Language Models (VLMs) have recently been proposed as promising tools for face recognition, as they can produce natural language explanations alongside similarity scores. This capability is considered appealing for face comparisons in forensic contexts, which require decisions to be transparent and auditable. However, existing evaluations of VLMs for that use case focus mostly on recognition accuracy, while the validity of generated explanations remains unquantified. In this work, we introduce a benchmarking framework for VLM-based face recognition that treats explanation quality as a core evaluation axis. We propose two criteria that explanations should satisfy: relevance, i.e., reliance on identity-stable facial features; and faithfulness, i.e., alignment with the visible image content without hallucinated features. We jointly develop a methodology enabling the quantification of relevance and faithfulness of evaluated models, based on constraining model outputs to a structured explanation format that supports automated querying and auditing. Using this framework, we benchmark several families of open-weight VLMs, jointly evaluating face verification accuracy and explanation quality. Our results highlight remaining shortcomings of produced explanations, and emphasize the need for such explanation quality metrics to get a complete picture of model performance. The proposed benchmark and open-source evaluation harness provide a foundation for proper benchmarking and future fine-tuning of explainable face recognition systems.

Laurent Colbois, Sébastien Marcel

Idiap Research Institute, Switzerland · Université de Lausanne, Switzerland

VLM Evaluation Face Recognition Faithfulness in VLMs

Ask PDF

Aug 21, 2026

EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment

Deep face recognition (FR) models reach near-saturated accuracy but remain opaque: a practitioner cannot ask which semantic attributes a similarity score relied upon. EXPL-FR answers this inside the FR model's own embedding space. A lightweight adapter aligns a vision-language model's (VLM) image encoder with the frozen FR space, trained on face images alone and never on text. Because the VLM's encoders share one space, the same adapter applies to the text encoder, turning 978 attribute prompts in 22 categories, also extendable, into FR-space anchors at no extra cost. We do not assume this transfer works: a face-verification protocol measures it, and an ablation changing only the adapter isolates its contribution. Not every concept survives, because an FR model earns its invariances by discarding the factors it must verify identities across. A label-free detectability measure compares each concept's separability in FR space against the VLM space, and the 100 most detectable form the model's readable semantic signature, which separates identities better than the full vocabulary. We cover four FR backbones and two VLM encoders, EXPL-FR needs no architecture access, and supports identity-level, per-image, and differential explanations. We benchmark attribute-level auditing under three supervision settings, human labels (current practice), VLM pseudo-labels, and our fully prompt-driven audit, against real verification behavior. With no labels, the prompt-driven audit ranks four FR models by their measured per-ethnicity RFW errors and ranks controlled attribute changes by their true verification cost.

Guray Ozgur, Mustafa Efe Tamyapar, Naser Damer +1

Fraunhofer IGD, Fraunhoferstr. 5, 64283 Darmstadt, Germany · TU Darmstadt, Karolinenpl. 5, 64289 Darmstadt, Germany

Face Recognition Concept-Based Explanations Vision-Language Alignment

Ask PDF

Mar 17, 2026

MLLM-based Textual Explanations for Face Comparison

Multimodal Large Language Models (MLLMs) have recently been proposed as a means to generate natural-language explanations for face recognition decisions. While such explanations facilitate human interpretability, their reliability on unconstrained face images remains underexplored. In this work, we systematically analyze MLLM-generated explanations for the unconstrained face verification task on the challenging IJB-S dataset, with a particular focus on extreme pose variation and surveillance imagery. Our results show that even when MLLMs produce correct verification decisions, the accompanying explanations frequently rely on non-verifiable or hallucinated facial attributes that are not supported by visual evidence. We further study the effect of incorporating information from traditional face recognition systems, viz., scores and decisions, alongside the input images. Although such information improves categorical verification performance, it does not consistently lead to faithful explanations. To evaluate the explanations beyond decision accuracy, we introduce a likelihood-ratio-based framework that measures the evidential strength of textual explanations. Our findings highlight fundamental limitations of current MLLMs for explainable face recognition and underscore the need for a principled evaluation of reliable and trustworthy explanations in biometric applications. Code is available at https://github.com/redwankarimsony/LR-MLLMFR-Explainability.

Redwan Sony, Anil K Jain, Arun Ross

Department of Computer Science and Engineering, Michigan State University, East Lansing, MI, USA

Faithfulness of Language Model Explanations Face Recognition Multimodal Large Language Models Hallucination Detection in VLMs

Ask PDF

Abstract

Face Recognition

Explore similar work

Benchmarking the Explanatory Quality of Open-Weight Vision-Language Models in Face Recognition

EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment

MLLM-based Textual Explanations for Face Comparison