Deep face recognition (FR) models reach near-saturated accuracy but remain opaque: a practitioner cannot ask which semantic attributes a similarity score relied upon. EXPL-FR answers this inside the FR model's own embedding space. A lightweight adapter aligns a vision-language model's (VLM) image encoder with the frozen FR space, trained on face images alone and never on text. Because the VLM's encoders share one space, the same adapter applies to the text encoder, turning 978 attribute prompts in 22 categories, also extendable, into FR-space anchors at no extra cost. We do not assume this transfer works: a face-verification protocol measures it, and an ablation changing only the adapter isolates its contribution. Not every concept survives, because an FR model earns its invariances by discarding the factors it must verify identities across. A label-free detectability measure compares each concept's separability in FR space against the VLM space, and the 100 most detectable form the model's readable semantic signature, which separates identities better than the full vocabulary. We cover four FR backbones and two VLM encoders, EXPL-FR needs no architecture access, and supports identity-level, per-image, and differential explanations. We benchmark attribute-level auditing under three supervision settings, human labels (current practice), VLM pseudo-labels, and our fully prompt-driven audit, against real verification behavior. With no labels, the prompt-driven audit ranks four FR models by their measured per-ethnicity RFW errors and ranks controlled attribute changes by their true verification cost.
Figures & tables
Figure 1: (a) EXPL-FR overview. The only learned component is the adapter gθ , mapping frozen VLM image embeddings onto frozen FR embeddings (Eq. 2 ), it never sees text. The same frozen gθ is then applied to the text encoder, turning each prompt into an FR-space anchor (Eq. 3 ). A label-free measure keeps the attributes still detectable after the mapping, and a face’s cosine similarities to those anchors form its semantic signature (Eq. 4 restricted to Kψ ). (b) Explanations at three levels: Identity-wise (rows 1-2): mean signature over 50 images of two CASIA-WebFace [ 53 ] identities, identity-specific and stable across capture conditions, the second dominated by eyewear. Per-image (rows 3-5): single images from DCMorph [ 4 ] . Differential (rows 6-8): the reference against a genuine image, an imposter, and the imposter morph. The genuine difference is near-flat, the imposter difference large, the morph in between.
Table 1: Faithfulness of the cross-modal alignment. Each row scores one representation pair under the protocols of Section 4.1 , cross-encoder rows (4, 5) use different representations on the two sides. Vocabulary-projection rows (6-8) use the full 978 -prompt vocabulary, testing the transfer before any selection, the deployed signature is its top- 100 subset, evaluated in Figure 2 b under a different protocol. Adapter architecture and training are identical throughout. Rows 9-17 establish generality: the SigLIP variant (rows 9-11, its 768 -d embeddings giving an asymmetric first adapter layer) and three further FR targets over two architecture families and two training sets, each by alignment fidelity and vocabulary-projection faithfulness. Their own FR self-verification upper bounds, not listed, are 97.43 , 97.55 , and 97.19 mean, together with row 1’s 97.44 the four span just 0.36 points, so the targets are comparably strong.
Figure 2: Identity-discriminating attributes via label-free detectability (WebFace4M identity holdout, primary target). (a) Per category, mean AUC of a mean-difference detector fit on one identity half and scored on the other. Open: VLM space (mean 0.976 ). Filled: adapter-mapped FR space, the selection criterion AUCF , with its median cut. Diamonds: the same detector rescored on true FR embeddings ( AUCF∗ , mean 0.740 ), more pessimistic while preserving the ordering ( ρ=0.63 - 0.97 ). All three are scored against the VLM’s pseudo-labels, so they rank prompts rather than fix levels (Section 3.3 ). Discounted means not recoverable as a named direction, not demonstrated invariance. (b) Identity separability (genuine/imposter AUC) of coordinate subsets of size m , scored on the signature as deployed (true FR embeddings on the mapped anchors) and ranked by AUCF : top- m beat matched random subsets and the full vocabulary ( 978 ), bottom- m fall clearly below.
Figure 3: Labeled cross-checks on CelebA (primary model). (a) Anchor-attribute agreement as per-attribute detectability: AUC of one signature coordinate separating label-positives from negatives, VLM-proxy anchors in FR space (Eq. 5 , filled) vs. VLM space (open, control), chance dashed. (b) Label-free dependence: per attribute, macroscale KS from labels (x) vs. pseudo-labeled groups (y). (c) Per-category dependence profiles of the four targets over the 978 -prompt vocabulary (groups shared across models, so column differences reflect FR geometry alone).
Figure 4: Top (selection, RFW): each point one (FR model, ethnicity group) pair, label-free ethnicity dependence (x) vs. real ten-fold verification error (y), rightmost, ranking by mean dependence vs. by mean real error. Bottom (diagnosis, GAN-Control, primary target): each point one varied attribute, matched-axis sensitivity (x) vs. the real EER of the protocol varying exactly that attribute ( 14,000 pairs, panels labeled / VLM-proxy / ours).
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Supervision levels for attribute-level FR auditing (main-paper Sec. 3.5). Three constructions of the same FR-space attribute axis, in decreasing supervision: (1) labeled group means, the supervision prior analyses; (2) VLM-pseudo-labeled FR group means, from prompts and unlabeled images; (3) the transported prompt axis (ours), using no image at construction. Right column: the cost of extending the audit by one new attribute under each level.
Figure 6: Differential signatures under controlled variation (GAN-Control), written prompt groups. (a) Attribution, primary model: per single-attribute traversal ( 500 held-out identities, anchors from the other 500 ), the share of ∣Δ∣ mass falling in each prompt group. Rows are the varied attribute, red boxes mark the matching group, and chance is 12.5% . Encoded attributes are attributed reliably (age 82% , expression 75% , hair color 64% ). Photometric rows are diffuse because the FR embedding erases them, an invariance, while orientation fails because CLIP cannot rank pose on aligned crops, an anchor limitation (labeled FR probe AUC 0.97 ). (b) Sensitivity: matched-group ∣Δ∣ per attribute and model, where smaller means stronger invariance. The ordering is consistent across all four models and matches the ordering reported in the main paper (Sec. 4.3). Traversal endpoints verify against the base image at mean minimum cosine ≈0.79 , or 0.55 - 0.59 for age, whose sensitivity is therefore an upper bound.
Figure 7: Identity-wise semantic signatures. For five randomly sampled CASIA-WebFace identities ( N=50 images each), every image is projected onto the 100 -prompt semantic signature of the AdaFace ViT-B / WebFace4M model (the top- 100 candidate prompts by FR-space detectability, main-paper Sec. 3.3; CLIP anchors). Each bar is the mean cosine of that identity’s 50 images to one signature prompt; the top- 20 most-activated prompts per identity are drawn in bold, the rest faded. Prompts are grouped by category along the x-axis, with categories ordered by detectability (most identity-discriminating first; categories dropped from the signature do not appear). Distinct, identity-specific patterns emerge, e.g . one identity dominated by eyewear , and they persist across all 50 capture conditions. EXPL-FR thus surfaces the concepts along which each FR model actually separates identities.
Figure 8: Per-image semantic signatures on DCMorph quadruplets. Two DCMorph quadruplets are shown (top and bottom panels). Each quadruplet holds a reference, a genuine same-identity capture, an imposter, and the reference × imposter morph, and all four images are projected onto the 100 -prompt semantic signature of AdaFace ViT-B / WebFace4M (CLIP anchors). Each bar is the cosine of that single image to one signature prompt, and the top- 10 prompts per image are bold. Prompts are grouped by category and ordered by detectability, as in Fig. 7 . These per-image signatures are the raw material differenced pairwise in Fig. 9 .
Figure 9: Image-wise signature differences on DCMorph quadruplets. Signed differences of the 100 -prompt semantic signature between selected image pairs, for two quadruplets (top and bottom; the underlying per-image signatures are in Fig. 8 ). AdaFace ViT-B / WebFace4M, CLIP anchors. Positive bars mean the first image activates the prompt more, and the top- 30 prompts by ∣diff∣ are highlighted. Prompts are grouped by category and ordered by detectability. Ref-Genuine is near-uniformly small, reflecting within-identity stability. Ref-Imposter shows large, category-concentrated differences that name the axes separating the two identities. Ref-Morph is consistently smaller than Ref-Imposter and closer to Ref-Genuine : it reflects the identity content the morph retains from the reference, while its residual, category-concentrated divergence names the attributes inherited from the imposter. Main-paper Fig. 1b (rows 6-8) shows these three differences for a single quadruplet.
Figure 10: Setting (3) audits age and pose against the standard benchmarks. Each panel is one benchmark and each point one FR model. The x-axis is the prompts-only age- or pose-dependence on an independent unlabeled pool (pooled RFW; CelebA agrees in sign). The y-axis is the benchmark’s penalty, its error minus the model’s LFW error (main-paper Table 1). For age, the audit ranks the models as their cross-age penalties do ( τ=+0.67 / +0.33 ). Pose is inconclusive ( 0.00 / −1.0 , the two pose benchmarks disagreeing). CLIP cannot comprehend pose on tightly aligned crops, so the adapter cannot transport what the VLM does not encode, and the pose audit values support no conclusion. The audit returns them regardless, and without prior knowledge of the limitation the CPLFW inversion would be accepted as a finding (Sec. 0.E ).
Figure 11: Replacing supervised auditing at scale (CelebA, 40 attributes ×4 models). No per-attribute verification ground truth exists, so both label-free settings are compared against the supervised audit (1) they would replace. Left, center: label-free versus labeled dependence per (model, attribute); per-model Spearman ρ annotated (proxy 0.26 - 0.48 , ours 0.21 - 0.51 , against setting (1)’s own holdout self-consistency of 0.34 - 0.52 ). Right: survival curve of per-attribute Kendall τ between the 4-model audit ranking and the supervised ranking, proxy: mean τ=0.63 , 90% consistent; ours: 0.45 , 78% . Color = FR model.
Figure 12: Model diagnosis on GAN-Control, all four FR targets (rows) under the three supervision settings (columns); the main paper shows the primary target. Per-model Spearman ρ between matched-axis sensitivity and real EER: labeled 0.90 - 1.0 , VLM-proxy 0.81 - 0.88 , ours 0.69 - 0.95 .
Figure 13: Per-category detectability, all four FR targets (WebFace4M identity holdout; the main paper shows the primary target). Mean detectability AUC per category in VLM space (open), adapter-mapped FR space (filled; blue marks categories at or above the model’s median, the selection criterion), and on true FR embeddings (diamonds, AUCF∗ ; main-paper Sec. 4.3). Category order is fixed by the primary target. Every target shows the same gap between the mapped and the true measure, so the reported discount is conservative on all four. Gender drops to the lowest category under AUCF∗ on three of them, the exception being R100/MS1MV2, where it stays mid-table. Every model keeps eyewear, hair color, hair style, and facial hair, and discounts distance, context, camera angle, face coverings, and lighting. The kept halves (top 489 ) overlap at Jaccard 0.60 - 0.78 over the six model pairs (Table 4 ), with model-specific deviations in the mid-table ( e.g . age and ethnicity fall below the median cut only for ViT-B).
Figure 14: Anchor-attribute agreement on CelebA, remaining FR targets (complements main-paper Fig. 3a; attribute order fixed by the primary target). Per-attribute AUC of one signature coordinate separating label-positives from negatives, VLM-proxy anchors in FR space (filled) vs. VLM space (open, control); chance dashed. Means over the 40 attributes are 0.65/0.65/0.61 . Every target reproduces the quadrant structure of the primary model: identity-stable attributes highest, transient accessories (eyeglasses, hat) discounted, and low/low attributes unrescued. The behavior is architecture-independent (main-paper Sec. 4.3).
Figure 15: Label-free dependence ranking, remaining FR targets (complements main-paper Fig. 3b). Per CelebA attribute, macroscale KS from ground-truth label groups (x) vs. pseudo-labeled groups (y) on the same one-image-per-identity subsample; Spearman ρ=0.64/0.65/0.70 (all p<10−4 , n=40 ), with the label-dominant attributes (bald, blond hair, male) recovered as dominant label-free on every target.
Figure 16: Attribution under controlled variation, remaining FR targets (complements Fig. 6 a). Share of ∣Δ∣ mass per written prompt group on single-attribute GAN-Control traversals (red boxes: matching group; chance 0.125 ); top-1 accuracy 0.36/0.37/0.36 . The keep/discount split is identical across targets: age, expression, and hair color attribute reliably; photometric rows stay diffuse (FR invariance); orientation stays near chance (VLM pose limitation, main-paper Sec. 4.3).
Top VLM-proxy anchors (FR space)
Blond hair
0.89
Male
0.82
Five-o’clock shadow
0.77
Black hair
0.76
Bald
0.75
Goatee
0.75
Appendix
Table 2: Anchor-attribute agreement on CelebA: exemplar values (single-coordinate AUC; complements main-paper Fig. 3a). Top: highest-scoring VLM-proxy anchors in FR space. Middle: high-VLM/attenuated-FR quadrant (VLM-space AUC → FR-space VLM-proxy AUC), attributes the FR model has discounted. Bottom: low/low quadrant (FR VLM-proxy / VLM), concepts the VLM cannot rank on aligned 112×112 crops. Negation: CelebA’s No_Beard under naive negated phrasing scores below chance in both spaces; phrased positively with flipped sign it recovers to 0.74 / 0.93 .
Table 3: SigLIP variant and generality of the cross-modal alignment. The full version of the generality evaluation that main-paper Table 1 summarises, which lists only alignment fidelity and vocabulary-projection faithfulness for these variants; rows here are numbered independently of that table. Rows 1-6 repeat the faithfulness protocol with SigLIP-B16 in place of CLIP; in row 3, SigLIP embeddings ( 768 -d) are PCA-reduced to 512 -d, since cosine across unequal dimensionalities is undefined, and in cross-encoder rows 3-4 the two sides use different representations. Rows 7-15: three further FR targets (two architecture families, two training sets), each scored by FR upper bound, alignment fidelity, and vocabulary-projection faithfulness; vocabulary-projection rows use the full candidate vocabulary; adapter architecture and recipe identical throughout.
Target (full)
m
Top- m
Random
Bottom- m
ViT-B/WF4M ( 0.910 )
25
0.903
0.900
0.874
100
0.931
0.909
0.884
200
0.932
0.908
0.886
ViT-S/WF4M ( 0.898 )
25
0.872
0.889
0.872
100
0.915
0.897
0.873
200
0.918
0.897
0.876
Appendix
Table 4: Identity separability of filtered signatures, all four FR targets (genuine/imposter AUC on the WebFace4M identity holdout; complements the main-paper detectability figure). Signature restricted to m of the 978 prompts, ranked by adapter-mapped detectability AUCF : top- m , mean of 10 matched-size random subsets, and bottom- m ; “full” uses all 978 . Subsets are scored on the signature as deployed , real FR embeddings projected onto the mapped anchors (main-paper Eq. 4). From m=100 on, top- m beats both random and the full vocabulary on every target, and from m=50 on three of the four (R100/WF4M is the exception); at m=25 top- m is inconsistent (below random on three of four targets: near-duplicate prompts dominate the top block). The main paper fixes the semantic signature at m=100 . Between models, the four targets’ kept halves (top 489 ) overlap at Jaccard 0.60 - 0.78 over the six model pairs; main-paper Sec. 4.3 reports the corresponding overlap of the selected top- 100 sets, 0.48 - 0.77 . The same sweep in the VLM-only variant, where the signature is read off adapter-mapped images instead (main-paper Table 1, row 8), gives the same ordering at lower absolute values, e.g . ViT-B/WF4M full 0.845 and top- 1000.879 , R100/WF4M full 0.887 and top- 1000.921 .
Category
#
Example value
Category
#
Example value
facial features
110
broken nose
jewelry
39
cheek piercings
expression
97
exhausted
facial hair
34
light stubble beard
headwear
86
sports helmet
head pose
32
turned slightly left
lighting
82
hair light
distance
31
tightly cropped
context
68
unposed
accessories
27
lavalier mic
skin features
63
smooth skin
face shape
25
wide face
Appendix
Table 5: The candidate vocabulary: 978 prompts, 22 categories. Prompt counts per category with one example attribute value each; every value is embedded in the template “A photo of (a person with) …”.
Responsible deployment of face verification systems requires more than accurate decisions: systems should also provide interpretable and auditable evidence that enables users to understand, assess, and challenge their decisions. Vision-language models (VLMs) provide a promising foundation for explainable face recognition by combining visual analysis with natural-language reasoning. However, relying on a single model may further limit the decision accuracy as well as provided explanations. This work therefore investigates whether multiple VLMs can be combined to improve recognition accuracy, and to enrich the explanations associated with those decisions. This work evaluates four VLMs as standalone face verification systems and subsequently proposes a fusion framework, where two source models provide similarity scores and textual justifications and a third VLM acts as a decider model. Four different fusion scenarios are considered, progressively providing the decider model with scores, justifications, face images, and combinations of these modalities. Overall, the findings suggest that the value of multi-VLM fusion extends beyond recognition performance. VLMs can provide complementary justifications and perspectives that enable richer explanations of face recognition decisions, supporting greater transparency, auditability, and error analysis. This is relevant to the development of responsible explainable face verification systems, where users and operators should be able to understand not only the final decision but also the evidence and potential sources underlying it. The proposed multimodal VLM, which combines decision scores, explanations, and face images, achieves higher recognition accuracy than state-of-the-art VLMs and domain-specific face recognition models, while also providing fused explanations that are expected to be more robust than those generated by individual VLMs.
Ana Estrada-Real, Lydia Alapatt, Christoph Busch +1
da/sec Biometrics and Security Research Group Hochschule Darmstadt, Germany
Vision-Language Models (VLMs) have recently been proposed as promising tools for face recognition, as they can produce natural language explanations alongside similarity scores. This capability is considered appealing for face comparisons in forensic contexts, which require decisions to be transparent and auditable. However, existing evaluations of VLMs for that use case focus mostly on recognition accuracy, while the validity of generated explanations remains unquantified. In this work, we introduce a benchmarking framework for VLM-based face recognition that treats explanation quality as a core evaluation axis. We propose two criteria that explanations should satisfy: relevance, i.e., reliance on identity-stable facial features; and faithfulness, i.e., alignment with the visible image content without hallucinated features. We jointly develop a methodology enabling the quantification of relevance and faithfulness of evaluated models, based on constraining model outputs to a structured explanation format that supports automated querying and auditing. Using this framework, we benchmark several families of open-weight VLMs, jointly evaluating face verification accuracy and explanation quality. Our results highlight remaining shortcomings of produced explanations, and emphasize the need for such explanation quality metrics to get a complete picture of model performance. The proposed benchmark and open-source evaluation harness provide a foundation for proper benchmarking and future fine-tuning of explainable face recognition systems.
Laurent Colbois, Sébastien Marcel
Idiap Research Institute, Switzerland · Université de Lausanne, Switzerland
Modern face recognition (FR) owes much of its success to deep neural networks that learn to extract compact identity embeddings from face images. These models are typically trained for identity discrimination, producing embeddings that are highly effective for biometric matching but largely opaque to semantic interpretation. In contrast, foundation models, pretrained on broad visual or vision--language tasks, provide rich interfaces for describing, retrieving, generating, and organizing visual content. This contrast raises a natural question: what capabilities become available when face embeddings from domain-specific FR models are made interoperable with foundation models? Building on recent work on embedding compatibility across models, we use simple pre-computed linear transformations, estimated from paired embeddings alone, to connect existing FR models with off-the-shelf foundation models. Once aligned with a foundation model, a face embedding can be 'unmasked' in multiple ways, without training or modifying either model: it can be read in natural language, enabling free-form text queries over a gallery of FR embeddings; rendered into a face image that recovers a person's appearance, using an unmodified diffusion decoder; and converted to a name, enabling identification even in the absence of an enrolled face gallery. In effect, one linear transformation turns an identity embedding into a rich embedding for web-scale foundation models. This interoperability exposes face embeddings as semantically and visually rich biometric representations, with direct implications for interpretability, retrieval, reconstruction, and template security.