SCOUT: Semantic Concept Discovery for Open-Vocabulary Editing of face Recognition Templates

Leon Todorov; Peter Rot; Peter Peer; Vitomir Štruc; Klemen Grm

cs.CVAug 17, 2026

SCOUT: Semantic Concept Discovery for Open-Vocabulary Editing of face Recognition Templates

Authors: Leon Todorov, Peter Rot, Peter Peer, Vitomir Štruc, Klemen Grm

Organizations: University of Ljubljana: 1Faculty of Electrical Engineering · Faculty of Computer and Information Science

Abstract

Face recognition templates are compact identity representations, yet they also encode rich semantic information about facial appearance. Prior work has shown that templates can be inverted to images or indirectly manipulated through image-editing pipelines, but direct semantic editing in template space remains largely unexplored. Existing interpretability methods for face recognition often rely on manual neuron inspection or predefined attribute labels, limiting scalability and semantic flexibility. To address this gap, we propose SCOUT (Semantic Concept Discovery for Open-VocabUlary Editing of Face Recognition Templates), an end-to-end framework for discovering and directly manipulating semantic concepts in face recognition templates using mechanistic interpretability. SCOUT learns sparse template representations, generates semantic hypotheses for latent features from natural-language descriptions, and validates their stability. The resulting features act as controllable semantic directions for direct editing, avoiding costly edit--re-encode pipelines. Experiments with face recognition models using CNN, ViT, and Swin backbones show that SCOUT discovers interpretable concepts beyond standard attribute labels and enables controllable, identity-aware template manipulation with negligible impact on identity matching. We further show that edited templates can subsequently be decoded with independent inversion models for visualization and evaluation.

Figures & tables

Figure 1: SCOUT automatically discovers and validates semantic interpretations of sparse neurons in the sparse autoencoder (SAE) using face attributes defined by free-form text prompts or predefined labels. Once stable neuron–attribute associations are identified through inversion-based verification, the corresponding sparse features are used to directly manipulate semantic attributes in template space for efficient template augmentation.

	Works	Open- vocabulary	Sparse decomposition	Automatic discovery	Validation / intervention	Direct editing
\Block [tikz=fill=blue!12,draw=none]7-1 General vision	Net Dissection [ 3 ]	✗	✗	✗	$\sim$	✗
	TCAV [ 18 ]	✗	✗	✗	$\sim$	✗
	MAIA [ 37 ]	✓	✗	✓	✓	✗
	Prisma [ 16 ]	✗	✓	$\sim$	$\sim$	✗
	MSAE [ 44 ]	✗	✓	$\sim$	$\sim$	✗
	Revelio [ 19 ]	✗	✓	$\sim$	$\sim$	✗

Table 1: Overview of methods for semantic concept discovery in face recognition templates. ✓: supported, ✗: not supported,

\sim

: partially supported or not the primary focus.

Figure 2: High-level overview of SCOUT, the proposed framework for discovering and manipulating semantic concepts directly in face recognition template space. The framework consists of four stages: (1) training a sparse autoencoder on face templates, (2) assigning latent neurons to semantic concepts using labeled or open-vocabulary attributes, (3) reconstructing faces via template inversion, and (4) validating the discovered concepts through neuron-level interventions, where individual sparse features are perturbed to measure their semantic effect.

Model	Loss	Backbone	Training set
ADA-iR50	AdaFace [ 20 ]	iResNet-50 [ 13 ]	MS1MV2 [ 9 ]
ARC-iR100	ArcFace [ 9 ]	iResNet-100 [ 13 ]	WebFace42M [ 45 ]
ADA-ViT	AdaFace [ 20 ]	ViT-Base [ 10 ]	WebFace4M [ 45 ]
SWIN-T	CosFace [ 43 ]	Swin-Tiny [ 24 ]	MS1MV2 [ 9 ]

Table 2: FR encoders used in our experiments. The four models span CNN and transformer backbones with distinct margin-based losses and are trained on different large-scale datasets.

Method	Backbone	$\boldsymbol{d_{\mathrm{SAE}}}$	$\boldsymbol{k}$	MSE	alive%	$\boldsymbol{\cos(\mathbf{x},\hat{\mathbf{x}})}$	$\boldsymbol{\|\cos\|_{\mathrm{dict}}}$
SCOUT	ADA-iR50	$8192$	$32$	$0.0008$	$100.00$	$0.7743$	$0.0465$
	ADA-ViT	$8192$	$32$	$0.0007$	$100.00$	$0.8099$	$0.0547$
	ARC-iR100	$8192$	$32$	$0.0008$	$100.00$	$0.7694$	$0.0450$
	SWIN-T	$8192$	$32$	$0.0006$	$100.00$	$0.8284$	$0.0579$
Face MINT	ADA-iR50	$65536$	$400$	$0.0003$	$99.82$	$0.9998$	$0.0767$
	SWIN-T	$65536$	$200$	$0.0000$	$88.15$	$1.0000$	$0.0365$

Table 3: SAE statistics on Glint360K (

\sim 17

M templates) for the four SCOUT models, alongside FaceMINT [ 33 ] TopK SAE reference results. MSE

=\mathbb{E}[\lVert\mathbf{x}-\hat{\mathbf{x}}\rVert^{2}]

, alive% is the fraction of features activated at least once, and

|\cos|_{\mathrm{dict}}

is the mean pairwise similarity between template-space dictionary vectors.

Panel of Figure 5 — Figure 5: Cross-decoder validation in the shared ARC-iR100 space. Left: one intervened template per concept is decoded by DFD and Arc2Face; $\Delta_{c}$ is averaged across the disjoint evaluation-prompt subsets, while the shared template-level $\theta_{c}$ is shown once per intervention strength. Metadata $n$ , $S_{n}(c)$ , and $A_{n}(c)$ are obtained in Stage 2. Right: attribute-gain curves over 100 random LFW templates. Thick solid and dashed lines denote the DFD and Arc2Face means across evaluation-prompt subsets, respectively; thin matching boundaries and shading denote the $\pm 1$ standard deviation across evaluation-prompt subsets.

		Perturbed ( $\alpha=1$ )
Model	Clean	Mean	p95	Worst
ADA-iR50	$0.33$	$0.43$	$0.53$	$0.67$
ADA-ViT	$0.23$	$0.41$	$0.53$	$0.73$
SWIN-T	$0.20$	$0.41$	$0.50$	$0.67$
ARC-iR100	$0.27$	$0.42$	$0.50$	$0.63$

Table 4: Fixed-FMR identity evaluation on LFW: FNMR (%) at a threshold calibrated on clean impostor pairs to FMR

=10^{-2}

and frozen after manipulation. Perturbed statistics summarize the full SAE-direction sweep.

Panel of Figure 6 — Figure 6: Identity preservation at matched mean attribute gain on CelebA-HQ (100 samples per attribute). Left: $\theta_{c}$ versus $\overline{\Delta}_{c}$ for SCOUT, StyleCLIP, and W $+$ Adapter; dots mark each method’s maximum population-supported $(\overline{\Delta}_{\max},\theta)$ , reported per concept in Section 5 . Right: representative matched-gain edits; target gains are shown below each image.

Face Recognition

Face recognition matches or identifies people from facial images, videos, and biometric representations across changing conditions.

See all 108 papers

Aug 31, 2026

Unmasking Face Embeddings: Reading, Rendering and Naming with Foundation Models

Modern face recognition (FR) owes much of its success to deep neural networks that learn to extract compact identity embeddings from face images. These models are typically trained for identity discrimination, producing embeddings that are highly effective for biometric matching but largely opaque to semantic interpretation. In contrast, foundation models, pretrained on broad visual or vision--language tasks, provide rich interfaces for describing, retrieving, generating, and organizing visual content. This contrast raises a natural question: what capabilities become available when face embeddings from domain-specific FR models are made interoperable with foundation models? Building on recent work on embedding compatibility across models, we use simple pre-computed linear transformations, estimated from paired embeddings alone, to connect existing FR models with off-the-shelf foundation models. Once aligned with a foundation model, a face embedding can be 'unmasked' in multiple ways, without training or modifying either model: it can be read in natural language, enabling free-form text queries over a gallery of FR embeddings; rendered into a face image that recovers a person's appearance, using an unmodified diffusion decoder; and converted to a name, enabling identification even in the absence of an enrolled face gallery. In effect, one linear transformation turns an identity embedding into a rich embedding for web-scale foundation models. This interoperability exposes face embeddings as semantically and visually rich biometric representations, with direct implications for interpretability, retrieval, reconstruction, and template security.

Fizza Rubab, Yiying Tong, Arun Ross

Michigan State University

Cross-Modal Alignment Face Recognition

Ask PDF

Aug 21, 2026

EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment

Deep face recognition (FR) models reach near-saturated accuracy but remain opaque: a practitioner cannot ask which semantic attributes a similarity score relied upon. EXPL-FR answers this inside the FR model's own embedding space. A lightweight adapter aligns a vision-language model's (VLM) image encoder with the frozen FR space, trained on face images alone and never on text. Because the VLM's encoders share one space, the same adapter applies to the text encoder, turning 978 attribute prompts in 22 categories, also extendable, into FR-space anchors at no extra cost. We do not assume this transfer works: a face-verification protocol measures it, and an ablation changing only the adapter isolates its contribution. Not every concept survives, because an FR model earns its invariances by discarding the factors it must verify identities across. A label-free detectability measure compares each concept's separability in FR space against the VLM space, and the 100 most detectable form the model's readable semantic signature, which separates identities better than the full vocabulary. We cover four FR backbones and two VLM encoders, EXPL-FR needs no architecture access, and supports identity-level, per-image, and differential explanations. We benchmark attribute-level auditing under three supervision settings, human labels (current practice), VLM pseudo-labels, and our fully prompt-driven audit, against real verification behavior. With no labels, the prompt-driven audit ranks four FR models by their measured per-ethnicity RFW errors and ranks controlled attribute changes by their true verification cost.

Guray Ozgur, Mustafa Efe Tamyapar, Naser Damer +1

Fraunhofer IGD, Fraunhoferstr. 5, 64283 Darmstadt, Germany · TU Darmstadt, Karolinenpl. 5, 64289 Darmstadt, Germany

Face Recognition Concept-Based Explanations Vision-Language Alignment

Ask PDF

Apr 20, 2026

LBFTI: Layer-Based Facial Template Inversion for Identity-Preserving Fine-Grained Face Reconstruction

In face recognition systems, facial templates are widely adopted for identity authentication due to their compliance with the data minimization principle. However, facial template inversion technologies have posed a severe privacy leakage risk by enabling face reconstruction from templates. This paper proposes a Layer-Based Facial Template Inversion (LBFTI) method to reconstruct identity-preserving fine-grained face images. Our scheme decomposes face images into three layers: foreground layers (including eyebrows, eyes, nose, and mouth), midground layers (skin), and background layers (other parts). LBFTI leverages dedicated generators to produce these layers, adopting a rigorous three-stage training strategy: (1) independent refined generation of foreground and midground layers, (2) fusion of foreground and midground layers with template secondary injection to produce complete panoramic face images with background layers, and (3) joint fine-tuning of all modules to optimize inter-layer coordination and identity consistency. Experiments demonstrate that our LBFTI not only outperforms state-of-the-art methods in machine authentication performance, with a 25.3% improvement in TAR, but also achieves better similarity in human perception, as validated by both quantitative metrics and a questionnaire survey.

Zixuan Shen, Zhihua Xia, Kaikai Gan +1

College of Cyber Security, Jinan University · Guangzhou, Guangdong, China

Biometric Identification Image Reconstruction Identity-Preserving Image Generation

Ask PDF

Abstract

Face Recognition

Explore similar work

Unmasking Face Embeddings: Reading, Rendering and Naming with Foundation Models

EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment

LBFTI: Layer-Based Facial Template Inversion for Identity-Preserving Fine-Grained Face Reconstruction