Variational Latent Entropy Estimation Disentanglement: Controlled Attribute Leakage for Face Recognition

Ünsal Öztürk; Vedrana Krivokuća Hahn; Sushil Bhattacharjee; Sébastien Marcel

cs.CVApr 13, 2026

Variational Latent Entropy Estimation Disentanglement: Controlled Attribute Leakage for Face Recognition

Authors: Ünsal Öztürk, Vedrana Krivokuća Hahn, Sushil Bhattacharjee, Sébastien Marcel

Organizations: Idiap Research Institute, Martigny, Switzerland · UNIL, Lausanne, Switzerland

Abstract

Face recognition embeddings encode identity, but they also encode other factors such as gender and ethnicity. Depending on how these factors are used by a downstream system, separating them from the information needed for verification is important for both privacy and fairness. We propose Variational Latent Entropy Estimation Disentanglement (VLEED), a post-hoc method that transforms pretrained embeddings with a variational autoencoder and encourages a distilled representation where the categorical variable of interest is separated from identity-relevant information. VLEED uses a mutual information-based objective realised through the estimation of the entropy of the categorical attribute in the latent space, and provides stable training with fine-grained control over information removal. We evaluate our method on IJB-C, RFW, and VGGFace2 for gender and ethnicity disentanglement, and compare it to various state-of-the-art methods. We report verification utility, predictability of the disentangled variable under linear and nonlinear classifiers, and group disparity metrics based on false match rates. Our results show that VLEED offers a wide range of privacy-utility tradeoffs over existing methods and can also reduce recognition bias across demographic groups.

Figures & tables

Panel of Figure 1 — Fig. 1: Top: standard face recognition (FR) embeddings $\bm{e}^{\text{e}}$ (enrolment) and $\bm{e}^{\text{p}}$ (probe) yield high cosine similarity but leak sensitive attributes (gender, ethnicity). Bottom: a disentanglement step produces privacy-preserving embeddings $\bm{e}^{\text{e}^{\prime}}$ , $\bm{e}^{\text{p}^{\prime}}$ that retain verification utility while reducing attribute leakage. VLEED, proposed in this paper, is one such disentanglement method. Values are illustrative.

(a) End-to-end and image-level methods
Method	Architecture	Disentanglement objective	Variational	Multi-attr.	Tunability	Open source
DebFace [ 4 ]	CNN + adversarial head	Adversarial min-max	×	✓	Continuous ( $\lambda$ )	✓
PASS [ 5 ]	CNN + adversarial head	Adversarial min-max	×	✓	Continuous ( $\lambda$ )	✓
AdvFace [ 11 ]	Perturbation net	Adversarial perturbation	×	×	Continuous ( $\epsilon$ )	×
SlerpFace [ 10 ]	Spherical interpolation	Adversarial on hypersphere	×	×	Continuous ( $\alpha$ )	✓
Face-CPFNet [ 22 ]	VAE + GAN	Conditional privacy funnel (MI)	✓	×	Continuous ( $\beta$ )	×

TABLE I: Qualitative comparison of soft-biometric privacy-enhancement methods. Methods are grouped into (a) end-to-end and image-level approaches that require retraining, and (b) embedding-level post-hoc methods applicable to frozen pretrained embeddings. “Variational” indicates use of a probabilistic latent model. “Multi-attr.” indicates native support for disentangling multiple sensitive attributes. “Tunability” reflects whether the privacy–utility tradeoff can be smoothly controlled via a continuous parameter. “Open source” indicates publicly available source code as of the time of writing.

Dataset	Split	Gender		Ethnicity				Total
Dataset	Split	Female	Male	African	Asian	Caucasian	Indian	w/ Gender	w/ Ethnicity
VGGFace2	Train	1,299,393 (41.4%)	1,842,891 (58.6%)	258,342 (8.3%)	196,259 (6.3%)	2,402,603 (77.3%)	250,304 (8.1%)	3,142,284	3,107,508
	Eval	34,815 (39.5%)	53,389 (60.5%)	5,867 (6.8%)	13,064 (15.1%)	60,125 (69.6%)	7,390 (8.5%)	88,204	86,446
RFW	Eval	9,939 (24.5%)	30,607 (75.5%)	10,415 (25.6%)	9,688 (23.9%)	10,196 (25.1%)	10,308 (25.4%)	40,546	40,607
IJB-C	Eval	173,495 (37.0%)	295,880 (63.0%)	47,492 (10.1%)	43,438 (9.3%)	323,868 (69.0%)	54,337 (11.6%)	469,375	469,135

TABLE II: Demographic distribution across datasets showing gender and ethnicity breakdowns. Percentages are relative to total samples with valid demographic labels.

Method		Verification (TMR $\uparrow$ )												Fairness (Gini $\downarrow$ )
		Gender Removal						Ethnicity Removal						Gender Removal						Ethnicity Removal
		IJB-C		RFW		VGGFace2		IJB-C		RFW		VGGFace2		IJB-C		RFW		VGGFace2		IJB-C		RFW		VGGFace2
		1e-3	1e-1	1e-3	1e-1	1e-3	1e-1	1e-3	1e-1	1e-3	1e-1	1e-3	1e-1	1e-3	1e-1	1e-3	1e-1	1e-3	1e-1	1e-3	1e-1	1e-3	1e-1	1e-3	1e-1
Baseline		.815	.971	.966	.997	.680	.947	.815	.971	.966	.997	.680	.947	.836	.468	.328	.002	.932	.472	.687	.223	.557	.288	.785	.167
INLP		.852	.976	.965	.997	.754	.946	.822	.969	.959	.996	.677	.940	.320	.122	.374	.008	.670	.128	.639	.184	.613	.292	.709	.063
PFRNet/	$\lambda$ =0	.289	.790	.182	.744	.174	.667	.788	.954	.840	.985	.643	.905	.108	.066	.064	.076	.044	.030	.148	.057	.224	.119	.120	.049

TABLE III: Joint verification–fairness comparison across methods. Left block: verification utility (TMR at FMR

10^{-3}

and

10^{-1}

; higher is better). Right block: fairness (Gini coefficient over all-pairs false positive differentials of FMR; lower is better). “Gender Removal” and “Ethnicity Removal” denote which attribute was removed to produce the embeddings. In each fairness block, the Gini coefficient is computed over groups of the removed attribute (gender groups for Gender Removal; ethnicity groups for Ethnicity Removal) for each evaluation dataset. Bold marks the best operating point within each method.

Panel of Figure 5 — Fig. 4: Visual analysis of VLEED behaviour and comparison with prior methods. (a) t-SNE visualisation of how disentanglement strength affects the residual latent space. (b) Aggregate privacy–utility tradeoff curves across datasets.

Attribute	$\lambda_{\mathrm{dis}}$	IJB-C	RFW	VGGFace2
Ethnicity	Baseline	.227	.350	.180
	$\lambda$ =0	.194	.373	.138
	$\lambda$ =0.1	.870	.825	.777
	$\lambda$ =1	.893	.846	.814
	$\lambda$ =10	.996	.992	.993
	$\lambda$ =100	.996	.973	.994

TABLE V: EER thresholds (cosine similarity) for VLEED across the

\lambda_{\mathrm{dis}}

sweep on IJB-C, RFW, and VGGFace2, reported for gender and ethnicity removal.

Fig. 5: ROC curves (TMR vs. FMR) for VLEED across $\lambda_{\mathrm{dis}}\in\{0,0.1,1,10,100,1000\}$ on IJB-C, RFW, and VGGFace2. Legend values denote AUC. Red markers denote the operating point at the equal error rate threshold (where FMR equals FNMR).

Representation Disentanglement

Representation disentanglement studies how to separate distinct factors of variation in learned features so models can interpret, control, and transfer them.

See all 98 papers

The Good, the Better, and the Best: Improving the Discriminability of Face Embeddings through Attribute-aware Learning
Mar 16, 2026Ana Dias, João Ribeiro Pinto, Hugo Proença +1Face Recognition Representation Learning
University of Beira Interior, Portugal · IT: Instituto de Telecomunicações · NOVA LINCS +1
DenseFace: Bias Mitigation in Face Recognition via Density-Aware Probabilistic Matching
Sep 14, 2026Mansur Bultygov, Vadim Seliutin, Dmitry Nekhaev +1Algorithmic Fairness Algorithmic Bias
VisionLabs · Amazon Web Services · Mohamed bin Zayed University of Artificial Intelligence
EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment
Aug 21, 2026Guray Ozgur, Mustafa Efe Tamyapar, Naser Damer +1Face Recognition Concept-Based Explanations
Fraunhofer IGD, Fraunhoferstr. 5, 64283 Darmstadt, Germany · TU Darmstadt, Karolinenpl. 5, 64289 Darmstadt, Germany

Mar 16, 2026J/K move · Enter open · S save

The Good, the Better, and the Best: Improving the Discriminability of Face Embeddings through Attribute-aware Learning

Ana Dias, João Ribeiro Pinto, Hugo Proença, João C. Neves

University of Beira Interior, Portugal · IT: Instituto de Telecomunicações · NOVA LINCS+1

Face Recognition Representation Learning Shortcut Learning

Open paper AskPDF arXiv

Despite recent advances in face recognition, robust performance remains challenging under large variations in age, pose, and occlusion. A common strategy to address these issues is to guide representation learning with auxiliary supervision from facial attributes, encouraging the visual encoder to focus on identity-relevant regions. However, existing approaches typically rely on heterogeneous and fixed sets of attributes, implicitly assuming equal relevance across attributes. This assumption is suboptimal, as different attributes exhibit varying discriminative power for identity recognition, and some may even introduce harmful biases. In this paper, we propose an attribute-aware face recognition architecture that supervises the learning of facial embeddings using identity class labels, identity-relevant facial attributes, and non-identity-related attributes. Facial attributes are organized into interpretable groups, making it possible to decompose and analyze their individual contributions in a human-understandable manner. Experiments on standard face verification benchmarks demonstrate that joint learning of identity and facial attributes improves the discriminability of face embeddings with two major conclusions: (i) using identity-relevant subsets of facial attributes consistently outperforms supervision with a broader attribute set, and (ii) explicitly forcing embeddings to unlearn non-identity-related attributes yields further performance gains compared to leaving such attributes unsupervised. Additionally, our method serves as a diagnostic tool for assessing the trustworthiness of face recognition encoders by allowing for the measurement of accuracy gains with suppression of non-identity-relevant attributes, with such gains suggesting shortcut learning from redundant attributes associated with each identity.

Abstract

Representation Disentanglement

Explore similar work

The Good, the Better, and the Best: Improving the Discriminability of Face Embeddings through Attribute-aware Learning

DenseFace: Bias Mitigation in Face Recognition via Density-Aware Probabilistic Matching

EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment