cs.CVMar 6, 2026

DiffInf: Influence-Guided Diffusion for Supervision Alignment in Facial Attribute Learning

Authors: Basudha Pal, Zhaoyang Wang, Rama Chellappa

Organizations: Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA · Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA · Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Facial attribute classification relies on large-scale annotated datasets in which many traits, such as age and expression, are inherently ambiguous and continuous but are discretized into categorical labels. Annotation inconsistencies arise from subjectivity and visual confounders such as pose, illumination, expression, and demographic variation, creating mismatch between images and assigned labels. These inconsistencies introduce supervision errors that impair representation learning and degrade downstream prediction. We introduce DiffInf, a self-influence--guided diffusion framework for mitigating annotation inconsistencies in facial attribute learning. We first train a baseline classifier and compute sample-wise self-influence scores using a practical first-order approximation to identify training instances that disproportionately destabilize optimization. Instead of discarding these influential samples, we apply targeted generative correction via a latent diffusion autoencoder to better align visual content with assigned labels while preserving identity and realism. To enable differentiable guidance during correction, we train a lightweight predictor of high-influence membership and use it as a surrogate influence regularizer. The edited samples replace the originals, yielding an influence-refined dataset of unchanged size. Across multi-class facial attribute classification, DiffInf consistently improves generalization compared with standard noisy-label training, robust optimization baselines, and influence-based filtering. Our results demonstrate that repairing influential annotation inconsistencies at the image level enhances downstream facial attribute classification without sacrificing distributional coverage.

Figures & tables

Figure 1: The DiffInf framework for influence-guided generative supervision alignment. The process begins with a semantic encoder and conditional DDIM encoding to project the input image into a latent representation. During decoding, iterative latent optimization is guided by a composite loss balancing identity preservation (via a face recognition/verification network), influence suppression (via a predictor of high-influence membership), and semantic alignment while maintaining realism. By targeting high-influence samples for generative correction, DiffInf realigns training images with their assigned attribute labels while preserving identity fidelity and structural realism.
Figure 2: Empirical validation of self-influence under synthetic label noise. Left: Distribution of log self-influence values for clean and corrupted training samples after classifier training. Corrupted samples consistently exhibit higher self-influence, indicating persistent optimization instability caused by annotation errors. Right: Recovery of corrupted samples and inclusion of correctly labeled samples as the self-influence threshold varies. Selecting approximately the top 20% highest self-influence samples recovers nearly all corrupted samples while retaining only a small fraction of correctly labeled instances.
Figure 3: Qualitative examples of influence-guided generative correction (DiffInf) under noisy supervision. Each pair shows a high self-influence training sample (left) and its generated correction (right). Labels correspond to the dataset-provided age-group class, where 0 = Young (0–18), 1 = Middle (25–40), and 2 = Old (50+). Several original samples exhibit semantic mismatch with their assigned labels (e.g., an older-looking face labeled as Young). DiffInf preserves facial identity, pose, and illumination while selectively modifying age-related features to align with the assigned class, producing label-consistent samples while maintaining realism and identity fidelity.
Figure 4: Qualitative examples of influence-guided generative correction (DiffInf) for facial expression classification under noisy supervision. Each pair shows an original high self-influence training sample (left) and its generated correction (right). Labels correspond to the dataset-provided expression class, where 0 = Happy, 1 = Neutral, 2 = Surprised, and 3 = Sad. Several original samples are semantically inconsistent with their assigned labels, resulting in high influence during training. DiffInf preserves facial identity, pose, and illumination while selectively modifying expression-related features (e.g., mouth, eyebrows, and eyes) to align with the assigned class, producing label-consistent samples without altering identity.
  Age (3-class)Expression (4-class)
DatasetMethodAccAUROCκ\kappaAccAUROCκ\kappa
FFHQOriginal noisy70.4485.480.5878.9594.430.69
Small-loss74.9288.540.6883.5596.840.77
ELR+80.5692.400.7681.9093.980.73
Self-inf removal81.2393.730.7693.4299.270.88
Self-inf gen (Ours)83.3794.940.7894.2499.380.90
Table 1: Performance on FFHQ under 30% symmetric label noise for age classification and 20% for expression classification. Metrics are evaluated on the clean test set. Best results are shown in bold.
Figure 5: Recovery of corrupted samples as progressively larger fractions of the highest self-influence training samples are selected. The blue curve shows the cumulative percentage of corrupted samples recovered, while the orange curve denotes the cumulative proportion of correctly labeled samples included. Nearly all corrupted samples are concentrated within the top 20% of the self-influence ranking, indicating that optimization-disruptive samples occupy the highest influence regions and motivating targeted generative correction.
ClassLPIPS
Age
Young (0–18)0.227
Middle (25–40)0.219
Old (50+)0.224
Expression
Happy0.244
Table 2: Class-wise LPIPS values between original and DiffInf-generated samples. Lower values indicate higher perceptual similarity.
Noise (%)MethodAcc.AUROCκ\boldsymbol{\kappa}
10R79.24692.5040.71
G80.08092.6090.73
20R80.37793.2340.74
G81.99293.9300.75
30R81.23093.7300.76
G83.37094.9400.78
Table 3: Performance under different noise levels.
More in this topic

Learning with Noisy Labels

Learning with noisy labels studies how to train reliable models when annotations are incorrect, incomplete, subjective, or weakly supervised.

See all 150 papers

Explore similar work

CardsList
Apr 23, 2026

AttDiff-GAN: A Hybrid Diffusion-GAN Framework for Facial Attribute Editing

Facial attribute editing aims to modify target attributes while preserving attribute-irrelevant content and overall image fidelity. Existing GAN-based methods provide favorable controllability, but often suffer from weak alignment between style codes and attribute semantics. Diffusion-based methods can synthesize highly realistic images; however, their editing precision is limited by the entanglement of semantic directions among different attributes. In this paper, we propose AttDiff-GAN, a hybrid framework that combines GAN-based attribute manipulation with diffusion-based image generation. A key challenge in such integration lies in the inconsistency between one-step adversarial learning and multi-step diffusion denoising, which makes effective optimization difficult. To address this issue, we decouple attribute editing from image synthesis by introducing a feature-level adversarial learning scheme to learn explicit attribute manipulation, and then using the manipulated features to guide the diffusion process for image generation, while also removing the reliance on semantic direction-based editing. Moreover, we enhance style-attribute alignment by introducing PriorMapper, which incorporates facial priors into style generation, and RefineExtractor, which captures global semantic relationships through a Transformer for more precise style extraction. Experimental results on CelebA-HQ show that the proposed method achieves more accurate facial attribute editing and better preservation of non-target attributes than state-of-the-art methods in both qualitative and quantitative evaluations.

Wenmin Huang, Weiqi Luo, Xiaochun Cao +1

GuangDong Province Key Lab of Information Security Technology and School of Computer Science and Engineering, Sun Yat-sen University, Guangdong 510000, China · School of Cyber Science and Technology, Shenzhen Campus, Sun Yat-sen University, Shenzhen 518107, China · Guangdong Laboratory of Machine Perception and Intelligent Computing, Faculty of Engineering, Shenzhen MSU-BIT University, Shenzhen, 518116, China

Diffusion Model GuidanceDiffusion-Based Image EditingImage EditingDiffusion Models
AskPDF
Feb 22, 2026

Controlled Face Manipulation and Synthesis for Data Augmentation

Deep learning vision models excel with abundant supervision, but many applications face label scarcity and class imbalance. Controllable image editing can augment scarce labeled data, yet edits often introduce artifacts and entangle non-target attributes. We study this in facial expression analysis, targeting Action Unit (AU) manipulation where annotation is costly and AU co-activation drives entanglement. We present a facial manipulation method that operates in the semantic latent space of a pre-trained face generator (Diffusion Autoencoder). Using lightweight linear models, we reduce entanglement of semantic features via (i) dependency-aware conditioning that accounts for AU co-activation, and (ii) orthogonal projection that removes nuisance attribute directions (e.g., glasses), together with an expression neutralization step to enable absolute AU edit. We use these edits to balance AU occurrence by editing labeled faces and to diversify identities/demographics via controlled synthesis. Augmenting AU detector training with the generated data improves accuracy and yields more disentangled predictions with fewer co-activation shortcuts, outperforming alternative data-efficient training strategies and suggesting improvements similar to what would require substantially more labeled data in our learning-curve analysis. Compared to prior methods, our edits are stronger, produce fewer artifacts, and preserve identity better.

Joris Kirchner, Amogh Gudi, Marian Bittner +1

Vicarious Perception Technologies (VicarVision) · Delft University of Technology

Synthetic Data AugmentationImage EditingControllable Image GenerationFacial Action Unit DetectionDisentangled Representation Learning
AskPDF
Apr 22, 2026

MD-Face: MoE-Enhanced Label-Free Disentangled Representation for Interactive Facial Attribute Editing

GAN-based facial attribute editing is widely used in virtual avatars and social media but often suffers from attribute entanglement, where modifying one face attribute unintentionally alters others. While supervised disentangled representation learning can address this, it relies heavily on labeled data, incurring high annotation costs. To address these challenges, we propose MD-Face, a label-free disentangled representation learning framework based on Mixture of Experts (MoE). MD-Face utilizes a MoE backbone with a gating mechanism that dynamically allocates experts, enabling the model to learn semantic vectors with greater independence. To further enhance attribute entanglement, we introduce a geometry-aware loss, which aligns each semantic vector with its corresponding Semantic Boundary Vector (SBV) through a Jacobian-based pushforward method. Experiments with ProGAN and StyleGAN show that MD-Face outperforms unsupervised baselines and competes with supervised ones. Compared to diffusion-based methods, it offers better image quality and lower inference latency, making it ideal for interactive editing.

Xuan Cui, Yunfei Zhao, Bo Liu +2

Disentangled Representation LearningUnsupervised Representation LearningImage EditingGenerative Adversarial Networks
AskPDF