Learning Late, Guiding Early: Timestep-Decoupled Semantic Guidance for Fair Face Generation
Authors: Subir Kumar Parida, Rajbabu Velmurugan, Ketan Kotwal, R. S. Sengar, Swati Hiremath
Organizations: Bhabha Atomic Research Centre (B.A.R.C.), Mumbai, India · Department of Electrical Engineering, Indian Institute of Technology Bombay, Mumbai, India
Demographic imbalance in synthetic face generation can propagate to downstream face recognition systems, making fairness an important consideration when diffusion models are used for data generation. Existing fairness-aware generation approaches often require model retraining, architectural modifications, or repeated guidance throughout the reverse diffusion process. In this work, we introduce Semantic Boundary Predictor (SBP), an inference-time framework that performs demographic guidance through a one-shot intervention during reverse denoising. Our approach is motivated by the observation that latent representations at different diffusion timesteps play distinct semantic roles: late-stage latents provide stronger demographic separability, whereas early-stage latents offer greater flexibility for semantic intervention. SBP leverages this timestep decoupling by learning linear semantic boundaries from late-stage latent representations while applying them only once at the initial noisy latent, allowing the remainder of the reverse denoising process to proceed unchanged. The method requires neither retraining nor fine-tuning of the underlying Latent Diffusion Model and operates without external balanced datasets. Experiments on CelebA-HQ demonstrate substantial improvements in demographic fairness, reducing fairness disparity by 98% for gender, 95% for binary race, and 15% for four-class race, while maintaining perceptual image quality across demographic groups. Owing to its one-shot inference strategy and model-agnostic design, SBP introduces only a small computational overhead and can be readily integrated with existing pre-trained latent diffusion models.
Figures & tables
Figure 1: Conceptual overview of the proposed timestep-decoupled semantic guidance. A semantic boundary is learned at the semantically discriminative latent space at late-stage and transferred to the initial noisy latent space for one-shot guidance, enabling balanced image generation without retraining.
Figure 2: Gender distribution of latent encodings across denoising timesteps, showing stronger demographic separation in later stages.
Figure 3: Overview of SBP learning pipeline. (1) A pre-trained frozen LDM generates synthetic facial images. (2) The attribute classifier predicts demographic labels for the generated images. The predicted labels are paired with the corresponding latent representations to construct attribute-specific training datasets. (3) An independent Semantic Boundary Predictor (SBP) is trained for each demographic attribute, producing one semantic boundary vector per attribute for inference-time guidance.
ZT={zT1,…,zTN},
Algorithm 1 Inference Using Timestep-Decoupled Semantic Guidance
Attributes
T = 50
T = 40
T = 25
T = 10
T = 0
gender
0.59
0.63
0.81
0.95
0.97
race (2-class)
0.71
0.77
0.88
0.91
0.92
race (4-class)
0.71
0.75
0.77
0.90
0.91
age
0.73
0.82
0.91
0.94
0.95
Table 1: SBP Classifier Accuracy vs. Denoising Timestep
Figure 4: Late-stage latent representations require substantially fewer principal components to preserve the dominant variance than early-stage representations, indicating stronger semantic compactness. This observation supports learning semantic boundaries from late diffusion timesteps in the proposed timestep-decoupled guidance framework.
Method
FD ↓
FID ↓
Pre-trained DM (FFHQ)
0.212
-
Gaussian Harmony [ 28 ]
0.127
-
Pre-trained DM (FFHQ)
0.109
78.44
Unbiased-Diff [ 38 ]
0.021
78.23
Pre-trained DM (FFHQ)
0.068
17.45
SBP (ours)
0.006
18.41
Table 2: Quantitative comparison of demographic fairness (FD ↓ ) and perceptual quality (FID ↓ ) for gender generation. Methods are compared with the corresponding pre-trained DM trained on the same dataset.
Method
Race (B+W)
Race (4-class)
FD ↓
FID ↓
FD ↓
FID ↓
Pre-trained DM (FairFace)
-
-
0.393
-
Gaussian Harmony [ 28 ]
-
-
0.177
-
Pre-trained DM (FairFace)
-
-
0.222
58.98
Unbiased-Diff [ 38 ]
-
-
0.014
59.24
Pre-trained DM (CelebA)
0.334
60.01
0.292
89.14
Table 3: Quantitative comparison of demographic fairness (FD ↓ ) and perceptual quality (FID ↓ ) for racial generation.
Figure 5: Samples generated with our approach effectively address gender bias by producing an equal distribution of faces: approximately 51% female above the yellow line and 49% male below. Additionally, the generated faces exhibit higher perceptual quality, as reflected by the FID score.
Method
FD ↓
FID ↓
Pre-trained DM (CelebA)
0.256
60.68
Balancing Act [ 37 ]
0.283
71.84
Pre-trained DM (CelebA)
0.450
46.01
SBP (ours)
0.135
36.98
Table 4: Quantitative comparison of demographic fairness (FD ↓ ) and perceptual quality (FID ↓ ) for age generation.
Attribute
Gender + Race
Method
FD ↓
FID ↓
Random Sampling
0.256
60.68
Balancing Act [ 37 ]
0.075
49.91
SBP (ours)
0.055
45.16
Table 5: Evaluation of FD and FID scores for multi-attribute generation. The best score is highlighted in bold .
Method
Sampling throughput
samples/sec. ↑
Random Sampling [ 14 ]
1.56
Diff-AE [ 40 ]
0.30 †
Unbiased-Diff [ 38 ]
0.26
Balancing Act [ 37 ]
1.45 †
SBP (ours)
1.41
Table 7: Computational Efficiency Comparison. Sampling throughput was measured in samples/sec. on a single NVIDIA A100.
Scale δ
0
1
2
3
4
5
FD ↓
0.051
0.039
0.024
0.013
0.001
0.001
FID ↓
34.66
34.78
32.52
30.17
31.54
33.2
Table 8: Effect of δ on Gender Debiasing.
Figure 6: (a) Generation percentage vs. δ , (b) FID vs. δ . Effect of the guidance strength δ on demographic fairness (FD) and perceptual quality (FID) for four-class race generation. Increasing δ strengthens semantic guidance, improving demographic balance while gradually reducing image fidelity beyond the optimal operating point.
Training Set Size
FD ↓
FID ↓
5,000
0.002
36.27
15,000
0.001
31.54
45,000
0.001
31.46
Table 9: Influence of SBP training sample size on the final generation performance measured using FD and FID.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The top row depicts image manipulation with the latent representations when t=0 , that is, at the stage of well formed facial semantics. Attribute changes mainly affect fine-grained features around the lips and eyes, like glossy lips or eyes with kajal. The bottom row shows the manipulation at t=T in which semantic separability is less, but transformations yield visually consistent results through coarser structural changes.
no. of samples
FD ↓
FID ↓
2,000
0.001
34.13
5,000
0.001
32.49
10,000
0.001
31.54
Appendix
Table 10: Impact of Sample Size on Gender Bias in Generated Samples.
Figure 8: Qualitative visualization of gender guidance. The guidance strength δ controls movement along the learned semantic boundary. Increasing δ progressively changes male faces to female faces (left), whereas decreasing δ produces the reverse transition (right), while preserving image quality.
Figure 9: Samples from the LDM [ 14 ] model trained on CelebA-HQ, demonstrating the model’s tendency to produce disproportionately female faces (66%) shown above the yellow line and male faces (34%) below, amplifying the gender bias found in the original data.
Figure 10: Samples generated with our approach effectively address gender bias by producing an equal distribution of faces: approximately 51% female above the yellow line and 49% male below. Additionally, the generated faces exhibit higher perceptual quality, as reflected by the FID score.
Figure 11: Samples from the LDM [ 14 ] model trained on CelebA-HQ, highlighting the model’s inherent bias towards generating White faces (91%), bounded by yellow line, disproportionately. Black faces (4%) bounded by the blue line, Asian faces (3%) bounded by the green line, and Indian faces (2%) bounded by the red line are underrepresented, reflecting the racial bias present in the original dataset.
Figure 12: Our LDM + SBP approach mitigates racial bias by upsampling underrepresented groups. The generated samples demonstrate a more balanced distribution: White faces (58%) bounded by the yellow line, Black faces (16%) bounded by the blue line, Asian faces (20%) bounded by the green line, and Indian faces (6%) bounded by the red line. Despite the upsampling, the generated faces maintain high perceptual quality.
Although diffusion models produce high-quality images, they also reproduce and amplify demographic imbalances in their training data. Debiasing their generation process post-training w.r.t. some sensitive attribute usually relies on classifier guidance or explicit text extra-conditioning, but this reduces methods' applicability and output diversity. Conversely, methods promoting diversity alone do not ensure fair attribute representation. In this paper, we propose a method tackling fairness and diversity jointly that is generally applicable to any diffusion model and any sensitive attribute. To this end, an adapter connects the frozen diffusion model to a pretrained vision-language embedding space, enabling fairness and diversity guidance without sensitive-attribute annotations. For fairness, pairs of text prompts define attribute directions which guide batch composition towards specific proportions. For diversity, we introduce a score measuring disagreement between the semantic estimates derived from this representation. The formulation supports unconditional and text-conditional diffusion models, while requiring no prior knowledge or data of sensitive attribute. Experiments confirm that our method improves quality and diversity scores at comparable fairness levels.
Diffusion models steer conditional generation with a tunable guidance scale to trade off prompt alignment and diversity. However, existing debiasing techniques are optimized for a single scale, degrading fairness when users adjust this parameter. We trace this behavior to a previously overlooked source by decomposing total bias into two components: a model bias and a guidance bias. While prior work primarily targets the former, we show that the guidance bias varies with the guidance scale, persisting even in models that have already been debiased. To address this, we extend Strong Demographic Parity to guidance and derive a condition under which the target distribution retains its group ratio across guidance scales. Building on this condition, we propose StayFair, which designs fair guidance algorithms in both regimes. For classifier guidance, it equalizes the classifier's output distributions across groups; for classifier-free guidance, it shifts the null embedding by a prompt-dependent offset. Because StayFair modifies only the guidance step, it is orthogonal to model debiasing and can be layered onto existing fair diffusion models. Across class-conditional and text-to-image generation, StayFair decouples fairness from the guidance scale without sacrificing image quality. Our decomposition also enables a more detailed analysis of the sources of bias in diffusion models.
Text-to-image diffusion models achieve impressive visual quality, yet demographic bias remains a challenge, as neutral prompts consistently produce stereotypical representations across gender and race. Existing approaches remain limited by costly retraining or by inference-time interventions that often degrade image quality and semantic alignment. We propose Text Embedding Steering (TES), a training-free framework that mitigates demographic bias by directly optimizing conditional text embeddings during the diffusion process. We show that a two-stage strategy - early-stage global alignment followed by iterative denoising-time refinement with CLIP-based feedback - enables stable and controllable attribute steering without modifying model parameters. Extensive experiments on Stable Diffusion demonstrate that TES outperforms existing training-free baselines in fairness while maintaining competitive image quality. These results highlight that inference-time text embedding optimization is a practical and scalable solution for fairness-aware generation in diffusion models.
Dain Kim, Jinseo Kim, Sungyong Baik
Dept. of Artificial Intelligence, Hanyang University, South Korea · Dept. of Data Science, Hanyang University, South Korea