Learning Late, Guiding Early: Timestep-Decoupled Semantic Guidance for Fair Face Generation
Authors: Subir Kumar Parida, Rajbabu Velmurugan, Ketan Kotwal, R. S. Sengar, Swati Hiremath
Organizations: Bhabha Atomic Research Centre (B.A.R.C.), Mumbai, India · Department of Electrical Engineering, Indian Institute of Technology Bombay, Mumbai, India
Demographic imbalance in synthetic face generation can propagate to downstream face recognition systems, making fairness an important consideration when diffusion models are used for data generation. Existing fairness-aware generation approaches often require model retraining, architectural modifications, or repeated guidance throughout the reverse diffusion process. In this work, we introduce Semantic Boundary Predictor (SBP), an inference-time framework that performs demographic guidance through a one-shot intervention during reverse denoising. Our approach is motivated by the observation that latent representations at different diffusion timesteps play distinct semantic roles: late-stage latents provide stronger demographic separability, whereas early-stage latents offer greater flexibility for semantic intervention. SBP leverages this timestep decoupling by learning linear semantic boundaries from late-stage latent representations while applying them only once at the initial noisy latent, allowing the remainder of the reverse denoising process to proceed unchanged. The method requires neither retraining nor fine-tuning of the underlying Latent Diffusion Model and operates without external balanced datasets. Experiments on CelebA-HQ demonstrate substantial improvements in demographic fairness, reducing fairness disparity by 98% for gender, 95% for binary race, and 15% for four-class race, while maintaining perceptual image quality across demographic groups. Owing to its one-shot inference strategy and model-agnostic design, SBP introduces only a small computational overhead and can be readily integrated with existing pre-trained latent diffusion models.
Figures & tables
Figure 1: Conceptual overview of the proposed timestep-decoupled semantic guidance. A semantic boundary is learned at the semantically discriminative latent space at late-stage and transferred to the initial noisy latent space for one-shot guidance, enabling balanced image generation without retraining.
Figure 2: Gender distribution of latent encodings across denoising timesteps, showing stronger demographic separation in later stages.
Figure 3: Overview of SBP learning pipeline. (1) A pre-trained frozen LDM generates synthetic facial images. (2) The attribute classifier predicts demographic labels for the generated images. The predicted labels are paired with the corresponding latent representations to construct attribute-specific training datasets. (3) An independent Semantic Boundary Predictor (SBP) is trained for each demographic attribute, producing one semantic boundary vector per attribute for inference-time guidance.
ZT={zT1,…,zTN},
Algorithm 1 Inference Using Timestep-Decoupled Semantic Guidance
Attributes
T = 50
T = 40
T = 25
T = 10
T = 0
gender
0.59
0.63
0.81
0.95
0.97
race (2-class)
0.71
0.77
0.88
0.91
0.92
race (4-class)
0.71
0.75
0.77
0.90
0.91
age
0.73
0.82
0.91
0.94
0.95
Table 1: SBP Classifier Accuracy vs. Denoising Timestep
Figure 4: Late-stage latent representations require substantially fewer principal components to preserve the dominant variance than early-stage representations, indicating stronger semantic compactness. This observation supports learning semantic boundaries from late diffusion timesteps in the proposed timestep-decoupled guidance framework.
Method
FD ↓
FID ↓
Pre-trained DM (FFHQ)
0.212
-
Gaussian Harmony [ 28 ]
0.127
-
Pre-trained DM (FFHQ)
0.109
78.44
Unbiased-Diff [ 38 ]
0.021
78.23
Pre-trained DM (FFHQ)
0.068
17.45
SBP (ours)
0.006
18.41
Table 2: Quantitative comparison of demographic fairness (FD ↓ ) and perceptual quality (FID ↓ ) for gender generation. Methods are compared with the corresponding pre-trained DM trained on the same dataset.
Method
Race (B+W)
Race (4-class)
FD ↓
FID ↓
FD ↓
FID ↓
Pre-trained DM (FairFace)
-
-
0.393
-
Gaussian Harmony [ 28 ]
-
-
0.177
-
Pre-trained DM (FairFace)
-
-
0.222
58.98
Unbiased-Diff [ 38 ]
-
-
0.014
59.24
Pre-trained DM (CelebA)
0.334
60.01
0.292
89.14
Table 3: Quantitative comparison of demographic fairness (FD ↓ ) and perceptual quality (FID ↓ ) for racial generation.
Figure 5: Samples generated with our approach effectively address gender bias by producing an equal distribution of faces: approximately 51% female above the yellow line and 49% male below. Additionally, the generated faces exhibit higher perceptual quality, as reflected by the FID score.
Method
FD ↓
FID ↓
Pre-trained DM (CelebA)
0.256
60.68
Balancing Act [ 37 ]
0.283
71.84
Pre-trained DM (CelebA)
0.450
46.01
SBP (ours)
0.135
36.98
Table 4: Quantitative comparison of demographic fairness (FD ↓ ) and perceptual quality (FID ↓ ) for age generation.
Attribute
Gender + Race
Method
FD ↓
FID ↓
Random Sampling
0.256
60.68
Balancing Act [ 37 ]
0.075
49.91
SBP (ours)
0.055
45.16
Table 5: Evaluation of FD and FID scores for multi-attribute generation. The best score is highlighted in bold .
Method
Sampling throughput
samples/sec. ↑
Random Sampling [ 14 ]
1.56
Diff-AE [ 40 ]
0.30 †
Unbiased-Diff [ 38 ]
0.26
Balancing Act [ 37 ]
1.45 †
SBP (ours)
1.41
Table 7: Computational Efficiency Comparison. Sampling throughput was measured in samples/sec. on a single NVIDIA A100.
Scale δ
0
1
2
3
4
5
FD ↓
0.051
0.039
0.024
0.013
0.001
0.001
FID ↓
34.66
34.78
32.52
30.17
31.54
33.2
Table 8: Effect of δ on Gender Debiasing.
Figure 6: (a) Generation percentage vs. δ , (b) FID vs. δ . Effect of the guidance strength δ on demographic fairness (FD) and perceptual quality (FID) for four-class race generation. Increasing δ strengthens semantic guidance, improving demographic balance while gradually reducing image fidelity beyond the optimal operating point.
Training Set Size
FD ↓
FID ↓
5,000
0.002
36.27
15,000
0.001
31.54
45,000
0.001
31.46
Table 9: Influence of SBP training sample size on the final generation performance measured using FD and FID.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The top row depicts image manipulation with the latent representations when t=0 , that is, at the stage of well formed facial semantics. Attribute changes mainly affect fine-grained features around the lips and eyes, like glossy lips or eyes with kajal. The bottom row shows the manipulation at t=T in which semantic separability is less, but transformations yield visually consistent results through coarser structural changes.
no. of samples
FD ↓
FID ↓
2,000
0.001
34.13
5,000
0.001
32.49
10,000
0.001
31.54
Appendix
Table 10: Impact of Sample Size on Gender Bias in Generated Samples.
Figure 8: Qualitative visualization of gender guidance. The guidance strength δ controls movement along the learned semantic boundary. Increasing δ progressively changes male faces to female faces (left), whereas decreasing δ produces the reverse transition (right), while preserving image quality.
Figure 9: Samples from the LDM [ 14 ] model trained on CelebA-HQ, demonstrating the model’s tendency to produce disproportionately female faces (66%) shown above the yellow line and male faces (34%) below, amplifying the gender bias found in the original data.
Figure 10: Samples generated with our approach effectively address gender bias by producing an equal distribution of faces: approximately 51% female above the yellow line and 49% male below. Additionally, the generated faces exhibit higher perceptual quality, as reflected by the FID score.
Figure 11: Samples from the LDM [ 14 ] model trained on CelebA-HQ, highlighting the model’s inherent bias towards generating White faces (91%), bounded by yellow line, disproportionately. Black faces (4%) bounded by the blue line, Asian faces (3%) bounded by the green line, and Indian faces (2%) bounded by the red line are underrepresented, reflecting the racial bias present in the original dataset.
Figure 12: Our LDM + SBP approach mitigates racial bias by upsampling underrepresented groups. The generated samples demonstrate a more balanced distribution: White faces (58%) bounded by the yellow line, Black faces (16%) bounded by the blue line, Asian faces (20%) bounded by the green line, and Indian faces (6%) bounded by the red line. Despite the upsampling, the generated faces maintain high perceptual quality.