Debias Anything: Fairness with Diversity without Supervision in Diffusion Models

Théau d'Audiffret; Mariia Vladimirova; Jean-Yves Franceschi

cs.LGOct 1, 2026

Debias Anything: Fairness with Diversity without Supervision in Diffusion Models

Authors: Théau d'Audiffret, Mariia Vladimirova, Jean-Yves Franceschi

Organizations: Criteo AI Lab, Paris, France · ESSEC Business School, France · CentraleSupélec, Université Paris-Saclay, France · FairPlay joint team

Abstract

Although diffusion models produce high-quality images, they also reproduce and amplify demographic imbalances in their training data. Debiasing their generation process post-training w.r.t. some sensitive attribute usually relies on classifier guidance or explicit text extra-conditioning, but this reduces methods' applicability and output diversity. Conversely, methods promoting diversity alone do not ensure fair attribute representation. In this paper, we propose a method tackling fairness and diversity jointly that is generally applicable to any diffusion model and any sensitive attribute. To this end, an adapter connects the frozen diffusion model to a pretrained vision-language embedding space, enabling fairness and diversity guidance without sensitive-attribute annotations. For fairness, pairs of text prompts define attribute directions which guide batch composition towards specific proportions. For diversity, we introduce a score measuring disagreement between the semantic estimates derived from this representation. The formulation supports unconditional and text-conditional diffusion models, while requiring no prior knowledge or data of sensitive attribute. Experiments confirm that our method improves quality and diversity scores at comparable fairness levels.

Figures & tables

Figure 1: Generations examples on CelebA-HQ. Top : the original model, 6 random draws of women; two of them appear with artifacts, the third looking away from the camera and the fourth turning her head. Middle : guidance with the fairness term. Three columns turn into men and the faces sharpen, but the pose collapses, every face now frontal and centered. Bottom : guidance with the fairness and diversity terms. The woman in the second column looks older, the man in the fourth turns his head away and gains a mustache.

	no attribute supervision	arbitrary proportion	arbitrary conditioning	no retraining	attribute
Fair Diffusion ( Friedrich et al., 2025 )	✓	✓	✗	✓	text
ITI-GEN ( Zhang et al., 2023 )	✗	✓	✗	✓	ref. images
Self-debiasing ( Vardhana et al., 2026 )	✓	✗	✓	✓	discovered
Latent editing ( Kwon et al., 2023 )	✓	✗	✓	✓	text, training per attribute
Latent direction ( Li et al., 2024 )	✓	✓	✗	✓	text, training per attribute
FairGen ( Jiang et al., 2025 )	✓	✓	✗	✗	text, training per attribute

Table 1: Positioning among debiasing methods for diffusion models. A method may need attribute supervision (e.g. labels, reference images), may not accept an arbitrary target proportion, may not have been applied to both unconditional and text-conditioned models, and may retrain the generator. (

*

) Score Guidance also presents an alternate supervision-free version, but with significantly diminished performance compared to the supervised version, which is the one we compare with.

Panel of Figure 2 — Figure 2: One guided sampling step. The prompts $t_{k}$ are set once and encoded into directions $\Delta_{k}$ . At each step, the adapter $\varphi$ maps an inner layer $h(x_{\sigma})$ of the frozen denoiser to $m(x_{\sigma};\sigma)$ for every image of the batch, where the fairness and diversity terms are computed.

Figure 3: CelebA-HQ, P2 backbone, six shared seeds per column. Top: unguided samples. Second row: Balancing Act steered towards a dark-skinned woman with its labelled predictor. Third row: Debias Anything on the same target, given as two sentences. Bottom: Debias Anything on a second attribute with no CelebA-HQ label, red hair, from the same seeds.

Panel of Table 2 — Table 2: CelebA-HQ, P2 backbone, protocol of Balancing Act ( $10\,000$ images, one seed); all rows are our runs. $\mathrm{FD}$ is CLEAM-corrected except on race, which has no label to correct against. Guided rows are shaded from best (dark, bold) to worst (light); the bracket marks the rows matched on discrepancy.

	Gender		Age		Race
Method	$\mathrm{FD}$ $\downarrow$	$\mathrm{FID}$ $\downarrow$	$\mathrm{FD}$ $\downarrow$	$\mathrm{FID}$ $\downarrow$	$\mathrm{FD}$ $\downarrow$	$\mathrm{FID}$ $\downarrow$
Original	0.564	120.1	0.752	120.1	0.558	120.1
Attribute-specific
Latent editing ( Kwon et al., 2023 )	0.408	166.1	0.682	200.9	0.524	153.1
$h$ -distribution ( Parihar et al., 2024 )	0.222	151.7	0.506	147.7	0.544	126.9
Latent direction ( Li et al., 2024 )	0.305	129.4	0.052	113.8	0.175	128.3

Table 3: Stable Diffusion 1.5, protocol of Shi et al. (2025) , whose reported numbers fill the baseline rows bar ICM. All baselines incorporate attribute knowledge (labeled data, attribute-specific training); ours only gets two sentences. Guided rows are shaded from best (dark, bold) to worst (light).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Data	Noise levels	Optimisation	Epochs	Time
EDM, CelebA	$162{,}770$ training images	$\sigma$ log-uniform on $[0.002,80]$	AdamW, batch 512, lr $10^{-3}$	$50$	$2.9$ h
P2, CelebA-HQ	$28{,}000$ ( $+2{,}000$ val.)	$t$ uniform on the 49 guided steps	AdamW, batch 32, lr $10^{-3}$ , wd $10^{-4}$	$47$ of $60$	$4.2$ h
SD 1.5	$49{,}000$ generated ( $+1{,}000$ val.)	$\sigma$ uniform on $[0.03,14.6]$	AdamW, batch 64, lr $5\cdot 10^{-4}$ , wd $0.05$ , warm-up and cosine	$47$ of $50$	$11$ h

Appendix

Table 4: Adapter training, on one A100 GPU.

Figure 4: What the CelebA adapter reads of the attribute as the noise grows. AUC of the logit $S(x_{\sigma};\sigma)$ along the direction $\Delta$ of Equation 15 , against the CelebA label, on $500$ images per class of the test split, for gender ( a photo of a man / a photo of a woman , left) and eyeglasses ( a photo of a person wearing eyeglasses / a photo of a person , right). Circles use the adapter output $\varphi_{\psi}(h(x_{\sigma});\sigma)$ , squares the encoder on the Tweedie estimate, $E_{\vartheta}^{\mathrm{img}}(\hat{x}_{0})$ ; the dotted line is the encoder on the clean image, $E_{\vartheta}^{\mathrm{img}}(x_{0})$ . Over the guidance window $\sigma\leq 10$ the adapter matches the Tweedie estimate on gender and exceeds it on eyeglasses, at no decoding cost.

Figure 5: Steering an attribute that has no label. Top: unguided samples from a diffusion model trained on CelebA. Bottom: the same eight initial noises, guided towards dark-skinned female with sunglasses . The attribute is handed to the sampler as this sentence and a contrasting one describing its absence. CelebA has no label for it, and neither the generator nor any classifier was trained.

Figure 6: Same compound target, dark-skinned female with sunglasses , three ways of reading the guidance term. Top: from $\varphi(h(x_{\sigma});\sigma)$ . Middle: from $E_{\vartheta}^{\mathrm{img}}(\hat{x}_{0})$ at the same sampling budget. Bottom: from $E_{\vartheta}^{\mathrm{img}}(\hat{x}_{0})$ with more than twice the budget. Eight samples per row.

Eyeglasses attribute ( $\mathrm{FD}$ $=0.614$ unguided)
	h-space guidance			pixel guidance
$w$	$\mathrm{MIND}$	$\mathrm{FD}$	$\mathrm{FID}$	$\mathrm{MIND}$	$\mathrm{FD}$	$\mathrm{FID}$
0	19.13	0.614	16.23	19.13	0.614	16.23
50	15.14	0.534	13.81	19.69	0.611	16.92
100	11.81	0.441	12.14	19.88	0.598	17.66
150	9.97	0.377	11.60	20.38	0.587	18.56

Appendix

Table 5:

h

-space against pixel-space fairness term at equal weight

w

;

\mathrm{FD}

on the steered attribute; both share the

w=0

run, 5000 images, one seed. Bold: the better

\mathrm{MIND}

of the two on each row.

Panel of Figure 7 — Figure 7: FD– $\mathrm{MIND}$ trade-off, h-space versus pixel-space fairness term; each polyline follows its own sweep in order of increasing guidance weight $w$ , and the white square is the shared unguided model. Left: on eyeglasses the two curves do not overlap, pixel guidance stays pinned in the top-right corner, drifting rightwards as $w$ grows, while h-space sweeps down and to the left, then turns back up past $w=200$ as fidelity starts to cost, so no pixel setting matches any h-space setting on both axes. Right: on gender the two sweeps trace the same short hook, pixel running marginally inside h-space; both bottom out in $\mathrm{MIND}$ around $w=5$ and then climb steeply as $\mathrm{FD}$ is driven to zero.

Method	$w_{\mathrm{div}}$	$\mathrm{MIND}$ $\downarrow$	$\mathrm{FD}$ $\downarrow$	$\mathrm{FID}$ $\downarrow$	Vendi $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	Cov. $\uparrow$	Dens. $\uparrow$
avgKNN	0	10.76	0.336	19.21	6.47	0.814	0.679	0.893	1.218
	2	10.56	0.332	19.21	6.52	0.812	0.689	0.893	1.208
	5 ⋆	10.44	0.330	19.39	6.58	0.805	0.690	0.893	1.190
	10	10.46	0.330	19.78	6.69	0.795	0.697	0.880	1.159
	20	11.24	0.317	20.77	6.88	0.778	0.692	0.870	1.131
	40	14.72	0.309	23.80	7.31	0.765	0.696	0.835	1.035

Appendix

Table 6: Diversity-weight sweep on eyeglasses, fairness weight fixed at

200

w_{\mathrm{div}}=0

is the fairness-only baseline, 2000 images, one seed;

\star

marks the setting carried to Table 8 .

Method	$w_{\mathrm{div}}$	$\mathrm{MIND}$ $\downarrow$	$\mathrm{FD}$ $\downarrow$	$\mathrm{FID}$ $\downarrow$	Vendi $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	Cov. $\uparrow$	Dens. $\uparrow$
avgKNN	0	2.76	0.0104	12.41	6.18	0.829	0.723	0.970	1.133
	2	2.65	0.0075	12.40	6.24	0.828	0.723	0.969	1.124
	5 ⋆	2.54	0.0039	12.39	6.32	0.832	0.732	0.973	1.113
	10	2.56	0.0017	12.54	6.45	0.820	0.734	0.963	1.094
	20	3.42	0.0019	13.28	6.72	0.824	0.732	0.967	1.060
	40	8.34	0.0121	16.81	7.26	0.792	0.749	0.902	0.936

Appendix

Table 7: Diversity-weight sweep, on gender, fairness weight fixed at

10

w_{\mathrm{div}}=0

is the fairness-only baseline, 2000 images, one seed;

\star

marks the setting carried to Table 8 .

Gender, $w_{\text{fair}}=5$
	time (s)	$\mathrm{FD}$ $\downarrow$	$\mathrm{FID}$ $\downarrow$	$\mathrm{MIND}$ $\downarrow$	Vendi $\uparrow$
Debias Anything	$1613\pm 665$	$0.031\pm 0.004$	$6.59\pm 0.04$	$2.52\pm 0.13$	$6.26\pm 0.03$
$+$ SGMS	$1380\pm 5$	$0.023\pm 0.005$	$6.46\pm 0.05$	$2.05\pm 0.14$	$6.46\pm 0.02$
$+$ avgKNN	$1635\pm 713$	$0.021\pm 0.003$	$6.40\pm 0.11$	$2.06\pm 0.15$	$6.42\pm 0.03$
$+$ SigLIPMS	$2179\pm 962$	$0.011\pm 0.004$	$6.48\pm 0.07$	$2.13\pm 0.17$	$6.41\pm 0.04$
$+$ PerturbProj	$1476\pm 592$	$0.006\pm 0.003$	$6.44\pm 0.10$	$2.03\pm 0.15$	$6.43\pm 0.04$

Appendix

Table 8: Diversity terms on top of the fairness term at the selected weights, 5000 images, 5 seeds; time is wall-clock generation of one run and is not ranked.

Gender, $w_{\text{fair}}=5$
	time (s)	$\mathrm{FD}$ $\downarrow$	$\mathrm{FID}$ $\downarrow$	$\mathrm{MIND}$ $\downarrow$	Vendi $\uparrow$
Debias Anything	11065	0.024	2.48	2.14	6.33
$+$ SGMS	15747	0.018	2.24	1.61	6.55
$+$ avgKNN	10942	0.015	2.22	1.63	6.50
$+$ SigLIPMS	15560	0.008	2.29	1.72	6.48
$+$ PerturbProj	11417	0.002	2.23	1.59	6.52

Appendix

Table 9: As Table 8 , with

50\,000

images per configuration and one seed.

Method	Setting	$\mathrm{MIND}$ $\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$	Cov. $\uparrow$	Dens. $\uparrow$	Vendi $\uparrow$
Original	—	124.10	0.972	0.389	0.809	3.010	5.94
avgKNN	$\kappa=50,\ w_{\mathrm{div}}=30$	62.71	0.954	0.425	0.874	2.875	7.33
SGMS	$\sigma_{s}=0.8,\ w_{\mathrm{div}}=40$	100.96	0.955	0.468	0.841	2.783	6.63
PerturbProj	$\sigma_{s}=0.4,\ w_{\mathrm{div}}=250$	92.34	0.958	0.386	0.837	2.909	6.56
SigLIPMS	$w_{\mathrm{div}}=250$	71.17	0.953	0.332	0.865	3.076	6.96

Appendix

Table 10: Each diversity term alone, at its lowest-

\mathrm{MIND}

setting with precision

\geq 0.95

in Tables 12 , 14 , 14 and 12 .

Panel of Figure 8 — Figure 8: MIND against the diversity guidance weight $w_{\mathrm{div}}$ , one curve per hyperparameter configuration. All four panels share the $\mathrm{MIND}$ axis. avgKNN is nearly insensitive to $\kappa$ over two orders of magnitude (seven superimposed curves, $\kappa=1$ to $\kappa=100$ ). For SGMS and PerturbProj the perturbation scale is non-monotone: $\sigma_{s}=0.2$ is markedly worse than $0.4$ , so the useful range is $\sigma_{s}\in[0.4,0.6]$ . SGMS is the only method reaching $\mathrm{MIND}$ below $60$ , and it does so at a precision below $0.9$ ( Table 14 ). SigLIPMS turns around: $\mathrm{MIND}$ bottoms out at $w_{\mathrm{div}}=250$ and rises again beyond it.

Table 19

Table 20

Figure 9: Gender on CelebA-HQ, six shared seeds: the unguided model, Balancing Act, and Debias Anything at matched discrepancy.

Figure 10: Sensitivity to the prompt pair, gender on CelebA-HQ, $w_{\text{fair}}=20$ , six shared seeds. Top: unguided. Then, one pair per row: a photo of a woman / a photo of a man ; a female person / a male person ; a girl’s portrait / a boy’s portrait ; a photo of a lady / a photo of a gentleman ; a woman’s close-up photo / a man’s close-up photo .

Panel of Figure 11 — Figure 11: Stable Diffusion 1.5, prompts A face of a doctor (top pair) and A face of a firefighter (bottom pair), six shared seeds each: the unguided model and Debias Anything at a gender target of one half. Two doctor columns and three firefighter columns change gender; clothing, pose, background and expression stay.

Occupation	$\mathrm{FD}$ gender $\downarrow$	$\mathrm{FD}$ age $\downarrow$	$\mathrm{FD}$ race $\downarrow$
doctor	0.021	0.092	0.208
firefighter	0.104	0.276	0.193
nurse	0.296	0.077	0.206
receptionist	0.378	0.381	0.174
mean ( Table 3 )	0.200	0.206	0.195

Appendix

Table 15: Debias Anything on Stable Diffusion 1.5, discrepancy per occupation. Each entry is the distance of the mean FairFace output to the uniform target on the

500

images of one prompt; the last row is their average, the

\mathrm{FD}

of Table 3 .

Feature or weight editing
		Gender				Age				Race
Method	Attribute	$\mathrm{FD}$ $\downarrow$	$\mathrm{FID}$ $\downarrow$	CLIP-I $\uparrow$	CLIP-T $\uparrow$	$\mathrm{FD}$ $\downarrow$	$\mathrm{FID}$ $\downarrow$	CLIP-I $\uparrow$	CLIP-T $\uparrow$	$\mathrm{FD}$ $\downarrow$	$\mathrm{FID}$ $\downarrow$	CLIP-I $\uparrow$	CLIP-T $\uparrow$
Original	—	0.564	120.06	–	0.6155	0.752	120.06	–	0.6155	0.558	120.06	–	0.6155
Latent editing ( Kwon et al., 2023 )	text, training per attribute	0.408	166.11	0.8253	0.6005	0.682	200.90	0.8527	0.6122	0.524	153.05	0.8804	0.6086
Latent direction ( Li et al., 2024 )	text, training per attribute	0.305	129.37	0.8058	0.6091	0.052	113.81	0.8151	0.6067	0.175	128.30	0.8211	0.6132
Fine-tuning ( Shen et al., 2024 )	attribute classifier	0.050	161.47	0.8779	0.6095	0.746	161.47	0.8779	0.6095	0.198	161.47	0.8779	0.6095

Appendix

Table 16: Stable Diffusion 1.5 ( Rombach et al., 2022 ) under the protocol of DiffLens ( Shi et al., 2025 ) , all four metrics. Baseline rows other than ICM ( Zaleska et al., 2026 ) are as reported by Shi et al. (2025) , which gives fine-tuning ( Shen et al., 2024 ) a single

\mathrm{FID}

, CLIP-I and CLIP-T for the three attributes. Guided rows are shaded from best (dark, bold) to worst (light); equal values share a shade. The unguided row has no CLIP-I by definition.

Explore similar work

May 27, 2026cs.CV

Stay Fair! Ensuring Group Fairness in Diffusion Models Across Guidance Scales

Diffusion models steer conditional generation with a tunable guidance scale to trade off prompt alignment and diversity. However, existing debiasing techniques are optimized for a single scale, degrading fairness when users adjust this parameter. We trace this behavior to a previously overlooked source by decomposing total bias into two components: a model bias and a guidance bias. While prior work primarily targets the former, we show that the guidance bias grows monotonically with the guidance scale, eventually dominating the high-guidance regimes users prefer. To address this, we extend Strong Demographic Parity to guidance and derive a condition under which the target distribution retains its group ratio across guidance scales. We propose StayFair, which leverages this condition to design fair guidance algorithms in both regimes. For classifier guidance, it equalizes the classifier's output distributions across groups; for classifier-free guidance, it shifts the null embedding by a prompt-dependent offset. Because StayFair modifies only the guidance step, it is orthogonal to model debiasing and can be layered onto existing fair diffusion models to extend their fairness across guidance scales. Across class-conditional and text-to-image generation, StayFair decouples fairness from the guidance scale without sacrificing image quality.

Myeongsoo Kim, Eunji Kim, Minwoo Chae +1

1POSTECH · 2Amazon

Algorithmic Fairness Diffusion Models

Ask PDF

Jul 1, 2026cs.CV

Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization

Text-to-image diffusion models achieve impressive visual quality, yet demographic bias remains a challenge, as neutral prompts consistently produce stereotypical representations across gender and race. Existing approaches remain limited by costly retraining or by inference-time interventions that often degrade image quality and semantic alignment. We propose Text Embedding Steering (TES), a training-free framework that mitigates demographic bias by directly optimizing conditional text embeddings during the diffusion process. We show that a two-stage strategy - early-stage global alignment followed by iterative denoising-time refinement with CLIP-based feedback - enables stable and controllable attribute steering without modifying model parameters. Extensive experiments on Stable Diffusion demonstrate that TES outperforms existing training-free baselines in fairness while maintaining competitive image quality. These results highlight that inference-time text embedding optimization is a practical and scalable solution for fairness-aware generation in diffusion models.

Dain Kim, Jinseo Kim, Sungyong Baik

Dept. of Artificial Intelligence, Hanyang University, South Korea · Dept. of Data Science, Hanyang University, South Korea

Text-To-Image Diffusion Models Diffusion Models Training-Free Algorithmic Fairness Stereotype Mitigation

Ask PDF

Jul 1, 2026cs.CV

EquiSteer: Cross-Attention Steering Towards a Fairer Text-Guided Image Generation

Text-to-image diffusion models power everyday creative tasks, but they still reproduce the demographic biases in their training data. On common prompts such as a photo of a nurse,'' a photo of a CEO'', they skew their outputs toward one gender, driven by the statistics of training data rather than anything in the text. Existing debiasing methods show promise in narrow settings but require retraining, batch-level control, or prompt-specific tuning, limiting their scalability. We propose \emph{EquiSteer}, a training-free method that works per sample by steering cross-attention (CA) activations at inference time. For each target attribute, EquiSteer precomputes steering vectors from contrastive prompts. Then at generation time, a prompt-aware gate leaves attribute-specific prompts untouched, while for neutral ones it clears existing attribute signals from the CA activations and injects a target attribute. Across SD-1.5, SD-2.1, SDXL, and SANA, EquiSteer reduces the average parity gap by up to

87\%

, with minimal effect on image quality and text-image alignment. Code is available at \href{https://github.com/Atmyre/EquiSteer}{https://github.com/Atmyre/EquiSteer}.%

Tatiana Gaintseva, Akshit Achara, Gregory Slabaugh +2

Queen Mary University of London, UK · Huawei Noah’s Ark, UK · King’s College London, UK +1

Text-To-Image Diffusion Models Text Generation Cross-Attention Image Quality Assessment

Ask PDF