Spatially Gated Demographic Steering for Debiasing Multi-Person Text-to-Image Generation

Venkatesh Thirugnana Sambandham; Torsten Schon

cs.CVOct 21, 2026

Spatially Gated Demographic Steering for Debiasing Multi-Person Text-to-Image Generation

Authors: Venkatesh Thirugnana Sambandham, Torsten Schon

Organizations: Technische Hochschule Ingolstadt, Ingolstadt, Germany

Abstract

Text-to-Image models reproduce societal stereotypes, and existing mitigation applies one demographic correction to the whole scene. In multi-person scene generation this creates a trade-off: global steering improves aggregate balance only by homogenizing the people within an image. Steering-based methods like Embedding Arithmetic drive the intra-image diversity below the unsteered baseline. Text- and latent-projection approaches shift the aggregate skew with only marginal gains in intra-image diversity. We present Spatially Gated Steering (SGS), which steers the demographics of each person independently by extracting per-person masks from the Diffusion Transformer's (DiT) internal attention maps at a critical denoising timestep and applying a distinct demographic vector within each mask for the remaining steps. SGS needs no fine-tuning, inversion or layout specifications, and uses a lightweight segmentor over frozen DiT attention features to localize subjects at early timesteps. On skewed occupational prompts, SGS reaches high intra-image racial and gender diversity while keeping aggregate balance competitive and image quality near baseline, a result we validate on two architecturally distinct diffusion transformers, FLUX.1-Dev and SD3.5-Large. Codebase and model weights are available at https://github.com/cvims/spatially-gated-steering

Figures & tables

Figure 1: SGS intuition. The base trajectory (black) reaches $x_{0}^{\text{base}}$ in a stereotyped region of $p_{\text{data}}$ ; global mitigation [ 39 ] (brown dashed) shifts it to one alternative mode. SGS instead intercepts at $x_{\text{start}}$ , extracts per-person masks, and steers each individual independently (orange/blue dashed) toward distinct demographic attractors within the same image $x_{0}$ .

Figure 2: Qualitative comparison across three profession types spanning different stereotype directions. Prompts contain no demographic descriptors. Base is demographically homogeneous; EA shifts everyone uniformly; FairImagen is statistically indistinguishable from baseline on FLUX; SGS steers each instance toward a distinct intersectional target while preserving occupational context and scene coherence. Rightmost column: segmentation masks (nearest-neighbor upsampled from $32\times 32$ ) with assigned demographic labels.

Figure 3: Instance segmentation analysis (

n{=}720

): segmentation performance across the FLUX denoising trajectory for four architectures. All methods peak around

t{\approx}0.7

–

0.8

; DETR consistently outperforms all baselines. Layer bin ablation motivating our choice of Bins 3+4 as

\mathbf{F}_{t}

(Eq. ( 3 )) is in supplementary Section B .

	FLUX.1-Dev					SD3.5-Large
Method	$H_{ix}$ $\uparrow$	CLIP $\uparrow$	$Aes\Delta$ $\uparrow$	DR ${}_{\text{race}}$ $\downarrow$	DR ${}_{\text{gender}}$ $\downarrow$	$H_{ix}$ $\uparrow$	CLIP $\uparrow$	$Aes\Delta$ $\uparrow$	DR ${}_{\text{race}}$ $\downarrow$	DR ${}_{\text{gender}}$ $\downarrow$
Baseline	0.524 $\pm$ 0.016	0.321 $\pm$ 0.001	–	0.748 $\pm$ 0.015	0.608 $\pm$ 0.022	0.549 $\pm$ 0.015	0.325 $\pm$ 0.001	–	0.615 $\pm$ 0.018	0.367 $\pm$ 0.026
EA [ 39 ]	0.285 $\pm$ 0.013	0.236 $\pm$ 0.002	+0.336 $\pm$ 0.011	0.079 $\pm$ 0.016	0.434 $\pm$ 0.027	0.319 $\pm$ 0.012	0.323 $\pm$ 0.001	+0.163 $\pm$ 0.010	0.069 $\pm$ 0.016	0.303 $\pm$ 0.029
SDID [ 25 ]	0.515 $\pm$ 0.016	0.314 $\pm$ 0.001	+0.118 $\pm$ 0.008	0.797 $\pm$ 0.013	0.593 $\pm$ 0.023	0.573 $\pm$ 0.015	0.322 $\pm$ 0.001	+0.006 $\pm$ 0.015	0.628 $\pm$ 0.017	0.456 $\pm$ 0.026
RespoDiff [ 40 ]	0.540 $\pm$ 0.017	0.315 $\pm$ 0.001	$-$ 0.055 $\pm$ 0.006	0.737 $\pm$ 0.017	0.624 $\pm$ 0.022	0.575 $\pm$ 0.015	0.326 $\pm$ 0.001	$-$ 0.042 $\pm$ 0.009	0.584 $\pm$ 0.018	0.458 $\pm$ 0.025
FairImagen [ 12 ]	0.534 $\pm$ 0.016	0.320 $\pm$ 0.001	+0.004 $\pm$ 0.005	0.744 $\pm$ 0.015	0.574 $\pm$ 0.025	0.539 $\pm$ 0.015	0.319 $\pm$ 0.001	+0.057 $\pm$ 0.011	0.385 $\pm$ 0.019	0.293 $\pm$ 0.028

Table 1: Quantitative results on the extended WinoBias occupation benchmark (30 occupations

\times

5 templates

\times

k\in\{2,3,4\}

\times

10 seeds,

n{=}4{,}500

/method/architecture), on FLUX.1-Dev and SD3.5-Large.

H_{ix}

: intra-image intersectional entropy. DR: deviation ratio from uniform (lower is better). Bold and Underlined values represent the best and second-best methods per metric, within each architecture.

vs.	Diversity	Faithful.	Preference
Baseline	88.5% ∗	46.2%	47.4%
EA	90.5% ∗	89.4% ∗	75.6% ∗
FairImagen	88.8% ∗	45.8%	46.8%
RespoDiff	89.0% ∗	51.9%	51.3%
SDID	90.5% ∗	52.3%	55.1%

Table 2: Human evaluation : share of judgments preferring SGS over each baseline (

n{=}65

raters). ∗ : statistically significant preference (

p{<}0.05

Method	FPs	Time (s)	Overhead
Baseline	21	7.8 $\pm$ 0.1	1.0
EA	21	7.9 $\pm$ 0.0	1.0
SGS (K=2)	33	12.3 $\pm$ 0.0	1.6
SGS (K=3)	39	14.6 $\pm$ 0.0	1.9
SGS (K=4)	45	16.8 $\pm$ 0.0	2.2

Table 3: Inference time per-image (A6000 GPU, 512

\times

512, 21 denoising steps). FPs: total forward passes through the DiT per image. Overhead relative to unsteered baseline.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Bin	mIoU	AP50	AP50-95
Bin 1	0.539	0.581	0.225
Bin 2	0.583	0.661	0.287
Bin 3	0.606	0.710	0.333
Bin 4	0.626	0.736	0.360
3+4	0.631	0.747	0.368

Appendix

Table S1: Layer bin ablation (

n{=}720

). Mean mIoU, AP50, and AP50-95 averaged across all denoising timesteps using DETR. Best in bold .

Figure 4: Segmentation performance of the SGS instance segmentor (DETR variant) broken down by scene difficulty ( left ) and person count

K

( right ). Left: mIoU, AP50, and AP50-95 averaged over all

K

for easy, medium, and hard scenes. Performance degrades with difficulty primarily due to increased occlusion and smaller per-person regions. Right: The same metrics averaged over difficulty levels per

K

. All three metrics decline as

K

increases, reflecting the greater spatial competition among instances at higher person counts. Results are reported at denoising step

t=4

Figure 5: Qualitative comparison of instance segmentation methods across four scenes of increasing difficulty. Each row shows the generated image, SAM2 oracle masks, and predictions from K-Means, Spectral clustering, CenterNet, and our DETR-based segmentor. K-Means and Spectral produce fragmented, noisy masks in complex scenes ( rows 3–4 ), while CenterNet sometimes over-predicts the instance count under occlusion. DETR consistently recovers the correct number of instances with coherent boundaries across all difficulty levels.

$K$	$\mathrm{Acc}_{\mathrm{race}}$	$\mathrm{Acc}_{\mathrm{gender}}$	$K_{\mathrm{detected}}$ (mean)	$H_{ix}$
2	0.733	0.748	2.050	0.662
3	0.683	0.766	3.075	0.861

Appendix

Table S2: Steering accuracy by person count at the selected operating point (

n{=}80

/K).

$K$	Mask source	$\mathrm{Acc}_{\mathrm{race}}$	$\mathrm{Acc}_{\mathrm{gender}}$	$H_{ix}$	$\mathrm{LPIPS}_{\mathrm{bg}}$
2	DETR	0.843	0.896	0.792	0.743
2	Oracle	0.812	0.849	0.799	0.750
3	DETR	0.722	0.806	0.835	0.738
3	Oracle	0.705	0.776	0.902	0.740
4	DETR	0.643	0.720	0.834	0.756
4	Oracle	0.701	0.764	0.874	0.753

Appendix

Table S3: DETR vs. oracle (SAM2) masks , matched pairs,

n{=}445

overall (149/148/148 at

K{=}2/3/4

	Race		Gender
Classifier	Accuracy	Abstain	Accuracy	Abstain
FairFace	95.7%	2.5%	99.7%	0.0%
CLIP	95.3%	0.0%	99.9%	0.0%
DeepFace	87.7%	7.5%	92.7%	0.0%

Appendix

Table S4: Classifier validation against known-correct labels (

n{=}1{,}832

face crops, both FLUX.1-Dev and SD3.5-Large).

Figure 6: SD3.5-Large DETR segmentor, timestep sweep (val split,

n{=}288

). mIoU, AP50, and AP50-95 across SD3.5’s 28-step denoising schedule, mirroring Figure 3 ’s FLUX sweep. Performance peaks at timestep index 17 and plateaus around it, the same qualitative shape as FLUX’s own sweep.

	$\mathrm{Acc}_{\mathrm{race}}$ $\uparrow$	$\mathrm{Acc}_{\mathrm{gender}}$ $\uparrow$	$\mathrm{Aes}\Delta$ $\uparrow$	$\mathrm{LPIPS}_{\mathrm{bg}}$ $\uparrow$	$\mathrm{CLIP}\Delta$ $(\rightarrow 0)$
(a) EA strength $\alpha$ [ $\lambda{=}4.0$ fixed]
1.5	0.573	0.722	+0.009	0.916	$-$ 0.026
2.0	0.708	0.757	+0.027	0.900	$-$ 0.033
2.5	0.723	0.740	+0.010	0.890	$-$ 0.042
3.0	0.636	0.738	$-$ 0.134	0.876	$-$ 0.052
3.5	0.438	0.566	$-$ 0.762	0.842	$-$ 0.104

Appendix

Table S5: Steering hyperparameter ablation. Bold = selected configuration.

\mathrm{Aes}\Delta

: aesthetic score delta vs. unsteered baseline (higher

=

better quality).

\mathrm{LPIPS}_{\mathrm{bg}}

: background preservation.

\mathrm{CLIP}\Delta

: prompt–image CLIP similarity delta (closer to 0

=

less semantic drift).

$Start$	$End$	$\mathrm{Acc}_{\mathrm{race}}$ $\uparrow$	$\mathrm{Acc}_{\mathrm{gender}}$ $\uparrow$	$\mathrm{Aes}\Delta$ $\uparrow$	$\mathrm{LPIPS}_{\mathrm{bg}}$ $\uparrow$	$\mathrm{CLIP}\Delta$ $\rightarrow 0$
2	8	0.713	0.784	$-$ 0.097	0.760	$-$ 0.097
	10	0.718	0.783	$-$ 0.123	0.759	$-$ 0.102
	12	0.749	0.791	$-$ 0.179	0.758	$-$ 0.105
	14	0.754	0.783	$-$ 0.237	0.757	$-$ 0.106
4	8	0.632	0.744	+0.008	0.904	$-$ 0.029
	10	0.708	0.757	+0.027	0.900	$-$ 0.033

Appendix

Table S6: Steering window placement ablation (FLUX.1-Dev,

\alpha{=}2.0

\lambda{=}4.0

). Bold = selected operating point. CLIP

\Delta

: prompt–image similarity delta vs. baseline.

$\alpha$	$\lambda$	$\mathrm{Acc}_{\mathrm{race}}$ $\uparrow$	$\mathrm{Acc}_{\mathrm{gender}}$ $\uparrow$	$\mathrm{LPIPS}_{\mathrm{bg}}$ $\uparrow$
1.0	1.0	0.443	0.620	0.958
1.0	2.0	0.526	0.646	0.938
1.0	4.0	0.707	0.693	0.904
1.5	1.0	0.561	0.653	0.946
1.5	2.0	0.694	0.677	0.912
1.5	4.0	0.809	0.722	0.877

Appendix

Table S7: SD3.5-Large steering-strength ablation.

\mathrm{LPIPS}_{\mathrm{bg}}

: background preservation.

$Start$	$End$	$\mathrm{Acc}_{\mathrm{race}}$	$\mathrm{Acc}_{\mathrm{gender}}$	$\mathrm{LPIPS}_{\mathrm{bg}}$
3	10	0.391	0.624	0.921
	12	0.402	0.624	0.920
	14	0.427	0.642	0.919
4	11	0.378	0.623	0.945
	13	0.397	0.617	0.944
	16	0.424	0.617	0.944

Appendix

Table S8: SD3.5-Large steering-window ablation ,

\alpha{=}2.0

\lambda{=}4.0

	Baseline			SGS
Occupation	$H_{ix}$	$DR_{r}$	$DR_{g}$	$H_{ix}$	$DR_{r}$	$DR_{g}$
Driver	0.480	0.544	1.000	0.578	0.324	0.680
Mover	0.406	0.486	0.920	0.717	0.297	0.627
Developer	0.482	0.794	0.987	0.843	0.279	0.787
Farmer	0.426	0.759	0.987	0.761	0.324	0.587
Guard	0.503	0.333	0.813	0.661	0.270	0.720

Appendix

Table S9: Per-occupation demographic metrics, baseline vs. SGS (FLUX.1-Dev, extended benchmark,

n{=}150

/occupation).

H_{ix}

: intra-image intersectional entropy (higher = more diverse).

DR_{\mathrm{race}}

DR_{\mathrm{gender}}

: deviation ratio from uniform (lower = more balanced). Left block: male-skewed occupations. Right block: female-skewed occupations.

Figure 7: Extended qualitative comparison across four occupations and three person counts $k\in\{2,3,4\}$ . Left : baseline FLUX outputs. Right : SGS outputs. Rows show female-dominant (Nurses, Housekeepers) and male-dominant (Sheriffs, Construction Workers) professions. Baseline outputs are demographically homogeneous across all $k$ values and both stereotype directions. SGS consistently produces intersectional demographic diversity — each person within the same scene is steered to a distinct demographic target — while also preserving occupational context (uniform, apron, badge, hard hat) and scene coherence across all conditions. Performance is consistent across $k\in\{2,3,4\}$ , demonstrating that the segmentor reliably separates and independently steers two, three, and four persons within the same image.

Figure 8: Qualitative comparison on SD3.5-Large , mirroring Figure 2 . Base, EA, FairImagen, RespoDiff, and SDID remain homogeneous or only mildly shifted across all four prompts; SGS independently diversifies each detected person while preserving occupational context.

Figure 9: The human evaluation interface. Raters see a captioned imagepair with method identity and position blinded, and answer three questions per pair: perceived diversity, prompt faithfulness, and overall preference.

Demographic Bias in T2I Models

Demographic bias in text-to-image models is the unequal or stereotyped representation of people across generated images.

See all 31 papers

EquiSteer: Cross-Attention Steering Towards a Fairer Text-Guided Image Generation
Jul 1, 2026Tatiana Gaintseva, Akshit Achara, Gregory Slabaugh +2Cross-Attention T2I Generation
Queen Mary University of London, UK · Huawei Noah’s Ark, UK · King’s College London, UK +1
Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization
Jul 1, 2026Dain Kim, Jinseo Kim, Sungyong BaikDiffusion Model Guidance Text Embeddings
Dept. of Artificial Intelligence, Hanyang University, South Korea · Dept. of Data Science, Hanyang University, South Korea
FairFlow: Demystifying and Mitigating Stereotype Bias in Text-to-Image Diffusion Transformers
Jul 3, 2026Chen Chen, Yuanmin Huang, Zhenfei Zhang +5Diffusion Model Alignment Diffusion Transformer
Fudan University Shanghai, China · East China University of Science and Technology Shanghai, China

Jul 1, 2026J/K move · Enter open · S save

EquiSteer: Cross-Attention Steering Towards a Fairer Text-Guided Image Generation

Tatiana Gaintseva, Akshit Achara, Gregory Slabaugh, Jiankang Deng+1

Queen Mary University of London, UK · Huawei Noah’s Ark, UK · King’s College London, UK+1

Cross-Attention T2I Generation Algorithmic Bias Demographic Bias in T2I Models T2I Diffusion Models

Open paper AskPDF arXiv

Text-to-image diffusion models power everyday creative tasks, but they still reproduce the demographic biases in their training data. On common prompts such as a photo of a nurse,'' a photo of a CEO'', they skew their outputs toward one gender, driven by the statistics of training data rather than anything in the text. Existing debiasing methods show promise in narrow settings but require retraining, batch-level control, or prompt-specific tuning, limiting their scalability. We propose EquiSteer, a training-free method that works per sample by steering cross-attention (CA) activations at inference time. For each target attribute, EquiSteer precomputes steering vectors from contrastive prompts. Then at generation time, a prompt-aware gate leaves attribute-specific prompts untouched, while for neutral ones it clears existing attribute signals from the CA activations and injects a target attribute. Across SD-1.5, SD-2.1, SDXL, and SANA, EquiSteer reduces the average parity gap by up to

87\%

, with minimal effect on image quality and text-image alignment. Code is available at https://github.com/Atmyre/EquiSteer.%

Abstract

Demographic Bias in T2I Models

Explore similar work

EquiSteer: Cross-Attention Steering Towards a Fairer Text-Guided Image Generation

Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization

FairFlow: Demystifying and Mitigating Stereotype Bias in Text-to-Image Diffusion Transformers