Exploring Attention Sinks in Text-to-Image Transformers from Diffusion and Autoregressive Models

Binglei Li; Mengping Yang; Zhiyu Tan; Wenxu Qian; Junping Zhang; Hao Li

cs.CVOct 9, 2026

Exploring Attention Sinks in Text-to-Image Transformers from Diffusion and Autoregressive Models

Authors: Binglei Li, Mengping Yang, Zhiyu Tan, Wenxu Qian, Junping Zhang, Hao Li

Organizations: Fudan University · Shanghai Innovation Institute · Shanghai Academy of AI for Science

Abstract

Attention sinks are tokens or positions that consistently receive disproportionate attention mass, often beyond what their semantic content would suggest. Despite extensive study in large language models, their behavior and functional relevance in text-to-image (T2I) Transformers remain largely unexplored. To fill this gap and better understand how text-to-image Transformers allocate and route attention, we explore this phenomenon of text-to-image transformers across bidirectional diffusion models and causal autoregressive models. Specifically, we explore whether attention sinks exist, where they emerge, and how they functionally influence generation. Through systematic attention analysis and targeted interventions, we identify that attention sinks emerge in both bidirectional diffusion and autoregressive models but exhibit distinct patterns. The autoregressive model concentrates attention on the first and global-scale tokens, whereas the bidirectional models develop sinks near modality boundaries, within text regions, and at spatial corners. Moreover, these patterns vary across generation stages and are localized to a sparse subset of layers and heads. To assess their functional relevance, we perturb sink-related heads and token positions. It turns out that perturbing sink-ranked heads produces larger average changes than non-sink comparisons, while some sink-position perturbations have limited effects on bidirectional model scores. We further introduce register tokens to redistribute attention sinks and find that several configurations preserve or modestly improve generation performance. Overall, attention sinks are architecture-, inference-stage-, and component-dependent phenomena, providing a basis for mechanistic analysis and principled attention design in text-to-image transformers.

Figures & tables

Model	Candidate $c$	$\rho_{1}(c)$	$\rho_{5}(c)$
FLUX.1	sequence pos 0	0.32%	0.56%
	text pos 7	12.95%	32.26%
	boundary pos 511	6.05%	23.06%
	image pos 4544	3.04%	18.33%
Z-Image	sequence pos 0	0.47%	0.65%
	boundary pos 4096	10.89%	21.31%

Table 1: Sink ratios of predefined candidate positions. Infinity exhibits pronounced concentration on the first token, while the diffusion models concentrate attention near modality boundaries, in text-token regions, and at spatial corners.

Figure 2: Attention mass by token position across models. We highlight the most prominent sink positions for each model. Panels show FLUX.1, Z-Image, and Infinity-2B, while vertical markers identify modality or scale boundaries and insets magnify local peaks. Diffusion models peak near modality boundaries or image corners, whereas Infinity exhibits a pronounced position-0 sink.

Figure 3: Layer–head maps of sink concentration $\kappa^{(l,h)}$ across models. Each cell represents one attention head, and higher values indicate stronger concentration on a single key token. Sink-dominant heads are sparse and occur at architecture-specific depths.

Image-Directed Ratio
Configurations		FLUX.1	Z-Image	Infinity-2B
Early Stage	$T\!\rightarrow\!0.7T$	41.21%	7.46%	81.24%
Middle Stage	$0.7T\!\rightarrow\!0.3T$	45.34%	9.70%	86.61%
Late Stage	$0.3T\!\rightarrow\!0$	50.47%	12.73%	89.95%
Sink Position Stability
Adjacent Step	Top-1	79.12%	76.03%	52.56%

Table 2: Attention sink dynamics over generation. Image-directed ratio is the

R_{img}

for FLUX.1 and Z-Image, and the

R_{fine}

for Infinity-2B. Sink Position Stability is the Jaccard overlap at adjacent steps or across each full stage. Larger values indicate greater image/fine-scale concentration or greater positional persistence.

Figure 4: Trajectories of

\rho_{1}^{(t)}(c)

for text/image positions during generation. Candidate positions

c

are from Fig. 2 .

Model	Sink Heads	$\mathrm{H}_{norm}$		$\mathrm{H}_{norm}\geq 0.5$
	Pearson r	$\mathcal{D}_{\text{Geneval}}$	$\mathcal{D}_{\text{DPG}}$	$\mathcal{D}_{\text{Geneval}}$	$\mathcal{D}_{\text{DPG}}$
FLUX.1	0.988	0.37	0.61	6.8%	92.2%
Z-Image	0.983	0.31	0.43	2.7%	49.4%
Infinity-2B	0.999	0.26	0.26	2.3%	2.4%

Table 3: Cross-dataset stability of sink heads and sink positions. Pearson

r

compares layer–head concentration maps between

\mathcal{D}_{\text{GenEval}}

and

\mathcal{D}_{\text{DPG}}

H_{\mathrm{norm}}

is the normalized entropy of top-1 sink positions across prompts, and

H_{\mathrm{norm}}\geq 0.5

gives the fraction of high-entropy heads. Higher entropy indicates greater positional variation.

Settings \ Models	FLUX.1					Z-Image					Infinity-2B
Metrics	GenEval	DPG	HPSv2	ImgRwd	$\bar{\Delta}$ (%)	GenEval	DPG	HPSv2	ImgRwd	$\bar{\Delta}$ (%)	GenEval	DPG	HPSv2	ImgRwd	$\bar{\Delta}$ (%)
Baseline	67.14	83.67	29.62	0.9539	-	76.18	87.41	29.28	0.9481	-	64.56	81.84	29.68	0.9761	-
Logit.	66.99	83.86	29.58	0.9478	-0.19	76.55	87.27	28.92	0.9429	-0.36	62.57	78.82	29.40	0.9424	-2.79
Value.	66.51	83.84	29.60	0.9343	-0.71	75.98	87.15	28.93	0.9427	-0.58	61.35	76.41	28.68	0.9096	-5.45
Head.	63.69	82.29	29.47	0.8636	-4.19	72.94	87.06	28.98	0.9125	-2.36	60.84	75.72	28.37	0.8923	-6.56

Table 4: Effects of three interventions on the top-50 sink heads across models. Logit penalizes attention to selected sink positions, Value zeros their value vectors, and Head removes the selected sink-dominant heads. GenEval and DPG are averages over four independent runs.

\bar{\Delta}

is the mean

\%

change from baseline over

4

metrics, where more negative values denote greater degradation.

Figure 5: Curves of

\bar{\Delta}

vs. number of perturbed sink tokens for logit and value intervention experiments. FLUX.1 and Z‑Image use left axis, Infinity‑2B right axis.

Settings	Geneval	DPG	HPSv2	ImgRwd	$\bar{\Delta}(\%)$
FLUX.1	67.14	83.67	29.62	0.9539	-
$[T,0]$	64.81	84.19	29.46	0.9638	-0.59
$[T,T//2]$	65.45	83.74	29.59	0.9631	-0.39
$[T//2,0]$	64.95	83.77	29.48	0.9617	-0.70
Infinity-2B	64.56	81.84	29.68	0.9761	-
$[s1,s12]$	11.37	20.31	22.56	-0.8055	-91.02

Table 5: Effects of sink-token removal over selected generation intervals. For FLUX.1, intervals denote denoising timestep ranges (e.g.,

[T,0]

= full trajectory). For Infinity-2B, intervals denote autoregressive scale ranges (e.g.,

[si,s12]

= remove the position-0 global token from scale

i

onwards).

\bar{\Delta}

is the mean percentage change across the reported metrics relative to baseline.

Settings	Geneval	DPG	HPSv2	ImgRwd	$\bar{\Delta}(\%)$
Baseline	76.18	87.41	29.28	0.9481	-
Top-20	75.42	87.21	29.24	0.9187	-1.12
Top-50	72.94	87.06	28.98	0.9125	-2.36
Top-100	71.44	86.87	28.69	0.8104	-5.84
Random-50 ∗	74.02	86.91	29.13	0.9156	-1.84
Bottom-50	74.32	87.02	29.07	0.9214	-1.60

Table 6: Ablation results of pruning selected attention heads on Z-Image. Top-

K

/Bottom-

50

removes the

K

heads with highest/lowest sink concentration, while Random-

50^{*}

randomly selects

50

heads for three runs.

\bar{\Delta}

is the mean percentage change across metrics relative to baseline.

Settings	GenEval	DPG	$\rho_{1}(7)$	$\rho_{1}(4544)$	$\rho_{1}(c_{\mathrm{reg}})$
FLUX.1	67.14	83.67	12.95%	3.04%	–
+ register token	68.85	84.01	12.44%	2.72%	0.53%

Table 7: Register-token probe on FLUX.1 generation quality and position-wise top-1 sink ratios. Adding the register improves GenEval and DPG while slightly reducing the dominance of native sink positions. The low but nonzero

\rho_{1}(c_{\mathrm{reg}})

supports partial redistribution of sink dominance rather than wholesale replacement of the original structural sinks.

Attention Sinks

Attention sinks are tokens that attract disproportionate Transformer attention, shaping information flow, long-context behavior, and model efficiency.

See all 55 papers

Attention Sinks in Diffusion Transformers: A Causal Analysis
May 10, 2026Fangzheng Wu, Brian SummaTransformer Interpretability Diffusion Transformer
Department of Computer Science, Tulane University, New Orleans, LA, USA.
A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions
Jun 6, 2026Lukas Fesser*, Mozes Jacobs*, Thomas Fel* +2Transformer Interpretability Vision Transformer
Kempner Institute Harvard University Cambridge, MA
A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation
Apr 16, 2026Yuval Ran-Milo, Hila Ofek, Shahar MendelSelf-Attention Decoder-Only Language Models
Tel Aviv University

May 10, 2026J/K move · Enter open · S save

Attention Sinks in Diffusion Transformers: A Causal Analysis

Fangzheng Wu, Brian Summa

Department of Computer Science, Tulane University, New Orleans, LA, USA.

Transformer Interpretability Diffusion Transformer Self-Attention Attention Sinks T2I Diffusion Models Diffusion Models

Open paper AskPDF arXiv

Attention sinks -- tokens that receive disproportionate attention mass -- are assumed to be functionally important in autoregressive language models, but their role in diffusion transformers remains unclear. We present a causal analysis in text-to-image diffusion, dynamically identifying dominant attention recipients per timestep and suppressing them via paired, training-free interventions on the score and value paths. Across 553 GenEval prompts on Stable Diffusion~3 (with SDXL corroboration), removing these sinks does not degrade text-image alignment (CLIP-T) or preference proxies (ImageReward, HPS-v2) at

k{=}1

; only under stronger interventions (

k\!\geq\!10

) does HPS-v2 exhibit a metric-dependent boundary, while CLIP-T remains robust throughout. The perceptual shifts induced by suppression are nonetheless sink-specific --

\sim\!6\times

larger than equal-budget random masking -- revealing an empirical dissociation between trajectory-level perturbation and semantic alignment in diffusion transformers. \footnote{Code available at https://github.com/wfz666/ICML26-attention-sink.}

Abstract

Attention Sinks

Explore similar work

Attention Sinks in Diffusion Transformers: A Causal Analysis

A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions

A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation