Exploring Attention Sinks in Text-to-Image Transformers from Diffusion and Autoregressive Models

Binglei Li; Mengping Yang; Zhiyu Tan; Wenxu Qian; Junping Zhang; Hao Li

cs.CVOct 9, 2026

Exploring Attention Sinks in Text-to-Image Transformers from Diffusion and Autoregressive Models

Authors: Binglei Li, Mengping Yang, Zhiyu Tan, Wenxu Qian, Junping Zhang, Hao Li

Organizations: Fudan University · Shanghai Innovation Institute · Shanghai Academy of AI for Science

Abstract

Attention sinks are tokens or positions that consistently receive disproportionate attention mass, often beyond what their semantic content would suggest. Despite extensive study in large language models, their behavior and functional relevance in text-to-image (T2I) Transformers remain largely unexplored. To fill this gap and better understand how text-to-image Transformers allocate and route attention, we explore this phenomenon of text-to-image transformers across bidirectional diffusion models and causal autoregressive models. Specifically, we explore whether attention sinks exist, where they emerge, and how they functionally influence generation. Through systematic attention analysis and targeted interventions, we identify that attention sinks emerge in both bidirectional diffusion and autoregressive models but exhibit distinct patterns. The autoregressive model concentrates attention on the first and global-scale tokens, whereas the bidirectional models develop sinks near modality boundaries, within text regions, and at spatial corners. Moreover, these patterns vary across generation stages and are localized to a sparse subset of layers and heads. To assess their functional relevance, we perturb sink-related heads and token positions. It turns out that perturbing sink-ranked heads produces larger average changes than non-sink comparisons, while some sink-position perturbations have limited effects on bidirectional model scores. We further introduce register tokens to redistribute attention sinks and find that several configurations preserve or modestly improve generation performance. Overall, attention sinks are architecture-, inference-stage-, and component-dependent phenomena, providing a basis for mechanistic analysis and principled attention design in text-to-image transformers.

Figures & tables

Model	Candidate $c$	$\rho_{1}(c)$	$\rho_{5}(c)$
FLUX.1	sequence pos 0	0.32%	0.56%
	text pos 7	12.95%	32.26%
	boundary pos 511	6.05%	23.06%
	image pos 4544	3.04%	18.33%
Z-Image	sequence pos 0	0.47%	0.65%
	boundary pos 4096	10.89%	21.31%

Table 1: Sink ratios of predefined candidate positions. Infinity exhibits pronounced concentration on the first token, while the diffusion models concentrate attention near modality boundaries, in text-token regions, and at spatial corners.

Figure 2: Attention mass by token position across models. We highlight the most prominent sink positions for each model. Panels show FLUX.1, Z-Image, and Infinity-2B, while vertical markers identify modality or scale boundaries and insets magnify local peaks. Diffusion models peak near modality boundaries or image corners, whereas Infinity exhibits a pronounced position-0 sink.

Figure 3: Layer–head maps of sink concentration $\kappa^{(l,h)}$ across models. Each cell represents one attention head, and higher values indicate stronger concentration on a single key token. Sink-dominant heads are sparse and occur at architecture-specific depths.

Image-Directed Ratio
Configurations		FLUX.1	Z-Image	Infinity-2B
Early Stage	$T\!\rightarrow\!0.7T$	41.21%	7.46%	81.24%
Middle Stage	$0.7T\!\rightarrow\!0.3T$	45.34%	9.70%	86.61%
Late Stage	$0.3T\!\rightarrow\!0$	50.47%	12.73%	89.95%
Sink Position Stability
Adjacent Step	Top-1	79.12%	76.03%	52.56%

Table 2: Attention sink dynamics over generation. Image-directed ratio is the

R_{img}

for FLUX.1 and Z-Image, and the

R_{fine}

for Infinity-2B. Sink Position Stability is the Jaccard overlap at adjacent steps or across each full stage. Larger values indicate greater image/fine-scale concentration or greater positional persistence.

Figure 4: Trajectories of

\rho_{1}^{(t)}(c)

for text/image positions during generation. Candidate positions

c

are from Fig. 2 .

Model	Sink Heads	$\mathrm{H}_{norm}$		$\mathrm{H}_{norm}\geq 0.5$
	Pearson r	$\mathcal{D}_{\text{Geneval}}$	$\mathcal{D}_{\text{DPG}}$	$\mathcal{D}_{\text{Geneval}}$	$\mathcal{D}_{\text{DPG}}$
FLUX.1	0.988	0.37	0.61	6.8%	92.2%
Z-Image	0.983	0.31	0.43	2.7%	49.4%
Infinity-2B	0.999	0.26	0.26	2.3%	2.4%

Table 3: Cross-dataset stability of sink heads and sink positions. Pearson

r

compares layer–head concentration maps between

\mathcal{D}_{\text{GenEval}}

and

\mathcal{D}_{\text{DPG}}

H_{\mathrm{norm}}

is the normalized entropy of top-1 sink positions across prompts, and

H_{\mathrm{norm}}\geq 0.5

gives the fraction of high-entropy heads. Higher entropy indicates greater positional variation.

Settings \ Models	FLUX.1					Z-Image					Infinity-2B
Metrics	GenEval	DPG	HPSv2	ImgRwd	$\bar{\Delta}$ (%)	GenEval	DPG	HPSv2	ImgRwd	$\bar{\Delta}$ (%)	GenEval	DPG	HPSv2	ImgRwd	$\bar{\Delta}$ (%)
Baseline	67.14	83.67	29.62	0.9539	-	76.18	87.41	29.28	0.9481	-	64.56	81.84	29.68	0.9761	-
Logit.	66.99	83.86	29.58	0.9478	-0.19	76.55	87.27	28.92	0.9429	-0.36	62.57	78.82	29.40	0.9424	-2.79
Value.	66.51	83.84	29.60	0.9343	-0.71	75.98	87.15	28.93	0.9427	-0.58	61.35	76.41	28.68	0.9096	-5.45
Head.	63.69	82.29	29.47	0.8636	-4.19	72.94	87.06	28.98	0.9125	-2.36	60.84	75.72	28.37	0.8923	-6.56

Table 4: Effects of three interventions on the top-50 sink heads across models. Logit penalizes attention to selected sink positions, Value zeros their value vectors, and Head removes the selected sink-dominant heads. GenEval and DPG are averages over four independent runs.

\bar{\Delta}

is the mean

\%

change from baseline over

4

metrics, where more negative values denote greater degradation.

Figure 5: Curves of

\bar{\Delta}

vs. number of perturbed sink tokens for logit and value intervention experiments. FLUX.1 and Z‑Image use left axis, Infinity‑2B right axis.

Settings	Geneval	DPG	HPSv2	ImgRwd	$\bar{\Delta}(\%)$
FLUX.1	67.14	83.67	29.62	0.9539	-
$[T,0]$	64.81	84.19	29.46	0.9638	-0.59
$[T,T//2]$	65.45	83.74	29.59	0.9631	-0.39
$[T//2,0]$	64.95	83.77	29.48	0.9617	-0.70
Infinity-2B	64.56	81.84	29.68	0.9761	-
$[s1,s12]$	11.37	20.31	22.56	-0.8055	-91.02

Table 5: Effects of sink-token removal over selected generation intervals. For FLUX.1, intervals denote denoising timestep ranges (e.g.,

[T,0]

= full trajectory). For Infinity-2B, intervals denote autoregressive scale ranges (e.g.,

[si,s12]

= remove the position-0 global token from scale

i

onwards).

\bar{\Delta}

is the mean percentage change across the reported metrics relative to baseline.

Settings	Geneval	DPG	HPSv2	ImgRwd	$\bar{\Delta}(\%)$
Baseline	76.18	87.41	29.28	0.9481	-
Top-20	75.42	87.21	29.24	0.9187	-1.12
Top-50	72.94	87.06	28.98	0.9125	-2.36
Top-100	71.44	86.87	28.69	0.8104	-5.84
Random-50 ∗	74.02	86.91	29.13	0.9156	-1.84
Bottom-50	74.32	87.02	29.07	0.9214	-1.60

Table 6: Ablation results of pruning selected attention heads on Z-Image. Top-

K

/Bottom-

50

removes the

K

heads with highest/lowest sink concentration, while Random-

50^{*}

randomly selects

50

heads for three runs.

\bar{\Delta}

is the mean percentage change across metrics relative to baseline.

Settings	GenEval	DPG	$\rho_{1}(7)$	$\rho_{1}(4544)$	$\rho_{1}(c_{\mathrm{reg}})$
FLUX.1	67.14	83.67	12.95%	3.04%	–
+ register token	68.85	84.01	12.44%	2.72%	0.53%

Table 7: Register-token probe on FLUX.1 generation quality and position-wise top-1 sink ratios. Adding the register improves GenEval and DPG while slightly reducing the dominance of native sink positions. The low but nonzero

\rho_{1}(c_{\mathrm{reg}})

supports partial redistribution of sink dominance rather than wholesale replacement of the original structural sinks.

Attention Sinks

Attention sinks are tokens that attract disproportionate Transformer attention, shaping information flow, long-context behavior, and model efficiency.

See all 55 papers

May 10, 2026

Attention Sinks in Diffusion Transformers: A Causal Analysis

Attention sinks -- tokens that receive disproportionate attention mass -- are assumed to be functionally important in autoregressive language models, but their role in diffusion transformers remains unclear. We present a causal analysis in text-to-image diffusion, dynamically identifying dominant attention recipients per timestep and suppressing them via paired, training-free interventions on the score and value paths. Across 553 GenEval prompts on Stable Diffusion~3 (with SDXL corroboration), removing these sinks does not degrade text-image alignment (CLIP-T) or preference proxies (ImageReward, HPS-v2) at

k{=}1

; only under stronger interventions (

k\!\geq\!10

) does HPS-v2 exhibit a metric-dependent boundary, while CLIP-T remains robust throughout. The perceptual shifts induced by suppression are nonetheless sink-specific --

\sim\!6\times

larger than equal-budget random masking -- revealing an empirical dissociation between trajectory-level perturbation and semantic alignment in diffusion transformers. \footnote{Code available at https://github.com/wfz666/ICML26-attention-sink.}

Fangzheng Wu, Brian Summa

Department of Computer Science, Tulane University, New Orleans, LA, USA.

Transformer Interpretability Diffusion Transformer Self-Attention Attention Sinks T2I Diffusion Models Diffusion Models

Ask PDF

Jun 6, 2026

A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions

When attention concentrates on a single token, a sink, what is the model actually computing? Attention sinks are ubiquitous in softmax transformers, yet this shared visual signature can hide fundamentally different algorithms. We show that visually similar sink patterns can reflect two distinct mechanisms: (i) adaptive nop, where a head suppresses its update by routing to a null token, and (ii) broadcast, where a sink aggregates and redistributes global information. Each mechanism leaves distinct traces (nop-sinks exhibit negligible value norms; broadcast sinks induce low-rank outputs), which we formalize on synthetic tasks and use to derive practical diagnostics. Applied to pretrained vision transformers, these diagnostics reveal that both mechanisms exist at scale: sinks transition from CLS in early layers to patches in deeper layers and concentrate in specialized heads. Causal interventions further connect these signatures to near-null suppression and shared residual contributions. We then use architectural interventions to show how these computations can be reorganized: gating eliminates detected nop-like sinks but increases broadcast-like sinks, registers relocate rather than remove sink computation, and our position-free global pathway provides an explicit route for shared communication that reduces the broadcast-like sinks induced by gating. On dense probes, combining gating with the global pathway gives the strongest results among the tested variants, despite retaining some broadcast-like sinks. Overall, we find that the same attention pattern can reflect two very different computations, and that effective intervention depends not only on identifying the computation, but also on providing architectural alternatives through which the model can reorganize it.

Lukas Fesser*, Mozes Jacobs*, Thomas Fel* +2

Kempner Institute Harvard University Cambridge, MA

Transformer Interpretability Vision Transformer Attention Sinks Multi-Head Attention Attention Head Specialization

Ask PDF

Apr 16, 2026

A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation

Transformers commonly exhibit an attention sink: disproportionately high attention to the first position. We study this behavior in GPT-2-style models with learned query biases and absolute positional embeddings. Combining structural analysis with causal interventions, validated across natural-language, mathematical, and code inputs, we find that the sink arises from the interaction among (i) a learned query bias, (ii) the first-layer MLP transformation of the positional encoding, and (iii) structure in the key projection. Crucially, each component we identify is individually dispensable: architectures omitting each of them robustly exhibit sinks. This indicates that attention sinks may arise through distinct circuits across architectures. These findings inform mitigation of sinks, and motivate broader investigation into why sinks emerge.

Yuval Ran-Milo, Hila Ofek, Shahar Mendel

Tel Aviv University

Self-Attention Decoder-Only Language Models Mechanistic Interpretability Attention Sinks Attention Mechanisms Circuits in Language Models

Ask PDF

Abstract

Attention Sinks

Explore similar work

Attention Sinks in Diffusion Transformers: A Causal Analysis

A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions

A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation