cs.CVApr 7, 2026

Selective Aggregation of Attention Maps Improves Diffusion-Based Visual Interpretation

Authors: Jungwon Park, Jungmin Ko, Dongnam Byun, Wonjong Rhee

Organizations: Department of Intelligence and Information, Seoul National University, Seoul, Republic of Korea · Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, Republic of Korea

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Numerous studies on text-to-image (T2I) generative models have utilized cross-attention maps to boost application performance and interpret model behavior. However, the distinct characteristics of attention maps from different attention heads remain relatively underexplored. In this study, we show that selectively aggregating cross-attention maps from heads most relevant to a target concept can improve visual interpretability. Compared to the diffusion-based segmentation method DAAM, our approach achieves higher mean IoU scores. We also find that the most relevant heads capture concept-specific features more accurately than the least relevant ones, and that selective aggregation helps diagnose prompt misinterpretations. These findings suggest that attention head selection offers a promising direction for improving the interpretability and controllability of T2I generation.

Figures & tables

 Mean IoU score
DAAM - 0.30.7490
DAAM - 0.40.7540
DAAM - 0.50.6261
Ours - 0.30.7698
Ours - 0.40.7765
Ours - 0.50.6785
Table 1: Quantitative comparison of diffusion-based object semantic segmentation. The values following “DAAM -” or “Ours -” (i.e., 0.3, 0.4, 0.5) indicate the thresholds applied to the aggregated attention maps when computing the mean Intersection over Union (IoU).
Figure 1: Qualitative comparison of diffusion-based object semantic segmentation. The original generated images are shown on the left, and the same images with aggregated attention maps overlaid are shown on the right. Red circles highlight regions with undesired focus or lack of focus.
 Most Relevant 30 HeadsLeast Relevant 30 Heads
Ours - 0.30.76980.6654
Ours - 0.40.77650.6172
Ours - 0.50.67850.4649
Table 2: Comparison of attention map aggregation using the most and least relevant attention heads. Aggregated attention maps are computed using either the 30 most or 30 least relevant attention heads for the concept ‘Animals.’ The values following “DAAM -” or “Ours -” (i.e., 0.3, 0.4, 0.5) indicate the thresholds applied to the aggregated attention maps when computing the mean Intersection over Union (IoU).
Figure 2: Visual comparison of attention map aggregation using the most and least relevant attention heads. The attention maps are computed using either the 30 most or 30 least relevant heads for the concept ‘Animals’ and are overlaid on the generated images to illustrate differences in focus regions.
Figure 3: Visual diagnosis of text-to-image misinterpretation with selectively aggregated attention maps. The T2I model mistakenly generates both meanings of the word ‘mouse’–as an animal and as an electronic device. ‘Ours-Animals’ shows the aggregated attention map from the 30 heads most relevant to the concept ‘Animals,’ while ‘Ours-Electronics’ shows the map from the 30 heads most relevant to the concept ‘Electronics’. Note that DAAM aggregates attention maps from all heads in the T2I model. All three visualizations show the aggregated attention maps of the word ‘mouse.’
 20 heads30 heads40 heads
Ours - 0.30.74360.76980.7669
Ours - 0.40.70010.77650.7412
Ours - 0.50.53860.67850.6315
Table 3: Ablation study on the number of selected attention heads. The values following “DAAM -” or “Ours -” (i.e., 0.3, 0.4, 0.5) indicate the thresholds applied to the aggregated attention maps when computing the mean Intersection over Union (IoU). Note that Stable Diffusion v1 contains 128 cross-attention heads.
More in this topic

T2I Diffusion Models

Text-to-image (T2I) diffusion models generate images from prompts by learning to reverse noise under text conditioning.

See all 402 papers

Explore similar work

CardsList

  1. Attention Dynamics in Diffusion Models: A Visual Analytics Framework for Human-AI Collaboration

    Jun 28, 2026Yiran Xiao, George LegradyCross-AttentionT2I Diffusion Models

    University of California, Santa Barbara

  2. DreamReader: An Interpretability Toolkit for Text-to-Image Models

    Mar 2, 2026Nirmalendu Prakash, Narmeen Oozeer, Michael Lan +6Diffusion ModelsT2I Diffusion Models

    Singapore University of Technology and Design · Martian · Thoughtworks +2

  3. AI-T2I: Aggregating-and-Isolating Cross-Attention to Diffusion Models for Text-to-Image Synthesis

    May 25, 2026Shipeng Cao, Biao Qian, Haipeng Liu +2Diffusion Model AlignmentCross-Attention

    Key Laboratory of Knowledge Engineering With Big Data, Ministry of Education, Hefei University of Technology, China · Department of Automation, Tsinghua University, China