Metonymic Circuits for Abstract Concept Grounding in Vision Transformers

Jing Ding; Ziqiao Ma; Jiayuan Mao; Joyce Chai; Freda Shi

cs.AIOct 3, 2026

Metonymic Circuits for Abstract Concept Grounding in Vision Transformers

Authors: Jing Ding, Ziqiao Ma, Jiayuan Mao, Joyce Chai, Freda Shi

Organizations: University of Michigan · Vector Institute · University of Pennsylvania · Amazon FAR ·

Abstract

We study how Vision Transformers ground abstract concepts (e.g., angry) when training data provide limited direct referential evidence. We hypothesize a metonymic grounding mechanism in which abstract predictions are driven by concrete, interpretable anchor concepts (e.g., fire) that bridge visual signals to abstract semantics. By applying Transcoders on CLIP and DINO vision encoders, we recover intermediate features that can be associated with semantic labels for more concrete concepts, and trace their contributions in circuits underlying abstract concept recognition. Experiments on a carefully curated icon dataset reveal structured metonymic circuits, in which perceptual primitives dominate early layers and object-like anchors precede abstract targets. Images containing rendered text instead recruit a distinct perceptual-to-textual route. Causal interventions further validate that metonymic intermediates are functionally involved in grounding abstract concepts.

Figures & tables

Figure 1: Word-frequency density in the OpenCLIP ( Ilharco et al., 2021 ) training captions ( Gadre et al., 2023 , DataComp-1B;) , stratified by lexical concreteness. Each word is assigned a concreteness score from 1 (abstract) to 5 (concrete) using the norms of Brysbaert et al. (2014) ; words contribute to the density weighted by their caption document frequency.

Figure 2: Left : the metonymic grounding hypothesis: abstract concepts (e.g., “angry”) are grounded via intermediate, more concrete visual concepts. Right : traced ViT circuits showing perceptual and object-level features mediating the abstract prediction across layers. Color coding is shown in the examples on the right.

Setting	Mean Spearman $r$	Mean $z$	Unadj. $p<0.05$	FDR $q<0.05$	Consistent Sig.	Consistent Sig. (FDR)
100 concepts $\times$ 10 images	0.066	0.57	14.1%	0.0%	22	0
50 concepts $\times$ 20 images	0.073	0.68	13.6%	5.2%	7	3
33 concepts $\times$ 30 images	0.075	0.71	27.3%	3.0%	9	1

Table 1: Correlation between circuit concept importance and training caption co-occurrence. We report correlations between concrete concept mass

M_{y}(w)

in a circuit for the abstract concept

y

and concrete–abstract concept association measured by NPMI on DataComp-1B captions, under different image-per-class settings. Unadj. : unadjusted

p

-value from a permutation null; FDR : Benjamini–Hochberg false-discovery-rate–corrected

q

-value; Mean

z

: observed Spearman

r

standardized against the permutation null; Consistent Sig. : number of distinct concepts significant in at least half of 10 resampling seeds. While some abstract concepts exhibit unadjusted significance, correlations are weak on average and rarely survive BH-FDR correction, suggesting limited alignment between circuit causation and caption co-occurrence.

Figure 3: Node-level circuit statistics across layers. Left: Layer-wise MassFlow, where MF sum denotes the sum of MassFlow over all circuit nodes in each layer. Right: Grounding-type composition within each layer. Grounding types are color-coded as Perceptual, Object, Textual, and Abstract.

Figure 4: Examples of grounding path clusters grouped by grounding-type transition patterns. For each pattern, we show an example cluster profile, an example icon, and an example metonymic path (sequence of decoded node concepts) leading to the target logit. Pattern names such as P-dominant, O-dominant, and P

\rightarrow

T are post-hoc descriptive summaries rather than cluster labels. Percentages indicate the displayed cluster’s share of total MassFlow within its subset. Clustering uses all paths while displayed examples are selected only for visualization.

Figure 5: Examples of grounding paths grouped by node–abstract-concept alignment patterns. For each cluster, we show (1) the characteristic alignment pattern together with the corresponding grounding types for the same paths; (2) an example icon image from the cluster; and (3) an example metonymic path illustrating the sequence of nodes leading to the target logit. Grounding types are color-coded as Perceptual, Object, Abstract, and Textual.

Figure 6: Path interventions across 4 Gaussian noise scales.

\Delta

NLL is aggregated over pooled image–seed observations at each intervention budget and noise scale. Error bars denote the corresponding standard error (

\mathrm{SE}\leq 0.03

\Delta

NLL).

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration item	Value
MLP input/output dimension ( $d_{\text{mlp}}$ )	768
Expansion factor	64
Hidden dimension ( $d_{\text{TC}}$ )	49,152
Activation	Top- $k$ , $k=1024$
Dataset	ImageNet-1K
Epochs	1

Appendix

Table 2: Transcoder training configuration. All settings are shared across CLIP and DINO unless otherwise specified. Learning rate was selected from the range 0.002–0.012.

Panel of Figure 7 — Figure 7: Global transcoder activation profile. Together, these plots establish that transcoder representations are highly sparse: the vast majority of features are inactive, and the few that activate follow a heavy-tailed distribution. This motivates the use of top- $K$ selection in subsequent analyses.

Subset	$N$	Mean	Med.	Std	Range
Visual Icon	1,000	148.8	126	94.9	16–603
Mix	1,000	176.1	148	106.3	28–643
Visual Text	1,000	161.7	138	98.3	19–625
All	3,000	162.2	138	100.5	16–643

Appendix

Table 3: Circuit nodes per image, over all 3,000 analyzed images.

Subset	$N$	Mean	Med.	Std	Range
Visual Icon	784	81.9	84	43.0	1–265
Mix	741	89.6	89	39.1	1–245
Visual Text	783	87.5	91	41.2	1–234
All	2,308	86.3	88	41.3	1–265

Appendix

Table 4: Clustered paths per image, restricted to the 2,308 images (of 3,000) that retain at least one path under the

\text{keep\_mass}=0.9

threshold, yielding 199,140 paths in total. Path length has mean 3.86 and median 4, ranging from 1 to 10.

Subset	Total
Visual Icon	261,949
Mix	3,926
Visual Text	3,735
All	269,610

Appendix

Table 5: Image counts per subset after concept selection and category partitioning.

Panel of Figure 8 — Figure 8: Grounding-type transition patterns for all CLIP subsets. We show the six largest clusters for each subset. Each panel corresponds to one cluster and shows its normalized transition profile across positions along the path, with the cluster identifier and its MassFlow proportion indicated in the title.

Panel of Figure 11 — Figure 11: Node-level circuit statistics across layers for the two DINO target constructions. Proto uses the class mean prototype as the target representation, while Proj uses the summed feature-projection target constructed from class-relevant Transcoder features.

Panel of Figure 12 — Figure 12: Full grounding-type transition patterns for all DINO subsets using the Proto target construction. We show the six largest clusters for each subset. Each panel corresponds to one cluster and shows its normalized transition profile across positions along the path, with the cluster identifier and its MassFlow proportion indicated in the title.

Panel of Figure 13 — Figure 13: Full grounding-type transition patterns for all DINO subsets using the Proj target construction. We show the six largest clusters for each subset. Each panel corresponds to one cluster and shows its normalized transition profile across positions along the path, with the cluster identifier and its MassFlow proportion indicated in the title.

Panel of Figure 14 — Figure 14: Full node–abstract-concept alignment results for DINO using the Proto target construction. For each subset, grounding paths are clustered into $K=5$ groups based on the shape of their normalized calibrated BERTScore alignment trajectories. Each panel shows the cluster-level mean alignment curve (top) and the corresponding grounding-type composition along the same paths (bottom). Rows correspond to the Visual Icon , Mix , and Visual Text subsets, and each panel title reports the cluster index together with its MassFlow proportion. Grounding types are color-coded as Perceptual, Object, Abstract, and Textual.

Panel of Figure 15 — Figure 15: Full node–abstract-concept alignment results for DINO using the Proj target construction. For each subset, grounding paths are clustered into $K=5$ groups based on the shape of their normalized calibrated BERTScore alignment trajectories. Each panel shows the cluster-level mean alignment curve (top) and the corresponding grounding-type composition along the same paths (bottom). Rows correspond to the Visual Icon , Mix , and Visual Text subsets, and each panel title reports the cluster index together with its MassFlow proportion. Grounding types are color-coded as Perceptual, Object, Abstract, and Textual.

Panel of Figure 16 — Figure 16: Recovered-circuit intervention versus a feature-count-matched random control. For each image and intervention threshold, the random condition perturbs the same number of feature nodes as the recovered-circuit condition, but samples them from features outside the recovered circuit under the identical Gaussian-noise protocol, with $\sigma\in\{1,2,3,4\}$ . In (b), the horizontal axis indexes the matched intervention by the MassFlow budget $\tau$ of the corresponding recovered-circuit condition. The perturbed features themselves lie outside the circuit and carry no MassFlow. Each image is evaluated under five noise seeds; means and standard errors are computed over the resulting pooled image–seed observations. Results give equal weight to Visual Icon, Mix, and Visual Text ( $N=300$ images per subset).

Panel of Figure 17 — Figure 17: Circuit recovery experiment. Top: retaining the real circuit plus $\sim$ 3–4% of top-activated non-circuit features nearly restores the full logit ratio, whereas a size-matched random circuit remains unrecoverable. Bottom: the full $2\times 2$ comparison shows that randomly restoring outside features also requires substantially more features. Recovery rises from 0.5038 at 3% retention to 0.9848 at 4%, while all controls remain at 0.2230–0.2412 at 4%. Results are aggregated over five seeds under Gaussian-noise ablation at $\sigma=4$ ; lines connect empirical measurements and no curve smoothing is applied.

Participant-facing response	Analysis category
Directly names it, or a type of it	Direct/literal
A specific cue within the same thing, event, or situation	Same-frame metonymic
Represents it by convention or comparison	Convention/comparison
Related, but no specific cue to the label	Related only
Cannot determine	Cannot determine

Appendix

Table 6: Mapping between participant-facing relation descriptions and the five analysis categories used in Section 5.4 . Technical linguistic labels were not shown to annotators.

VLM Interpretability

Vision-language model interpretability explains how image and language representations shape predictions, evidence use, and multimodal reasoning.

See all 256 papers

"World Knowledge" in the Weights: Reading Concept Circuits of Vision Transformers
Sep 8, 2026Yanlin Chen, Tang Li, Xi PengTransformer Interpretability Vision Foundation Models
Department of Computer & Information Sciences, University of Delaware, USA · Department of Computer Science, University of Virginia, USA
TRACE: Transparent Retrieval for Abstract Concept Evaluation
Aug 11, 2026Joseph BinghamVision-Language Grounding VLM Evaluation
Dept. of Biology, Technion University
VisionLogic: Discovering and Grounding Decision-Relevant Visual Concepts
Mar 13, 2025Chuqin Geng, Yuhe Jiang, Li Zhang +5Explainable Artificial Intelligence Concept-Based Explanations
School of Computer Science, McGill University, Montreal, Canada · Department of Computer Science, University of Toronto, Toronto, Canada

Sep 8, 2026J/K move · Enter open · S save

"World Knowledge" in the Weights: Reading Concept Circuits of Vision Transformers

Yanlin Chen, Tang Li, Xi Peng

Department of Computer & Information Sciences, University of Delaware, USA · Department of Computer Science, University of Virginia, USA

Transformer Interpretability Vision Foundation Models Vision Transformer Mechanistic Interpretability Concept-Based Explanations Spurious Correlation Robustness

Open paper AskPDF arXiv

Vision transformers (ViTs) have achieved remarkable generalization across visual domains, yet little is known about how they internally represent the structure of the world. To address this gap, we use Cross-Layer Transcoders (CLTs) to read concept circuits from ViTs: directed graphs whose nodes correspond to sparse, interpretable concepts and edges capture concept interactions across layers. Our method yields two complementary views of model behavior. The global concept circuit is input-invariant and can be recovered directly from learned cross-layer weights, exposing the reusable "world knowledge" encoded in the model. The instance concept circuit is input-dependent and identifies the concepts and pathways actually used for a specific prediction, enabling faithful example-level explanations. We demonstrate the utility of concept circuits in three ways: (1) Automatic spurious correlation discovery: leveraging the statistics of our global concept circuits to identify shortcut dependencies within the model. (2) Spurious correlation removal: intervening on the instance concept circuit to steer the model towards correct predictions. Empirical results show that our method outperforms existing counterparts by 11.0% on the Waterbird dataset. (3) Model comparison: contrasting the global concept circuits of different foundation models (e.g., CLIP vs. DINO) to reveal how supervision paradigms shape representational structure. Our code is available at https://github.com/deep-real/VisionCLT

Abstract

VLM Interpretability

Explore similar work

"World Knowledge" in the Weights: Reading Concept Circuits of Vision Transformers

TRACE: Transparent Retrieval for Abstract Concept Evaluation

VisionLogic: Discovering and Grounding Decision-Relevant Visual Concepts