Label-Free Localization and Test-Time Suppression of CLIP's Typographic Circuits

Bohan Liu; Wenqian Ye; Guangzhi Xiong; Zhenghao He; Sanchit Sinha; Aidong Zhang

cs.CVNov 1, 2026

Label-Free Localization and Test-Time Suppression of CLIP's Typographic Circuits

Authors: Bohan Liu, Wenqian Ye, Guangzhi Xiong, Zhenghao He, Sanchit Sinha, Aidong Zhang

Organizations: University of Virginia Charlottesville VA 22903, USA

Abstract

CLIP-based Vision-Language Models have become cornerstones of modern multimodal systems. The cross-modal contrastive objective equips the CLIP vision encoder to both see objects and read text. These two abilities induce an entanglement between the encoding of lexical and semantic shapes, creating a distinctive vulnerability to typographic attacks, in which rendered text overrides an image's semantic content with its literal meaning. The black-box nature of these encoders has prevented a mechanistic interpretation of such entanglement. We introduce a head-decomposed Sparse Dictionary Learning (SDL) architecture that, by design, decomposes individual attention heads into interpretable, head-specific latents. Based on this inductive disentanglement, we re-purpose Clarity Score and propose normalized Text Attribution Score, which are both label-free, to identify attention heads that carry typographic information. As a causal validation of successful circuit separation, we perform a mechanistic intervention that reweighs attention away from the patch tokens in the identified lexical heads at test-time. Experiments show that the intervention effectively recovers object classification under typographic attack and, across five CLIP variants and four typographic benchmarks, achieves up to 46.9% improvement while incurring up to 1% accuracy loss in general image perception, outperforming prior supervised and training-free methods. Expanding on the mechanistic intervention, we propose a test-time adaptive attention head routing to further reduce the impact on performance at matched robustness, pushing beyond the Pareto frontier of current robustness-perception tradeoff.

Figures & tables

Figure 1: Motivating example. A superimposed “Goose” text causes CLIP to label a cat as a goose.

Figure 2: Method Overview. (a) Head-decomposed sparse decomposition. We employ Sparse Dictionary Learning to decompose each attention module ( $\text{Attn}_{1}\ldots\text{Attn}_{4}$ ) into per-head sparse latent neurons. (b) Lexical-head localization and causal intervention. The decomposition surfaces a separation between heads encoding semantic shapes and heads encoding lexical shapes. Identifying the lexical heads (e.g., $\text{Attn}_{3}$ ) and reweighting their attention away from patch tokens onto the class token causally suppresses the lexical-processing circuit, recovering semantic perception under typographic attacks.

Method	IN-100 (Clean)	RTA-100	Disentangling	PAINT	IN-100-Text-D
No intervention	81.5	55.1	45.6	55.1	57.7
Defense-Prefix	81.4 (-0.1)	63.6	67.8	67.8	70.1
Dyslexify	81.3 (-0.3)	66.1	63.7	70.9	71.4
Dyslexify ∗	81.0 (-0.5)	68.5	70.7	72.2	–
Ours (CS)	80.5 (-1.0)	69.9	82.4	70.9	75.3
Ours (nTAS)	81.1 (-0.4)	70.8	75.8	71.6	76.7

Table 1: Comparison of intervention strategies, macro-averaged over the five ViT variants . Numbers are zero-shot Object Accuracy (%) averaged over ViT-B/16, L/14, H/14, g/14, and bigG/14; ImageNet-100 (Clean) shows the mean difference to the un-intervened model in parentheses. Per-model breakdown in the supplement. Defense-Prefix and Dyslexify rows from their released evaluation artifacts under matched benchmarks. Bold and italic mark first and second place per column. ∗ indicates numbers reported by the original authors with larger data budget; “–” marks benchmarks absent from the original report.

Figure 3: Robustness–accuracy trade-off. Mean attacked accuracy (RTA-100, Disentangling, PAINT) vs. clean accuracy; gray staircase: Pareto frontier over matched-protocol points. Seven of its eight members are ours.

Method	IN-100 (Clean)	RTA-100	Disentangling	PAINT	IN-100-Text-D
CS	80.5 (-1.0)	69.9	82.4	70.9	75.3
nTAS	81.1 (-0.4)	70.8	75.8	71.6	76.7
CS+Adapt.	81.4 (-0.2)	67.7	76.8	68.4	74.8
nTAS+Adapt.	81.2 (-0.3)	69.4	73.0	70.5	76.2

Table 2: Static versus adaptive intervention, averaged over the five ViT variants. Bold and italic mark first and second place per column. Gating narrows the clean-data cost while retaining most attacked-accuracy recovery.

Panel of Figure 4 — Figure 4: CS–nTAS convergence across depth and scale. (a,b) Per-head CS–nTAS scatter in late layers of ViT-B/16 and ViT-G/14: each point is one attention head; lexical heads cluster at high nTAS / low CS, semantic heads at the opposite corner. (c,d) The anti-correlation strengthens across all CLIP scales: Pearson $r$ becomes more negative, and the regression slope steepens in the final $20\%$ of blocks.

Panel of Figure 5 — Figure 5: Typographic and Semantic Neurons. Example activating images of neurons and their parent attention head. The top 4 activating images are shown with corresponding neuron attribution overlaid in a red gradient. Neurons with high nTAS scores are activated by the images that contain text with no significant semantic focus, whereas neurons with low nTAS scores lead to image clusters with strong semantic meaning, e.g., “Mallard” in the first row.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Prompt for Typographic Image Editing. text_type refers to one of the seven text types we defined. text_word is chosen randomly from the class label set, excluding the ground truth label of the input image. gt_text refers to the ground truth label of the input image.

Figure 7: Sampled Images from ImageNet-100-Text-D.

Figure 8: Intervention Strength vs. Performance. We sweep 5 values across [0,1] of

\alpha

, plot clean ImageNet-100 accuracy, and aggregate the Object Accuracy and the Attack Accuracy among all 5 tested CLIP models. Increasing

\alpha

gives consistent performance gain with minimal effect on clean data accuracy.

Panel of Figure 9 — Figure 9: Changes in object and attack text similarity. The bar plot shows the changes in cosine similarities of the CLIP image embedding to both the object text (Obj) and attack text (Atk). (a) The ImageNet-100-Text-D case. (b) The RTA-100 case.

Panel of Figure 10 — Figure 10: Case visualizations. Each panel shows the text-injected image overlaid with the patch attention map of the <cls> token (rows 1, 3) and the average attribution maps of the Top-3 activations in each head (rows 2, 4). Post-intervention visualizations are omitted: patch tokens do not contribute to the <cls> representation at reweighting parameter $\alpha=1$ .

	CLIP space		DINOv2 space
Corpus text	B/16	H/14	B/16	H/14
none (clean)	$0.611$	$0.280$	$0.522$	$0.206$
one word	$0.479$	$0.135$	$0.590$	$0.230$
5-word set	$0.166$	$\mathbf{0.000}$	$0.592$	$0.284$
100-word set	$0.329$	$0.190$	$0.480$	$0.306$
scrambled glyphs	$0.528$	$0.401$	$0.700$	$0.313$

Appendix

Table 3: Red-teaming the Clarity Score. Spearman

\rho

between the CS head ranking (scored on each contaminated corpus) and held-out causal effect. In CLIP space low-diversity text destroys the signal (H/14 CI

[-0.04,+0.06]

contains zero); in a language-free space the same top-activating sets remain predictive throughout.

Figure 11: Red-teaming the Clarity Score.

95\%

bootstrap CIs: CLIP-space scoring collapses under low-diversity text; DINOv2-space scoring of the same top-activating sets does not.

Selector	$\rho$ vs. causal		Group $\Delta$ (pt)
	B/16	H/14	B/16	H/14
Head-dec. nTAS	$0.569$	$0.350$	$+12.5$	$+15.3$
Monolithic nTAS	$0.576$	$0.340$	$+14.4$	$+17.6$
Direct grad. ratio	$0.654$	$0.403$	n/a	n/a
Head-dec. CS (50k)	$\mathbf{0.712}$	$\mathbf{0.404}$	$+19.1$	$+14.9$
Monolithic CS	$0.494$	$0.116$	n/a	n/a

Appendix

Table 4: Architecture ablation at equal capacity. Head ranking accuracy and group recovery on ImageNet-100-Text. Group

\Delta

here uses the held-out causal-evaluation split ; the main table reports full-benchmark absolute accuracies, hence the different magnitudes. “n/a”: ranking-only baselines, evaluated for selection accuracy but not run as group interventions. nTAS rows here use the neutral-vocabulary scoring protocol ( Appendix E ). CS row:

\rho

at the 50k scoring budget; its group

\Delta

uses the clean-corpus selection.

Figure 12: Concept panels, ViT-B/16. Two lexical (top rows) and two semantic (bottom rows) latents — the four highest-activating images of each latent, with its analytic attribution overlaid in red. Row labels: layer:head, population, and nTAS / CS of the latent.

Figure 17: Concept panels, Qwen3-VL-8B vision tower. Scores are TAS / CS evaluated with a CLIP ViT-B/16 reference encoder.

Figure 18: Concept panels, InternVL3.5-8B vision tower. Scores are TAS / CS evaluated with a CLIP ViT-B/16 reference encoder.

Figure 19: Concept panels, Gemma-3 12B SigLIP tower. Scores are TAS / CS evaluated with a CLIP ViT-B/16 reference encoder.

Method	Model	IN-100 (Clean)	RTA-100	Disentangling	PAINT	IN-100-Text-D
No intervention	ViT-B/16	74.8	56.4	46.2	59.1	54.6
	ViT-L/14	79.9	54.6	46.2	60.9	58.2
	ViT-H/14	83.8	53.4	39.4	50.9	56.6
	ViT-g/14	83.8	50.2	53.8	54.5	56.9
	ViT-bigG/14	85.4	60.9	42.5	50.0	62.4
Defense-Prefix	ViT-B/16	75.4 (+0.7)	65.6	84.4	68.0	69.9

Appendix

Table 5: Per-model comparison. All intervention strategies; conventions as in the main text; ∗ rows as reported by the original authors (gray; their own evaluation protocol rather than ours).

Model	Quantity	RTA-100	Disent.	PAINT	IN100-T	Clean
B/16	Base acc. (%)	56.4	46.2	59.1	54.6	74.8
	CS $\Delta$ [CI]	$+9.7$ [ $+7.5,+11.9$ ]	$+40.0$ [ $+32.5,+47.5$ ]	$+8.2$ [ $+0.9,+15.5$ ]	$+18.6$ [ $+17.4,+19.9$ ]	$-0.4$ [ $-1.0,+0.2$ ]
	nTAS $\Delta$ [CI]	$+12.3$ [ $+10.0,+14.6$ ]	$+40.6$ [ $+33.1,+48.1$ ]	$+11.8$ [ $+3.6,+20.0$ ]	$+19.6$ [ $+18.4,+20.9$ ]	$-0.5$ [ $-1.2,+0.1$ ]
	CS+Adaptive $\Delta$ [CI]	$+9.9$ [ $+7.8,+12.0$ ]	$+38.8$ [ $+31.2,+46.2$ ]	$+8.2$ [ $+1.8,+15.5$ ]	$+17.6$ [ $+16.4,+18.9$ ]	$-0.3$ [ $-0.6,+0.0$ ]
	nTAS+Adaptive $\Delta$ [CI]	$+11.1$ [ $+8.9,+13.4$ ]	$+40.0$ [ $+32.5,+47.5$ ]	$+11.8$ [ $+4.5,+19.1$ ]	$+19.2$ [ $+18.0,+20.5$ ]	$-0.3$ [ $-0.8,+0.2$ ]
L/14	Base acc. (%)	54.6	46.2	60.9	58.2	79.9

Appendix

Table 6: Main-table detail. Baseline accuracies and per-cell paired-bootstrap deltas with

95\%

CIs (

10^{4}

resamples) for both selection signals and their gated variants, computed against each run’s own paired baseline. Holm correction over the 20-cell CS attack family (five models

\times

four attack sets, one-sided): 19/20 cells remain significant at

0.05

. Random-suppression distributions: Table 8 .

$N$	B/16		H/14		Stab. (min)
	$\rho$	p@ $k$	$\rho$	p@ $k$
1k	0.553	0.72	0.374	0.39	0.903
2k	0.607	0.83	0.370	0.39	0.956
5k	0.658	0.83	0.387	0.43	0.978
10k	0.670	0.83	0.378	0.43	0.986
25k	0.711	0.83	0.391	0.47	0.995

Appendix

Table 7: Data-budget sweep. Clean ImageNet-1k val (3 subsample seeds per budget; mean

\rho

and precision@

k

; minimum pairwise cross-seed rank stability across both models).

Panel of Figure 20 — Figure 20: Per-layer CS–nTAS agreement. Spearman correlation at every scored transformer block, one panel per backbone. Each point is the correlation, across the $16$ heads of that block, between the two per-head lexicality scores; the shaded band is the $95\%$ bootstrap CI over heads, the dashed line marks $\rho=0$ , and the orange region is the last $20\%$ of depth — the region interventions draw from, and the 29-block family of the sign test.

	IN100-T $\Delta$ (pts)		Clean $\Delta$ (pts)
Model	mean $\pm$ sd	[min, max]	mean $\pm$ sd	[min, max]
B/16	$-5.1\pm 12.4$	$[-35.9,+11.8]$	$-11.6\pm 15.1$	$[-62.1,+0.1]$
L/14	$-0.8\pm 12.7$	$[-45.3,+11.7]$	$-8.2\pm 13.0$	$[-55.5,-0.6]$
H/14	$+7.9\pm 7.0$	$[-14.1,+16.5]$	$-4.3\pm 4.6$	$[-23.3,-1.5]$
g/14	$+4.8\pm 9.5$	$[-19.5,+16.4]$	$-3.9\pm 5.4$	$[-24.0,+0.2]$
bigG	$-3.7\pm 17.0$	$[-45.6,+12.6]$	$-8.9\pm 13.8$	$[-54.3,-0.6]$

Appendix

Table 8: Random head suppression over twenty draws. Head sets of the CS selection’s size; change in object accuracy versus no intervention on ImageNet-100-Text and on clean ImageNet-100 (points).

Model	Clean		Easy		Medium		Hard
	Base	Ours	Base	Ours	Base	Ours	Base	Ours
Gemma-3 4B	93.7	94.4	48.0	49.8	46.5	47.6	38.6	40.7
Gemma-3 12B	94.6	94.6	49.9	52.3	48.7	49.8	46.1	47.5
InternVL3.5 4B	92.0	91.6	55.1	54.8	47.4	47.7	36.1	35.9
InternVL3.5 8B	95.3	95.3	63.6	63.7	60.7	60.7	50.3	50.2
InternVL3.5 14B	92.0	91.8	56.9	57.0	55.7	55.8	46.6	46.7

Appendix

Table 9: VQA accuracy under typographic attacks. Accuracy in %. Base : vision encoder without intervention; Ours : vision encoder with intervention. Bold marks the higher value within each pair. Clean, Easy, Medium, Hard denote the splits, with Clean split free of lexical interference.

	Accuracy (%)			ASR (%)
Split	Base	Ours	Rand.	Base	Ours	Rand.
clean	94.76	94.63	94.31	–	–	–
easy	50.69	51.67	51.83	48.42	47.41	47.28
medium	48.61	49.65	49.62	50.66	49.40	49.53
hard	46.65	46.90	46.02	52.65	52.40	53.22

Appendix

Table 10: Selected heads vs. an equal-size random draw. Gemma-3 12B. Base columns are identical between the two runs (bit-stable).

Tower	Variant	clean	easy	medium	hard
Gemma-3 12B	Base	3000	1605	1539	1477
	Static	2996	1636	1572	1485
	Adaptive	2998	1653	1577	1495
Qwen3-VL 8B	Base	3010	2349	2255	2055
	Static	3011	2416	2319	2109
	Adaptive	3010	2406	2306	2090

Appendix

Table 11: Adaptive gating on three LVLM vision towers. RIO-Bench, correct counts out of

3166

;

\theta^{*}{=}0.25

on every tower). Bold marks the best intervention per attack split where any intervention beats base.

nTAS selection
Model	Variant	$\theta^{*}$	RTA	Disent.	PAINT	IN100-T †	Clean
ViT-B/16	Static	–	$+12.3$	$+40.6$	$+11.8$	$+20.0$	$-1.2$
	Gated	$0.25$	$+11.1$	$+40.0$	$+11.8$	$+19.6$	$-0.6$ ∘
ViT-L/14	Static	–	$+13.1$	$+18.1$	$+4.5$	$+16.1$	$-0.0$ ∘
	Gated	$1.0$	$+11.0$	$+16.2$	$+2.7$	$+16.0$	$+0.0$ ∘
ViT-H/14	Static	–	$+22.2$	$+40.0$	$+27.3$	$+22.6$	$-0.4$ ∘

Appendix

Table 12: Adaptive gating details. Five models reported, on both selection signals, with tuned

\theta^{*}

per configuration, under the tuning-split protocol. Cells are accuracy deltas (points) against each run’s paired un-intervened baseline.

Panel of Figure 21 — Figure 21: CKA across normalized depth. The horizontal axis is normalized depth (block index divided by model depth, so all five backbones share one axis). Left: mean pairwise CKA between the output representations of the heads within a block — falling values toward the right mean late-layer heads compute increasingly dissimilar functions. Right: CKA between each head’s output and the model’s final [cls] image embedding — rising values mean individual late heads each carry final-representation content. Together: late layers host specialized, individually influential heads, the regime the per-head dictionary is built for.

	ViT Fwd	ViT Bwd	Dict. F+B
Defense-Prefix ( Azuma and Matsui 2023 )	$E_{\text{ViT}}\cdot N_{\text{train}}$	$E_{\text{ViT}}\cdot N_{\text{train}}$	—
Dyslexify ( Hufe et al. 2025 )	$L\cdot H\cdot N_{\text{train}}$	—	—
Ours	$N_{\text{train}}$	—	$E_{\text{dict}}\cdot N_{\text{train}}$

Appendix

Table 13: Computational cost. Per-image cost decomposed across ViT forward (Fwd), ViT backward (Bwd), and head-decomposed transcoder forward+backward (Dict. F+B).

N_{\text{train}}

is the size of the training set;

E_{\text{ViT}}

and

E_{\text{dict}}

are the training epochs of the ViT and the dictionary, respectively;

L

and

H

are the number of ViT layers and per-layer attention heads.

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 460 papers

Towards Robustness against Typographic Attack with Training-free Concept Localization
Jul 2, 2026Bohan Liu, Wenqian Ye, Guangzhi Xiong +3VLM Robustness Adversarial Attacks on VLMs
University of Virginia, Charlottesville VA 22903, USA
Beyond Native Success: Auditing Deployment-Interface Exposure of CLIP Backdoors
Jun 16, 2026Kunlan Xiang, Haomiao Yang, Wenbo JiangVLM Robustness Adversarial Attacks on VLMs
University of Electronic Science and Technology of China
ReACT-CLIP: Response-Aware Test-Time Defense for Vision--Language Models
Aug 2, 2026Hashmat Shadab Malik, Toluwani Aremu, Samuele Poppi +2VLM Robustness Adversarial Attacks on VLMs
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE · Khalifa University, UAE · University of Western Australia, Australia +1

Jul 2, 2026J/K move · Enter open · S save

Towards Robustness against Typographic Attack with Training-free Concept Localization

Bohan Liu, Wenqian Ye, Guangzhi Xiong, Zhenghao He+2

University of Virginia, Charlottesville VA 22903, USA

VLM Robustness Adversarial Attacks on VLMs Large Vision-Language Models VLM Interpretability Mechanistic Interpretability

Open paper AskPDF arXiv

Models trained via Contrastive Language-Image Pretraining (CLIP) serve as the foundational vision encoders for most modern Large Vision Language Models (LVLMs). Despite their widespread adoption, CLIP models exhibit a critical yet underexplored failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics. This robustness issue, commonly described as a Typographic Attack (TA), exposes a vulnerability that poses a significant risk to safety-critical applications such as autonomous driving. To achieve interpretable and effective robustness against TA, we propose a novel, training-free mechanistic interpretability method. Our method provides sampling-based interpretations of hidden state representations and quantitatively attributes semantic versus lexical focus to individual attention heads. Through probabilistic analysis and circuit mining, we isolate specific Vision Transformer (ViT) components that disproportionately encode lexical information, thereby identifying the mechanistic source of TA. We further show that simple interventions applied directly to the identified circuits, without any additional training, can substantially improve robustness against Typographic Attacks in object classification. These interventions, such as selective adjustment of attention weights, also outperform both supervised and training-free defense methods. Our experiments demonstrate that applying the proposed intervention to the vision encoders of several state-of-the-art LVLMs yields substantial gains in Visual Question Answering accuracy under Typographic Attack interference on RIO-Bench. These results confirm both the efficacy and the generalizability of our mechanistic approach. Code is released at https://github.com/Liu-524/SamplingTAR.

Abstract

VLM Robustness

Explore similar work

Towards Robustness against Typographic Attack with Training-free Concept Localization

Beyond Native Success: Auditing Deployment-Interface Exposure of CLIP Backdoors

ReACT-CLIP: Response-Aware Test-Time Defense for Vision--Language Models