Mind the Spike: Mechanisms and Brittleness of Visual Massive Activations in Large Vision-Language Models

Jonas Ngnawé; Yann Pequignot; Sabyasachi Sahoo; Christian Gagné; Frédéric Precioso; Sanmi Koyejo

cs.LGSep 26, 2026

Mind the Spike: Mechanisms and Brittleness of Visual Massive Activations in Large Vision-Language Models

Authors: Jonas Ngnawé, Yann Pequignot, Sabyasachi Sahoo, Christian Gagné, Frédéric Precioso, Sanmi Koyejo

Organizations: Université Laval (IID) · Mila – Quebec AI Institute · Stanford University · Canada CIFAR AI Chair ·

Abstract

Large vision-language models (LVLMs) inherit massive activations from their text-only bases: spikes where a few fixed hidden channels receive values thousands of times above the typical magnitude. The text spike systematically appears in early layers at a fixed initial position, independently of input content. Visual spikes vary across images, but whether their formation follows a consistent pattern across LVLMs and how they respond to image perturbations remain open questions. We find that some LVLMs do not form visual spikes, while others spike at different rates, typically in deeper layers. We identify the trigger direction from model weights and an interpretable location rule: before the language model decoder runs, eventual spike tokens are largely restricted to those sharing least with the rest of the image. Crucially, visual spikes are strikingly brittle. Common corruptions frequently create and relocate spikes, and less often remove them, raising overall incidence. Our trigger-guided spike attack deliberately creates or removes spikes under a small $\ell_\infty$ budget, with 1/255 enough in nine of the ten models that spike. Finally, our preventive intervention removes only the trigger component before spikes erupt, eliminating or substantially reducing spikes on clean and perturbed images while leaving the other image tokens nearly unchanged. Our study spans 25 adapter-based LVLMs built on 18 released text-only bases from 10 families, ranging from 2B to 72B parameters.

Figures & tables

Figure 1: Visual spikes share the text spike’s channels but differ in when and where they form. An image processed by Qwen3-VL-8B illustrates this distinction. Red and blue mark two sky patches and their corresponding image tokens. Right, each token’s activation magnitude on the inherited spike channel, $|x[d_{\mathrm{spike}}]|$ , across hidden states. The text token (purple) becomes massive early, at block 6. The red image token initially has an ordinary activation, which abruptly rises at block 16, making it a visual spike token . Although both tokens come from the sky background, the blue token decodes to sky and remains non-massive, while the red token decodes to a repetitive pattern and later spikes.

Figure 2: A visual switch writes the spike in only some LVLMs. The spike-channel magnitude

|x[d_{\mathrm{spike}}]|

across hidden states on a log scale for four LVLMs, one image each. Grey lines are the image tokens, the dash-dot line the text spike token. Blue is the largest image token in two non-spiking models, and red the spike token in two spiking ones. Figure 7 shows all 25 LVLMs.

Figure 3: The common definition flags most image tokens in nearly every LVLM, the original flags spikes in only ten, and attention sinks largely track the original. Left, the mean share of image tokens passing

\phi\geq 20

over 300 images for each model. For 23 of the 25 models, every image has a token passing this threshold. Right,

\mathrm{VSI}_{\kappa}

\kappa=1{,}000

, the fraction of images that spike under the original massive-activation criterion, alongside independently measured attention-sink incidence. Ten models spike and fifteen do not. The numerical values, confidence intervals, and related measurements are in Tables 3 and 6 and Appendices B.1 and C.1 .

Panel of Figure 4 — Figure 4: Eventual spike tokens lie among the least-shared tokens, and editing a small region can relocate the spike. Left, for each spike token, the share of its image’s other tokens ranked above it against the share of spike tokens found, with the shared part, the norm or the direction at the decoder’s input. Thin blue lines are the shared part in each of the ten spiking models, bold lines the medians over models, and the legend gives the median AUC / FPR@95. Right, two images under InternVL3.5-8B with the same $5\times 5$ block placed at three locations. The block covers $25/256\approx 10\%$ of image tokens, whose decoder-input vectors are scaled to a tenth. Red marks the resulting spike, amber the clean spike, and blue outlines the edited block. Tables 22 and 26 give the location statistics; Figure 13 and Tables 33 and 34 give further relocation results.

Figure 5: Visual spikes are brittle to image changes, even when the answer stays the same. (a) Across ten spiking models and six corruptions, with 200 questions per model and operator, the forwards on which the corruption creates a spike, removes one, or relocates it, the last among forwards that spike both clean and corrupted, on a different image token. (b, c) One VQAv2 image under InternVL3.5-14B: a trigger-guided perturbation bounded by $\nicefrac{{1}}{{255}}$ raises $\Psi_{\mathrm{peak}}$ from $204$ to $19{,}628$ without changing the answer. Red marks the spike token. Across models, the attack creates or removes spikes at $\nicefrac{{1}}{{255}}$ in nine of the ten spiking LVLMs, with yes/no answers unchanged on $93$ to $100\%$ of attempts. Tables 37 and 38 give the corruption results and the full attack sweep.

	$\mathrm{VSI}_{\kappa}$ before $\to$ after (%)									Task-score change
LVLM	Clean			Corrupted			Attacked			$\Delta$ VQA (pct. points)	$\Delta$ CIDEr
Qwen2.5-VL-32B	49.0	$\to$	0.0	90.0	$\to$	0.0	100.0	$\to$	0.0	$+0.0$	$-0.007$
Qwen3-VL-4B	71.0	$\to$	0.0	89.7	$\to$	0.0	95.0	$\to$	0.0	$-0.6$	$+0.003$
InternVL3.5-4B	16.0	$\to$	3.0	42.0	$\to$	11.3	80.0	$\to$	27.5	$-0.4$	$+0.010$
Qwen3-VL-8B	9.0	$\to$	0.0	53.0	$\to$	0.0	82.5	$\to$	0.0	$+0.1$	$+0.008$
InternVL3.5-8B	59.5	$\to$	32.5	86.7	$\to$	62.0	100.0	$\to$	52.5	$+0.0$	$+0.000$

Table 1: Removing the trigger component suppresses spikes on clean and perturbed images, with small task-score changes in nine of the ten models.

\mathrm{VSI}_{\kappa}

(%), the fraction of images that spike, is shown before

\to

after intervention: clean POPE uses 200 questions per model, corruption is the mean over six operators on 50 questions each, and attack uses 40 saved creation-attack images where available. These are separate evaluation sets. Score changes are intervention minus original on 297 VQAv2 questions and 300 COCO captions. A random direction leaves every model’s reported clean spike rate unchanged. Tables 40 , 42 , 43 and 41 give the detailed results.

edit at the switch	spikes $\downarrow$	at zero $\uparrow$	answers same $\uparrow$	$\cos F$ $\uparrow$	$\lVert F\rVert$ ratio
no edit	0.54 (0.04–1.00)	0/10	1.00	1.000	1.000
ours, the trigger	0.00 (0.00–0.45)	6/10	1.00 (0.94–1.00)	0.999 (0.980–1.000)	1.000 (0.948–1.004)
the read zeroed, $\mathbf{x}^{\prime}=\mathbf{0}$	0.00	10/10	0.99 (0.94–1.00)	0.000	0.000
random $\mathbf{r}$	0.54 (0.04–1.00)	0/10	1.00	1.000 (0.999–1.000)	1.000

Table 2: The trigger intervention suppresses the spike without disabling the switch. The ten models that raise a visual spike, 200 evaluation questions each, every edit on all image tokens at the switch’s read. Each cell is the median over the ten models with its range, and at zero counts the models left with no spiking forward.

\cos F

and the ratio are medians over image tokens of

\cos(F(\bar{\vb{x}}'), F(\bar{\vb{x}}))

and

\lVert F(\bar{\mathbf{x}}^{\prime})\rVert/\lVert F(\bar{\mathbf{x}})\rVert

. Bold marks the best among the edits that remove the spike.

Appendix figures & tables51 assets

Supplementary material from the paper’s appendix.

Appendix

base LLM	spike channel(s)	LVLM	model line	clean $\mathrm{VSI}_{\kappa}$	max $\Psi$
Qwen2-7B	$\{458,2570\}$	Qwen2-VL-7B	Qwen2-VL	0.0% [0.0, 1.3]	388
		Molmo-7B-D	Molmo	0.0% [0.0, 1.3]	392
		LLaVA-OneVision-7B	LLaVA-OV	0.0% [0.0, 1.3]	378
Qwen2.5-3B	$\{318,1874\}$	Qwen2.5-VL-3B	Qwen2.5-VL	0.0% [0.0, 1.3]	171
Qwen2.5-7B	$\{458,2570\}$	Qwen2.5-VL-7B	Qwen2.5-VL	0.0% [0.0, 1.3]	237
		Qwen2.5-Omni-7B	Qwen2.5-Omni	0.0% [0.0, 1.3]	172

Appendix

Table 3: Within a shared base the adaptation moves the rate but never flips the spiking status of a model. The 25 LVLMs grouped by base language model, with the base’s inherited spike channels: each LVLM’s model line, clean

\mathrm{VSI}_{\kappa}

\kappa=1{,}000

n=300

COCO images at the default input size with the Wilson

95\%

interval, and the largest clean peak.

model	peak / runner-up	peak / 99th pct	tokens above	exactly one	ordinary $\Psi$
Qwen2-VL-7B	1.0	1.3	7, 12	0%	39, 261, 325
Molmo-7B-D	1.0	4.9	6, 14	1%	44, 71, 335
LLaVA-OV-7B	1.0	5.4	4, 4	0%	23, 62, 328
Qwen2.5-VL-3B	1.1	1.4	2, 36	31%	58, 100, 131
Qwen2.5-VL-7B	1.0	1.1	233, 425	2%	110, 172, 184
Qwen2.5-Omni-7B	1.1	1.3	0, 80	0%	51, 108, 128

Appendix

Table 4: At the criterion one token stands apart, except in Molmo2-8B. At the hundreds none does. 300 census images per model at each image’s peak state, spiking images in the spiking models. Columns: peak token’s

\Psi

over the runner-up’s and over the 99th percentile of the other tokens; tokens at or above the criterion per image, median and maximum; share with exactly one; ordinary tokens’

\Psi

, median, 99th percentile and maximum. Entries are per-image values, median over images. The criterion is

100

in the first block,

1{,}000

in the second.

LVLM	questions	spike, SDPA, eager	one alone, SDPA, eager	alignment, spiking, other	$\Psi$
Qwen3-VL-4B	200	142, 136	7, 1	0.38, 0.04	14,605
Qwen3-VL-8B	200	16, 16	5, 5	0.27, 0.02	11,320
Qwen3-VL-32B	200	70, 81	12, 23	0.25, 0.05	15,329
Qwen2.5-VL-32B	200	98, 100	0, 2	0.16, 0.06	2,891
InternVL3.5-4B	200	30, 30	2, 2	0.27, 0.23	2,509
InternVL3.5-8B	200	106, 105	8, 7	0.18, 0.09	2,990

Appendix

Table 5: Attention implementations change the spiking status of a few forwards, through the alignment with the trigger. Per LVLM, the 200 POPE evaluation questions, each read under scaled-dot-product and eager attention: forwards that spike under each, those that spike under one alone, over the forwards that change side the median largest cosine of an image token with the trigger at the switch’s feed-forward input where the image spikes and under the other implementation, and the median

\Psi

where it spikes.

Figure 6: The two groups are separated by a wide range of levels, not by a tuned one.

\mathrm{VSI}_{\kappa}

, the share of census images whose peak reaches

\Psi\geq\kappa

, over

\kappa

from 50 to

10^{5}

, the 300 census images of each of the 25 models, the ten with a visual switch on the left and the fifteen without one on the right, dotted rules at

10^{2}

10^{3}

and

10^{4}

LVLM	base LLM	spike channel(s)	$\phi\geq 20$	$\Psi\geq 100$	attention	$\mathrm{VSI}_{\kappa}$ , 300 images
Qwen2-VL-7B	Qwen2-7B	$\{458,2570\}$	100 (99)	100 (11)	0	0.0 [0.0, 1.3]
Molmo-7B-D	Qwen2-7B	$\{458,2570\}$	100 (83)	100 (1.5)	10 ( $<$ 0.1)	0.0 [0.0, 1.3]
LLaVA-OneVision-7B	Qwen2-7B	$\{458,2570\}$	100 (99)	100 (19)	0	0.0 [0.0, 1.3]
Qwen2.5-VL-3B	Qwen2.5-3B	$\{318,1874\}$	100 (100)	100 (6.8)	0	0.0 [0.0, 1.3]
Qwen2.5-VL-7B	Qwen2.5-7B	$\{458,2570\}$	100 (100)	100 (68)	0	0.0 [0.0, 1.3]
Qwen2.5-Omni-7B	Qwen2.5-7B	$\{458,2570\}$	100 (98)	100 (9.3)	0	0.0 [0.0, 1.3]

Appendix

Table 6: Only the criterion at

1{,}000

separates the two groups. The two lower criteria pass in 23 of the 25 models. Four criteria on the 300 census images per model: the share of images on which some image token passes, with the mean share of the image’s tokens passing in parentheses. The three activation criteria are read on the census forward, the attention on the captioning forward over the same images, and on the census forward for Idefics3-8B.

LVLM	POPE	VQAv2	captions	top share	single pair	outcome
Qwen2.5-VL-32B	99/99, 0/101	166/167, 0/130	169/169, 0/131	0.30 / 0.32 / 0.24	4, 7, 0	0.870, 0.715, 0.23
Qwen3-VL-4B	133/136, 1/64	231/233, 0/64	225/234, 0/66	0.25 / 0.23 / 0.21	0, 0, 0	0.915, 0.722, 0.19
InternVL3.5-4B	30/32, 0/168	57/58, 2/239	57/58, 0/242	0.40 / 0.38 / 0.37	0, 5, 0	0.845, 0.658, 0.61
Qwen3-VL-8B	15/16, 3/184	31/32, 2/265	30/34, 0/266	0.21 / 0.18 / 0.19	0, 0, 0	0.890, 0.734, 0.35
InternVL3.5-8B	107/110, 0/90	173/181, 1/116	171/182, 0/118	0.41 / 0.34 / 0.36	0, 6, 0	0.860, 0.676, 0.71
Qwen3-VL-32B	74/81, 0/119	119/127, 3/170	118/127, 4/173	0.20 / 0.19 / 0.16	1, 3, 6	0.895, 0.739, 0.15

Appendix

Table 7: An attention sink forms on the spike token, and rarely without one. Molmo2-8B does on a third at most. Per LVLM and task: forwards with an attention sink, an image token at

0.3

or more of the image’s head-mean attention in a deep block, over spiking then non-spiking forwards in the spiking models, all forwards in the others; top image token’s median share; forwards passing

0.3

at one layer-head pair of raw attention; POPE and VQAv2 accuracy and CIDEr. Read on the text queries after the image.

Figure 7: Every model’s switch, or its absence, on one page. Figure 2 for all 25 LVLMs, the ten spiking models first, each on the census image whose peak is that model’s median. The spike-channel magnitude across hidden states on a log scale. Every image token is a grey line. The spike token is red in the ten spiking models, and the largest image token is blue in the fifteen without one. The dash-dot line is the text spike token, and the dashed line the criterion’s floor at $100$ .

Figure 8: How much of the spike token is the spike. Figure 7 fixes the spike channel and runs over the tokens. This one fixes the token that spikes and runs over its $4{,}096$ channels, under Qwen3-VL-8B on Figure 1 ’s image. Left, every channel along the decoder on a log scale, the spike channel in red. Right, the same token at the state the visual switch writes into, where the spike channel reaches $7{,}008$ against the next channel’s $436$ and the median’s $4.6$ .

base	block, channel	$\lambda_{\lvert\max\rvert}$	$\lambda_{+}$	$\cos(\vb{t}, v_{+})$	write along $v_{+}$ , $\mathbf{t}$	top-1 by $v_{+}$	$\mathbf{t}$ across adaptations
Qwen3-4B	16, 4	$-38.2$	$+14.8$	1.000	37,893, 37,895	0.73, 0.98	0.995, 0.995
Qwen3-8B	16, 2276	$-39.5$	$+19.5$	1.000	79,630, 79,635	1.00, 0.85	0.997 to 0.998
Qwen3-8B, its text spike token	6, 2276	$+17.6$	$+17.6$	1.000	70,707, 70,732	first token at cosine 0.46	n/a
Qwen3-14B	19, 731	$-41.3$	$+36.7$	1.000	186,333, 186,369	0.86	0.995
Qwen3-32B	43, 731	$-118.8$	$+71.7$	1.000	365,735, 365,776	0.94	0.938, 0.999
Qwen2.5-32B	38, 4675	$-32.3$	$+16.7$	1.000	85,273, 85,281	1.00	0.999

Appendix

Table 8: The trigger and the published eigenvector agree at the visual switch, and the trigger survives adaptation. The quadratic form at the visual switch of the six bases, the norm’s weight folded in. Columns: largest eigenvalues, cosine of the trigger with the top positive eigenvector, real write along each, share of spiking images whose spike token tops that eigenvector, and the trigger’s cosine between the base and each of its LVLMs, estimated on the LVLM’s own weights at the same block, channel and sign. Third row: Qwen3-8B’s text switch, top-1 read on the first token. ∗ Against the most negative eigenvalue’s eigenvector, the sign of Mistral-NeMo-12B’s eruption.

LVLM	base, block	spiking, non-spiking	spike	others	spike on top	non-spiking best	non-spiking above
Qwen3-VL-4B	Qwen3-4B, 16	226, 74	0.295, 0.199	$-0.011$ , 0.015	0.75	0.022	0.00
InternVL3.5-4B	Qwen3-4B, 16	49, 251	0.216, 0.169	$-0.008$ , 0.020	1.00	0.026	0.00
Qwen3-VL-8B	Qwen3-8B, 16	38, 262	0.210, 0.138	$-0.012$ , 0.003	0.89	0.008	0.00
InternVL3.5-8B	Qwen3-8B, 16	185, 115	0.220, 0.138	$-0.015$ , 0.018	0.82	0.034	0.00
InternVL3.5-14B	Qwen3-14B, 19	207, 93	0.119, 0.061	0.001, 0.027	0.87	0.031	0.02
Qwen3-VL-32B	Qwen3-32B, 43	140, 160	0.084, 0.056	0.002, 0.024	0.91	0.029	0.09

Appendix

Table 9: At the switch’s input the eventual spike token is aligned with the trigger, and no other token is. Per LVLM at its visual switch’s input, with its base’s trigger. Columns: spiking and non-spiking census images; the eventual spike token’s cosine, median and tenth percentile; every other token’s, median and 99th percentile; share of spiking images whose spike token is the top token; non-spiking images’ best token, median; share of non-spiking images whose best token passes the spike tokens’ tenth percentile. † Five spiking images, indicative.

LVLM	base LLM	text switch: text spike token	image write	image peak	later block, spiking images, write
Qwen3-VL-4B	Qwen3-4B	6: 12 $\to$ 11,072	18	8, 15	16, 100%, 6,736
InternVL3.5-4B	Qwen3-4B	6: 16 $\to$ 15,616	17	3, 11	16, 100%, 5,786
Qwen3-VL-8B	Qwen3-8B	6: 20 $\to$ 13,440	26	5, 12	16, 100%, 6,092
InternVL3.5-8B	Qwen3-8B	6: 26 $\to$ 24,448	22	5, 16	16, 100%, 12,275
InternVL3.5-14B	Qwen3-14B	6: 278 $\to$ 13,888	25	68, 179	19, 100%, 10,494
Qwen2.5-VL-32B	Qwen2.5-32B	6: 36 $\to$ 3,280	95	165, 142	38, 100%, 3,919

Appendix

Table 10: The text switch leaves image tokens small, and only the spiking models write one into the thousands, Pixtral-12B at the text switch itself. Per LVLM: the text switch’s block, the text spike token before and after it, that block’s largest write on an image token, and the image’s peak after it as

\lvert\mathbf{x}[d]\rvert

and

\Psi

. Then the largest later write’s block, the share of spiking images on which it is that block, and that write. Read on block outputs, over image tokens entering below the floor.

vector at position 0	norm	$n$	reach the level (%)	channel after block 6
every vocabulary token (LVLM)	1.1	151,936	100.0	7,712
every vocabulary token (base)	1.1	151,936	100.0	10,176
image tokens, the projector’s outputs	39.6	2,048	0.5	4
the same image tokens at the text norm	1.1	2,048	100.0	7,936
the same at ten times the text norm	11.1	2,048	16.4	18
text embeddings at the image norm	39.6	4,096	3.0	104

Appendix

Table 11: At position 0 any small enough vector is carried onto the trigger. Qwen3-VL-4B’s language model, the base in the second row: one-token sequences of the vectors named, the share reaching the level and the spike channel after block 6.

model	LVLM	base	own norm	text norm	text norm	$20\times$	image norm	norms
	vocabulary		image tokens		Gaussian
Qwen3-VL-4B	100.0	100.0	0.5	100.0	100.0	14.0	0.5	1.1, 39.4
InternVL3.5-4B	99.8	100.0	0.0	77.8	99.9	5.0	9.5	1.1, 19.9
Qwen3-VL-8B	99.9	99.9	4.4	96.1	100.0	30.7	88.7	1.4, 19.0
InternVL3.5-8B	95.3	99.9	0.0	62.5	100.0	0.1	0.0	1.4, 23.8
Molmo2-8B	99.7	99.9	0.0	94.2	100.0	3.4	0.0	1.5, 85.2

Appendix

Table 12: At position 0 small vectors reach the text spike in eight of the ten models, and image tokens in none. Each spiking LVLM’s language model on one-token sequences, as in Table 11 : the share (%) reaching the level after the text switch. Columns: every vocabulary token in the LVLM and in its base, the projector’s image tokens at their own norm and at the text norm, Gaussian vectors at the text norm, at twenty times it and at the image norm, then the text and the image norm.

block	along the trigger	on an image token	on ordinary text	on the text spike token
1	8,850	3	1	0
2	7,635	3	0	4
6	28,879	5	5	11,072
16	37,895	7,328	5	211
30	7,128	26	31	-2
31	7,333	26	25	-3

Appendix

Table 13: Many blocks could write the channel. One writes an image token. The Qwen3-4B base, channel 4, the ten blocks whose write along the trigger exceeds

5{,}000

: per block the largest write along any unit direction in the eruption’s sign (Equation 1 ) and the write the block actually makes on an image token, on an ordinary text token and on the text spike token.

base	block	along the trigger	spike, Qwen3-VL	closest, Qwen3-VL	spike, InternVL3.5	closest, InternVL3.5
Qwen3-4B	1	8,850	$-0.30$	$-0.13$	$-0.40$	$-0.27$
Qwen3-4B	2	7,635	$-0.33$	$-0.17$	$-0.42$	$-0.35$
Qwen3-4B	6	28,880	$-0.02$	$+0.01$	$-0.02$	$-0.01$
Qwen3-4B	16	37,895	$+0.40$	$+0.09$	$+0.42$	$+0.04$
Qwen3-4B	30	7,128	$0.00$	$-0.09$	$0.00$	$-0.09$
Qwen3-4B	31	7,333	$-0.01$	$-0.09$	$-0.01$	$-0.08$

Appendix

Table 14: Only the visual switch’s trigger is reached. For the blocks of the two bases whose write along the trigger passes

5{,}000

: that write, and the cosine with the block’s trigger of the eventual spike token and of the closest other image token, in each adaptation of the base. Read on the residual entering the block’s feed-forward, median over the spiking images, 49 and 5 of 60 on Qwen3-4B and 6 and 40 of 60 on Qwen3-8B. Read under

\Psi

with the floor, tokens entering at

100

or above excluded.

input	peak $\Psi$	$\geq 1{,}000$	on the spike token	$\Psi$ there
Qwen3-VL-4B as it runs	16,384	78%	100%	16,384
its LM, 1-D positions, no DeepStack	13,243	73%	7%	56
2-D positions, no DeepStack	13,092	57%	11%	52
1-D positions, DeepStack	15,763	87%	29%	60
1-D, from block 13, its own state	15,767	85%	73%	14,415
from block 13, the base’s state	204 (250)	1% (9%)	1% (1%)	122 (132)

Appendix

Table 15: The base does not spike on the adaptation’s vectors until its own visual switch. Qwen3-VL-4B’s decoder-0 input run through the text-only Qwen3-4B, 150 POPE images: the peak

\Psi

over the deep layers, the share of images at or above the level, the share on which the peak is the adapted model’s own spike token, and

\Psi

at that token. In parentheses, the same run on Qwen3-4B-Base, the pretrained checkpoint the instruction-tuned base is built from.

after block	base, its token	adapted, that token	base, position 0	adapted, position 0
2	6	11	10	107
3	1,748	9	7,605	8,992
4	2,655	16	8,777	10,561
22	3,137	54	9,058	7,956

Appendix

Table 16: Here the base spikes on the adaptation’s vectors and the adaptation does not. LLaVA-OV-7B’s decoder-0 input run through the text-only Qwen2-7B, both models with one-dimensional positions. Top:

\Psi

after each block at the base’s peak image token and at position 0, median over the 128 of 150 images on which the base spikes. Bottom: on 150 images, the peak

\Psi

over the deep layers and where it lands.

			decoder input		from block $k$
LVLM	base	spiking	base spikes	on its token	$S-3$	$S-1$	$S$
Qwen3-VL-4B	Qwen3-4B	114/150	0%	0%	4% (13)	48% (15)	90% (16)
InternVL3.5-4B	Qwen3-4B	25/150	0%	0%	8% (13)	76% (15)	100% (16)
Qwen3-VL-8B	Qwen3-8B	13/150	0%	0%	0% (13)	31% (15)	100% (16)
InternVL3.5-8B	Qwen3-8B	89/150	0%	0%	7% (13)	53% (15)	93% (16)
Molmo2-8B	Qwen3-8B	150/150	3%	0%	45% (13)	54% (15)	66% (16)

Appendix

Table 17: Across the pairs, the base spikes on the LVLM’s token only once given the state entering the visual switch. Each LVLM with its text-only base, 150 POPE images. Columns: the LVLM’s spiking images; the share of all images on which the base, run on the LVLM’s decoder-input vectors, spikes, and the share of spiking images on which it does so on the LVLM’s own spike token; then, for each block

k

from which the base is given the LVLM’s state (in parentheses, the last is the visual switch), the share of spiking images on which the base spikes on the LVLM’s spike token. Lower block, the non-spiking LVLMs: the share of images on which the base spikes on the decoder-input vectors, and its median peak

\Psi

there. Read under

\Psi

with the floor, tokens entering at

100

or above excluded.

base	text switch	visual switch	last third	spike channel on top
Qwen3-4B	6: 1st, $\times 3.5^{*}$	16: 1st, $\times 8.6^{*}$	35: 1st ∗	13 of 36
Qwen3-8B	6: 1st, $\times 5.0^{*}$	16: 1st, $\times 22.9^{*}$	34: 1st ∗	13 of 36
Qwen3-14B	6: 1st, $\times 7.3^{*}$	19: 1st, $\times 6.3^{*}$	38: 1st	39 of 40
Qwen3-32B	6: 1st, $\times 10.1^{*}$	43: 1st, $\times 6.5^{*}$	62: 1st ∗	63 of 64
Qwen2.5-32B	5: 1st, $\times 1.4^{*}$	38: 1st, $\times 1.7^{*}$	60: 1st ∗	55 of 64
Qwen3-1.7B	2: 1st, $\times 15.0^{*}$	(18: 1st, $\times 1.3$ )	27: 1st ∗	24 of 28

Appendix

Table 18: At the switches the spike channel’s writer writes hardest of the block’s rows. Per base, at the text switch, the visual switch and the last third’s top block: the writer’s rank among the block’s rows, four starts per row, and its ratio to the best other row. Parentheses: a stand-in where no LVLM on the base spikes. Last column: blocks topped by a spike row, one start per row. ∗ largest single weight on a spike row. † Before the norm these two apply to the feed-forward’s output.

base	text switch	visual switch (largest write)	candidate blocks between	spike channel on top
Qwen3-4B	6	16 ( $97{,}859$ )	0	0
Qwen3-8B	6	16 ( $161{,}622$ )	0	0
Qwen3-14B	6	19 ( $211{,}300$ )	1	1
Qwen3-32B	6	43 ( $608{,}414$ )	15	15
Qwen2.5-32B	5	38 ( $165{,}490$ )	6	4
Mistral-NeMo-12B	2	2 ( $6{,}672$ )	0	0

Appendix

Table 19: Image tokens can pass candidate blocks before their visual switch. The six bases carrying a spiking LVLM: the text switch, the visual switch with its largest write on the spike channel, and the candidate blocks between the two, with how many of them have the spike channel as their top row.

model	template	text spike token	text switch	peak	position 1	position 2
Qwen2.5-32B	its own	1	5 ( $12\rightarrow 14{,}976$ )	$18{,}176$	$18{,}176$	796
Qwen2.5-32B	the LVLM’s	1	5 ( $12\rightarrow 14{,}976$ )	$18{,}176$	$18{,}176$	796
Qwen2.5-VL-32B	its own	2	6 ( $36\rightarrow 3{,}280$ )	$5{,}728$	452	$5{,}728$
Qwen2.5-VL-32B	the base’s	2	6 ( $36\rightarrow 3{,}280$ )	$5{,}728$	452	$5{,}728$

Appendix

Table 20: On the same tokens the adapted model carries its text spike token elsewhere, on the same channels. Qwen2.5-32B and Qwen2.5-VL-32B text only, on the same token ids of the 12 chat prompts under each model’s template, channels 4675 and 3094: the text spike token’s position, counted from 0, its text switch with the median value before and after, its peak, and the peaks at positions 1 and 2.

Bases carrying a spiking LVLM
base	blocks	text switch	largest write (block)	LVLM	its text switch	$\mathrm{VSI}_{\kappa}$ (%)	past $0.2$ (%, block)
Qwen3-4B	36	6	$97{,}858$ (16)	Qwen3-VL-4B	6	$75.3$	$80$ (16)
				InternVL3.5-4B	6	$16.3$	$17$ (16)
Qwen3-8B	36	6	$161{,}627$ (16)	Qwen3-VL-8B	6	$12.7$	$15$ (16)
				InternVL3.5-8B	6	$61.7$	$67$ (16)
Qwen3-14B	40	6	$211{,}303$ (19)	InternVL3.5-14B	6	$69.0$	$65$ (19)

Appendix

Table 21: An early text switch goes with no visual switch in every base but Mistral-NeMo-12B, and a late one does not decide. Per base, from its text forward on 12 chat prompts: blocks, text switch, largest spike-channel write of a middle-third block, that block in parentheses. Per LVLM: text switch, clean

\mathrm{VSI}_{\kappa}

, and share of census images, 60 or 300 per model, whose best read block holds an image token at cosine

0.2

or more with the base’s trigger. Read blocks: text switch, each third’s top block, visual switch.

model	$n$ spiking	lowest decile, $p$	lowest quarter, $p$	lowest decile, norm	lowest quarter, norm
Qwen3-VL-8B	38	92.1%	97.4%	97.4%	97.4%
Qwen3-VL-4B	226	67.7%	98.2%	69.5%	93.4%
Qwen3-VL-32B	140	45.7%	70.7%	49.3%	74.3%
Qwen2.5-VL-32B	156	97.4%	100.0%	0.0%	7.1%
InternVL3.5-4B	49	42.9%	89.8%	12.2%	53.1%
InternVL3.5-8B	185	60.5%	87.0%	39.5%	70.3%

Appendix

Table 22: In every model the spike token sits in the least-shared quarter of its image on at least

71\%

of images. The share of images whose spike token lies in the image’s lowest decile and lowest quarter, by the shared part

p

and by the norm, over the 300 census images per model. The ten spiking models, their spiking images counted. † Fewer than ten images, indicative.

Figure 9: The spike token is an ordinary-sized token with an unusually small shared part. The ten spiking models, their spiking census images, image tokens entering below the floor. Left: the spike token over its image’s median for the norm, the shared part and the rejection, median and interquartile range over spiking images. Right: the spike token’s percentile in its image by each of the three,

0

the lowest token.

LVLM	spiking	$\rho$ , spiking images	negative	$\rho$ , non-spiking	cosine, lowest tenth, highest tenth, rest
Qwen3-VL-4B	226	$-0.33$ ( $-0.40$ , $-0.27$ )	100%	$-0.28$	$0.010$ , $-0.014$ , $-0.010$
InternVL3.5-4B	49	$-0.54$ ( $-0.61$ , $-0.48$ )	100%	$-0.50$	$0.006$ , $-0.016$ , $-0.008$
Qwen3-VL-8B	38	$-0.05$ ( $-0.09$ , $0.03$ )	65.8%	$0.01$	$-0.005$ , $-0.011$ , $-0.012$
InternVL3.5-8B	185	$-0.31$ ( $-0.37$ , $-0.23$ )	100%	$-0.29$	$-0.003$ , $-0.018$ , $-0.015$
InternVL3.5-14B	207	$-0.38$ ( $-0.45$ , $-0.31$ )	99.5%	$-0.34$	$0.011$ , $-0.002$ , $0.001$
Qwen3-VL-32B	140	$-0.26$ ( $-0.31$ , $-0.19$ )	99.3%	$-0.26$	$0.008$ , $0.001$ , $0.003$

Appendix

Table 23: The less a token shares with its image at layer 0, the more it aligns with the trigger at the switch. Per LVLM, the Spearman correlation across each census image’s tokens between the layer-0 shared part and the cosine with the base’s trigger at the visual switch’s input. Columns: median and interquartile range over spiking images, share negative, median over non-spiking images, and the mean cosine of the image’s lowest tenth by the shared part, its highest tenth and the rest, pooled over spiking images. † Five spiking images, indicative.

Figure 10: At equal coverage the shared part locates more spike tokens than background does. Left: one image under Qwen3-VL-8B, COCO objects tinted, the shared part’s lowest decile in blue, tokens meeting the criterion marked. Right: the spiking models, their spiking images, the share whose spike token falls in the given fraction of the image’s tokens, taken from the smallest shared part up. Molmo2-8B is absent because its overlapping crops have no single token grid on which to place background, and Pixtral-12B is read on the census. Each square is background, token share against spike-token share; the dot of its colour is the curve there.

model	$n$	spike in bg	base rate	lift	$z$	$p$
Qwen2.5-VL-32B	158	93.0%	69.2%	$+23.9$ pp	7.48	$<10^{-4}$
Qwen3-VL-32B	112	85.7%	68.6%	$+17.2$ pp	4.56	$<10^{-4}$
InternVL3.5-14B	180	87.8%	70.8%	$+17.0$ pp	5.79	$<10^{-4}$
InternVL3.5-4B	53	84.9%	68.3%	$+16.6$ pp	3.06	0.0011
Pixtral-12B	296	85.5%	69.2%	$+16.2$ pp	7.21	$<10^{-4}$
InternVL3.5-8B	174	82.8%	67.9%	$+14.9$ pp	4.98	$<10^{-4}$

Appendix

Table 24: The spike token lands in background more often than a random token would. Over the spiking images of each spiking model: the share of spike tokens under no COCO object mask, the background’s share of the image’s tokens, the difference in points, and the

z

of that count against a random-landing null with its one-sided

p

. Pixtral-12B is read on the census, on its

32\times 32

token grid; Molmo2-8B’s overlapping crops have no single grid. † Fewer than ten images, indicative.

model	$n$	$P(D\mid S,B)$	$P(D\mid B)$	the shared part adds	$P(B\mid S,D)$	$P(B\mid D)$	background adds
Qwen3-VL-8B	26	94.7%	11.2%	$+83.5$	75.0%	68.5%	$+6.5$
Qwen3-VL-4B	227	73.3%	11.7%	$+61.6$	80.5%	77.6%	$+2.9$
Qwen3-VL-32B	112	50.0%	11.9%	$+38.1$	88.9%	81.5%	$+7.4$
Qwen2.5-VL-32B	158	100.0%	12.0%	$+88.0$	93.0%	83.5%	$+9.5$
InternVL3.5-4B	53	51.1%	12.0%	$+39.1$	82.1%	84.1%	$-1.9$
InternVL3.5-8B	174	55.6%	12.2%	$+43.3$	82.5%	85.0%	$-2.6$

Appendix

Table 25: Given background the shared part still locates the spike token. Given the shared part, background adds little. For a token of a spiking image,

S

is being the spike token,

D

being among the tenth of the image’s tokens with the smallest shared part, and

B

being background. Columns 5 and 8 are the differences

P(D\mid S,B)-P(D\mid B)

and

P(B\mid S,D)-P(B\mid D)

, in points. Pixtral-12B is read on the census. † Fewer than ten images, indicative.

model	spike tokens	shared part $p$	direction	norm	rejection $r$
		AUC / FPR@95	AUC / FPR@95	AUC / FPR@95	AUC / FPR@95
Qwen3-VL-4B	366	0.926 / 20%	0.721 / 81%	0.919 / 27%	0.900 / 37%
InternVL3.5-4B	55	0.880 / 28%	0.894 / 35%	0.755 / 53%	0.647 / 69%
Qwen3-VL-8B	39	0.957 / 14%	0.572 / 98%	0.950 / 9%	0.943 / 13%
InternVL3.5-8B	282	0.873 / 41%	0.811 / 79%	0.792 / 64%	0.727 / 81%
InternVL3.5-14B	341	0.880 / 51%	0.845 / 71%	0.866 / 44%	0.820 / 53%

Appendix

Table 26: The shared part beats both the token’s size and its direction in five of the ten models, and neither of those two holds everywhere. The ten spiking models on their spiking census forwards, each score read at the language model’s input. Positives are the spike tokens, counted in the second column. AUC is the within-image area, higher better; FPR@95 the share of non-spike tokens above the threshold covering

95\%

of spike tokens. Last row: median over the ten models. † Five spiking images, indicative.

real tokens	the mean token	the most-shared token	the least-shared token	the spike token
A vintage steam locomotive numbered 71 is illuminated at dusk, standing proudly on…	a dark, vintage-style steam locomotive with multiple large driving wheels and a prominent… lens: kai (a name), end, seat	a long, dark street illuminated by a series of evenly spaced, warm orange… lens: lighting, lamp fixture, lamp	a repetitive, monotonous pattern of the same placeholder or empty square icon, stacked… lens: more, @example, lé	a repetitive, identical pattern of a single small, indistinct square or pixelated block,… lens: yne, a set, more
A colorful plate of stir-fried chicken, broccoli, carrots, and cauliflower served with a…	a close-up of a plate of food, likely a stir-fry or curry, featuring… lens: dining table, dining, bar	a single fork with multiple tines, repeatedly duplicated in a row, creating a… lens: yd, clip, kiss	a repetitive, monotonous pattern of the word “tonon” in a uniform, unchanging font… lens: ayne, jou, jon	a close-up of a single, slightly blurry, spiral-shaped food item, possibly a cooked… lens: shape, Shapes, shape
A young boy in a Star Wars t-shirt walks a brown and white…	a black dog with a white patch on its chest, standing on a… lens: qi (qilin), column, trs	a person wearing a beige, long-sleeved, button-up shirt and matching beige pants, standing… lens: brown, uder, kiss	a series of identical, empty, and slightly blurred square frames arranged in a… lens: lé, more, yne	a repetitive, visually striking pattern of the word “save” written in a bold,… lens: aid, ej, aid
A person holds a white Nintendo Wii remote in front of a monitor…	a row of identical white, rectangular objects with a small circular detail on… lens: kai (a name), áo, roe deer	a row of identical white Wii controllers, each with a slightly different angle… lens: pale, transparent, simple	a long horizontal line of repeated text reading “MAX” in a bold, sans-serif… lens: thwart, lesson, yla	a single, repeated, and identical pattern of a small, square-shaped icon or symbol,… lens: more, plain, wid
A cozy, well-lit hotel room features two neatly made beds with plush towels,…	a row of identical dark brown wooden doors with a simple, modern design,… lens: beds, lodging, beside	a stack of neatly folded, dark-colored towels arranged in a repeating pattern, with… lens: OfClass, pillow, .same	a repetitive, stacked pattern of identical, empty, square-shaped placeholders, creating a visually uniform… lens: more, composition, a group	a single, stylized, repeating pattern of a dark green leaf or leaf-like shape… lens: lodging, hotel, bar
A freshly baked pizza with tomatoes, spinach, and onions sits on a metal…	a close-up of a round, thin-crust pizza with a golden-brown surface, topped with… lens: end, bar, canteen	a stack of identical, empty, square coasters arranged in a neat, repeating pattern. lens: beside, porter, thick	a repetitive, static pattern of the same word “tomato” written multiple times in… lens: persuasion, goosefoot, ying (Britain)	a single, slightly blurry, unremarkable photograph of a plain white surface with no… lens: bar, dining table, beer

Appendix

Table 27: In Qwen3-VL-4B the spike token decodes to a repetitive pattern rather than to its own position’s content. Qwen3-VL-4B, eight images on the

16\times 16

token grid the model reads, the most-shared token outlined in blue, the least-shared in grey and the spike token in red. Beside each, the description with the real tokens and with every image position filled by the image’s mean token or its most-shared, least-shared or spike token, each followed, in blue italics, by the three highest tokens of that vector’s layer-0 logit lens. The eighth image does not spike, and its red cell and last column read its peak token.

		repetitive, scene (%)
model	images	real	mean	most-shared	least-shared	spike
Qwen3-VL-4B	50	10, 100	38, 74	66, 20	100, 0	98, 4
Qwen3-VL-8B	34	3, 94	53, 56	74, 15	100, 0	100, 6
Qwen3-VL-32B	50	10, 100	46, 32	70, 20	98, 0	66, 12
Qwen2.5-VL-32B	50	8, 96	18, 56	32, 40	30, 2	4, 0
InternVL3.5-4B	50	4, 96	16, 32	26, 8	8, 16	2, 10

Appendix

Table 28: The spike token rarely carries the scene. Per spiking model, up to 50 spiking census images, the whole spiking pool for Qwen3-VL-8B and InternVL3.5-38B: the description with the real image tokens and with every image position filled by the mean, most-shared, least-shared or spike token at the language model’s input. Each cell gives the share of descriptions naming a repetitive pattern, then the share naming the scene, with at least two content words shared with COCO’s five reference captions.

model	images	switch	at the input	near block 12	one block before
Qwen3-VL-4B	100	16	0.35	0.61	0.86
Qwen3-VL-8B	33	16	0.15	0.36	0.73
Qwen3-VL-32B	60	43	0.22	0.20	0.77
InternVL3.5-4B	49	16	0.35	0.57	0.98
InternVL3.5-8B	80	16	0.11	0.33	0.63
InternVL3.5-14B	80	19	0.24	0.44	0.89

Appendix

Table 29: The choice of spike token is settled late. The spike token’s whole state is exchanged with a random other image token’s at the input of one block, every other token untouched. Columns: the spiking census images, the visual switch, and the share of images on which the moved state spikes at its new slot, exchanged at the language model’s input, near block 12, and one block before the switch. Qwen3-VL-32B is exchanged from block 4, its visual features entering over the first three blocks. Pixtral-12B’s switch is block 2. InternVL3.5-38B spikes on 5 of its 300 census images, 3 of which enter this reading.

model	forwards	$S$	$S-4$	$S-1$	$S$	attention	feed-forward	attention
			cosine with the trigger			writes at $S-1$		at $S$
Qwen3-VL-4B	80	16	-0.01	+0.01	+0.27	3.8	9.5	2.4
Qwen3-VL-8B	35	16	-0.02	+0.00	+0.22	4.8	11.3	0.8
Qwen3-VL-32B	60	43	+0.01	+0.03	+0.09	0.5	8.6	20.1
InternVL3.5-4B	49	16	-0.01	+0.06	+0.22	-0.6	6.4	7.5
InternVL3.5-8B	80	16	-0.01	+0.00	+0.21	3.6	12.6	15.5

Appendix

Table 30: The alignment with the trigger is written in the last blocks. Medians over spiking census forwards under the description prompt, up to 80 per model. Columns: forwards, the visual switch

S

, the spike token’s cosine with the trigger entering blocks

S-4

S-1

and

S

, its attention and feed-forward writes along the trigger at

S-1

, and its attention write at

S

. Qwen2.5-VL-32B’s spike token is already aligned at

S-4

, and Pixtral-12B’s switch at block 2 leaves no block

S-4

model	images	text spike, before, after	clean, blocked	text spike	its entries	sources	random	feed-forward	attention	both
		attention to the	alignment,	denied			control	trigger component removed
Qwen3-VL-4B	60	0.05, 0.65	0.60, -0.01	0.05	0.35	0.05	1.00	0.42	0.90	0.00
Qwen3-VL-8B	33	0.12, 0.60	0.55, 0.00	0.00	0.27	0.12	1.00	0.55	0.91	0.00
Qwen3-VL-32B	60	0.31, 0.77	0.51, -0.03	0.00	0.18	0.45	0.95	0.87	0.90	0.02
InternVL3.5-4B	49	0.10, 0.67	0.58, 0.03	0.00	0.82	0.86	1.00	0.02	1.00	0.00
InternVL3.5-8B	60	0.17, 0.61	0.63, 0.11	0.25	0.32	0.23	1.00	0.72	0.80	0.00

Appendix

Table 31: What the eruption needs. The share of spiking census images whose spike token still spikes under one intervention. Columns: images; the spike token’s head-mean attention to the text spike token, averaged over the blocks up to the text switch and over those after it up to

S

, median over images; its cosine with the trigger at the input the switch’s feed-forward reads, clean and with that attention blocked, median over images; the spike token denied the text spike token from the text switch to

S

; that token’s massive entries zeroed; the spike token denied, at

S-1

and

S

, the prompt position and five image tokens whose values aim most along the trigger; as many random positions; and the trigger component removed from its feed-forward writes, its attention writes, and both. n/a: Pixtral-12B’s visual switch is its text switch.

model	images	clean	denied	values read	writes	state	still spikes
		attention on image tokens
Qwen3-VL-4B	60	0.26	0.93	3.55	1.50	0.65	0.05
Qwen3-VL-8B	34	0.34	0.94	3.68	1.53	0.70	0.00
Qwen3-VL-32B	60	0.20	0.49	2.74	2.25	0.55	0.00
InternVL3.5-4B	50	0.23	0.92	3.73	1.70	0.63	0.00
InternVL3.5-8B	60	0.24	0.91	3.98	1.53	0.71	0.25

Appendix

Table 32: Without the text spike token, the eventual spike token reads larger values and loses its state. The eventual spike token denied the text spike token from the block after the text switch to the visual switch

S

, on spiking census images, medians over images. Columns: images; its head-mean attention on image tokens at the first denied block, clean and denied; the attention-weighted norm of the values it reads there over the text spike token’s, denied; its attention writes over the denied blocks, denied over clean; its state entering

S

, cosine with the clean state; the share of images on which it still spikes.

Figure 11: Moving where the spike lands. InternVL3.5-8B on three images whose clean spike token sits outside the outlined quadrant. Columns: the clean forward, then the quadrant’s tokens with their shared part cut to a tenth and with it quadrupled, the norm restored after each, and scaled to a tenth. The red square is the spike token, the amber square is the clean one, and the outline is the target quadrant.

		$p$ down	$p$ up	scaled
model	clean	norm held	norm held	to a tenth
Qwen3-VL-4B	0.17 (116)	0.28 (125)	0.15 (115)	0.56 (140)
Qwen3-VL-8B	0.11 (9)	0.30 (10)	0.40 (10)	0.79 (28)
Qwen3-VL-32B	0.11 (57)	0.10 (58)	0.35 (71)	0.40 (67)
Qwen2.5-VL-32B	0.11 (72)	0.68 (109)	0.00 (64)	0.00 (64)
InternVL3.5-4B	0.19 (27)	0.29 (31)	0.05 (22)	0.35 (23)

Appendix

Table 33: The relocation interventions. Per model, 150 images and one quadrant per image, the share of spiking images whose spike token lies in that quadrant, with the number of spiking images in parentheses. The quadrant is the one whose tokens have the largest median norm on the clean image, so the clean share lies under the

25\%

of a uniform landing. Columns: the clean forward, then the quadrant’s shared part cut to a tenth and quadrupled, the norm restored after each, and its tokens scaled to a tenth. Read under

\Psi

with the floor, tokens entering at

100

or above excluded.

Figure 12: The spike can be placed in a tenth of the image, or in one token. Left, InternVL3.5-8B on three images whose clean spike sits outside the block. Columns: a block of 1 token, $2\times 2$ , $3\times 3$ and $5\times 5$ of a $16\times 16$ grid, scaled to a tenth at the input. Red is the spike token, amber the clean one, grey a peak without a spike. Right, the share of spiking images whose spike lands in the block, against the block’s share of the tokens.

model	1	9	16	25	36	64
Qwen3-VL-4B	0.03	0.08	0.32	0.38	0.44	0.56
Qwen3-VL-8B	0.00	0.13	0.18	0.33	0.43	0.79
Qwen3-VL-32B	0.00	0.02	0.11	0.14	0.20	0.40
Qwen2.5-VL-32B	0.00	0.00	0.00	0.00	0.00	0.00
InternVL3.5-4B	0.00	0.00	0.03	0.18	0.20	0.35
InternVL3.5-8B	0.05	0.22	0.41	0.47	0.61	0.72

Appendix

Table 34: The spike comes to the block in proportion to its size. Per model, 150 images, a

k\times k

block around the centre of the target quadrant scaled to a tenth at the language model’s input, and the share of spiking images whose spike lands inside it. Columns: the block’s size in tokens. The last row is the share a token landing at random gives on the

16\times 16

grid of seven of the models. The grid is

18\times 18

in Qwen2.5-VL-32B and

32\times 32

in Pixtral-12B, where the largest block covers

20\%

and

6\%

of the tokens. † At most eleven spiking images per size and none at one token, indicative. Read under

\Psi

with the floor, tokens entering at

100

or above excluded.

Figure 13: The place is ours to choose. InternVL3.5-8B on two images, the same block of $25$ tokens moved to the centre of each quadrant, to the image’s centre, and to a position drawn at random, scaled to a tenth at the input. Red is the spike token, amber the clean one, and the outline the block. Right, the share of spiking images whose spike lands in the block at each position, over 150 images, the dotted line a token landing at random.

model	grid	top left	top right	bottom left	bottom right	centre	random	at random
Qwen3-VL-4B	$16\times 16$	0.64 (139)	0.60 (136)	0.18 (120)	0.16 (118)	0.17 (121)	0.33 (125)	0.098
Qwen3-VL-8B	$16\times 16$	0.68 (22)	0.67 (27)	0.20 (10)	0.25 (12)	0.44 (16)	0.21 (14)	0.098
Qwen3-VL-32B	$16\times 16$	0.46 (65)	0.19 (58)	0.09 (55)	0.16 (62)	0.28 (67)	0.22 (58)	0.098
Qwen2.5-VL-32B	$18\times 18$	0.00 (72)	0.00 (72)	0.00 (72)	0.00 (72)	0.00 (72)	0.00 (72)	0.077
InternVL3.5-4B	$16\times 16$	0.00 (26)	0.07 (28)	0.22 (36)	0.21 (34)	0.00 (26)	0.08 (26)	0.098
InternVL3.5-8B	$16\times 16$	0.68 (127)	0.66 (122)	0.34 (99)	0.40 (107)	0.39 (99)	0.50 (115)	0.098

Appendix

Table 35: Where the block moves the spike, its place can be chosen. Per model, 150 images, a

5\times 5

block of image tokens scaled to a tenth at the language model’s input and placed at the centre of each quadrant, the grid’s centre or a random position: the share of spiking forwards whose spike token lies inside the block, with their number in parentheses, and the share a token landing at random gives. Read under

\Psi

with the floor, tokens entering at

100

or above excluded. Molmo2-8B has no row, its overlapping crops having no single token grid.

Figure 14: The corruption sweep on the same 200 forwards per model. Left, the share of forwards that spike, clean and at the five noise grades. Middle, the forwards that spike clean and not under the noise. Right, the forwards that spike under the noise and not clean. Eight spiking models rise overall with the grade, Molmo2-8B and Pixtral-12B sit at every forward, and the fifteen non-spiking models at none. Table 36 carries the numbers.

	spikes
LVLM	clean	$\sigma\,0.05$	$0.10$	$0.19$	$0.30$	$0.45$	accuracy, clean to $0.45$
Qwen3-VL-4B	71.0%	82.0%	82.5%	84.0%	91.5%	96.0%	90.5% to 79.0%
Qwen3-VL-8B	8.0%	14.5%	17.0%	25.0%	50.5%	76.5%	89.0% to 78.0%
Qwen3-VL-32B	35.0%	39.0%	32.0%	34.5%	53.0%	78.5%	89.5% to 78.0%
Qwen2.5-VL-32B	49.5%	66.0%	79.5%	86.0%	90.5%	96.0%	87.0% to 69.5%
InternVL3.5-4B	16.0%	19.5%	20.5%	15.5%	18.5%	54.5%	84.0% to 79.0%

Appendix

Table 36: Destroying the image raises the incidence in every spiking LVLM below saturation and creates it in none of the others. Per LVLM, share of the same 200 POPE forwards that spike, clean and under Gaussian noise of standard deviation

\sigma

on the

0

1

image scale, then POPE accuracy clean and at

\sigma=0.45

. Read on the 200 POPE questions of the evaluation manifest, not the 300-image census, so clean shares differ from Table 3 by a few points. The six models of Appendix G close each group. The separate run of Table 37 has clean shares within six points of these.

model	operator	clean	corrupted	gained	lost	moved	unchanged	clean	corrupted
		spikes		questions		spike	answer	accuracy
Qwen3-VL-4B	Gaussian noise, $\sigma\,0.45$	0.68	0.97	61	3	0.99	0.83	0.915	0.795
	shot noise	0.68	0.96	57	2	0.97	0.87	0.915	0.810
	Gaussian blur	0.68	0.97	60	3	0.98	0.91	0.915	0.855
	JPEG	0.68	0.83	41	12	0.91	0.93	0.915	0.870
	brightness	0.68	0.79	37	16	0.85	0.96	0.915	0.895

Appendix

Table 37: Six kinds of corruption all switch the state. Per model and operator, the POPE manifest’s 200 questions at one grade each, read as in Table 36 on a separate run. Columns: the share of forwards that spike clean and corrupted, the number of questions that gain and lose the state, the share of the forwards spiking in both whose spike sits on a different image token, the share whose yes-or-no decision is unchanged, and accuracy clean and corrupted.

Figure 15: Two causes, three effects. Qwen3-VL-4B. Top row, the trigger-guided attack at one pixel level, which the page cannot show. Bottom row, a common corruption at the grade of Table 37 . Columns: the state created where the clean image had none, suppressed where it stood, and relocated to another token. The red square is the token carrying the spike after the change, and the amber square the token that carried it on the clean image. Under each panel, the criterion’s peak before and after.

LVLM	budget	create	abolish	no change	median KL	median $L_{\infty}$
Qwen3-VL-4B	$\nicefrac{{1}}{{255}}$	100% (40)	95% (40)	99% (80)	1e-6	1 / 1
Qwen3-VL-4B	$\nicefrac{{2}}{{255}}$	100% (40)	93% (40)	100% (80)	3e-6	1 / 2
Qwen3-VL-8B	$\nicefrac{{1}}{{255}}$	95% (40)	100% (16)	93% (56)	3e-4	1 / 1
Qwen3-VL-8B	$\nicefrac{{2}}{{255}}$	93% (40)	100% (16)	95% (56)	3e-4	2 / 1
Qwen3-VL-32B	$\nicefrac{{1}}{{255}}$	100% (40)	100% (40)	100% (80)	1e-2	1 / 1
Qwen3-VL-32B	$\nicefrac{{2}}{{255}}$	100% (40)	100% (40)	99% (80)	1e-2	2 / 1

Appendix

Table 38: One pixel level over the image already switches the massive state. Per LVLM and budget, 40 POPE images a direction. Create and abolish are success rates over the eligible images in parentheses, no change the share of answers that hold, then median KL and

L_{\infty}

. Up to 300 steps of

\nicefrac{{\varepsilon}}{{4}}

, of

\nicefrac{{0.25}}{{255}}

for InternVL3.5-38B, whose cells count each image at the smallest budget where a run succeeds. Molmo2-8B and Pixtral-12B are read on the tensor,

\ast

after 60 steps and unstarred 600, others on the saved image.

			VQAv2 accuracy		captioning CIDEr
model	budget	$n$	clean	attacked	clean	attacked
Qwen3-VL-4B	$\nicefrac{{1}}{{255}}$	264, 264	72.0%	70.4%	0.191	0.184
Qwen3-VL-8B	$\nicefrac{{1}}{{255}}$	277, 277	73.9%	70.9%	0.348	0.319
Qwen2.5-VL-32B	$\nicefrac{{1}}{{255}}$	293, 293	72.6%	70.1%	0.232	0.200
Qwen3-VL-32B	$\nicefrac{{1}}{{255}}$	258, 258	75.1%	75.7%	0.143	0.136
InternVL3.5-4B	$\nicefrac{{1}}{{255}}$	151, 151	63.8%	58.9%	0.588	0.578

Appendix

Table 39: On the free-form tasks the attack costs the answer a few points. The eight Qwen and InternVL3.5 models at one pixel level in both directions, and Pixtral-12B below the rule on the abolishing side only, under a cap of four pixel levels. Each image is attacked once, with the VQAv2 prompt, and both tasks are scored on that image. Pixtral-12B’s captions come from their own attack at the same budget. Columns: the budget, the

n

VQAv2 questions and captions attacked, the pool less the forwards whose clean side moved, then the clean and the attacked score. Qwen3-VL-8B’s abolishing side covers 23 of its 32 spiking questions. Every attempt is scored. The attack succeeds on

90

100\%

of attempts in seven models and on

66

of InternVL3.5-38B’s

101

. VQAv2 is the official accuracy, leave one annotator out, and captioning is corpus CIDEr-D against COCO’s five references.

model	clean	nulled	clean	nulled	clean	nulled	accuracy	direction	modified	clean to nulled
	spikes		attention sink		noised spikes			random	tokens	peak $\Psi$
Qwen3-VL-4B	0.71	0.00	0.68	0.00	0.85	0.00	0.910	0.71	1	16,343 to 90
Qwen3-VL-8B	0.09	0.00	0.08	0.00	0.27	0.00	0.885	0.09	1	9,376 to 120
Qwen3-VL-32B	0.35	0.00	0.38	0.00	0.34	0.00	0.890	0.35	1	16,024 to 495
Qwen2.5-VL-32B	0.49	0.00	0.49	0.00	0.95	0.00	0.870	0.49	1	4,004 to 208
InternVL3.5-4B	0.16	0.03	0.14	0.00	0.14	0.03	0.845	0.16	2	10,372 to 208

Appendix

Table 40: Removing the trigger component prevents the eruption. Per model, 200 evaluation questions. Columns: share spiking and share reaching the attention-sink threshold, each without and with the intervention, and again under Gaussian noise at

\sigma=0.15

; clean accuracy, and the intervened one where it differs; spike rate under a random direction of the same rank; image tokens the conditional form modifies; median peak

\Psi

clean and under it, over questions that spike clean. Each model is run with the trigger of its own base.

	spikes, VQAv2		spikes, captions		VQA accuracy		CIDEr		answer	caption unchanged
model	clean	nulled	clean	nulled	clean	nulled	clean	nulled	unchanged	nulled	random
Qwen3-VL-4B	0.79	0.00	0.79	0.00	0.729	0.723	0.191	0.194	0.99	0.79	0.82
Qwen3-VL-8B	0.11	0.00	0.12	0.00	0.736	0.737	0.361	0.369	0.99	0.81	0.86
Qwen3-VL-32B	0.41	0.00	0.41	0.00	0.741	0.741	0.145	0.145	1.00	0.87	0.88
Qwen2.5-VL-32B	0.56	0.00	0.56	0.00	0.720	0.720	0.234	0.227	0.98	0.79	0.81
InternVL3.5-4B	0.16	0.04	0.16	0.04	0.655	0.651	0.588	0.598	0.97	0.86	0.89

Appendix

Table 41: The intervention holds on free-form answers. Per model, 297 VQAv2 questions and 300 COCO captions, greedy decoding, with Equation 3 applied to every image token. Columns: the share of forwards that spike without and with the intervention on each task, VQA accuracy, corpus CIDEr against COCO’s references, the share of VQAv2 answers whose text is identical to the original up to case, punctuation and articles, and the same share for captions under the intervention and under a random direction of the same rank.

		spikes		accuracy
model	image	plain	interv.	plain	interv.
Qwen3-VL-4B	clean	0.58	0.00	0.86	0.86
	Gaussian noise	0.98	0.00	0.80	0.80
	shot noise	0.94	0.00	0.82	0.82
	Gaussian blur	0.94	0.00	0.90	0.90
	JPEG	0.76	0.00	0.84	0.84

Appendix

Table 42: The intervention holds under corruption. Ten models, 50 evaluation questions each, the six operators of Table 37 at the same grades. Columns: the share of images that spike without and with the intervention, then accuracy without and with it. Read under

\Psi

with the floor, tokens entering at

100

or above excluded.

		spikes			accuracy			peak $\Psi$
model	$n$	clean	attacked	intervened	clean	attacked	intervened	attacked	intervened
Qwen3-VL-4B	40	0.00	0.95	0.00	0.93	0.90	0.90	19,626	100
Qwen3-VL-8B	40	0.03	0.83	0.00	0.88	0.78	0.78	23,121	156
Qwen3-VL-32B	40	0.10	0.93	0.00	0.80	0.80	0.80	21,749	691
Qwen2.5-VL-32B	40	0.10	1.00	0.00	0.88	0.88	0.88	5,542	304
InternVL3.5-4B	40	0.00	0.80	0.28	0.88	0.88	0.88	23,740	2,009

Appendix

Table 43: The intervention holds on the attack’s own images. Every image on which the creating attack succeeded at one pixel level, read from the 8-bit file its success was judged on. Molmo2-8B and Pixtral-12B spike on nearly every clean image, so the attack has no spike to create in them. Columns: the images, the share that spike clean, under the attack, and under the attack with Equation 3 in place, then the accuracy of each, then the largest criterion value over the images under the attack and under the intervention. Read under

\Psi

with the floor, tokens entering at

100

or above excluded.

Large Vision-Language Models

Large vision-language models (LVLMs) connect image and video perception with language generation, reasoning, and interactive tasks.

See all 404 papers

May 20, 2026

Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating

Large vision-language models (LVLMs) often hallucinate content that is fluent yet unsupported by the image, limiting their reliability in real-world deployment. We show that a key failure mode arises from route competition: even when visual tokens receive attention, the final token decision can be dominated by the textual pathway, causing the decoder to follow linguistic priors over visual evidence. To mitigate this, we propose a training-free, decision-aligned intervention that decomposes each attention head into a visual route and a text route, and estimates their token-level effects using an efficient one-forward/one-gradient approximation. These estimates reveal route conflict within heads and identify prior-dominant ones, enabling selective suppression of only the text route while keeping the visual route intact. Across five benchmarks spanning discriminative and generative settings, our method consistently reduces hallucination-related errors across models with limited impact on overall multimodal performance, while incurring a modest inference-time overhead.

Zhe Cheng, Wenyu Chen, Fode Zhang +1

Center of Statistical Research, School of Statistics and Data Science, Southwestern University of Finance and Economics, Chengdu, China. · Department of Biomedical Engineering, College of Design and Engineering, National University of Singapore, Singapore.

VLM Robustness Hallucination in Language Models VLM Hallucination Large Vision-Language Models Faithfulness in VLMs VLM Hallucination Mitigation

Ask PDF

Oct 11, 2026

Attention Sinks in VLMs Are Borrowed, Not Seen

We study how to detect and mitigate visual attention sinks in vision-language models (VLMs). Visual attention sinks are image tokens that absorb a large share of attention over the image regardless of the query or task, and they mislead methods that treat attention maps as a record of where the model ``looked.'' Prior work has detected sinks through massive activation dimensions and proposed redistributing their attention, but none have addressed what makes a token become a sink. Comparing six candidate detectors on ten VLMs from four families, we find that only the centered cosine similarity between an image token's key and the key of the language model's text-sink detects visual sinks. Thus, a visual sink is an image token whose key aligns with the text-sink's key. Moving a sink's key component along this direction to the median of the image keys sends its attention back to the text-sink (Are Borrowed) while leaving model answers unchanged~(Not Seen). Building on this, we propose a training-free, entropy-triggered re-look that first cleans the attention map, then crops the most-attended region of the cleaned map and inserts it into the reasoning trace. Cleaning adds negligible computation to the forward pass, and the re-look improves accuracy with a mean gain of

+4.3

points on fine-detail VQA benchmarks across model families.

Dengjia Zhang, Alexander Martin, Pengyu Yan +6

Johns Hopkins University · University of Pittsburgh · University at Buffalo +2

Attention Intervention in VLMs VLM Interpretability Faithfulness in VLMs

Ask PDF

Dec 11, 2024

Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models

Large Vision-Language Models (LVLMs) have achieved remarkable performance on multimodal tasks but remain highly vulnerable to small adversarial perturbations in input images. Existing attacks typically target the vision encoder's final output embeddings, implicitly treating the encoder as a uniform attack surface, while a systematic analysis of which internal components are most vulnerable has remained largely unexplored. We show such analysis is essential, as adversarial vulnerability in LVLM vision encoders is structurally concentrated rather than uniformly distributed. Building on this, we propose Vision Encoder Vulnerable-Component-Targeted Universal Adversarial Perturbation (VEV-UAP), a task-agnostic and cost-efficient attack framework. Through a component- and layer-wise analysis of attention mechanisms, we identify the value components in middle layers as critical vulnerabilities that strongly influence downstream language model behavior. VEV-UAP selectively targets these components to generate a single universal perturbation shared across images, without involving textual inputs or the language model during optimization. Experiments across multiple LVLMs and tasks show VEV-UAP achieves state-of-the-art attack success rates with reduced computational overhead. Moreover, a single VEV-UAP transfers across LVLMs sharing the same vision encoder, even when paired with different language models, making it a practical framework for scalable robustness evaluation.

Hee-Seon Kim, Minbeom Kim, Seokil Ham +1

School of Electrical Engineering, KAIST, Daejeon 34141, Republic of Korea

VLM Robustness Adversarial Attacks on VLMs Large Vision-Language Models Adversarial Perturbations Transfer-Based Adversarial Attacks

Ask PDF

Abstract

Large Vision-Language Models

Explore similar work

Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating

Attention Sinks in VLMs Are Borrowed, Not Seen

Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models