Taking Shortcuts for Categorical VQA Using Super Neurons

Pierre Musacchio; Jaeyi Jeong; Dahun Kim; Jaesik Park

cs.CVMar 11, 2026

Taking Shortcuts for Categorical VQA Using Super Neurons

Authors: Pierre Musacchio, Jaeyi Jeong, Dahun Kim, Jaesik Park

Organizations: Seoul National University · EPFL · Google Deepmind

Abstract

Sparse Attention Vectors (SAVs) have emerged as an excellent training-free alternative to supervised finetuning or low-rank adaptation to improve the performance of Vision Language Models (VLMs). At their heart, SAVs select a few accurate attention heads for a task of interest and use them as classifiers, rather than relying on the model's prediction. In a similar spirit, we find that directly probing the raw activations of the VLM, in the form of scalar values, is sufficient to yield accurate classifiers on diverse visually grounded downstream tasks. Shifting focus from attention vectors to scalar activations dramatically increases the search space for accurate parameters, allowing us to find more discriminative neurons immediately from the first generated token. We call such activations Super Neurons (SNs). In this probing setting, we discover that enough SNs appear in the shallower layers of the large language model to allow for extreme early exiting from the first layer of the model at the first generated token. Compared to the original network, SNs robustly improve the classification performance while achieving a speedup of up to 5.10x.

Figures & tables

Figure 1: Overview of our approach. Our training-free scheme uncovers Super Neurons (SNs) via probing data. They robustly outperform the base model on a variety of categorical VQA datasets. As a byproduct, they enable extreme early exiting at the first layer of the LLM on the first generated token. Colored boxes in the LLM represent SNs for their data types.

Panel of Figure 2 — Figure 2: Comparison between SAVs and ours. We show architectural divergences in Fig. 2(a) and resulting search spaces of the two approaches for LLaVA-v1.5-7b in Fig. 2(b) .

Panel of Figure 3 — Figure 3: Empirical analysis of SNs. Figure 3(a) shows the maximum accuracy on the probing set with respect to different $\alpha$ . We evaluate $\alpha$ over the range $\alpha\in[-3,3]$ . The maximum accuracy peaks around $\alpha=0$ . Figure 3(b) records the number of found SNs that obtain a better accuracy than the model in each layer. We use LLaVA-v1.5-7b on VizWiz for both figures.

		Dataset
Metric	Method	Pope	InstaOrder (Depth)	InstaOrder (Occ.)	VizWiz	Clevr	A-OKVQA	ScienceQA
	LLaVA-v1.5-7b	90.7	61.7	53.9	64.8	52.3	67.8	62.2
	SN	92.5 (+1.8)	63.5 (+1.8)	62.7 (+8.8)	71.9 (+7.1)	54.4 (+2.1)	68.0 (+0.2)	63.8 (+1.6)
	Qwen3-VL-4b-Instruct	95.0	61.8	50.8	78.3	84.7	82.7	81.3
Accuracy	SN	96.1 (+1.1)	62.8 (+1.0)	62.6 (+11.8)	81.0 (+2.7)	88.6 (+3.9)	83.0 (+0.3)	82.3 (+1.0)
	LLaVA-v1.5-7b	91.1	67.7	46.1	72.0	61.7	74.5	70.0

Table 1: Best probed SN on diverse categorical VQA datasets. We report accuracy and F1. We display the results obtained by the single best performing SN in the model. Best results are in bold . Baselines are highlighted in gray . We use a probing set size of 3,000 samples except for VizWiz , which only contains 942 binary questions. Accuracy and F1 are optimized separately. SNs outperform the base models in all cases.

		Dataset
Metric	Method	Pope	InstaOrder (Depth)	InstaOrder (Occ.)	VizWiz	Clevr	A-OKVQA	ScienceQA
	SNs LLaVA-v1.5-7b	92%	63%	62%	71%	53%	67%	63%
SN $t$	SNs Qwen3-VL-4b-Instruct	95.8%	62.6%	62.0%	80.9%	88.3%	80.0%	82.0%
	LLaVA-v1.5-7b	89.8	65.0	64.9	65.6	51.3	54.8	53.5
	SNs (mean)	90.9 (+1.1)	66.1 (+1.1)	64.5 (-0.4)	72.6 (+7.0)	51.5 (+0.2)	61.4 (+6.6)	57.4 (+6.9)
	SNs (maj. voting)	90.9 (+1.1)	65.2 (+0.2)	78.2 (+13.3)	69.9 (+4.3)	51.4 (+0.1)	72.0 (+17.2)	48.4 (-5.1)

Table 2: Evaluation of probed SNs on diverse categorical VQA validation datasets. We report both the mean of SNs and the majority voting with respect to the SN

t

found from the probing data. All SNs were optimized for accuracy on the probing set. For a given metric, best results are in bold and second best are underlined . Baselines are highlighted in gray .

Figure 4: Agreement rate with respect to different SN

t

. We compute AR on Pope using LLaVA-v1.5-7b. At lower accuracy, SNs largely agree with the model prediction. However, for SNs to obtain better results than the model, they have to disagree on some answers.

Method	$N$	Acc.	Prec.	Recall	F1
SAVs [ 27 ]	40	51.8	50.6	95.7	66.1
SNs (ours)	40	60.2	56.8	84.9	68.1

Table 3: Comparison of SNs and SAVs. We benchmark LLaVA-v1.5-7b on VizWiz using the same number of probing samples

N

. Best performing method is in yellow .

Table 4: Performances of SNs for scaled-up models. After probing SNs following our method, we report the validation results of LLaVA-v1.5-13b and Qwen3-VL-32b-Instruct on Pope and ScienceQa respectively. Best performances for a given optimized metric are in bold, baseline performance is in gray .

Figure 5: Super Neurons performances with respect to different probing set sizes. We compute accuracy, precision, recall and F1 on diverse benchmarks using LLaVA-v1.5-7b. Dashed lines (colors matching their respective datasets) indicate the performance of the vanilla model. Overall, more data leads to performance improvements.

Inference Strategy		Autoreg.	$L^{\star}$	Acc.	F1	Runtime (s.)
Model	LLaVA-v1.5-7b	✓	32	89.8	90.2	0.78
	$\text{SN$ t ${}}=0.92$ (maj. vote)	✗	15	90.9	91.2	0.16 (-4.81 $\times$ )
	$\text{SN$ t ${}}=0.91$ (mean)	✗	12	90.6	90.8	0.16 (-4.87 $\times$ )
SNs	$\text{SN$ t ${}}=0.90$ (maj. vote)	✗	1	89.8	90.2	0.15 (-5.10 $\times$ )

Table 5: Runtime benchmark on Pope . We benchmark the average runtime of LLaVA-v1.5-7b on the full validation set using its official inference code .

L^{\star}

indicates the inference exit layer. All numbers are recorded using a single NVIDIA RTX A6000. Best performances are in bold . Baseline is in gray , and best runtime is in yellow .

Token Pos.	$L^{\star}$	Acc.	Prec.	Recall	F1	Runtime (s.)
Last	14	90.9	92.6	89.5	91.0	0.76
First	15	90.9	91.0	91.4	91.2	0.16 (-4.75 $\times$ )

Table 6: Impact of token selection for SNs discovery on Pope . We use LLaVA-v1.5-7b.

L^{\star}

is the first possible early stopping layer. Best performing method is highlighted in yellow . Early exiting after the first token improves runtime and performances.

Method	Probe	Val.	Acc.	Prec.	Recall	F1
LLaVA-v1.5-7b	–	Voc	91.8	88.1	96.7	92.2
SNs	Voc	Voc	91.9	88.1	96.8	92.3
SNs	Coco	Voc	91.6	87.2	97.6	92.1

Table 7: Transfer capabilities of SNs to a novel distribution. We use the Pope format.

Figure 6: Qualitative results of SNs. We show examples on diverse validation sets. We use LLaVA-v1.5-7b on Pope and InstaOrder (Occ.) and Qwen3-VL-4b-Instruct on InstaOrder (Depth) and A-OKVQA. The robot icon symbolizes the model output. Green indicate that the answer matches the ground-truth and red indicates an incorrect answer.

Num. shots		0	1	3	5	SNs
	Acc.	89.8	82.3	63.0	60.8	90.9
	Prec.	89.5	84.0	84.0	81.6	91.0
	Recall	90.8	81.3	81.3	31.0	91.4
Pope	F1	90.2	83.6	82.6	44.9	91.2
	Acc.	65.0	60.7	58.9	58.9	65.2
	Prec.	66.4	60.3	58.6	58.7	66.4

Table 8: Comparing

n

-shot prompting and SNs. We benchmark LLaVA-v1.5-7b. SNs are consistently more accurate.

		Metric optimization ( $\mu$ )
Metric	Method	Accuracy	F1
	SNs (mean)	64.5	25.6
Accuracy	SNs (maj. voting)	78.2	67.0
	SNs (mean)	23.5	16.4
Precision	SNs (maj. voting)	30.4	24.6
	SNs (mean)	61.8	98.4

Table 9: Impact of metric optimization for SNs discovery. We use LLaVA-v1.5-7b on InstOrder (Occ.) . Willingly optimizing for a given metric

\mu

results in significant performance improvements in the subsequent measure.

Dataset	Ref.	Prompt
Pope	[ 21 ]	Is there a {obj} in the image?
InstaOrder (Occ.)	[ 28 ]	Is the {obj} at position {loc} obstructing the {obj} at position {loc}? Answer the question using a single word.
InstaOrder (Depth.)	[ 28 ]	Is the {obj} in front of the {obj} at position {loc} from the viewer’s perspective? Answer the question using a single word.
A-OKVQA	[ 34 ]	{q} Is it { $a_{i}$ }? Answer with yes or no.
ScienceQa	[ 26 ]	{q} Is it { $a_{i}$ }? Answer with yes or no.

Table 10: Prompt templates. We detail the templates used for some of the VQA datasets. {} indicates variables, obj indicates the object category, loc indicates the location of the object in a bounding box format and. For MCQ data, q indicates the original question and

a_{i}

indicates the

i

-th MCQ answer.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Table 11: Default configurations of the LLaVA-v1.5-7b and Qwen3-VL-4b-Instruct we use for all the experiments throughout our work.

				Occlusion performances				Depth performances
Method	$N$	Training time	Data generalization	Acc.	Precision	Recall	F1	Acc.	Precision	Recall	F1
LLaVA-v1.5-7b	753k	8 h. [ 23 ]	✗	64.9	18.5	40.2	25.3	65.0	66.4	80.2	72.6
LLaVA-v1.5-7b -sft	3k	2m.	✗	14.8	14.8	100.0	25.8	57.9	57.9	100.0	73.4
LLaVA-v1.5-7b -sft	10k	5m.	✗	82.1	43.5	71.2	54.0	75.1	83.1	71.5	76.9
LLaVA-v1.5-7b -sft	100k	49m.	✗	83.7	47.1	81.8	59.8	79.8	83.8	80.8	82.3
LLaVA-v1.5-7b -LoRA	3k	2m.	✗	67.5	23.9	54.7	33.3	55.4	58.8	76.4	66.5

Appendix

Table 12: SNs compared to finetuned LLaVA alternatives on InstaOrder . Given an annotation budget

N

, we compare both training time and task performances. We use the pretrained model of [ 23 ] as our baseline and their official finetuning script without altering any of the hyperparameters. For our approach, we report the results from the method obtaining the best F1 based on Tab. 2 . Best results are in bold . Baseline is highlighted in gray , while our super neurons are in yellow .

Model	Max new toks.	Embedding	Prefill	Decoding	Wall time
LLaVA-v1.5-7b	128	0.032	0.085	0.025	1.01
LLaVA-v1.5-7b	1	0.032	0.086	0.024	0.223
SNs	1	0.032	0.085	0.002	0.119 (-1.9 $\times$ )

Appendix

Table 13: Profiling LLaVA-v1.5-7b on an NVIDIA A100 GPU using the huggingface’s inference routine (in s.). Ours is benchmarked on the first layer of the LLM.

Comparative word		obstructing		hiding		occluding		Random
Split	Metric	Base	SNs	Base	SNs	Base	SNs	Base	SNs
Probe	Acc.	53.9	62.7	45.6	62.6	48.8	64.2	49.5	60.0
Probe	F1	46.1	69.0	61.6	67.9	64.3	67.6	65.0	67.0
Validation	Acc.	64.9	78.2	16.2	68.1	18.3	66.8	17.9	66.9
	Prec.	18.5	30.4	13.9	24.2	14.6	23.8	14.7	22.2
	Recall	40.2	36.5	89.8	54.1	92.8	56.5	94.3	49.2

Appendix

Table 14: Prompt robustness. We use LLaVA-v1.5-7b on InstaOrder (Occ.) . We replace the comparative word in the prompt template with alternative words and evaluate compare the base model performances to the SNs performances.

		Dataset
Metric	Method	Pope - Im	Pope - Txt	Pope
	SNs (mean)	60.4	60.4	90.9
Accuracy	SNs (maj. voting)	60.3	61.6	90.9
	SNs (mean)	71.2	71.4	92.6
Precision	SNs (maj. voting)	68.2	69.9	91.0
	SNs (mean)	38.9	38.9	89.5

Appendix

Table 15: Adversarial prompting. We evaluate LLaVA-v1.5-7b on adversarial Pope - Im and Pope - Txt validation datasets containing shuffled VQA pairs. SNs results on the original dataset is highlighted in gray .

Panel of Figure 7 — Figure 7: Visualization of SNs on different datasets. We use LLaVA-v1.5-7b. We only visualize SNs that obtain better performances than the model itself. Few SNs emerge on the datasets that are already very well answered by the model, while we observe a significant amount of SNs on harder datasets.

Panel of Figure 8 — Figure 8: SNs overlap between different datasets. We visualize the probed neurons that surpass the model’s performance on all the indicated datasets. We use LLaVA-v1.5-7b. $x$ -axis indicate neuron index while $y$ -axis indicates layer index. Depending on the dataset pair, many SNs can overlap. We also find a quadruplet of datasets sharing SNs.

Explore similar work

ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Models
Jun 28, 2026Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen +2Visual Spatial Reasoning VLM Adaptation
Northeastern University, Boston, USA · EmbodyX Inc., USA · Zhejiang University, China
NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning
Sep 29, 2026Ruiyu Yan, Bowen Chen, Shaowen Wan +1VLM Interpretability VLM Reasoning
New York University · New Jersey Institute of Technology
AVIS: Adaptive Test-Time Scaling for Vision-Language Models
Jun 10, 2026Ahmadreza Jeddi, Minh Ngoc Le, Amirhossein Kazerouni +8Test-Time Adaptation of VLMs Efficient VLM Inference
AI Center-Toronto, Samsung Electronics · University of Toronto · Vector Institute +1

Jun 28, 2026J/K move · Enter open · S save

ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Models

Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen, Pu Zhao+1

Northeastern University, Boston, USA · EmbodyX Inc., USA · Zhejiang University, China

Visual Spatial Reasoning VLM Adaptation VLM Reasoning Spatial Reasoning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Spatial reasoning remains a persistent challenge for many vision language models (VLMs), and improving it typically requires fine-tuning with substantial additional parameters. Our preliminary analysis reveals that rescaling activations in selected transformer layers-without modifying pretrained weights-can significantly influence downstream performance. Motivated by this observation, we propose ScAle, an ultra-lightweight adaptation method that learns a small set of scalar coefficients to modulate last-token attention and MLP activations in a fully frozen backbone. We evaluate our method on the synthetic spatial reasoning benchmark SpatialEval and on real-world VQA datasets (COCOQA and VGQA) across multiple model families. Our method, ScAle, achieves up to 134.1% relative accuracy gains using only 1K trainable parameters without requiring millions of trainable parameters as in standard PEFT methods such as LoRA. Despite its extreme compactness, our approach recovers a substantial fraction of standard PEFT performance while preserving strong non-spatial VQA accuracy. These results demonstrate that bounded activation reweighting provides a simple, architecture-agnostic, and highly parameter-efficient alternative for adapting pretrained VLMs.

Abstract

Explore similar work

ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Models

NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning

AVIS: Adaptive Test-Time Scaling for Vision-Language Models