Sparse Attention Vectors (SAVs) have emerged as an excellent training-free alternative to supervised finetuning or low-rank adaptation to improve the performance of Vision Language Models (VLMs). At their heart, SAVs select a few accurate attention heads for a task of interest and use them as classifiers, rather than relying on the model's prediction. In a similar spirit, we find that directly probing the raw activations of the VLM, in the form of scalar values, is sufficient to yield accurate classifiers on diverse visually grounded downstream tasks. Shifting focus from attention vectors to scalar activations dramatically increases the search space for accurate parameters, allowing us to find more discriminative neurons immediately from the first generated token. We call such activations Super Neurons (SNs). In this probing setting, we discover that enough SNs appear in the shallower layers of the large language model to allow for extreme early exiting from the first layer of the model at the first generated token. Compared to the original network, SNs robustly improve the classification performance while achieving a speedup of up to 5.10x.
Figures & tables
Figure 1: Overview of our approach. Our training-free scheme uncovers Super Neurons (SNs) via probing data. They robustly outperform the base model on a variety of categorical VQA datasets. As a byproduct, they enable extreme early exiting at the first layer of the LLM on the first generated token. Colored boxes in the LLM represent SNs for their data types.
Figure 2: Comparison between SAVs and ours. We show architectural divergences in Fig. 2(a) and resulting search spaces of the two approaches for LLaVA-v1.5-7b in Fig. 2(b) .
Figure 3: Empirical analysis of SNs. Figure 3(a) shows the maximum accuracy on the probing set with respect to different α . We evaluate α over the range α∈[−3,3] . The maximum accuracy peaks around α=0 . Figure 3(b) records the number of found SNs that obtain a better accuracy than the model in each layer. We use LLaVA-v1.5-7b on VizWiz for both figures.
Dataset
Metric
Method
Pope
InstaOrder (Depth)
InstaOrder (Occ.)
VizWiz
Clevr
A-OKVQA
ScienceQA
LLaVA-v1.5-7b
90.7
61.7
53.9
64.8
52.3
67.8
62.2
SN
92.5 (+1.8)
63.5 (+1.8)
62.7 (+8.8)
71.9 (+7.1)
54.4 (+2.1)
68.0 (+0.2)
63.8 (+1.6)
Qwen3-VL-4b-Instruct
95.0
61.8
50.8
78.3
84.7
82.7
81.3
Accuracy
SN
96.1 (+1.1)
62.8 (+1.0)
62.6 (+11.8)
81.0 (+2.7)
88.6 (+3.9)
83.0 (+0.3)
82.3 (+1.0)
LLaVA-v1.5-7b
91.1
67.7
46.1
72.0
61.7
74.5
70.0
Table 1: Best probed SN on diverse categorical VQA datasets. We report accuracy and F1. We display the results obtained by the single best performing SN in the model. Best results are in bold . Baselines are highlighted in gray . We use a probing set size of 3,000 samples except for VizWiz , which only contains 942 binary questions. Accuracy and F1 are optimized separately. SNs outperform the base models in all cases.
Dataset
Metric
Method
Pope
InstaOrder (Depth)
InstaOrder (Occ.)
VizWiz
Clevr
A-OKVQA
ScienceQA
SNs LLaVA-v1.5-7b
92%
63%
62%
71%
53%
67%
63%
SN t
SNs Qwen3-VL-4b-Instruct
95.8%
62.6%
62.0%
80.9%
88.3%
80.0%
82.0%
LLaVA-v1.5-7b
89.8
65.0
64.9
65.6
51.3
54.8
53.5
SNs (mean)
90.9 (+1.1)
66.1 (+1.1)
64.5 (-0.4)
72.6 (+7.0)
51.5 (+0.2)
61.4 (+6.6)
57.4 (+6.9)
SNs (maj. voting)
90.9 (+1.1)
65.2 (+0.2)
78.2 (+13.3)
69.9 (+4.3)
51.4 (+0.1)
72.0 (+17.2)
48.4 (-5.1)
Table 2: Evaluation of probed SNs on diverse categorical VQA validation datasets. We report both the mean of SNs and the majority voting with respect to the SN t found from the probing data. All SNs were optimized for accuracy on the probing set. For a given metric, best results are in bold and second best are underlined . Baselines are highlighted in gray .
Figure 4: Agreement rate with respect to different SN t . We compute AR on Pope using LLaVA-v1.5-7b. At lower accuracy, SNs largely agree with the model prediction. However, for SNs to obtain better results than the model, they have to disagree on some answers.
Method
N
Acc.
Prec.
Recall
F1
SAVs [ 27 ]
40
51.8
50.6
95.7
66.1
SNs (ours)
40
60.2
56.8
84.9
68.1
Table 3: Comparison of SNs and SAVs. We benchmark LLaVA-v1.5-7b on VizWiz using the same number of probing samples N . Best performing method is in yellow .
Table 4: Performances of SNs for scaled-up models. After probing SNs following our method, we report the validation results of LLaVA-v1.5-13b and Qwen3-VL-32b-Instruct on Pope and ScienceQa respectively. Best performances for a given optimized metric are in bold, baseline performance is in gray .
Figure 5: Super Neurons performances with respect to different probing set sizes. We compute accuracy, precision, recall and F1 on diverse benchmarks using LLaVA-v1.5-7b. Dashed lines (colors matching their respective datasets) indicate the performance of the vanilla model. Overall, more data leads to performance improvements.
Inference Strategy
Autoreg.
L⋆
Acc.
F1
Runtime (s.)
Model
LLaVA-v1.5-7b
✓
32
89.8
90.2
0.78
\text{SNt{}}=0.92 (maj. vote)
✗
15
90.9
91.2
0.16 (-4.81 × )
\text{SNt{}}=0.91 (mean)
✗
12
90.6
90.8
0.16 (-4.87 × )
SNs
\text{SNt{}}=0.90 (maj. vote)
✗
1
89.8
90.2
0.15 (-5.10 × )
Table 5: Runtime benchmark on Pope . We benchmark the average runtime of LLaVA-v1.5-7b on the full validation set using its official inference code . L⋆ indicates the inference exit layer. All numbers are recorded using a single NVIDIA RTX A6000. Best performances are in bold . Baseline is in gray , and best runtime is in yellow .
Token Pos.
L⋆
Acc.
Prec.
Recall
F1
Runtime (s.)
Last
14
90.9
92.6
89.5
91.0
0.76
First
15
90.9
91.0
91.4
91.2
0.16 (-4.75 × )
Table 6: Impact of token selection for SNs discovery on Pope . We use LLaVA-v1.5-7b. L⋆ is the first possible early stopping layer. Best performing method is highlighted in yellow . Early exiting after the first token improves runtime and performances.
Method
Probe
Val.
Acc.
Prec.
Recall
F1
LLaVA-v1.5-7b
–
Voc
91.8
88.1
96.7
92.2
SNs
Voc
Voc
91.9
88.1
96.8
92.3
SNs
Coco
Voc
91.6
87.2
97.6
92.1
Table 7: Transfer capabilities of SNs to a novel distribution. We use the Pope format.
Figure 6: Qualitative results of SNs. We show examples on diverse validation sets. We use LLaVA-v1.5-7b on Pope and InstaOrder (Occ.) and Qwen3-VL-4b-Instruct on InstaOrder (Depth) and A-OKVQA. The robot icon symbolizes the model output. Green indicate that the answer matches the ground-truth and red indicates an incorrect answer.
Num. shots
0
1
3
5
SNs
Acc.
89.8
82.3
63.0
60.8
90.9
Prec.
89.5
84.0
84.0
81.6
91.0
Recall
90.8
81.3
81.3
31.0
91.4
Pope
F1
90.2
83.6
82.6
44.9
91.2
Acc.
65.0
60.7
58.9
58.9
65.2
Prec.
66.4
60.3
58.6
58.7
66.4
Table 8: Comparing n -shot prompting and SNs. We benchmark LLaVA-v1.5-7b. SNs are consistently more accurate.
Metric optimization ( μ )
Metric
Method
Accuracy
F1
SNs (mean)
64.5
25.6
Accuracy
SNs (maj. voting)
78.2
67.0
SNs (mean)
23.5
16.4
Precision
SNs (maj. voting)
30.4
24.6
SNs (mean)
61.8
98.4
Table 9: Impact of metric optimization for SNs discovery. We use LLaVA-v1.5-7b on InstOrder (Occ.) . Willingly optimizing for a given metric μ results in significant performance improvements in the subsequent measure.
Dataset
Ref.
Prompt
Pope
[ 21 ]
Is there a {obj} in the image?
InstaOrder (Occ.)
[ 28 ]
Is the {obj} at position {loc} obstructing the {obj} at position {loc}? Answer the question using a single word.
InstaOrder (Depth.)
[ 28 ]
Is the {obj} in front of the {obj} at position {loc} from the viewer’s perspective? Answer the question using a single word.
A-OKVQA
[ 34 ]
{q} Is it { ai }? Answer with yes or no.
ScienceQa
[ 26 ]
{q} Is it { ai }? Answer with yes or no.
Table 10: Prompt templates. We detail the templates used for some of the VQA datasets. {} indicates variables, obj indicates the object category, loc indicates the location of the object in a bounding box format and. For MCQ data, q indicates the original question and ai indicates the i -th MCQ answer.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Table 11: Default configurations of the LLaVA-v1.5-7b and Qwen3-VL-4b-Instruct we use for all the experiments throughout our work.
Occlusion performances
Depth performances
Method
N
Training time
Data generalization
Acc.
Precision
Recall
F1
Acc.
Precision
Recall
F1
LLaVA-v1.5-7b
753k
8 h. [ 23 ]
✗
64.9
18.5
40.2
25.3
65.0
66.4
80.2
72.6
LLaVA-v1.5-7b -sft
3k
2m.
✗
14.8
14.8
100.0
25.8
57.9
57.9
100.0
73.4
LLaVA-v1.5-7b -sft
10k
5m.
✗
82.1
43.5
71.2
54.0
75.1
83.1
71.5
76.9
LLaVA-v1.5-7b -sft
100k
49m.
✗
83.7
47.1
81.8
59.8
79.8
83.8
80.8
82.3
LLaVA-v1.5-7b -LoRA
3k
2m.
✗
67.5
23.9
54.7
33.3
55.4
58.8
76.4
66.5
Appendix
Table 12: SNs compared to finetuned LLaVA alternatives on InstaOrder . Given an annotation budget N , we compare both training time and task performances. We use the pretrained model of [ 23 ] as our baseline and their official finetuning script without altering any of the hyperparameters. For our approach, we report the results from the method obtaining the best F1 based on Tab. 2 . Best results are in bold . Baseline is highlighted in gray , while our super neurons are in yellow .
Model
Max new toks.
Embedding
Prefill
Decoding
Wall time
LLaVA-v1.5-7b
128
0.032
0.085
0.025
1.01
LLaVA-v1.5-7b
1
0.032
0.086
0.024
0.223
SNs
1
0.032
0.085
0.002
0.119 (-1.9 × )
Appendix
Table 13: Profiling LLaVA-v1.5-7b on an NVIDIA A100 GPU using the huggingface’s inference routine (in s.). Ours is benchmarked on the first layer of the LLM.
Comparative word
obstructing
hiding
occluding
Random
Split
Metric
Base
SNs
Base
SNs
Base
SNs
Base
SNs
Probe
Acc.
53.9
62.7
45.6
62.6
48.8
64.2
49.5
60.0
F1
46.1
69.0
61.6
67.9
64.3
67.6
65.0
67.0
Validation
Acc.
64.9
78.2
16.2
68.1
18.3
66.8
17.9
66.9
Prec.
18.5
30.4
13.9
24.2
14.6
23.8
14.7
22.2
Recall
40.2
36.5
89.8
54.1
92.8
56.5
94.3
49.2
Appendix
Table 14: Prompt robustness. We use LLaVA-v1.5-7b on InstaOrder (Occ.) . We replace the comparative word in the prompt template with alternative words and evaluate compare the base model performances to the SNs performances.
Dataset
Metric
Method
Pope - Im
Pope - Txt
Pope
SNs (mean)
60.4
60.4
90.9
Accuracy
SNs (maj. voting)
60.3
61.6
90.9
SNs (mean)
71.2
71.4
92.6
Precision
SNs (maj. voting)
68.2
69.9
91.0
SNs (mean)
38.9
38.9
89.5
Appendix
Table 15: Adversarial prompting. We evaluate LLaVA-v1.5-7b on adversarial Pope - Im and Pope - Txt validation datasets containing shuffled VQA pairs. SNs results on the original dataset is highlighted in gray .
Figure 7: Visualization of SNs on different datasets. We use LLaVA-v1.5-7b. We only visualize SNs that obtain better performances than the model itself. Few SNs emerge on the datasets that are already very well answered by the model, while we observe a significant amount of SNs on harder datasets.
Figure 8: SNs overlap between different datasets. We visualize the probed neurons that surpass the model’s performance on all the indicated datasets. We use LLaVA-v1.5-7b. x -axis indicate neuron index while y -axis indicates layer index. Depending on the dataset pair, many SNs can overlap. We also find a quadruplet of datasets sharing SNs.
Spatial reasoning remains a persistent challenge for many vision language models (VLMs), and improving it typically requires fine-tuning with substantial additional parameters. Our preliminary analysis reveals that rescaling activations in selected transformer layers-without modifying pretrained weights-can significantly influence downstream performance. Motivated by this observation, we propose ScAle, an ultra-lightweight adaptation method that learns a small set of scalar coefficients to modulate last-token attention and MLP activations in a fully frozen backbone. We evaluate our method on the synthetic spatial reasoning benchmark SpatialEval and on real-world VQA datasets (COCOQA and VGQA) across multiple model families. Our method, ScAle, achieves up to 134.1% relative accuracy gains using only 1K trainable parameters without requiring millions of trainable parameters as in standard PEFT methods such as LoRA. Despite its extreme compactness, our approach recovers a substantial fraction of standard PEFT performance while preserving strong non-spatial VQA accuracy. These results demonstrate that bounded activation reweighting provides a simple, architecture-agnostic, and highly parameter-efficient alternative for adapting pretrained VLMs.
Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen +2
Northeastern University, Boston, USA · EmbodyX Inc., USA · Zhejiang University, China
Current vision-language models (VLMs) encode visual information in dense hidden states where object identity, spatial layout, and local attributes are implicitly entangled rather than explicitly disentangled, limiting their ability to isolate and modulate the specific visual evidence required by a given language query. Inspired by sparse population coding and top-down modulation in biological vision, we introduce NeuronEye, a plug-in framework that constructs a sparse, concept-level neuron vocabulary from intermediate VLM representations and selectively activates query-relevant visual concepts during inference. NeuronEye decomposes vision-token states into an overcomplete sparse basis organized by concept-level clusters, uses the language query to activate relevant clusters and localize the patches where selected concepts are expressed, and injects the focused evidence back into vision tokens. A complementary suppression mechanism attenuates dominant perceptual directions to preserve weaker but relevant cues. All operations run in a single forward pass over a frozen VLM backbone. On Qwen2.5-VL-7B, NeuronEye raises CV-Bench overall accuracy by +3.1 with gains of +9.5 on Distance, and improves BLINK Multi-view by +8.3, with similar trends on LLaVA-1.6-7B. These results suggest that sparse neuron vocabularies can serve not only as post-hoc interpretability tools but also as active interfaces for concept-level visual reasoning.
Ruiyu Yan, Bowen Chen, Shaowen Wan +1
New York University · New Jersey Institute of Technology
Modern Vision-Language Models (VLMs) benefit from chain-of-thought prompting and test-time scaling, but these gains often come with prohibitive inference cost due to large visual contexts and long decoding chains. We view this cost through two coupled axes: Visual Context Scaling (VCS), which controls how much visual evidence is passed to the language model, and Visual Reasoning Scaling (VRS), which controls how much inference-time reasoning search is performed. Existing methods typically optimize one axis at a time, leaving the joint allocation of compute across these axes underexplored. We introduce Adaptive Visual Inference Scaling (AVIS), a lightweight policy that adapts both VCS and VRS per query. AVIS realizes VCS through Key Diversity Visual (KDV) pruning, a training-free O(N) key-based rule for removing redundant visual tokens before prefilling, and realizes VRS through adaptive self-consistency, using a learned difficulty predictor to select the number of reasoning rollouts. AVIS is deployment-friendly and compatible with shared-prefill inference, where all rollouts reuse a single prefilling pass and KV cache. Across diverse image and video reasoning benchmarks, AVIS improves the accuracy--compute trade-off relative to VCS-only and VRS-only baselines, and remains effective on top of RL post-trained VLMs while keeping compute and latency low.
Ahmadreza Jeddi, Minh Ngoc Le, Amirhossein Kazerouni +8
AI Center-Toronto, Samsung Electronics · University of Toronto · Vector Institute +1