The Neural Thickets hypothesis posits that pretrained language models lie in dense neighborhoods of task-specific experts in weight space, which random Gaussian perturbations can uncover. We reproduce the reported gains, but show that they do not require this geometric explanation. Perturbing a single decoder block already recovers much of the improvement, while keeping the model entirely frozen and adding a random bias to its output logits matches or exceeds full-weight perturbations on most reasoning tasks. This suggests an alternative explanation based on output diversity, validation-based selection, and voting. We formalize this view as Perturb-Select-Vote (PSV), a general procedure that generates perturbed predictors, selects high-performing members on a small validation set, and aggregates their predictions. This interpretation connects Neural Thickets to established inference-time methods, including sampling, self-consistency, and verifier-guided selection. A first-order analysis further shows that small weight perturbations act primarily through their induced logit shifts. We extend this analysis to vision, where logit PSV substantially improves performance under domain shift, including on Earth observation, while full-weight perturbations provide little additional benefit. Together, these results suggest that the gains attributed to nearby weight-space experts can instead be obtained through output diversity, selection, and voting. Logit PSV realizes these benefits with a fully frozen model, providing a simple, training-free alternative to full-weight perturbation ensembles across language and vision.
Figures & tables
Figure 1: Full-weight PSV versus logit PSV. Both methods share the same selection and voting stages ( right ), but differ in the perturbation ( left ). Neural Thickets ( top ) perturb the full dθ -dimensional parameter vector, producing N distinct networks. Logit PSV ( bottom ) keeps one frozen network and adds a V -dimensional bias to the output logits, fixed across decoding steps and inputs.
Figure 2: Effect of perturbing individual decoder blocks. Countdown accuracy with OLMo-3-7B-Instruct ( K=50 ), perturbing one block at a time while keeping all other weights fixed.
Figure 3: Single-component perturbations within a decoder block. Fraction of the full-weight PSV gain recovered by perturbing one component of the first decoder block of OLMo-3-7B-Instruct on Countdown with K=50 . All components recover at least 85% of the full-weight gain (base 66.6% , full-weight PSV 90.8% ).
Figure 4: Logit PSV matches full-weight PSV across tasks and model scales. (a) On OLMo-3-7B-Instruct, logit PSV matches or exceeds full-weight PSV on all evaluated tasks. (b) Across model scales and tasks, logit PSV closely tracks full-weight PSV.
Figure 5: Temperature-tuned majority voting versus full-weight PSV. ( a ) On MATH-500 with a strong model, tuned TT-MV matches or exceeds full-weight PSV. ( b ) On USPTO with a weaker model, TT-MV remains far below PSV, showing the value of validation-based selection; both remain below the majority-class baseline. ( c ) Across tasks and model scales, TT-MV at T=0.7 closely tracks full-weight PSV, with larger gaps for weaker models and harder tasks.
task
base
soup
thicket
TT-MV
logit PSV
countdown
66.6
68.4
91.7
92.1
90.7
GSM8K
87.5
87.6
89.8
89.4
91.9
MATH-500
63.3
62.0
80.7
77.7
81.3
OlympiadBench
38.0
36.5
48.3
49.8
52.5
MBPP
64.3
66.1
76.1
74.4
73.2
ROCStories
65.2
65.3
65.6
65.3
65.8
Table 1: Comparison on OLMo-3-7B-Instruct. Accuracy at K=50 for the base model, model soup, full-weight PSV ( N=5000 ), TT-MV ( T=0.7 ), and logit PSV ( N=1000 ); the model soup also uses N=1000 . Despite a 5× smaller search budget, logit PSV matches or exceeds full-weight PSV on five of seven tasks; TT-MV does so on two.
Perturbation scale σ
Agreement
2.5×10−4
97.5±1.9%
5×10−4
94.7±2.0%
1×10−3
89.7±2.1%
2×10−3
75.9±2.6%
4×10−3
47.2±2.1%
6×10−3
25.3±1.4%
Table 2: Weight perturbations are well approximated by first-order logit shifts. On OLMo-3-7B-Instruct for Countdown, we compare exact weight perturbations to their first-order logit approximation. We report the fraction of argmax-token flips induced by the exact perturbation that are reproduced by the linearized logits. The perturbation scale used in our experiments is shown in bold.
DINOv3
weight-space PSV
output-space
Dataset (metric)
base
thicket
patch-PSV
logit PSV
closed-form
ImageNet (bal. top-1)
88.4
0.0
+0.1
−0.2
n/a
FMoW-OOD (worst-region)
35.1
+1.0
+0.3
+6.2
+3.7
Waterbirds (worst-group)
78.9
+0.3
+0.0
+5.0
+5.0
Table 3: Vision PSV under domain shift. We report improvements relative to linear probing on the frozen DINOv3-7B backbone using worst-region accuracy for FMoW, worst-group accuracy for Waterbirds, and balanced top-1 accuracy for ImageNet. PSV variants use validation-selected ensembles with K=10 ; closed-form denotes a single deterministic output recalibration. Weight-space perturbations yield little improvement, whereas output-space methods produce substantial gains on FMow, Waterbirds, which present strong train-to-test shift.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
σ
Agreement
2.5×10−4
97.5±1.9%
5×10−4
94.7±2.0%
1×10−3
89.7±2.1%
2×10−3
75.9±2.6%
4×10−3
47.2±2.1%
6×10−3
25.3±1.4%
Appendix
Table 4: Accuracy of the first-order approximation. Fraction of perturbation-induced arg-max changes reproduced by the linearized logits on OLMo-3-7B-Instruct for Countdown. Bold indicates the perturbation scale used in our experiments.
task
base
thicket
logit bias
embed bias
Qwen2.5-0.5B-Instruct
GSM8K
44.3
56.0
56.6
55.2
MATH-500
29.0
39.0
40.7
35.7
OlympiadBench
9.5
14.3
16.7
13.7
MBPP
27.0
28.8
26.4
22.1
ROCStories
31.8
31.9
31.9
32.1
Appendix
Table 5: Cross-scale comparison of full-weight and output-space perturbations. Accuracy at K=50 (%) for the greedy base, full-weight PSV, logit PSV, and an input-embedding-bias variant on Qwen2.5 and OLMo-3-7B-Instruct. The input-embedding variant adds a Gaussian bias to token embeddings and is evaluated on Qwen2.5 only. The logit-bias scale is fixed per model rather than chosen per task ( σ=0.5 for Qwen2.5-0.5B, 1.0 for 1.5B and 3B, 2.0 for OLMo-3-7B), and full-weight PSV uses σ=0.001 . Every method uses N=1000 , except full-weight PSV on Countdown ( N=2000 ). Degenerate Qwen2.5-0.5B cells on Countdown and USPTO are omitted. † A constant predictor of the majority reaction class scores 31.9% on USPTO; these cells are at or below it and are not bolded. ‡ Above that baseline, but with lower balanced accuracy than full-weight PSV ( 26.7% against 29.7% ).
Figure 6: Performance as a function of ensemble size. Gain over the greedy base, in accuracy points, as a function of K for full-weight PSV and logit PSV on Qwen2.5-0.5B, 1.5B, and 3B-Instruct ( N=1000 , N=2000 for full-weight Countdown; σ fixed per model as in Table 5 ). Countdown and USPTO are omitted at 0.5B because the base model cannot perform these tasks.
Figure 7: Single-component perturbations transfer across architectures. Gain over base, in accuracy points, when perturbing one component of the first decoder block of Qwen2.5-0.5B-Instruct on GSM8K ( K=50 ; base 44.3% , full-block gain +11.8 ). Every component improves over base, with the largest gains from the value, up, and output projections. Qwen2.5 uses pre-norm blocks and includes QKV biases, unlike OLMo.
Figure 8: Single-component perturbations remain effective at larger scales. Gain over base, in accuracy points, when perturbing one component of the first decoder block on Qwen2.5-1.5B and 3B-Instruct for Countdown ( K=32 ). All components improve over base, with the output and value projections among the strongest at both scales.
1.5B
3B
single weight perturbed
gain
reward std
gain
reward std
residual writes
self_attn.o_proj
+26.9
2.6
+39.9
3.0
mlp.down_proj
+21.7
2.2
+37.5
2.4
LayerNorm gains (pre-norm)
input_layernorm
+12.3
1.4
+29.8
2.2
Appendix
Table 6: Component-wise gains and selection-reward variability across scales. For Qwen2.5-1.5B and 3B-Instruct on Countdown ( K=32 ), we perturb one component of the first decoder block and report its gain over base and the standard deviation of member rewards on the selection set. Every component produces nonzero reward variability and improves the final vote, with the largest effects generally obtained from the value and output projections.
task
base
full-weight PSV
logit PSV
TT-MV
logit − full-weight [95% CI]
Qwen2.5-0.5B-Instruct
countdown
0.3
12.6
12.8
7.2
+ 0.2 [ − 0.8, + 1.1]
GSM8K
44.3
55.3
54.4
57.7
− 1.0 [ − 2.7, + 0.7]
MATH-500
29.0
39.3
39.3
43.0
+ 0.0 [ − 3.7, + 4.0]
OlympiadBench
7.2
15.2
14.8
14.8
− 0.4 [ − 2.7, + 1.7]
MBPP
27.0
27.2
28.5
11.8
+ 1.3 [ − 1.3, + 3.8]
Appendix
Table 7: Matched comparison under the original protocol ( N=5000 , K=50 , authors’ released code). Accuracy (%) of the greedy base, full-weight PSV, logit PSV, and TT-MV ( T=0.7 ), and the paired difference between logit and full-weight PSV with its 95% bootstrap confidence interval over test items; bold intervals exclude zero.
Figure 9: Accuracy as a function of ensemble size K . Plurality-vote accuracy for full-weight PSV, logit PSV, and TT-MV across model scales (rows) and tasks (columns). The dashed line indicates the greedy-base accuracy.
Figure 10: Gain over the greedy base as a function of ensemble size K . Same runs as Figure 9 , shown as the accuracy gain Δ over the greedy base, across model scales (rows) and tasks (columns). The dashed line marks zero gain.
Figure 11: Accuracy gains from individual and stacked perturbation mechanisms. Peak gain over greedy decoding across ensemble sizes K , comparing full-weight PSV, logit PSV, their combination, and a same-mechanism control that stacks two logit biases with a smaller noise range. The symbol “ ∘ ” denotes successive application of two perturbations. Combining weight and logit perturbations rarely improves upon the stronger individual method or the logit-only control, except on 7B MATH-500.
Figure 12: Peak accuracy of stacked perturbations versus individual methods and a logit-only control. Full-weight PSV stacked on logit PSV is compared against (a) the stronger individual method and (b) the same-mechanism control, which stacks two logit biases. The dashed diagonal indicates equal accuracy. Most points lie on or below it. Above it are 3B and 7B MATH-500 ( +1.3 , +1.0 ) and 7B GSM8K ( +0.2 ) in (a), and 7B Countdown ( +0.9 ), 7B MATH-500 ( +0.3 ), and 3B Countdown ( +0.2 ) in (b); only 7B MATH-500 is above it in both.
Pretraining produces a learned parameter vector that is typically treated as a starting point for further iterative adaptation. In this work, we instead view the outcome of pretraining as a distribution over parameter vectors, whose support already contains task-specific experts. We show that in small models such expert solutions occupy a negligible fraction of the volume of this distribution, making their discovery reliant on structured optimization methods such as gradient descent. In contrast, in large, well-pretrained models the density of task-experts increases dramatically, so that diverse, task-improving specialists populate a substantial fraction of the neighborhood around the pretrained weights. Motivated by this perspective, we explore a simple, fully parallel post-training method that samples N parameter perturbations at random, selects the top K, and ensembles predictions via majority vote. Despite its simplicity, this approach is competitive with standard post-training methods such as PPO, GRPO, and ES for contemporary large-scale models.
Adapting a language model to a task no longer requires training all of its weights, and a line of parameter-efficient methods has driven the trainable count from billions down to a handful of scalars. Gradient-free adaptation, which samples random weight perturbations and keeps the ones that score well, has not followed that trajectory and still perturbs every entry of the weight tensor. It is unknown whether that full-weight search is necessary, and more fundamentally which property of a perturbation makes it work at all, because existing methods vary the search space, the perturbation scale, and the aggregation together. We resolve this by intervening on one factor at a time inside a fixed pipeline, holding candidate scoring and voting constant while we vary the search dimension, the subspace that carries the perturbation, and its norm. Perturbing a frozen frame of 12 to 16 scalars stays 1.8 accuracy points behind full-weight search on average across 49 model-benchmark cells, trailing it in 36 of them. Neither the dimension nor the choice of basis explains that performance. A random frame whose Grassmann overlap with the SVD frame is at chance level performs identically once a single scale factor is matched, and at large scales the SVD directions collapse first. What survives is the perturbation norm, whose usable range closes within a factor of five across seven models and stays flat inside. The perturbation norm is therefore the one factor with a failure mode, and its safe region transfers across scale and family. The design question narrows from which subspace to perturb to how hard to shake.
Taeyeong Kim, Ahhyun Kim, TaeHyeon Kim +1
Chosun University · Korea University Sejong Campus
Test-time scaling lifts large language model reasoning by sampling many candidate solutions and selecting among them, yet the same recipe transfers poorly to vision-language models (VLMs): recent work shows that simple majority voting beats selection methods built on the model's own self-verification, apparently because at the selection layer an image-grounded answer and a confident guess from the language prior look the same. A natural fix is to make the selection signal one that cannot be computed without the image. We study Perturbation Grounded Selection (Pgs), a label-free, training-free rule that scores each candidate by whether the model re-derives it under label-preserving perturbations of the input (cropping, background masking, mild photometric or geometric jitter); Pgs recovers majority voting when the perturbation set is empty. The decisive question is not whether Pgs beats chain-of-thought only majority voting, but whether the perturbation term adds anything once decoding format and budget are controlled. We therefore introduce a format-matched control (MatchedCtrl): the same short, no-CoT draws spent on the original image. Across TextVQA, MATH-Vision, MMMU, and ViLP, with a Qwen headline (three-seed means) and LLaVA-OneVision coverage in matched-budget selector tables, Pgs appears to beat plain majority voting by up to +31.8 points on TextVQA (Qwen), but MatchedCtrl tracks or exceeds Pgs within noise on every benchmark, including the vision-required ViLP; no Qwen category shows a significant gain over this control. The stability gap is real and image-dependent (up to +0.48), yet does not predict per-instance wins. The result is negative and diagnostic: perturbation consistency is at best a partial diagnostic of visual dependence and, on its own, not a usable selection signal once format is controlled; gains reported against CoT-only majority voting overstate such methods.
Puzhuo Zheng, Hasan Kurban
College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar