Rethinking Neural Thickets Without Weight Perturbations

Nicolas Dufour; Elouan Gardès; Amil Dravid; Patrick Pérez; Loic Landrieu

cs.LGOct 29, 2026

Rethinking Neural Thickets Without Weight Perturbations

Authors: Nicolas Dufour, Elouan Gardès, Amil Dravid, Patrick Pérez, Loic Landrieu

Organizations: Kyutai · Meta FAIR · LIGM, CNRS, Univ Gustave Eiffel, ENPC, IP Paris · UC Berkeley ·

Abstract

The Neural Thickets hypothesis posits that pretrained language models lie in dense neighborhoods of task-specific experts in weight space, which random Gaussian perturbations can uncover. We reproduce the reported gains, but show that they do not require this geometric explanation. Perturbing a single decoder block already recovers much of the improvement, while keeping the model entirely frozen and adding a random bias to its output logits matches or exceeds full-weight perturbations on most reasoning tasks. This suggests an alternative explanation based on output diversity, validation-based selection, and voting. We formalize this view as Perturb-Select-Vote (PSV), a general procedure that generates perturbed predictors, selects high-performing members on a small validation set, and aggregates their predictions. This interpretation connects Neural Thickets to established inference-time methods, including sampling, self-consistency, and verifier-guided selection. A first-order analysis further shows that small weight perturbations act primarily through their induced logit shifts. We extend this analysis to vision, where logit PSV substantially improves performance under domain shift, including on Earth observation, while full-weight perturbations provide little additional benefit. Together, these results suggest that the gains attributed to nearby weight-space experts can instead be obtained through output diversity, selection, and voting. Logit PSV realizes these benefits with a fully frozen model, providing a simple, training-free alternative to full-weight perturbation ensembles across language and vision.

Figures & tables

Panel of Figure 1 — Figure 1: Full-weight PSV versus logit PSV. Both methods share the same selection and voting stages ( right ), but differ in the perturbation ( left ). Neural Thickets ( top ) perturb the full $d_{{\bm{\theta}}}$ -dimensional parameter vector, producing $N$ distinct networks. Logit PSV ( bottom ) keeps one frozen network and adds a $V$ -dimensional bias to the output logits, fixed across decoding steps and inputs.

Panel of Figure 2 — Figure 2: Effect of perturbing individual decoder blocks. Countdown accuracy with OLMo-3-7B-Instruct ( $K=50$ ), perturbing one block at a time while keeping all other weights fixed.

Panel of Figure 3 — Figure 3: Single-component perturbations within a decoder block. Fraction of the full-weight PSV gain recovered by perturbing one component of the first decoder block of OLMo-3-7B-Instruct on Countdown with $K=50$ . All components recover at least $85\%$ of the full-weight gain (base $66.6\%$ , full-weight PSV $90.8\%$ ).

Panel of Figure 4 — Figure 4: Logit PSV matches full-weight PSV across tasks and model scales. (a) On OLMo-3-7B-Instruct, logit PSV matches or exceeds full-weight PSV on all evaluated tasks. (b) Across model scales and tasks, logit PSV closely tracks full-weight PSV.

Panel of Figure 5 — Figure 5: Temperature-tuned majority voting versus full-weight PSV. ( a ) On MATH-500 with a strong model, tuned TT-MV matches or exceeds full-weight PSV. ( b ) On USPTO with a weaker model, TT-MV remains far below PSV, showing the value of validation-based selection; both remain below the majority-class baseline. ( c ) Across tasks and model scales, TT-MV at $T{=}0.7$ closely tracks full-weight PSV, with larger gaps for weaker models and harder tasks.

task	base	soup	thicket	TT-MV	logit PSV
countdown	66.6	68.4	91.7	92.1	90.7
GSM8K	87.5	87.6	89.8	89.4	91.9
MATH-500	63.3	62.0	80.7	77.7	81.3
OlympiadBench	38.0	36.5	48.3	49.8	52.5
MBPP	64.3	66.1	76.1	74.4	73.2
ROCStories	65.2	65.3	65.6	65.3	65.8

Table 1: Comparison on OLMo-3-7B-Instruct. Accuracy at

K=50

for the base model, model soup, full-weight PSV (

N=5000

), TT-MV (

T=0.7

), and logit PSV (

N=1000

); the model soup also uses

N=1000

. Despite a

5\times

smaller search budget, logit PSV matches or exceeds full-weight PSV on five of seven tasks; TT-MV does so on two.

Perturbation scale $\sigma$	Agreement
$2.5\times 10^{-4}$	$97.5\pm 1.9\%$
$5\times 10^{-4}$	$94.7\pm 2.0\%$
$\mathbf{1\times 10^{-3}}$	$\mathbf{89.7\pm 2.1\%}$
$2\times 10^{-3}$	$75.9\pm 2.6\%$
$4\times 10^{-3}$	$47.2\pm 2.1\%$
$6\times 10^{-3}$	$25.3\pm 1.4\%$

Table 2: Weight perturbations are well approximated by first-order logit shifts. On OLMo-3-7B-Instruct for Countdown, we compare exact weight perturbations to their first-order logit approximation. We report the fraction of argmax-token flips induced by the exact perturbation that are reproduced by the linearized logits. The perturbation scale used in our experiments is shown in bold.

Dataset (metric)	base	thicket	patch-PSV	logit PSV	closed-form
	DINOv3	weight-space PSV		output-space
ImageNet (bal. top-1)	88.4	$0.0$	$+0.1$	$-0.2$	n/a
FMoW-OOD (worst-region)	35.1	$+1.0$	$+0.3$	$\mathbf{+6.2}$	$+3.7$
Waterbirds (worst-group)	78.9	$+0.3$	$+0.0$	$\mathbf{+5.0}$	$+5.0$

Table 3: Vision PSV under domain shift. We report improvements relative to linear probing on the frozen DINOv3-7B backbone using worst-region accuracy for FMoW, worst-group accuracy for Waterbirds, and balanced top-1 accuracy for ImageNet. PSV variants use validation-selected ensembles with

K=10

; closed-form denotes a single deterministic output recalibration. Weight-space perturbations yield little improvement, whereas output-space methods produce substantial gains on FMow, Waterbirds, which present strong train-to-test shift.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

$\sigma$	Agreement
$2.5\times 10^{-4}$	$97.5\pm 1.9\%$
$5\times 10^{-4}$	$94.7\pm 2.0\%$
$\bm{1\times 10^{-3}}$	$\bm{89.7\pm 2.1\%}$
$2\times 10^{-3}$	$75.9\pm 2.6\%$
$4\times 10^{-3}$	$47.2\pm 2.1\%$
$6\times 10^{-3}$	$25.3\pm 1.4\%$

Appendix

Table 4: Accuracy of the first-order approximation. Fraction of perturbation-induced arg-max changes reproduced by the linearized logits on OLMo-3-7B-Instruct for Countdown. Bold indicates the perturbation scale used in our experiments.

Qwen2.5-0.5B-Instruct
task	base	thicket	logit bias	embed bias
GSM8K	44.3	56.0	56.6	55.2
MATH-500	29.0	39.0	40.7	35.7
OlympiadBench	9.5	14.3	16.7	13.7
MBPP	27.0	28.8	26.4	22.1
ROCStories	31.8	31.9	31.9	32.1

Appendix

Table 5: Cross-scale comparison of full-weight and output-space perturbations. Accuracy at

K{=}50

(%) for the greedy base, full-weight PSV, logit PSV, and an input-embedding-bias variant on Qwen2.5 and OLMo-3-7B-Instruct. The input-embedding variant adds a Gaussian bias to token embeddings and is evaluated on Qwen2.5 only. The logit-bias scale is fixed per model rather than chosen per task (

\sigma{=}0.5

for Qwen2.5-0.5B,

1.0

for 1.5B and 3B,

2.0

for OLMo-3-7B), and full-weight PSV uses

\sigma{=}0.001

. Every method uses

N{=}1000

, except full-weight PSV on Countdown (

N{=}2000

). Degenerate Qwen2.5-0.5B cells on Countdown and USPTO are omitted. † A constant predictor of the majority reaction class scores

31.9\%

on USPTO; these cells are at or below it and are not bolded. ‡ Above that baseline, but with lower balanced accuracy than full-weight PSV (

26.7\%

against

29.7\%

Panel of Figure 6 — Figure 6: Performance as a function of ensemble size. Gain over the greedy base, in accuracy points, as a function of $K$ for full-weight PSV and logit PSV on Qwen2.5-0.5B, 1.5B, and 3B-Instruct ( $N{=}1000$ , $N{=}2000$ for full-weight Countdown; $\sigma$ fixed per model as in Table 5 ). Countdown and USPTO are omitted at 0.5B because the base model cannot perform these tasks.

Panel of Figure 7 — Figure 7: Single-component perturbations transfer across architectures. Gain over base, in accuracy points, when perturbing one component of the first decoder block of Qwen2.5-0.5B-Instruct on GSM8K ( $K{=}50$ ; base $44.3\%$ , full-block gain $+11.8$ ). Every component improves over base, with the largest gains from the value, up, and output projections. Qwen2.5 uses pre-norm blocks and includes QKV biases, unlike OLMo.

Panel of Figure 8 — Figure 8: Single-component perturbations remain effective at larger scales. Gain over base, in accuracy points, when perturbing one component of the first decoder block on Qwen2.5-1.5B and 3B-Instruct for Countdown ( $K{=}32$ ). All components improve over base, with the output and value projections among the strongest at both scales.

residual writes
	1.5B		3B
single weight perturbed	gain	reward std	gain	reward std
self_attn.o_proj	$+26.9$	$2.6$	$+39.9$	$3.0$
mlp.down_proj	$+21.7$	$2.2$	$+37.5$	$2.4$
LayerNorm gains (pre-norm)
input_layernorm	$+12.3$	$1.4$	$+29.8$	$2.2$

Appendix

Table 6: Component-wise gains and selection-reward variability across scales. For Qwen2.5-1.5B and 3B-Instruct on Countdown (

K{=}32

), we perturb one component of the first decoder block and report its gain over base and the standard deviation of member rewards on the selection set. Every component produces nonzero reward variability and improves the final vote, with the largest effects generally obtained from the value and output projections.

task	base	full-weight PSV	logit PSV	TT-MV	logit $-$ full-weight [95% CI]
Qwen2.5-0.5B-Instruct
countdown	0.3	12.6	12.8	7.2	$+$ 0.2 [ $-$ 0.8, $+$ 1.1]
GSM8K	44.3	55.3	54.4	57.7	$-$ 1.0 [ $-$ 2.7, $+$ 0.7]
MATH-500	29.0	39.3	39.3	43.0	$+$ 0.0 [ $-$ 3.7, $+$ 4.0]
OlympiadBench	7.2	15.2	14.8	14.8	$-$ 0.4 [ $-$ 2.7, $+$ 1.7]
MBPP	27.0	27.2	28.5	11.8	$+$ 1.3 [ $-$ 1.3, $+$ 3.8]

Appendix

Table 7: Matched comparison under the original protocol (

N{=}5000

K{=}50

, authors’ released code). Accuracy (%) of the greedy base, full-weight PSV, logit PSV, and TT-MV (

T{=}0.7

), and the paired difference between logit and full-weight PSV with its

95\%

bootstrap confidence interval over test items; bold intervals exclude zero.

Panel of Figure 9 — Figure 9: Accuracy as a function of ensemble size $K$ . Plurality-vote accuracy for full-weight PSV, logit PSV, and TT-MV across model scales (rows) and tasks (columns). The dashed line indicates the greedy-base accuracy.

Panel of Figure 10 — Figure 10: Gain over the greedy base as a function of ensemble size $K$ . Same runs as Figure 9 , shown as the accuracy gain $\Delta$ over the greedy base, across model scales (rows) and tasks (columns). The dashed line marks zero gain.

Panel of Figure 11 — Figure 11: Accuracy gains from individual and stacked perturbation mechanisms. Peak gain over greedy decoding across ensemble sizes $K$ , comparing full-weight PSV, logit PSV, their combination, and a same-mechanism control that stacks two logit biases with a smaller noise range. The symbol “ $\circ$ ” denotes successive application of two perturbations. Combining weight and logit perturbations rarely improves upon the stronger individual method or the logit-only control, except on 7B MATH-500.

Panel of Figure 12 — Figure 12: Peak accuracy of stacked perturbations versus individual methods and a logit-only control. Full-weight PSV stacked on logit PSV is compared against (a) the stronger individual method and (b) the same-mechanism control, which stacks two logit biases. The dashed diagonal indicates equal accuracy. Most points lie on or below it. Above it are 3B and 7B MATH-500 ( $+1.3$ , $+1.0$ ) and 7B GSM8K ( $+0.2$ ) in (a), and 7B Countdown ( $+0.9$ ), 7B MATH-500 ( $+0.3$ ), and 3B Countdown ( $+0.2$ ) in (b); only 7B MATH-500 is above it in both.

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 931 papers

Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights
Mar 12, 2026Yulu Gan, Phillip IsolaLLM Post-Training Weight-Space Analysis
MIT CSAIL
Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Models
Aug 3, 2026Taeyeong Kim, Ahhyun Kim, TaeHyeon Kim +1Zeroth-Order Optimization LLM Fine-Tuning
Chosun University · Korea University Sejong Campus
It's the Decoding Format, Not the Perturbation: Auditing Consistency-Based Selection for Vision-Language Test-Time Scaling
Aug 2, 2026Puzhuo Zheng, Hasan KurbanVLM Evaluation Self-Consistency Decoding
College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar

Mar 12, 2026J/K move · Enter open · S save

Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

Yulu Gan, Phillip Isola

MIT CSAIL

LLM Post-Training Weight-Space Analysis

Open paper AskPDF arXiv

Pretraining produces a learned parameter vector that is typically treated as a starting point for further iterative adaptation. In this work, we instead view the outcome of pretraining as a distribution over parameter vectors, whose support already contains task-specific experts. We show that in small models such expert solutions occupy a negligible fraction of the volume of this distribution, making their discovery reliant on structured optimization methods such as gradient descent. In contrast, in large, well-pretrained models the density of task-experts increases dramatically, so that diverse, task-improving specialists populate a substantial fraction of the neighborhood around the pretrained weights. Motivated by this perspective, we explore a simple, fully parallel post-training method that samples

N

parameter perturbations at random, selects the top

K

, and ensembles predictions via majority vote. Despite its simplicity, this approach is competitive with standard post-training methods such as PPO, GRPO, and ES for contemporary large-scale models.

Abstract

LLM Reasoning

Explore similar work

Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Models

It's the Decoding Format, Not the Perturbation: Auditing Consistency-Based Selection for Vision-Language Test-Time Scaling