Pretrained models are typically viewed as generalists, requiring bespoke post-training to achieve task-specific expertise. Gan & Isola (2026) recently proposed a striking alternative: that pretrained models lie in a dense neural thicket of task specialists, which can be surfaced with simple Gaussian weight perturbations. We examine this neural thicket of task-specific experts and identify several key findings that question this perspective. First, we find that the apparent lift of the best-of-N selected perturbation decays quickly with selection-set size m, strongly suggesting overfitting to the selection questions. In fact, for 10 of the 12 model-task pairs we study, a simple null model of pure selection noise predicts more than 75 percent of the top expert's reward lift. Second, the top experts' advantage often vanishes on held-out questions or remains within the run-to-run variability of the unperturbed pretrained model. Third, we find that majority voting over the top-K selected perturbations, termed RandOpt(N, K) by Gan & Isola (2026), is often matched or outperformed by majority voting over K samples from the base model. Overall, our results suggest that random weight perturbations may be ineffective at uncovering task experts around pretrained models, and that task adaptation requires more sophistication than random search. Code and artifacts are available at github.com/asil012/RevisitingNeuralThickets.
Figures & tables
Figure 1: OLMo-3-7B-Instruct on Countdown. Left: Performance of N=1,000 perturbations at σ=0.001 ; the anchor line/band displays its five-run mean/range on a fixed runtime. Note little relation between selection and held-out performance. Top right: Reward difference between the best-performing perturbation (“top-1 expert”) at N=1,000 and the perturbation population mean on the selection set of size m . A null model of pure evaluation noise (shown with ±2 SD) predicts most observed lift, e.g., 85% at m=200 . Bottom right: RandOpt versus anchor majority-of- K , at N=5,000 and σ∈{0.5,1,2}×10−3 . Anchor voting leads at every K≥3 .
Figure 2: Selection-size sweeps for OLMo-3-7B-Instruct, N=5,000 . Blue: rank-1 selection reward lift over the perturbation population, with ±1 SD across selection subsets. Gray: empirical noise null with ±2 simulation SD. Orange: held-out rank-1 accuracy gain over the perturbation population. Left: GSM8K shows little transfer. Right: MBPP has a positive effect at larger m on the official 500-task test. ( m=400 uses the entire selection pool, thus lacks selection-subset bands.)
Figure 3: Held-out performance of the top-20 selected perturbations, expressed as percentage point accuracy difference from the anchor. Anchor baselines are averaged over five runs per model–task. All searches use N=5,000 candidates, with the exception of Qwen-32B, which uses N=1,000 . GSM8K and MBPP generally see little advantage from top experts, while Countdown sees substantial gains for Qwen-3B and 7B, but not for OLMo or Qwen-32B.
Answer choices
Top-1
Top-5
Top-10
Top-20
Original
+7.79
+5.21
+5.42
+5.06
Shuffled
+3.20
+1.48
+1.22
+1.12
Table 4: OLMo-3-7B-Instruct at N=5,000 perturbations on the USPTO-50k dataset. The first row reports top experts’ held-out gain over the anchor in accuracy points. The second reports the same for held-out data that has its category numbers shuffled. Roughly 60 – 80% of the top experts’ gains vanishes under shuffling, suggesting that experts primarily learn to abuse label imbalance.
K=10
K=20
K=50
Model
Task
Difference
Significant
Difference
Significant
Difference
Significant
OLMo 7B
GSM8K
+0.41
No
+0.30
No
+0.56
No
MBPP
+0.87
No
+0.72
No
+0.53
No
Countdown
−1.57
Yes
−1.24
Yes
−0.71
No
Qwen 3B
GSM8K
−0.71
No
0.00
No
−0.61
No
MBPP
+0.66
No
−0.03
No
+0.87
No
Table 5: Held-out accuracy of RandOpt minus that of anchor TT-MV, in percentage points at the voting budgets K={10,20,50} . Statistical significance columns record whether a two-sided 95% paired-question bootstrap interval excludes zero. On GSM8K and MBPP, TT-MV matches or outperforms RandOpt. On Countdown, Qwen models prefer RandOpt and OLMo prefers TT-MV.
Figure 6: Accuracy density versus evaluation-set size for OLMo and Qwen models. Each row gives the fraction of accuracies exceeding the indicated percentage of their population mean on the same questions. Specifically, the top row uses 95% and the bottom row 105%. Accuracy values are clipped to [0.1%,99.9%] for display. Solid curves use perturbations at σ=10−3 with greedy decoding; dashed curves use 100 rollouts from the anchor model with temperature 0.6 and top- p=0.95 . Accuracy density often trends toward 0 or 1 with increasing m , suggesting that perturbations concentrate around their population mean, rather than exhibiting a persistent advantage. Stochastic rollouts from the anchor model display qualitatively similar accuracy trends, suggesting that the observed behavior may be due to mere sampling noise.
Figure 7: Accuracy density versus Qwen model size on m=200 questions. Accuracy density trends toward 0 (respectively 1) for thresholds τ>100% (respectively τ<100% ), suggesting simple concentration of perturbations around their population mean. Similar trends are observed for simple stochastic rollouts from the anchor model.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Countdown
GSM8K
MBPP
OLMo-3-7B-Instruct
338
1,717
1,717
Qwen2.5-0.5B-Instruct
100
100
100
Qwen2.5-1.5B-Instruct
100
100
100
Qwen2.5-3B-Instruct
1,717
1,717
1,717
Qwen2.5-7B-Instruct
1,717
1,717
1,717
Qwen2.5-32B-Instruct
100
338
338
Appendix
Table 8: Unselected perturbation counts in the fixed-scale accuracy-density analysis (Figures 6 and 7 ). Each setting uses 100 stochastic anchor rollouts.
Model
Task / analysis
Candidates N
Questions Q
v
OLMo 7B
Countdown, headline sweep
1,000
1,600
0.0915
Countdown, transfer diagnostic
1,000
1,000
0.0904
GSM8K
5,000
1,000
0.0276
MBPP
5,000
400
0.1004
Qwen 3B
Countdown
5,000
1,000
0.0677
GSM8K
5,000
1,000
0.0817
Appendix
Table 9: Mean within-question reward variance in every selection-noise comparison. Values use raw reward-squared units, before multiplying lifts by 100 to express points. All rows use the original three-scale candidate populations. Qwen-32B Countdown has a fixed 200-question comparison rather than a selection-size sweep.
Task
m
Apparent lift
Noise null
Held-out rank-1
Countdown ( N=1,000 )
25
18.40
18.03±3.07
0.67±2.51
50
13.37
12.89±1.66
0.76±2.50
100
10.08
9.38±1.09
1.05±2.25
200
7.69
6.73±0.68
1.04±2.17
400
5.98
4.78±0.52
1.07±2.00
800
4.80
3.42±0.34
0.65±1.51
Appendix
Table 10: OLMo selection-and-transfer sweeps. Lift and null entries are reward points above the perturbation population; null entries give mean ± simulation SD. Held-out entries are accuracy points with the symmetrized 95% half-width of the selection-subset percentile interval shown after ± .
Model
Task
m=25
m=200
m=400
m=800
Qwen 3B
GSM8K
14.97
9.11
7.77
6.87
MBPP
21.74
10.12
7.92
—
Countdown
31.42
24.08
23.89
23.83
Qwen 7B
GSM8K
9.04
5.65
4.75
3.98
MBPP
16.65
6.87
5.51
—
Countdown
31.66
18.16
16.76
16.14
Appendix
Table 11: Qwen2.5-Instruct rank-1 selection reward lift over the perturbation population (points) as selection-set size m increases. Searches use N=5,000 candidates for 3B/7B and N=1,000 for 32B. Entries average 200 uniform question subsets drawn without replacement from 1,000 Countdown or GSM8K questions, or 400 MBPP tasks; the MBPP m=400 endpoint uses the full pool once (and m=800 is unavailable).
Model
Task
Lift
Noise null
Null / lift
Top-50 transfer
OLMo 7B
Countdown
7.69
6.73
87.6%
0.93
GSM8K
4.58
4.06
88.6%
0.00
MBPP
8.75
8.03
91.8%
1.18
Qwen 3B
Countdown
24.08
7.41
30.8%
4.98
GSM8K
9.11
6.90
75.8%
3.90
MBPP
10.12
8.38
82.7%
2.83
Appendix
Table 12: Selection-noise comparisons at m=200 . Lift and null use reward points; top-50 transfer averages individual held-out accuracy gains over the perturbation population. Entries average 200 selection subsets, except Qwen-32B Countdown, which uses the fixed first 200 questions and lacks a population-wide holdout comparison. Countdown transfer uses the auxiliary 600-question holdout; other tasks use their primary tests. OLMo Countdown and Qwen-32B use N=1,000 ; other settings use N=5,000 .
Model
Task
N
Rank 1 / anchor
Difference [95% interval]
OLMo 7B
GSM8K
5,000
85.44/87.41
−1.97[−3.15,−0.76]
OLMo 7B
MBPP
5,000
63.80/61.60
+2.20[−1.40,5.80]
Qwen 3B
GSM8K
5,000
81.12/80.67
+0.45[−1.52,2.28]
Qwen 3B
MBPP
5,000
56.00/56.60
−0.60[−4.01,3.40]
Qwen 3B
Countdown
5,000
26.79/15.43
+11.36[8.93,14.07]
Qwen 7B
GSM8K
5,000
87.95/88.02
−0.08[−1.36,1.14]
Appendix
Table 13: Historical matched rank-1 solo performance on the same questions as the anchor. Countdown uses the common 1,400-question test and MBPP the official 500-question test. The anchor column averages the recorded executions within each comparison. Differences are accuracy points, with paired-question 95% bootstrap intervals conditional on the saved executions. Figure 3 instead uses the five-run means in Table 20 .
Generation seed
Anchor
Candidate 930
Difference
42
68.07
67.93
−0.14
43
67.86
67.79
−0.07
44
68.14
67.64
−0.50
45
68.07
67.57
−0.50
46
67.36
67.57
+0.21
Mean
67.90
67.70
−0.20
Appendix
Table 14: Matched OLMo Countdown repeats on the same 1,400 questions. Accuracy is in percent; differences are candidate 930 minus anchor, in points.
Group
Total gain
Validity
Conditional correctness
Interaction
Qwen-3B ( N=5,000 )
Population
−2.01
−0.15
−1.88
+0.02
Top 20
+4.32
+4.15
+0.13
+0.04
Top 50
+3.32
+3.79
−0.37
−0.09
Selected rank 1
+11.36
+6.91
+3.07
+1.38
Qwen-7B ( N=5,000 )
Appendix
Table 15: Qwen Countdown primary 1,400-question test gain decomposition, in accuracy points. Top- K rows average individual experts, not their majority vote.
Model
Both valid
Expert wins / losses
Shared-set gain
Valid-set change
Per-run p range
Qwen 3B
177.8
31.0 / 20.6
+0.74
+11.41
0.025–0.401
Qwen 7B
481.4
61.0 / 34.2
+1.91
+9.50
0.002–0.045
Qwen 32B
571.4
25.0 / 12.0
+0.93
+0.81
0.024–0.090
Appendix
Table 16: Rank-1 Countdown gain split by shared versus changed valid-answer sets. Counts and gains average comparisons against five fresh greedy anchor runs. Both gain columns use accuracy points over all 1,400 questions and sum, before rounding, to the expert–anchor gap. Wins/losses count discordant correctness on shared valid-answer questions. The final column gives the range of unadjusted two-sided exact McNemar p -values computed separately for each run.
Figure 17: Density versus Qwen model size under alternative metrics and reference means ( m=200 ). Top: Corrected task reward, with each family compared against its own population mean. Middle: Accuracy, using a greedy seed-42 anchor evaluation on the same question subset as the reference. Bottom: Corrected task reward using the same greedy-anchor reference. Solid curves show σ=10−3 perturbations under greedy decoding; dashed curves show 100 stochastic anchor samples at T=0.6 and top- p=0.95 , thresholded against the greedy anchor. Colors mark the threshold multiple of the reference mean, and ties receive half credit. Reward and accuracy coincide on MBPP.
Quantity
Drift-on
Exact restore
Selected candidate
930
2948
Selection reward, m=200 (points)
Population mean
74.027
73.984
Across-candidate SD
3.436
3.424
Rank-1 reward
82.840
82.715
Top-50 mean lift over population
6.549
6.428
Appendix
Table 18: OLMo-3-7B-Instruct Countdown restoration control, with N=5,000 candidates per search. Selection statistics multiply raw rewards by 100; SD is across candidate mean rewards. Test accuracies use the same 1,400 questions. The final row gives paired-question 95% bootstrap intervals conditional on the saved executions, using the original matched anchors rather than the later five-run baselines.
K
Anchor TT-MV
RandOpt drift-on
RandOpt exact restore
1
67.02
67.71
68.14
2
74.84
74.57
74.57
3
78.47
77.21
77.29
5
81.81
80.14
80.29
10
85.71
84.14
83.93
20
89.02
87.79
87.86
Appendix
Table 19: OLMo Countdown vote curves. Figure 1 (bottom right) shows the anchor and released restoration (drift-on) columns on the common 1,400-question test. Anchor values use the stored 50-sample bank during figure generation. The exact-restoration control uses the same candidate seed list.
Model
Task
Mean
Minimum
Maximum
SD
OLMo 7B
Countdown
67.27
66.71
68.21
0.57
GSM8K
87.48
87.26
87.64
0.16
MBPP
61.72
60.40
62.60
0.88
Qwen 3B
Countdown
14.63
14.36
15.00
0.24
GSM8K
80.55
80.29
81.12
0.36
MBPP
55.48
55.40
55.80
0.18
Appendix
Table 20: Accuracy across five fresh greedy anchor executions, in percent. SD uses divisor four; the minimum and maximum describe the observed runs, not a confidence interval. Each execution evaluates 1,400 Countdown, 1,319 GSM8K, or 500 MBPP questions.
Model
Anchor accuracy
Rank-1 gap
Top-50 enrichment
Voting gap ( K=50 )
OLMo 7B
87.41/92.46
−1.97/−2.24
−0.05/−0.13
+0.56/+0.25
Qwen 3B
80.67/85.52
+0.45/−0.61
+3.91/+1.26
−0.61/−0.47
Qwen 7B
88.02/91.81
−0.08/−0.08
+2.20/+0.69
−0.28/−0.64
Qwen 32B
90.07/95.75
+1.44/+0.61
+1.18/+0.01
+0.32/+0.11
Appendix
Table 21: GSM8K reference/tolerant scoring on the same 1,319 test questions. Anchor accuracy is in percent; the other entries are accuracy-point differences. Rank-1 and voting gaps are relative to the anchor; top-50 enrichment is relative to the perturbation population.