Does Random Search Discover Task Experts? Revisiting Neural Thickets
Organizations: University of Oxford
Abstract
Pretrained models are typically viewed as generalists, requiring bespoke post-training to achieve task-specific expertise. Gan & Isola (2026) recently proposed a striking alternative: that pretrained models lie in a dense neural thicket of task specialists, which can be surfaced with simple Gaussian weight perturbations. We examine this neural thicket of task-specific experts and identify several key findings that question this perspective. First, we find that the apparent lift of the best-of-N selected perturbation decays quickly with selection-set size m, strongly suggesting overfitting to the selection questions. In fact, for 10 of the 12 model-task pairs we study, a simple null model of pure selection noise predicts more than 75 percent of the top expert's reward lift. Second, the top experts' advantage often vanishes on held-out questions or remains within the run-to-run variability of the unperturbed pretrained model. Third, we find that majority voting over the top-K selected perturbations, termed RandOpt(N, K) by Gan & Isola (2026), is often matched or outperformed by majority voting over K samples from the base model. Overall, our results suggest that random weight perturbations may be ineffective at uncovering task experts around pretrained models, and that task adaptation requires more sophistication than random search. Code and artifacts are available at github.com/asil012/RevisitingNeuralThickets.
Figures & tables
| Answer choices | Top-1 | Top-5 | Top-10 | Top-20 |
|---|---|---|---|---|
| Original | ||||
| Shuffled |
| Model | Task | Difference | Significant | Difference | Significant | Difference | Significant |
|---|---|---|---|---|---|---|---|
| OLMo 7B | GSM8K | No | No | No | |||
| MBPP | No | No | No | ||||
| Countdown | Yes | Yes | No | ||||
| Qwen 3B | GSM8K | No | No | No | |||
| MBPP | No | No | No | ||||
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Countdown | GSM8K | MBPP |
|---|---|---|---|
| OLMo-3-7B-Instruct | 338 | 1,717 | 1,717 |
| Qwen2.5-0.5B-Instruct | 100 | 100 | 100 |
| Qwen2.5-1.5B-Instruct | 100 | 100 | 100 |
| Qwen2.5-3B-Instruct | 1,717 | 1,717 | 1,717 |
| Qwen2.5-7B-Instruct | 1,717 | 1,717 | 1,717 |
| Qwen2.5-32B-Instruct | 100 | 338 | 338 |
| Model | Task / analysis | Candidates | Questions | |
| OLMo 7B | Countdown, headline sweep | 1,000 | 1,600 | 0.0915 |
| Countdown, transfer diagnostic | 1,000 | 1,000 | 0.0904 | |
| GSM8K | 5,000 | 1,000 | 0.0276 | |
| MBPP | 5,000 | 400 | 0.1004 | |
| Qwen 3B | Countdown | 5,000 | 1,000 | 0.0677 |
| GSM8K | 5,000 | 1,000 | 0.0817 |
| Task | Apparent lift | Noise null | Held-out rank-1 | |
|---|---|---|---|---|
| Countdown ( ) | 25 | 18.40 | ||
| 50 | 13.37 | |||
| 100 | 10.08 | |||
| 200 | 7.69 | |||
| 400 | 5.98 | |||
| 800 | 4.80 |
| Model | Task | ||||
|---|---|---|---|---|---|
| Qwen 3B | GSM8K | 14.97 | 9.11 | 7.77 | 6.87 |
| MBPP | 21.74 | 10.12 | 7.92 | — | |
| Countdown | 31.42 | 24.08 | 23.89 | 23.83 | |
| Qwen 7B | GSM8K | 9.04 | 5.65 | 4.75 | 3.98 |
| MBPP | 16.65 | 6.87 | 5.51 | — | |
| Countdown | 31.66 | 18.16 | 16.76 | 16.14 |
| Model | Task | Lift | Noise null | Null / lift | Top-50 transfer |
|---|---|---|---|---|---|
| OLMo 7B | Countdown | 7.69 | 6.73 | 0.93 | |
| GSM8K | 4.58 | 4.06 | 0.00 | ||
| MBPP | 8.75 | 8.03 | 1.18 | ||
| Qwen 3B | Countdown | 24.08 | 7.41 | 4.98 | |
| GSM8K | 9.11 | 6.90 | 3.90 | ||
| MBPP | 10.12 | 8.38 | 2.83 |
| Model | Task | Rank 1 / anchor | Difference [95% interval] | |
|---|---|---|---|---|
| OLMo 7B | GSM8K | 5,000 | ||
| OLMo 7B | MBPP | 5,000 | ||
| Qwen 3B | GSM8K | 5,000 | ||
| Qwen 3B | MBPP | 5,000 | ||
| Qwen 3B | Countdown | 5,000 | ||
| Qwen 7B | GSM8K | 5,000 |
| Generation seed | Anchor | Candidate 930 | Difference |
|---|---|---|---|
| 42 | 68.07 | 67.93 | |
| 43 | 67.86 | 67.79 | |
| 44 | 68.14 | 67.64 | |
| 45 | 68.07 | 67.57 | |
| 46 | 67.36 | 67.57 | |
| Mean | 67.90 | 67.70 |
| Group | Total gain | Validity | Conditional correctness | Interaction |
|---|---|---|---|---|
| Qwen-3B ( ) | ||||
| Population | ||||
| Top 20 | ||||
| Top 50 | ||||
| Selected rank 1 | ||||
| Qwen-7B ( ) | ||||
| Model | Both valid | Expert wins / losses | Shared-set gain | Valid-set change | Per-run range |
|---|---|---|---|---|---|
| Qwen 3B | 177.8 | 31.0 / 20.6 | 0.025–0.401 | ||
| Qwen 7B | 481.4 | 61.0 / 34.2 | 0.002–0.045 | ||
| Qwen 32B | 571.4 | 25.0 / 12.0 | 0.024–0.090 |
| Quantity | Drift-on | Exact restore |
|---|---|---|
| Selected candidate | 930 | 2948 |
| Selection reward, (points) | ||
| Population mean | 74.027 | 73.984 |
| Across-candidate SD | 3.436 | 3.424 |
| Rank-1 reward | 82.840 | 82.715 |
| Top-50 mean lift over population | 6.549 | 6.428 |
| Anchor TT-MV | RandOpt drift-on | RandOpt exact restore | |
|---|---|---|---|
| 1 | 67.02 | 67.71 | 68.14 |
| 2 | 74.84 | 74.57 | 74.57 |
| 3 | 78.47 | 77.21 | 77.29 |
| 5 | 81.81 | 80.14 | 80.29 |
| 10 | 85.71 | 84.14 | 83.93 |
| 20 | 89.02 | 87.79 | 87.86 |
| Model | Task | Mean | Minimum | Maximum | SD |
|---|---|---|---|---|---|
| OLMo 7B | Countdown | 67.27 | 66.71 | 68.21 | 0.57 |
| GSM8K | 87.48 | 87.26 | 87.64 | 0.16 | |
| MBPP | 61.72 | 60.40 | 62.60 | 0.88 | |
| Qwen 3B | Countdown | 14.63 | 14.36 | 15.00 | 0.24 |
| GSM8K | 80.55 | 80.29 | 81.12 | 0.36 | |
| MBPP | 55.48 | 55.40 | 55.80 | 0.18 |
| Model | Anchor accuracy | Rank-1 gap | Top-50 enrichment | Voting gap ( ) |
|---|---|---|---|---|
| OLMo 7B | ||||
| Qwen 3B | ||||
| Qwen 7B | ||||
| Qwen 32B |
Transfer Learning
Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.
See all 440 papers