Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

Yulu Gan; Phillip Isola

cs.LGMar 12, 2026

Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

Authors: Yulu Gan, Phillip Isola

Organizations: MIT CSAIL

Abstract

Pretraining produces a learned parameter vector that is typically treated as a starting point for further iterative adaptation. In this work, we instead view the outcome of pretraining as a distribution over parameter vectors, whose support already contains task-specific experts. We show that in small models such expert solutions occupy a negligible fraction of the volume of this distribution, making their discovery reliant on structured optimization methods such as gradient descent. In contrast, in large, well-pretrained models the density of task-experts increases dramatically, so that diverse, task-improving specialists populate a substantial fraction of the neighborhood around the pretrained weights. Motivated by this perspective, we explore a simple, fully parallel post-training method that samples $N$ parameter perturbations at random, selects the top $K$ , and ensembles predictions via majority vote. Despite its simplicity, this approach is competitive with standard post-training methods such as PPO, GRPO, and ES for contemporary large-scale models.

Figures & tables

Figure 1: (a) Schematic of the main effects we observe (see Fig 2 for a version with real data). Left: Small models live in a needle in a haystack regime, where good solutions to downstream tasks occupy a tiny fraction of the surrounding weights. In this regime, it is important to have a smart search algorithm, such as gradient descent or other forms of iterative optimization. Right: Large models are surrounded by a veritable thicket of task-specific solutions. In this regime, random sampling is sufficient to quickly land on promising adaptations, which can then be ensembled to yield strong behavior, an approach we call RandOpt . (b) Solution density – i.e. density of task-improving weights in a Gaussian neighborhood of the pretrained weights – scales with model size. (c) RandOpt is

\mathcal{O}(1)

in training steps, FLOP-efficient, and competitive in converged accuracy with GRPO and ES. Results are shown on the Countdown task with Olmo-3-7B-Instruct ; RandOpt uses 5000 random weight guesses and ensembles the top

K

; K-pass baselines use Test-time Majority Vote (TT-MV). More results are shown in Fig. 6 and Table 4 .

Figure 2: Accuracy landscapes in weight space across model scales and reasoning tasks. We perturb the pretrained Qwen2.5 models (from 0.5B to 32B) with 1000 random weight perturbations and project the perturbed models into 2D using random projection. Colors show relative accuracy change $(\mathrm{acc}-\mathrm{base})/\mathrm{base}\times 100$ ( blue : degraded, white : equivalent, red : improved.) Dashed circles indicate the mean perturbation distance and stars mark the best-performing perturbations. Larger models have warmer landscapes, with more high-performing neighborhoods. The last column shows an RGB visualization where GSM8K, Olympiad and Countdown accuracies are mapped to R,G,B channels; richer colors indicate more task experts.

Figure 3: Scaling laws of solution density and diversity (using Qwen-2.5 instruction tuned models). (a) Solution density increases with model scale, showing that larger models have a higher fraction of good solutions. (b) Spectral discordance across model scales, measuring solution diversity. Together, these results demonstrate that larger models have both denser and more diverse solution landscapes in the neighborhood around their pretrained weights.

Figure 4: Performance spectra and clustering of random seeds. Sampled vectors possess diverse areas of expertise, with individual seeds specializing in specific tasks. (Left) Performance of 100 random seeds across seven evaluation datasets. Each line represents a specific seed, with four lines highlighted as examples. (Right) PCA visualization of these performance vectors, where seeds with similar behavior cluster together into different groups.

Panel of Figure 5 — Figure 5: Pretraining a model of 1D signals, then probing the local neighborhood around the pretrained weights by random guessing $N=1000$ Gaussian perturbations. The plot shows the autoregressive predictions of a particular linear function (dashed blue line), given an observed context (solid blue line). Gray lines: random $f_{\theta}$ ’s; Red lines: top-K $f_{\theta}$ ’s. The figure shows three regimes: (a) No pretraining leads to needle-in-the-haystack search, (b) pretraining on several signal types leads to a thicket, (c) pretraining on just linear functions achieves nearly perfect predictions at pretraining time, hence post-training is at ceiling.

Figure 6: RandOpt vs. baselines on post-training LLMs. Marker size represents model scale (0.5-8B), shape indicates task family, and transparency distinguishes benchmarks within each task family. RandOpt matches or exceeds baselines in most settings. RandOpt is run with K=50 and ES+TT-MV also uses 50 test-time samples ensembled via majority vote. 1-pass PPO/GRPO use a single test-time sample; this setting disadvantages the baseline but reflects current standard usage of these models, in which there is no test-time ensembling. Full experimental results can be found in Appendix Table 4 .

Model	Method	GQA
Qwen2.5-VL-3B-Inst	Base	56.6
Qwen2.5-VL-3B-Inst	RandOpt	69.0

Table 1: RandOpt improves Accuracy (%) on GQA.

Model	Method	GSM8K
Qwen2.5-1.5B-Inst.	Base	58.8
	Distill	74.9
	RandOpt	76.4
Qwen2.5-3B-Inst.	Base	79.8
	Distill	84.3
	RandOpt	87.1

Table 2: Distilling the top-K RandOpt ensemble into a single model achieves performance comparable to the ensemble on GSM8K.

Figure 9: Accuracy decomposition on GSM8k using Qwen2.5-3B-Instruct (N=3000, K=50). Grey denotes strictly correct answers (both format and answer correct), light blue denotes the questions originally answered incorrectly that are corrected after training, and purple denotes the gains are just from fixing format. For both GRPO and RandOpt, a large portion of the gains come from format correction, while another portion is from solving the problem correctly.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Scaling curves for population size

K

and ensemble size

N

on Countdown task.

Figure 11: Model performance comparison on Countdown, GSM8k, MATH-500, and OlympiadBench using the Qwen2.5-1.5B-Instruct model. More results for additional models and baselines can be found in Appendix Table 4 .

Figure 12: Performance distributions of 500 randomly perturbed models relative to base model accuracy on the math reasoning task GSM8k (left) and Countdown (right). The fraction of perturbations matching or exceeding base performance (shaded region) increases with model size:

0\%\to 64\%

on GSM8K and

8\%\to 60\%

on Countdown as model size grows from 0.5B to 32B parameters. Note: x-axis shows relative performance; longer tails in smaller models do not equate to higher absolute accuracy.

Figure 13: Batch/group size vs. accuracy under one-step training. Scaling baseline parallelism does not get RandOpt’s performance. Task: GSM8K, Model: Qwen2.5-3B-Instruct, N=5000, K=50)

Category	Parameter	GRPO	PPO	ES	RandOpt
Budget & Scale	Batch Size	1024	128	–	–
	Group/Population Size	8	1	30	5000
	Iterations	200	600	167	1
Optimization Complexity	Actor Learning Rate	1e-6	1e-6	5e-4 ( $\alpha$ )	–
	Critic Learning Rate	–	1e-5	–	–
	Optimizer	AdamW	AdamW	-	–

Appendix

Table 3: Hyperparameters. ’ – ’ indicates that the hyperparameter is not required for the corresponding algorithm.

Model	Method	Countdown	GSM8k	MATH-500	OlyBench	MBPP	ROCStories	USPTO
		(math) $\uparrow$	(math) $\uparrow$	(math) $\uparrow$	(math) $\uparrow$	(prog.) $\uparrow$	(writing) $\uparrow$	(chemistry) $\uparrow$
Qwen2.5- 0.5B-Inst	Base	0.1 ${}_{\pm\text{0.1}}$	39.9 ${}_{\pm\text{0.0}}$	19.8 ${}_{\pm\text{0.0}}$	4.2 ${}_{\pm\text{0.4}}$	30.9 ${}_{\pm\text{0.2}}$	22.0 ${}_{\pm\text{0.0}}$	29.9 ${}_{\pm\text{0.3}}$
	TT-MV †	0.3 ${}_{\pm\text{0.3}}$	41.0 ${}_{\pm\text{0.1}}$	29.2 ${}_{\pm\text{0.5}}$	6.7 ${}_{\pm\text{0.2}}$	37.0 ${}_{\pm\text{0.3}}$	19.5 ${}_{\pm\text{0.2}}$	30.2 ${}_{\pm\text{0.2}}$
	Best-of-N ‡	5.9 ${}_{\pm\text{2.0}}$	40.5 ${}_{\pm\text{1.4}}$	34.5 ${}_{\pm\text{0.5}}$	10.5 ${}_{\pm\text{0.3}}$	34.9 ${}_{\pm\text{1.3}}$	20.3 ${}_{\pm\text{0.3}}$	26.8 ${}_{\pm\text{1.2}}$
	PPO	14.8 ${}_{\pm\text{2.8}}$	43.2 ${}_{\pm\text{1.2}}$	33.4 ${}_{\pm\text{1.7}}$	16.1 ${}_{\pm\text{0.0}}$	37.8 ${}_{\pm\text{3.0}}$	19.1 ${}_{\pm\text{0.8}}$	31.2 ${}_{\pm\text{0.6}}$
	GRPO	13.0 ${}_{\pm\text{0.0}}$	48.4 ${}_{\pm\text{0.8}}$	33.7 ${}_{\pm\text{0.9}}$	6.9 ${}_{\pm\text{0.1}}$	42.8 ${}_{\pm\text{3.7}}$	30.9 ${}_{\pm\text{1.1}}$	31.7 ${}_{\pm\text{0.2}}$

Appendix

Table 4: Experimental results on reasoning benchmarks. We compare RandOpt against RL-based methods (PPO, GRPO), Evolution Strategies (ES), Best-of-N and test-time majority voting (TT-MV) across model scales. Results are averaged over 3 runs with standard deviation shown in gray . Bold indicates best performance, underlined indicates runner-up. Details of the hyperparameters for all methods are provided in Appendix E.3 .

Panel of Table 5 — Table 5: RandOpt on 1D signals (approximation). Each cell corresponds to a pretraining–post-training pair. In this experiment we plot results on the same function that we fit to during post-training; this is a test of apprpoximation only, not generalization to new functions. We also compare to Xavier initialization ( Glorot & Bengio, 2010 ) and Kaiming initialization He et al. (2015) . For each pretraining method, we pick a perturbation noise scale, $\sigma$ that is large enough to show functional variation; this value is shown underneath the pretraining method name.

Panel of Table 6 — Table 6: RandOpt on 1D signals (generalization). Each row corresponds to a pretraining–post-training pair. For all rows, the test set functions are of the post-trained type, and we show three random test examples. Rightmost column shows the average mean squared error over the entire test set, for each method.

Panel of Table 7 — Table 7: RandOpt on Text-to-Image models (Train Set). We use the Stable Diffusion XL model ( Podell et al., 2023 ) . Images are generated from a text prompt. RandOpt selects the top-K models by scoring generated images with a target text (e.g., “blue”) using GPT-5.2, and performs mean ensembling over the K models at each denoising step.

Panel of Table 8 — Table 8: RandOpt on Text-to-Image models (Test Set). Top-K selected on the training set and evaluated on the test set.

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 246 papers

Sequential Pretraining Favors Large Models
Oct 7, 2026Mohnish Harwani, Yujia ZhengLanguage Model Pretraining Continued Pretraining
Purdue University · University of Illinois Urbana–Champaign
Early Data Exposure Improves Robustness to Subsequent Fine-Tuning
May 12, 2026Lawrence Feng, Gaurav R. Ghosal, Jacob Mitchell Springer +2Language Model Pretraining Continual Learning for LLMs
Carnegie Mellon University
TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts
Jun 3, 2026Jiangyang He, Shaolin Zhu, Deyi XiongMixture-of-Experts Pruning LLM Pruning
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China

Oct 7, 2026J/K move · Enter open · S save

Sequential Pretraining Favors Large Models

Mohnish Harwani, Yujia Zheng

Purdue University · University of Illinois Urbana–Champaign

Language Model Pretraining Continued Pretraining

Open paper AskPDF arXiv

Large neural networks often acquire capabilities that small models fail to learn. Does this stem from large models learning more representative features, or from being more robust to unaccounted-for adverse effects introduced during training? We define and quantify one such adverse effect, primacy bias, as the extent to which exposure to early data distributions impairs later learning. We show that small models can allocate learning capacity inefficiently toward early distributions, whereas sufficiently overparameterized models are robust to this effect. This inefficiency is particularly consequential in pretraining, where foundation models often encounter heterogeneous data distributions sequentially rather than jointly. As a result, small foundation models can struggle to learn distributions encountered late in training, which is particularly harmful when later data emphasizes desirable capabilities such as code, mathematics, and reasoning. Motivated by these findings, we introduce Exposure Therapy (ET), a simple regularization that promotes more efficient allocation of learning capacity during sequential pretraining. We demonstrate that ET improves foundation models' performance on late data distributions as well as overall capability in models up to the billion-parameter scale. Overall, our results suggest that some benefits of large foundation models may arise from greater robustness to adverse training effects, rather than from learning more representative features, and that improved training algorithms can recover some of these advantages in smaller models.

Abstract

LLM Post-Training

Explore similar work

Sequential Pretraining Favors Large Models

Early Data Exposure Improves Robustness to Subsequent Fine-Tuning

TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts