PRISM: Promoting Sampling Diversity of Large Language Models with Implicit Stochastic Modulation

Qixin Xu; Zihan Zhu; Xiaojiang Liu; Bohan Lyu

cs.LGOct 23, 2026

PRISM: Promoting Sampling Diversity of Large Language Models with Implicit Stochastic Modulation

Authors: Qixin Xu, Zihan Zhu, Xiaojiang Liu, Bohan Lyu

Organizations: Tsinghua University. · Apple.

Abstract

Large language models (LLMs) exhibit mode collapse, where their output distribution concentrates on familiar semantic modes. When diversity disappears, repeated sampling becomes semantic repetition rather than discovery, making rare but stronger solutions unlikely to surface and leaving self-evolution and continual learning difficult to sustain. Most remedies intervene only at inference time, shaping sam- ples without making diversity a model capability. We propose PRISM, a training-time framework that internalizes diversity within the model. PRISM combines (i) submodular curation that pairs each prompt with diverse-yet-correct multi-teacher solutions, and (ii) stochastic embedding modulation, which turns special-token perturbations into implicit signals for sampling distinct outputs. A theoretical analysis establishes the stability of this mechanism. On WritingPrompts and BookMIA, PRISM moves the quality-diversity frontier outward, achieving the best generation quality in all model-benchmark settings while maintaining the strongest or near-strongest semantic diversity across two open-weight model families. It also improves sampling efficiency on math and code benchmarks. Beyond inference, PRISM yields a stronger foundation for reinforcement learning, sustaining diverse rollouts and mitigating the diversity collapse commonly observed during RL training.

Figures & tables

Figure 1: Overview of PRISM . Left: Traditional SFT collapses onto a single dominant trajectory. Middle: PRISM internalizes diversity through submodular curation of diverse-yet-correct trajectories and stochastic embedding modulation that perturbs the prompt embedding with Gaussian noise. Right: At inference, different noise samples steer the model toward distinct modes, producing diverse trajectories from the same prompt.

Datasets	$\geq 3$	$2$	$1$
Code	$34.4$	$37.5$	$28.1$
w/o submodular optimization	$25.0$	$37.5$	$37.5$
Math	$42.8$	$39.3$	$17.9$
w/o submodular optimization	$28.1$	$37.5$	$34.4$

Table 1: Human-evaluated solution diversity in training data with different sampling methods.

General Baselines
Method	WritingPrompts		BookMIA
	Div	Qual	Div	Qual
Qwen2.5-72B-Instruct	$34.1$	$49.1$	$24.5$	$45.7$
llama-3.1-70b-instruct	$36.8$	$49.7$	$22.8$	$44.1$
Qwen2.5-7B-Instruct
Vanilla	$30.2$	$36.1$	$18.5$	$32.2$

Table 2: Creative writing results on WritingPrompts and BookMIA. Div is the within-prompt semantic diversity

1-\bar{s}

(%) computed on OpenAI text-embedding-3-small embeddings. Qual is the Creative Writing v3 rubric score (0–100).

	Math				Code
	MATH	AMC 23	AIME 24	AIME 25	HumanEval	HumanEval+	MBPP	MBPP+
Base	$57.40$	$62.50$	$6.67$	$10.00$	$87.73$	$79.88$	$80.16$	$74.34$
SFT	$76.60$	$\underline{65.00}$	$10.00$	$\mathbf{23.33}$	$93.25$	$87.80$	$89.95$	$82.01$
NEFTune	$78.20$	$\mathbf{75.00}$	$\underline{20.00}$	$\mathbf{23.33}$	$\underline{94.48}$	$\underline{89.24}$	$90.87$	$\mathbf{83.42}$
PRISM	$\mathbf{79.00}$	$\mathbf{75.00}$	$\mathbf{23.33}$	$\mathbf{23.33}$	$\mathbf{95.09}$	$\mathbf{89.63}$	$\underline{91.01}$	$\underline{82.28}$
w/o Noise	$77.40$	$\mathbf{75.00}$	$16.67$	$\underline{20.00}$	$93.25$	$87.80$	$\mathbf{91.53}$	$81.48$

Table 3: Pass@N performance on math and code benchmarks. Only the highest-

N

pass rate is reported for each benchmark(

N=64

for math,

N=256

for code). Results ranked first and second are highlighted.

Figure 2: Pass@N performance of PRISM , SFT, and Base models under different temperatures. PRISM ’s gain is consistent across temperatures.

		Temperature = 0.3				Temperature = 0.7
Base	Phase	AMC23	AIME	HumanEval	HumanEval+	AMC23	AIME	HumanEval	HumanEval+
SFT	SFT	$65.00$	$16.67$	$89.12$	$83.65$	$85.00$	$30.00$	93.66	$87.12$
SFT	RL	$92.50^{{\color[rgb]{1,0,0}\uparrow}}$	$28.33^{{\color[rgb]{1,0,0}\uparrow}}$	$88.34^{{\color[rgb]{0,0,1}\downarrow}}$	$82.32^{{\color[rgb]{0,0,1}\downarrow}}$	$87.50^{{\color[rgb]{1,0,0}\uparrow}}$	$33.33^{{\color[rgb]{1,0,0}\uparrow}}$	$92.02^{{\color[rgb]{0,0,1}\downarrow}}$	$84.76^{{\color[rgb]{0,0,1}\downarrow}}$
PRISM	SFT	$75.00$	$23.33$	$91.39$	85.86	$\mathbf{92.50}$	$28.33$	$93.01$	$87.20$
PRISM	RL	$\mathbf{95.00}^{{\color[rgb]{1,0,0}\uparrow}}$	$\mathbf{35.00}^{{\color[rgb]{1,0,0}\uparrow}}$	$\mathbf{92.02}^{{\color[rgb]{1,0,0}\uparrow}}$	$85.37^{{\color[rgb]{0,0,1}\downarrow}}$	$\mathbf{92.50}^{{\color[rgb]{0.5,0.5,0.5}\parallel}}$	$\mathbf{38.33}^{{\color[rgb]{1,0,0}\uparrow}}$	$93.25^{{\color[rgb]{1,0,0}\uparrow}}$	$\mathbf{89.02}^{{\color[rgb]{1,0,0}\uparrow}}$

Table 4: Models’ pass@64 performance before and after RL. AIME reports the average over AIME 2024 and AIME 2025.

{\color[rgb]{1,0,0}\uparrow}

{\color[rgb]{0,0,1}\downarrow}

, and

{\color[rgb]{0.5,0.5,0.5}\parallel}

denote increase, decrease, and no change after RL relative to the corresponding pre-RL checkpoint.

Method	HumanEval	HumanEval+	MBPP	MBPP+
Base	$81.14$	$72.89$	$75.42$	$69.99$
w/ DBS	$62.58$	$56.10$	$66.40$	$61.64$
Instruct	$\underline{88.49}$	$\underline{83.27}$	$\underline{86.05}$	$\underline{76.42}$
w/ DBS	$79.75$	$73.17$	$85.45$	$74.87$
GEM	$\mathbf{89.57}$	$82.32$	$84.13$	$75.66$
PRISM	$87.81$	$\mathbf{83.82}$	$\mathbf{86.11}$	$\mathbf{77.26}$

Table 5: Pass@64 performance on code benchmarks. PRISM demonstrates superior sampling efficiency compared with both GEM and DBS.

Figure 3: Distribution of unique algorithm counts per query for SFT and PRISM . PRISM ’s shifts toward higher counts, indicating greater algorithmic diversity.

Figure 4: Change in Pass@N versus change in unique algorithm count per query (both PRISM minus SFT), grouped into decrease (-), unchanged (0), and increase (+). Error bars show standard error of the mean.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

$\sigma$	Noise norm	Diversity $\uparrow$	Quality $\uparrow$
no noise	$0\%$	$30.2$	$37.1$
$0.001$	$\approx 8\%$	$32.2$	$38.9$
$0.005$	$\approx 38\%$	$41.4$	$\mathbf{51.0}$
$0.01$	$\approx 76\%$	$43.6$	$45.3$
$0.05$	$\approx 380\%$	$\mathbf{45.7}$	$34.0$

Appendix

Table 6: Sensitivity of PRISM to

\sigma

on WritingPrompts ( Qwen2.5-7B-Instruct ). Diversity rises monotonically while quality follows an inverted-U;

\sigma=0.005

is used in the main experiments.

Parameter	Value
Train batch size	64
Learning rate	5.0e-5
Number of epochs	1.0
LR scheduler	cosine
Warmup ratio	0.1
Precision	bf16

Appendix

Table 7: Hyperparameters for supervised fine-tuning.

Output Diversity in Language Models

See all 34 papers

Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity
Oct 1, 2025Jiayi Zhang, Simon Yu, Derek Chong +4LLM Prompting Model Collapse
Northeastern University · Stanford University · West Virginia University
Large Language Models Explore by Latent Distilling
Apr 27, 2026Yuanhao Zeng, Ao Lu, Lufei Li +3Language Model Decoding Test-Time Training
School of Information Science and Technology, ShanghaiTech University, Shanghai, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China.
Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs
Sep 15, 2026Kirill Skobelev, Eric Fithian, X. Y. HanOutput Diversity in Language Models Mode Collapse
Northwestern University · University of Chicago

Oct 1, 2025J/K move · Enter open · S save

Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity

Jiayi Zhang, Simon Yu, Derek Chong, Anthony Sicilia+3

Northeastern University · Stanford University · West Virginia University

LLM Prompting Model Collapse Diverse Text Generation Language Model Post-Training Output Diversity in Language Models

Open paper AskPDF arXiv

Post-training alignment often reduces LLM diversity, leading to a phenomenon known as mode collapse. Unlike prior work that attributes this effect to algorithmic limitations, we identify a fundamental, pervasive data-level driver: typicality bias in preference data, whereby annotators systematically favor familiar text as a result of well-established findings in cognitive psychology. We formalize this bias theoretically, verify it on preference datasets empirically, and show that it plays a central role in mode collapse. Motivated by this analysis, we introduce Verbalized Sampling, a simple, training-free prompting strategy to circumvent mode collapse. VS prompts the model to verbalize a probability distribution over a set of responses (e.g., "Generate 5 jokes about coffee and their corresponding probabilities"). Comprehensive experiments show that VS significantly improves performance across creative writing (poems, stories, jokes), dialogue simulation, open-ended QA, and synthetic data generation, without sacrificing factual accuracy and safety. For instance, in creative writing, VS increases diversity by 1.6-2.1x over direct prompting. We further observe an emergent trend that more capable models benefit more from VS. In sum, our work provides a new data-centric perspective on mode collapse and a practical inference-time remedy that helps unlock pre-trained generative diversity.

Abstract

Output Diversity in Language Models

Explore similar work

Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity

Large Language Models Explore by Latent Distilling

Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs