EliteReplay: Preserving Diverse Solutions in RL Post-Training via Quality-Diversity Search

Hanane Nour Moussa; Doeun Lee; Sachin Kumar

cs.AIOct 15, 2026

EliteReplay: Preserving Diverse Solutions in RL Post-Training via Quality-Diversity Search

Authors: Hanane Nour Moussa, Doeun Lee, Sachin Kumar

Organizations: The Ohio State University

Abstract

Reinforcement learning has become a common post-training technique to improve the accuracy of LLMs on verifiable tasks. However, such training can narrow the set of solutions the model can produce. Several approaches have attempted to address this behavior-sharpening effect, but they often optimize metrics that do not reflect the diversity of correct solutions. Ideally, we want a model that can produce multiple good solutions, not only a single reliable one. We propose EliteReplay, a post-training recipe that pairs GRPO with an "elite" archive built by quality-diversity search. Before training, we use the frozen base model to collect solutions that span distinct approaches; during training, GRPO is combined with a replay loss on this archive. The result is a post-trained model that improves accuracy while preserving more of the base model's solution diversity. On multiple text-to-SQL tasks, EliteReplay achieves a higher pass@128 than all baselines, producing up to 77% more distinct correct strategies than the strongest baselines. It preserves most of the base model's correct strategies, outperforming existing methods in strategy retention. This diversity translates into practical benefits on ambiguous questions, where EliteReplay covers both interpretations nearly twice as often as the strongest baseline on AmbiQT.

Figures & tables

Figure 1: Accuracy versus strategy diversity at

k=128

. Each point is a post-trained model on BIRD (in-distribution) and Spider (out-of-distribution). EliteReplay reaches the highest pass@128 while producing far more distinct correct strategies.

Figure 2: Overview of EliteReplay . (a) Before training, MAP-Elites with the frozen base model builds a per-question elite archive, keeping one verified solution per behavior cell. (b) During training, GRPO is combined with a replay loss on the fixed archive. The figure illustrates the method on text-to-SQL.

Figure 3: Pass@

k

and U@

k

as the sampling budget grows on BIRD and Spider. EliteReplay improves accuracy relative to the base model while producing far more distinct correct strategies than any baseline.

Figure 4: MATH-500 pass@

k

of BIRD-trained models. Baselines barely improve or degrade relative to the base model, whereas EliteReplay substantially improves it, despite not being trained on math.

Figure 5: Effective diversity , the probability (%) that two samples are both correct and structurally different. EliteReplay scores highest on both benchmarks.

Figure 6: Share of the base model’s correct strategies retained by each method , over questions on which the base model uses at least two strategies. EliteReplay has the highest retention rate.

Figure 7: How correct samples split across strategies on three BIRD questions. Each row shows the percentage of a method’s correct samples (out of 128 rollouts) that use each strategy. The baselines concentrate on one strategy, while EliteReplay keeps several.

$k$	2	4	8	16	32	64	128
Base model	2.3	5.5	10.3	16.3	23.0	30.1	37.0
GRPO	1.6	2.0	2.4	3.0	3.5	3.9	4.2
SetPO	2.5	5.0	5.5	6.7	7.8	8.8	10.0
DPH-F	2.4	4.1	5.9	7.6	9.3	10.8	12.0
MaxRL	1.9	3.7	6.0	8.5	10.8	12.7	14.2
EliteReplay	3.0	5.9	9.5	13.5	17.9	22.5	27.0

Table 1: Both@

k

on AmbiQT (%) , the share of questions for which

k

samples include correct queries for both valid readings. EliteReplay covers both readings most often among post-trained models.

Figure 8: SFT on the archive alone. SFT on the elite archive beats the IID archive at large

k

, but neither reaches EliteReplay ’s pass@1.

	pass@ $k$ (%)		U@ $k$
Variant	$k=1$	$k=128$	$k=16$	$k=128$
EliteReplay	52.5	74.8	3.0	9.1
IID replay	51.7	75.6	2.4	7.4
Elite SFT $\rightarrow$ GRPO	56.9	68.5	1.7	3.0

Table 2: Ablations of the archive and of replay during RL on BIRD. IID replay replaces the elite archive with verified samples from the base model, and Elite SFT

\rightarrow

GRPO fine-tunes on the elite archive before GRPO, without replay. EliteReplay produces the most distinct correct strategies.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Approach label	Typical construction
Join with filter	Join relations, then restrict rows
Nested query structure	Solve through multiple subqueries
Conditional aggregation	Aggregate only selected records
Existence check	Test whether related records exist
Group-by aggregate	Aggregate separately within groups
Sort and limit	Rank results and retain a subset

Appendix

Table 3: Examples from the induced approach vocabulary. The full vocabulary contains 24 approaches. Archive cells additionally distinguish structural features within these broad labels.

Questions searched	1,000
Candidates generated	72,000
Verified candidates	38,139
Occupied cells after search	2,708
Replay examples	1,500
Questions represented	780

Appendix

Table 4: Elite archive statistics on BIRD. Questions without a verified solution are not represented in the replay set.

Setting	Value
Base model	Qwen3.5-4B-Base
Training steps	100
Questions / step	128
On-policy responses / question	8
Optimizer	AdamW
Learning rate	$10^{-6}$

Appendix

Table 5: Principal training settings. All methods use the same training questions and sampling budget; objective-specific parameters vary by baseline.

Stage	Candidates	Occupied cells
Seeds	8,000	1,132
After round 2	24,000	1,744
After round 4	40,000	2,125
After round 8	72,000	2,708

Appendix

Table 6: Growth of verified behavior coverage during archive search.

	p@1	p@8	p@32	p@128
$\mu=0.02$	53.4	66.2	70.2	73.1
$\mu=0.05$ ∗	52.5	66.2	71.0	74.8
$\mu=0.10$	51.0	65.8	71.0	74.3
$\mu=0.15$	50.3	65.7	70.8	74.0

Appendix

Table 7: Effect of the replay weight

\mu

on BIRD (pass@

k

, %). ∗ Default used in all other experiments.

Figure 9: Change in strategy coverage relative to the base model on BIRD , in percentage points of questions; column labels give the base model’s coverage. EliteReplay loses less than every baseline on every strategy.

Figure 10: Change in strategy coverage relative to the base model on Spider. EliteReplay ’s losses are again the smallest on every strategy.

Figure 11: Distribution of correct samples over strategies on three Spider questions , in the format of Figure 7 .

Language Model Post-Training

Language model post-training adapts pretrained models to follow instructions, reason, use tools, and meet task-specific or human-defined goals.

See all 182 papers

Recovering Diversity Without Losing Alignment: A DPO Recipe for Post-Trained LLMs
May 28, 2026Vinay Samuel, Yapei Chang, Mohit IyyerLLM Alignment Direct Preference Optimization
University of Maryland, College Park
Measuring and Mitigating Solution Mode Collapse in RLVR
Oct 8, 2026Liv G. d'Aliberti, Marwa Abdulhai, Sofiia Druchyna +2Mode Collapse RL Post-Training
Princeton University
Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay
Oct 16, 2026Wen-Zhi Li, Sainyam Galhotra, Claire CardieReinforcement Learning RL for Language Model Reasoning
Cornell University

May 28, 2026J/K move · Enter open · S save

Recovering Diversity Without Losing Alignment: A DPO Recipe for Post-Trained LLMs

Vinay Samuel, Yapei Chang, Mohit Iyyer

University of Maryland, College Park

LLM Alignment Direct Preference Optimization Output Diversity in Language Models LLM Post-Training

Open paper AskPDF arXiv

Many open-ended instructions have multiple valid answers that users can benefit from seeing, but post-training often narrows an LLM's output space toward a small set of canonical responses. We introduce REDIPO, an offline DPO data-construction pipeline for recovering distinct valid answer modes while preserving the alignment benefits of the instruct model. For each prompt, REDIPO samples responses from both base and instruct models, rewrites base-model responses with the instruct model, filters candidates for safety and instruction-following quality, and builds preference pairs that favor marginally diverse responses among candidates with similar instruction-following reward. Across Qwen3-4B, OLMo-3-7B, and LLaMA-3.1-8B, REDIPO improves NoveltyBench distinct_k by 134%, 33%, and 44% relative to the instruct checkpoints, while DivPO changes diversity by 0%, -6%, and -4% on the same models. These gains largely maintain MTBench, IFEval, and Arena-Hard performance, and reduce direct-category HarmBench attack success rate. Ablations show that marginal-diversity pair selection and base-response rewriting drive the diversity gains, while filtering and quality-bounded pairing help maintain alignment. Overall, our results show that diverse valid answers from base-model generations can be reintroduced through carefully constructed preference data while retaining the alignment benefits of post-training. We release our code and data at https://github.com/vsamuel2003/ReDiPO.

Abstract

Language Model Post-Training

Explore similar work

Recovering Diversity Without Losing Alignment: A DPO Recipe for Post-Trained LLMs

Measuring and Mitigating Solution Mode Collapse in RLVR

Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay