ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining
Organizations: Caltech · Caltech, Living Intelligence
Abstract
Supervised fine-tuning (SFT) paired with Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant recipe for post-training LLMs on complex reasoning tasks, but has been shown to collapse the policy into a low-diversity mode. Repeated LLM outputs are nearly identical, and pass@k accuracy converges or degrades as a result. We propose ROAM-GRPO (Routed Off-mode Approach Mining for GRPO), an augmentation to the GRPO algorithm which incentivizes exploration of distinct reasoning strategies and aims to increase output creativity during evaluation. ROAM-GRPO detects well-solved problems during training and transfers them to an "off-mode" track. The model is prompted to produce distinct solution strategies to such problems, which are verified for logical validity and distinctness by an LLM judge and subsequently self-distilled into the policy. Training Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset, ROAM-GRPO implementation increases peak pass@16 by 4.7% and correct embedding cosine distance by approximately 1.5-2x on a difficult evaluation set.
Figures & tables
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value |
|---|---|
| Learning rate | (constant) |
| Prompts per step | 2 |
| Max new tokens | 3072 |
| Temperature | 1.0 |
| Top- | 1.0 |
| Judge max tokens (distinctness) | 512 |
| Benchmark | GRPO | ROAM | GRPO ROAM | GRPO ROAM |
|---|---|---|---|---|
| MATH-L5 | 91.0 | 89.8 | 88.0 | 92.8 |
| Omni-MATH-L5/6 | 41.3 | 46.0 | 36.5 | 50.8 |
RL for Language Model Reasoning
Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.
See all 1,171 papers