cs.AINov 19, 2026

ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining

Authors: Jeff Duan, Matt Thomson

Organizations: Caltech · Caltech, Living Intelligence

Abstract

Supervised fine-tuning (SFT) paired with Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant recipe for post-training LLMs on complex reasoning tasks, but has been shown to collapse the policy into a low-diversity mode. Repeated LLM outputs are nearly identical, and pass@k accuracy converges or degrades as a result. We propose ROAM-GRPO (Routed Off-mode Approach Mining for GRPO), an augmentation to the GRPO algorithm which incentivizes exploration of distinct reasoning strategies and aims to increase output creativity during evaluation. ROAM-GRPO detects well-solved problems during training and transfers them to an "off-mode" track. The model is prompted to produce distinct solution strategies to such problems, which are verified for logical validity and distinctness by an LLM judge and subsequently self-distilled into the policy. Training Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset, ROAM-GRPO implementation increases peak pass@16 by 4.7% and correct embedding cosine distance by approximately 1.5-2x on a difficult evaluation set.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Transformation-Augmented GRPO for Enhancing Exploration in Reasoning of Large Language Models

    Jan 30, 2026Khiem Le, Phuc Nguyen, Youssef Mroueh +4RL for Language Model ReasoningGroup Relative Policy Optimization

  2. Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data

    Apr 20, 2026Zhenwen Liang, Yujun Zhou, Sidi Lu +3Reinforcement LearningRL for Language Model Reasoning