RAGEN-2: Reasoning Collapse in Agentic RL

Zihan Wang; Chi Gui; Xing Jin; Qineng Wang; Licheng Liu; Kangrui Wang; Shiqi Chen; Linjie Li

cs.LGApr 7, 2026

RAGEN-2: Reasoning Collapse in Agentic RL

Authors: Zihan Wang, Chi Gui, Xing Jin, Qineng Wang, Licheng Liu, Kangrui Wang, Shiqi Chen, Linjie Li,

Organizations: Northwestern University · UIUC · Independent · Imperial College London ·

Abstract

RL training of multi-turn LLM agents is inherently unstable, and reasoning quality directly determines task performance. Entropy is widely used to track reasoning stability. However, entropy only measures diversity within the same input, and cannot tell whether reasoning actually responds to different inputs. In RAGEN-2, we find that even with stable entropy, models can rely on fixed templates that look diverse but are input-agnostic. We call this template collapse, a failure mode invisible to entropy and all existing metrics. To diagnose this failure, we decompose reasoning quality into within-input diversity (Entropy) and cross-input distinguishability (Mutual Information, MI), and introduce a family of mutual information proxies for online diagnosis. Across diverse tasks, mutual information correlates with final performance much more strongly than entropy, making it a more reliable proxy for reasoning quality. We further explain template collapse with a signal-to-noise ratio (SNR) mechanism. Low reward variance weakens task gradients, letting regularization terms dominate and erase cross-input reasoning differences. To address this, we propose SNR-Aware Filtering to select high-signal prompts per iteration using reward variance as a lightweight proxy. Across planning, math reasoning, web navigation, and code execution, the method consistently improves both input dependence and task performance.

Figures & tables

Figure 1: Left: input-driven reasoning adapts to the current state; templated reasoning produces nearly identical responses across different inputs. Right: four reasoning regimes characterized along two axes: conditional entropy $H(Z\mid X)$ (within-input diversity) and mutual information $I(X;Z)$ (input dependence). Details in Section 2 .

Type	Proxy	Formula	Notes
Discrete	Retrieval-Acc	$\frac{1}{PG}\sum_{i,k}\mathbf{1}[\arg\max_{j}\mathbf{L}_{i,k,j}=i]$	Chance level $1/P$ under template collapse
bi lilin k	Recall@ $k$	$\frac{1}{PG}\sum_{i,k}\mathbf{1}[i\in\mathrm{top}\text{-}k_{j}(\mathbf{L}_{i,k,j})]$	$k\in\{2,4,8\}$
Continuous (raw)	MI-Est	$\frac{1}{PG}\sum_{i,k}(\mathrm{matched}_{i,k}-\mathrm{marginal}_{i,k})$	Per-token; approaches 0 under collapse
Continuous (raw)	MI-Seq-Est	$\frac{1}{PG}\sum_{i,k}\bigl(\mathbf{L}_{i,k,i}-\log\tfrac{1}{P}\sum_{j}e^{\mathbf{L}_{i,k,j}}\bigr)$	Per-sequence; no length normalization
Continuous (z-score)	MI-ZScore	$\frac{1}{PG}\sum_{i,k}\frac{\mathrm{matched}_{i,k}-\mathrm{marginal}_{i,k}}{\sigma_{\mathrm{batch}}+\epsilon}$	Normalized by current-batch marginal std
Continuous (z-score)	MI-ZScore-EMA	$\frac{1}{PG}\sum_{i,k}\frac{\mathrm{matched}_{i,k}-\mathrm{marginal}_{i,k}}{\sigma_{\mathrm{EMA}}+\epsilon}$	$\sigma_{\mathrm{EMA}}^{(t)}=\alpha\,\sigma_{\mathrm{EMA}}^{(t-1)}+(1{-}\alpha)\,\sigma_{\mathrm{batch}}^{(t)}$

Table 1: MI proxy family. All variants are derived from in-batch cross-scoring of reasoning traces against prompts, using matched (per-token log-prob under the true prompt) and marginal (per-token log-prob under the uniform prompt mixture) as base quantities. First-turn variants use only the first agent turn; trajectory variants sample across all turns.

Figure 2: Schematic Signal-to-Noise Ratio (SNR) view of RL updates. Left: total gradient decomposes into task gradient (sharpens with higher within-input reward variance) and regularization gradient. Right: high reward variance yields strong task gradient and better convergence (high SNR); low reward variance makes regularization gradient dominate, producing erratic updates and input-agnostic reasoning (low SNR).

Figure 3: Prompts sorted into six equal-sized reward-variance buckets Q1–Q6. We find: (a) Task gradient norm increases monotonically with bucket RV; (b) When RV near 0, substantial task gradients persist despite carrying almost no useful signal; (c) Regularizer gradient norm (KL + entropy) is flat across buckets. This directly supports the SNR mechanism under both algorithms.

Component	Source	Level	Ctrl.	Mitigation
$g_{\text{signal}}$	Meaningful reward differences across same-prompt trajectories	Prompt	No	SNR-Aware Filtering
$g_{\text{task-noise}}$	Sampling and environment stochasticity	Prompt	No	Filter high-noise prompts
$g_{\text{reg}}$	Uniform contraction per chain, independent of input (KL, entropy)	Chain	Yes	Tune $\lambda_{\text{KL}}$ , $\lambda_{\text{ent}}$

Table 2: Three-noise decomposition of the policy update gradient.

Figure 4: SNR-Aware Filtering workflow. At each training iteration: (1) rollout generation collects trajectories; (2) within-prompt reward variance is computed as SNR proxy; (3) prompts are ranked by RV and top-p fraction retained, policy update performed only on high-signal subset. This filtering loop can prevent updating on noisy rollouts and requires no additional models/rollouts beyond standard RL.

Task	Stochastic	Multi-turn	State	Reward
Sokoban	✗	✓	Grid	Dense
FrozenLake	✓	✓	Grid	Binary
MetaMathQA	✗	✓	Text	Dense
Countdown	✗	✗	Text	Binary
SearchQA	✗	✓	Text	Dense
WebShop	✗	✓	Text	Dense

Table 3: Summary of the features of the environments used.

Figure 5: Training dynamics under different intervention strategies. (a) Task success rate, (b) MI proxy (retrieval accuracy), and (c) reasoning entropy. Without filtering, MI degrades early while entropy spikes, signaling template collapse. Filtering effectively mitigates the decline in retrieval accuracy, with top-p SNR-Aware filtering best preserving both task performance and reasoning diversity.

Figure 6: Comparison of filtering strategies showing Top- $p$ consistently outperforming Top- $k$ and no-filter baselines across four environments.

Experiment Variants	Sokoban	FrozenLake	MetaMathQA	Countdown	Average
Baseline
PPO ( Schulman et al., 2017b ) ,
Qwen2.5-3B ( Qwen Team, 2024 )	12.9 (+16.0)	67.0 (+10.9)	92.6 (+0.6)	97.9 (+0.0)	67.6 (+6.9)
Algorithm
DAPO ( Yu et al., 2025 )	16.2 (+5.1)	66.8 (+2.1)	90.8 (+2.8)	95.7 (+1.6)	67.4 (+2.9)
GRPO ( Shao et al., 2024 )	12.1 (+9.0)	70.9 (-3.0)	91.2 (+1.2)	95.7 (+2.2)	67.5 (+3.7)

Table 4: SNR-Aware Filtering results (%) across algorithms, model scales, types, and modalities. Each cell reports baseline peak with filter delta in parentheses; Qwen2.5-VL-3B includes text (T) and image (V) inputs. Filtering improves average score across all variants.

Figure 7: Reasoning length decline across eight environments, showing systematic compression as a behavioral signature of template collapse.

$P$ (prompts) $\times$ $G$ (traj/prompt)	Task Perf. (%)			Step Time (s)			VRAM (GB)
$P$ (prompts) $\times$ $G$ (traj/prompt)	NF	F	$\Delta$	NF	F	$\Delta$ %	NF	F	$\Delta$
$128\times 1$	23.6	–	–	89.8	–	–	201.80	–	–
$64\times 2$	18.8	27.3	$+8.6$	91.8	64.9	$-29\%$	201.39	201.83	$+0.44$
$32\times 4$	24.2	27.4	$+3.2$	89.8	52.6	$-41\%$	202.11	201.67	$-0.44$
$8\times 16$	15.6	23.6	$+8.0$	89.2	65.9	$-26\%$	201.54	201.90	$+0.36$

Table 5: Sweep over prompt batch size

P

and group size

G

(trajectories per prompt) on Sokoban (Qwen2.5-3B). NF = no filtering; F = SNR-Aware Filtering (

\rho{=}0.9

). Total rollout budget is fixed at 128 trajectories across all configurations.

Figure 8: Spearman correlations showing MI-family metrics positively predict performance while entropy metrics are near-zero or negative.

Figure 9: In FrozenLake, median success rates for both Top-p filtering (orange) and no filtering (gray) decrease as environment stochasticity increases from 0% to 100%. SNR-Aware Filtering maintains a clear advantage from 0% to 50% stochasticity, but the gap closes at 80%–100%, where high transition noise weakens reward variance as an informative signal proxy.

Quartile	RV Range	Task Perf (%)	MI Proxy	Entropy
Q1 (highest RV)	[4.4–5.6]	21.1	0.95	2.02
Q2	[1.5–4.2]	19.5	0.93	1.53
Q3	[0.0–0.2]	10.7	0.81	1.41
Q4 (lowest RV)	[0.0–0.1]	11.0	0.73	1.87

Table 6: Quartile ablation on Sokoban (Qwen2.5-3B,

P{=}8

G{=}16

, keeping 25% of prompts per step). Task performance and MI degrade monotonically from Q1 to Q4.

Method	Prompts Used	Traj/Update	Task Perf (%)	MI Proxy
No filter	8/8	128	12.9	0.83
Prompt-level RV ( $\rho{=}0.9$ )	3.2/8	50.6	23.6	1.80
Trajectory-level	8/8	64	16.8	0.20

Table 7: Trajectory-level vs. prompt-level filtering on Sokoban (Qwen2.5-3B). Prompt-level SNR-Aware Filtering provides the largest gains; trajectory-level filtering confirms that the benefit is not due to prompt-distribution bias.

Setting	Filter $\Delta$	P	G	RV Mean	RV Std	RV Var	RV Min	RV Max	Std/Mean
Sokoban, 14B	$+4.6\%$	8	8	2.24	2.88	8.32	0.10	6.00	1.29
Sokoban, 3B	$+3.2\%$	32	4	2.49	2.89	8.35	0.05	6.52	1.16
FrozenLake, 3B (GRPO)	$-5.0\%$	32	8	0.54	0.18	0.03	0.22	0.76	0.33

Table 8: Per-setting RV statistics and filtering effectiveness. Std/Mean of RV predicts whether SNR-Aware Filtering helps: high ratio means bimodal RV and effective filtering; low ratio means uniform RV and random discarding.

Figure 10: Effective kept ratio and zero-variance prompt count, showing adaptive selection pressure as variance collapses during training.

Figure 11: Prompt-level reward distribution across training phases, showing RV collapse as prompts shift toward uniform reward structures.

Figure 12: Format validity versus MI and entropy diagnostics, showing that high validity does not guarantee high input dependence.

Figure 13: Training dynamics under three interventions. For each setting, we choose two checkpoints (steps 10/400) and connect them into a trajectory (arrows point to later steps). Color intensity indicates weaker to stronger intervention.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

EXPERIMENT SETUP	TASK PERF	MI PROXY	ENTROPY	STABLE
Sampling Strategies
Top-p = 1.0 (Baseline)	0.17	0.54	2.76	✗
Top-p = 0.9	0.38 (+0.20)	0.84 (+0.29)	1.64 (-1.12)	✓
Top-p = 0.5	0.29 (+0.12)	0.83 (+0.29)	1.88 (-0.88)	✓
Min-p = 0.05	0.42 (+0.25)	0.67 (+0.13)	1.64 (-1.12)	✓
Min-p = 0.2	0.45 (+0.27)	0.36 (-0.18)	3.01 (+0.26)	✓

Appendix

Table 9: Ablation results for sampling strategies, filtering metrics, and keep strategies. Values in parentheses denote the change relative to the corresponding baseline in each block. A crossmark in the Stable column indicates training collapse.

Figure 14: Six MI proxy metrics over training steps. (a) MI Estimate, (b) MI Z-Score (EMA), (c) Retrieval Above Chance, (d) Retrieval Accuracy @4, (e) Retrieval Accuracy @8, (f) Conditional Entropy

H(Z|X)

. Consistent with Figure 5 , all proxies confirm that without filtering, MI degrades early, signaling reasoning collapse. Filtering effectively preserves information retention across all metrics, with top-

p

SNR-aware filtering best maintaining reasoning diversity throughout training.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 705 papers

Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework
May 21, 2026Shourov Joarder, Diganta Sikdar, Ahsan Habib Akash +2KL-Regularized RL Intrinsic Reward Methods
Bangladesh University of Engineering and Technology, Bangladesh · West Virginia University, USA · University of Aberdeen, UK +2
SAGE: Multi-Agent Self-Evolution for LLM Reasoning
Mar 16, 2026Yulin Peng, Xinxin Zhu, Chenxing Wei +4LLM Reasoning Self-Training for Language Models
College of Computer Science and Software Engineering, Shenzhen University, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), China · School of Information Technology, Carleton University, Canada
Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
Apr 8, 2026Rohan Surana, Gagan Mundada, Xunyi Jiang +19RL for Language Model Reasoning RL for Language Models
University of California, San Diego · University of Toronto · UIUC +3

May 21, 2026J/K move · Enter open · S save

Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework

Shourov Joarder, Diganta Sikdar, Ahsan Habib Akash, Binod Bhattarai+1

Bangladesh University of Engineering and Technology, Bangladesh · West Virginia University, USA · University of Aberdeen, UK+2

KL-Regularized RL Intrinsic Reward Methods Reinforcement Learning Multi-Objective Reinforcement Learning RL for Language Model Reasoning Reward Design for RL+1

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning ability of LLMs, but often depends on external supervision from human annotations or gold-standard solutions. Reinforcement learning from internal feedback (RLIF) has recently emerged as a scalable unsupervised alternative, using signals extracted from the model itself. However, existing RLIF methods typically rely on a single internal reward, which can lead to reward hacking, entropy collapse, and degraded reasoning structure. We propose a multi-reward RLIF framework that decomposes the training signal into two complementary components: an answer-level reward based on cluster voting and a completion-level reward based on token-wise self-certainty. To combine these signals robustly, we apply GDPO-based normalization to reduce reward-scale imbalance. We further introduce KL-Cov regularization, which targets low-entropy token distributions responsible for disproportionate entropy reduction, preserving exploration and preventing late-stage collapse. Across mathematical reasoning and code-generation benchmarks, our method improves stability and robustness over prior unsupervised RL approaches, while achieving performance close to supervised RLVR methods. These results show that complementary internal rewards, combined with targeted regularization, can support stable long-horizon reasoning without relying on external ground-truth supervision. Code will be released soon.

Abstract

LLM Agents

Explore similar work

Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework

SAGE: Multi-Agent Self-Evolution for LLM Reasoning

Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning