Entropy Collapse in RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Latest papers

11 papers1 paper in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

SCARL: Sequence Confidence-Aligned Reinforcement Learning
Oct 29, 2026Zhenghong Ke, Jingbo Zhou, Hua WuEntropy Collapse in RL Reinforcement Learning
Baidu Frontier Research Department · Nanjing University
GRPODropout: Less is More for Online Reinforcement Learning Rollouts
Oct 8, 2026Hexuan Deng, Zihao Yan, Xuebo Liu +8Reinforcement Learning RL for Language Model Reasoning
Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen) · Beijing Zhongguancun Academy · XinzhuAI
Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratification
Sep 7, 2026Yimeng Ye, Shuang Chen, Wenxuan Huang +8Entropy Collapse in RL Reinforcement Learning
Columbia University · CUHK · THU +1
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Aug 27, 2026Xingyu Shen, Huishuai Zhang, Peng Li +2Entropy Collapse in RL RL for Language Model Reasoning
Wangxuan Institute of Computer Technology, Peking University · National Engineering Research Center of New Electronic Publishing Technologies
STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability
Jun 17, 2026Haipeng Luo, Qingfeng Sun, Songli Wu +4RL for Language Model Reasoning Maximum Entropy RL
Shenzhen International Graduate School, Tsinghua University · Tencent Hunyuan
Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization
May 12, 2026Huimin Xu, Shuai Zhao, Xiaobao Wu +1Entropy Collapse in RL Reinforcement Learning
Nanyang Technological University, Singapore · Shanghai Jiao Tong University, China · 3VinUniversity, Vietnam
Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control
Apr 29, 2026Bolian Li, Yifan Wang, Yi Ding +3Entropy Collapse in RL RL for Language Models
Purdue University, West Lafayette, IN, USA
Targeted Exploration via Unified Entropy Control for Reinforcement Learning
Apr 16, 2026Chen Wang, Lai Wei, Yanzhi Zhang +5Entropy Collapse in RL Reinforcement Learning
College of Software, Nankai University · Zhongguancun Academy · Shanghai Jiao Tong University +3
Entropy-Preserving Reinforcement Learning
Mar 12, 2026Aleksei Petrenko, Ben Lipkin, Kevin Chen +4Policy Gradient Methods Entropy Collapse in RL
Apple · MIT
SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training
Oct 9, 2025Chen Wang, Zhaochun Li, Jionghao Bai +4Entropy Collapse in RL Reinforcement Learning
College of Software, Nankai University · Zhongguancun Academy · Beijing Institute of Technology +2
Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models
Sep 30, 2025Jaesung R. Park, Junsu Kim, Gyeongman Kim +4Entropy Collapse in RL RL for Language Models
Department of Mathematics, UCLA · Department of Mathematical Sciences, Seoul National University · KRAFTON +2

Oct 29, 2026J/K move · Enter open · S save

SCARL: Sequence Confidence-Aligned Reinforcement Learning

Zhenghong Ke, Jingbo Zhou, Hua Wu

Baidu Frontier Research Department · Nanjing University

Entropy Collapse in RL Reinforcement Learning RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Confidence Estimation in Language Models

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning by scoring sampled responses with a rule-based verifier and reinforcing those that pass. Where a classifier's training couples predicted probability to labels, nothing ties an RLVR policy's confidence in a response to its reward: aligned training would drive responses with the same advantage toward equal confidence, yet the advantage is blind to confidence. Under this sequence-confidence misalignment, a success the policy barely found earns the same update as one it produces with near certainty, and a confident error is corrected no harder than an exploratory one. We propose SCARL (Sequence Confidence-Aligned Reinforcement Learning), which reshapes, but never re-signs, each advantage with the alignment gap between the response's group-relative confidence and the outcome its advantage certifies, normalized into shares of each class's advantage budget. Misaligned samples absorb the force, responses with the same advantage are driven toward equal confidence, and repairing the misalignment doubles as a remedy for the entropy collapse that erodes exploration. SCARL provably reduces to its base estimator when a class's confidences agree, adds no extra passes and one hyperparameter. On Qwen3-4B-Base it lifts average Pass@32 by 2.4 points over DAPO and is best among nine compute-matched methods on both coverage and per-draw accuracy; applied unchanged across four estimators and four backbones, it improves 14 of 16 settings while preserving policy entropy and widening the confidence-correctness separation. Our code is available at https://github.com/kezhh163/scarl.