Reinforcement Learning with Verifiable Rewards

Also known as RLVR

This field studies how to improve language and multimodal models by rewarding outputs whose correctness or task progress can be checked, for example with unit tests, mathematical answers, formal proofs, or simulated environments. Common approaches use policy-gradient methods such as PPO and GRPO, with outcome rewards supplemented by step-level feedback, rollout exploration, or replay to improve credit assignment and data efficiency. Work spans math and coding benchmarks as well as search, GUI, and other interactive agents. A central tension is that rewards must be scalable yet faithful: imperfect or narrow verifiers can be exploited, and a correct final answer does not by itself establish sound reasoning, visual grounding, or robust generalization.

Latest papers

692 papers137 papers in the last four weeks, up 111% on the four weeks before. 0.8% of all new papers.

Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning
Feb 24, 2026Yuanda Xu, Hejian Sang, Zhengze Zhou +2RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
LinkedIn Corporation, CA, USA
Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
Feb 24, 2026Xu Wan, Yansheng Wang, Wenqi Huang +1RL for Language Model Reasoning RL Post-Training
Zhejiang University · Bytedance Seed Robotics · Peking University +1
ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Models
Feb 23, 2026Andre He, Nathaniel Weir, Kaj Bostrom +4Synthetic Data Generation for Language Models RL for Language Model Reasoning
Carnegie Mellon University · Amazon Web Services
DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning
Feb 23, 2026Zhongwei Wan, Yun Shen, Zhihao Dou +9Reinforcement Learning RL for Language Model Reasoning
The Ohio State University, USA · Case Western Reserve University, USA · The Chinese University of Hong Kong, Hong Kong +4
How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization
Feb 22, 2026Yangyi Fang, Jiaye Lin, Xiaoliang Fu +6Reinforcement Learning RL for Language Model Reasoning
Meituan · Tsinghua University · Fudan University +1
Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning
Feb 22, 2026Zhuoxu Huang, Mengxi Jia, Hao Sun +2RL Exploration RL for VLMs
Aberystwyth University · Institute of Artificial Intelligence (TeleAl), China Telecom · Tsinghua University
Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
Feb 20, 2026Johannes Ackermann, Michael Noukhovitch, Takashi Ishida +1Reward Hacking Reinforcement Learning with Verifiable Rewards
The University of Tokyo · Mila · RIKEN AIP
MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning
Feb 19, 2026Xiaoliang Fu, Jiaye Lin, Yangyi Fang +7RL for Language Models RL for Language Model Reasoning
Meituan · Fudan University · Tsinghua University +2
DeepVision-103K: A Visually Diverse, Broad-Coverage, and Verifiable Mathematical Dataset for Multimodal Reasoning
Feb 18, 2026Haoxiang Sun, Lizhen Xu, Bing Zhao +5Multimodal Reasoning Mathematical Reasoning
Alibaba Group · Shanghai Jiao Tong University
The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probes
Feb 17, 2026Mohammad Taufeeque, Stefan Heimersheim, Adam Gleave +1Deception in Language Models LLM Honesty
FAR.AI.
On the Emergence of Implicit Curriculum in RLVR Learning Dynamics
Feb 16, 2026Yu Huang, Zixin Wen, Yuejie Chi +4RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Department of Statistics and Data Science, Wharton School, University of Pennsylvania · Machine Learning Department, Carnegie Mellon University · Department of Statistics and Data Science, Yale University +1
To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models
Feb 13, 2026Haoqing Wang, Xiang Long, Ziheng Li +3RL for Language Models Gradient Interference
Samsung Research, Beijing, China · Peking University
Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models
Feb 12, 2026Xin Xu, Clive Bai, Kai Yang +8RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
HY, Tencent · The Hong Kong University of Science and Technology · The University of Hong Kong
Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
Feb 10, 2026Pei-Chi Pan, Yingbin Liang, Sen LinReward Modeling Reward Hacking
Department of Computer Science University of Houston · The Ohio State University · University of Houston
Process Rewards for Outcome-Guided Reasoning Steps in LLM Reinforcement Learning
Feb 8, 2026Mohammad Rezaei, Jens Lehmann, Sahar VahdatiLLM Mathematical Reasoning Reinforcement Learning
Faculty of Computer Science, Technical University of Dresden, Dresden, Germany · Amazon Science · Technical University of Dresden, Dresden, Germany +1
F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
Feb 6, 2026Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov +4Reinforcement Learning RL for Language Model Reasoning
T-Tech · Saint Petersburg Electrotechnical University “LETI”
Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation
Feb 5, 2026Zhiqi Yu, Zhangquan Chen, Mengting Liu +2RL for Language Model Reasoning Group Relative Policy Optimization
University of Hong Kong · Tsinghua University · Sun Yat-sen University
CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Models
Feb 4, 2026Xiao Zhu, Xinyu Zhou, Boyu Zhu +5RL for Code Generation Reward Modeling
LARK, HKUST(GZ) · Kuaishou Technology · UCL +3
HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learning
Jan 30, 2026Weiqi Wang, Xin Liu, Binxuan Huang +13RL for Language Models RL for Language Model Reasoning
Stores Foundational AI, Amazon Inc., Palo Alto, CA, USA · Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China
Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs
Jan 16, 2026Lecheng Yan, Ruizhe Li, Guanhua Chen +5Reward Hacking Memorization in Language Models
Southern University of Science and Technology · University of Science and Technology of China · University of Aberdeen +5
X-Coder: Advancing Competitive Programming with Synthetic Tasks, Solutions, and Tests
Jan 11, 2026Jie Wu, Haoling Li, Xin Zhang +8RL for Code Generation Synthetic Data Generation
Tsinghua University · Microsoft
GDEPO: Group Dual-dynamic and Equal-right Advantage Policy Optimization for Sample-Efficient Reinforcement Learning
Jan 11, 2026Zhengqing Yan, Fan Guo, Xinyang Liu +5Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards
State Key Laboratory of Engines, Tianjin University, 300354 Tianjin, China. · Artificial Intelligence Center, Cylingo Group, 100080 Beijing, China.
ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth
Dec 2, 2025Salman Rahman, Sruthi Gorantla, Arpit Gupta +3Process Reward Models Reinforcement Learning with Verifiable Rewards
2UCLA · Work done while as an intern at Amazon AGI. · 1Amazon AGI
Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation
Nov 14, 2025Mohamad Amin Mohamadi, Tianhao Wang, Zhiyuan LiLLM Abstention LLM Hallucination Mitigation
Toyota Technological Institute at Chicago · University of California, San Diego
Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models
Oct 24, 2025Hoang Phan, Xianjun Yang, Yuanshun Yao +6RL Post-Training Catastrophic Forgetting
Meta Superintelligence Labs · New York University · Johns Hopkins University
Group Contrastive Policy Optimization: Learning Beyond the Self-Exploration Limit with Golden Answers
Oct 9, 2025Hao Wu, Wei Liu, Shizhe Xiang +3RL for Language Model Reasoning LLM Reasoning
Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
Oct 5, 2025Zishang Jiang, Jinyi Han, Tingyun Li +7RL for Language Model Reasoning LLM-Guided RL
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers
Oct 1, 2025Xin-Qiang Cai, Wei Wang, Feng Liu +3Reinforcement Learning RL for Language Model Reasoning
RIKEN AIP, Tokyo, Japan · The University of Tokyo, Tokyo, Japan · The University of Melbourne, Melbourne, Australia
Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners
Sep 30, 2025Xin Xu, Clive Bai, Kai Yang +4Efficient Language Model Reasoning RL for Language Model Reasoning
LLM Department, Tencent · The Hong Kong University of Science and Technology · The University of Hong Kong
Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models
Sep 30, 2025Jaesung R. Park, Junsu Kim, Gyeongman Kim +4RL for Language Models Reinforcement Learning
Department of Mathematics, UCLA · Department of Mathematical Sciences, Seoul National University · KRAFTON +2

Feb 24, 2026J/K move · Enter open · S save

Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning

Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He+1

LinkedIn Corporation, CA, USA

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has become the leading paradigm for enhancing reasoning in Large Language Models (LLMs). However, standard RLVR algorithms suffer from a well-documented pathology: while they improve Pass@1 accuracy through sharpened sampling, they simultaneously narrow the model's reasoning boundary and reduce generation diversity. We identify a root cause that existing methods overlook: the uniform penalization of errors. Current approaches -- whether data-filtering methods that select prompts by difficulty, or advantage normalization schemes -- treat all incorrect rollouts within a group identically. We show that this uniformity allows overconfident errors (incorrect reasoning paths that the RL process has spuriously reinforced) to persist and monopolize probability mass, ultimately suppressing valid exploratory trajectories. To address this, we propose the Asymmetric Confidence-aware Error Penalty (ACE). ACE introduces a per-rollout confidence shift metric, c_i = log(pi_theta(y_i|x) / pi_ref(y_i|x)), to dynamically modulate negative advantages. Theoretically, we demonstrate that ACE's gradient can be decomposed into the gradient of a selective regularizer restricted to overconfident errors, plus a well-characterized residual that partially moderates the regularizer's strength. We conduct extensive experiments fine-tuning Qwen2.5-Math-7B, Qwen3-8B-Base, and Llama-3.1-8B-Instruct on the DAPO-Math-17K dataset using GRPO and DAPO within the VERL framework. Evaluated on MATH-500 and AIME 2025, ACE composes seamlessly with existing methods and consistently improves the full Pass@k spectrum across all three model families and benchmarks.