Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning

Yuanda Xu; Hejian Sang; Zhengze Zhou; Ran He; Zhipeng Wang

cs.LGFeb 24, 2026

Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning

Authors: Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang

Organizations: LinkedIn Corporation, CA, USA

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become the leading paradigm for enhancing reasoning in Large Language Models (LLMs). However, standard RLVR algorithms suffer from a well-documented pathology: while they improve Pass@1 accuracy through sharpened sampling, they simultaneously narrow the model's reasoning boundary and reduce generation diversity. We identify a root cause that existing methods overlook: the uniform penalization of errors. Current approaches -- whether data-filtering methods that select prompts by difficulty, or advantage normalization schemes -- treat all incorrect rollouts within a group identically. We show that this uniformity allows overconfident errors (incorrect reasoning paths that the RL process has spuriously reinforced) to persist and monopolize probability mass, ultimately suppressing valid exploratory trajectories. To address this, we propose the Asymmetric Confidence-aware Error Penalty (ACE). ACE introduces a per-rollout confidence shift metric, c_i = log(pi_theta(y_i|x) / pi_ref(y_i|x)), to dynamically modulate negative advantages. Theoretically, we demonstrate that ACE's gradient can be decomposed into the gradient of a selective regularizer restricted to overconfident errors, plus a well-characterized residual that partially moderates the regularizer's strength. We conduct extensive experiments fine-tuning Qwen2.5-Math-7B, Qwen3-8B-Base, and Llama-3.1-8B-Instruct on the DAPO-Math-17K dataset using GRPO and DAPO within the VERL framework. Evaluated on MATH-500 and AIME 2025, ACE composes seamlessly with existing methods and consistently improves the full Pass@k spectrum across all three model families and benchmarks.

Figures & tables

Figure 1: ACE Method Overview. Top: Incorrect rollouts fall into three regimes based on the confidence shift

c_{i}=\log(\pi_{\theta}(y_{i}|x)/\pi_{\mathrm{ref}}(y_{i}|x))

. Bottom-left: Standard GRPO assigns a uniform penalty

|\hat{A}^{-}|

to all errors regardless of regime. Bottom-right: ACE modulates the penalty via

\text{Softplus}(c_{i})

, strongly penalizing overconfident errors while leaving self-correcting errors nearly untouched.

Model	@1	@2	@4	@8	@16	@32
Qwen2.5-Math-7B	63.0	76.3	83.2	88.1	91.2	93.5
Qwen2.5-Math-7B + GRPO	73.4 $\pm$ 0.8	79.5 $\pm$ 0.7	83.2 $\pm$ 0.7	86.2 $\pm$ 0.5	89.7 $\pm$ 0.5	91.3 $\pm$ 0.5
Qwen2.5-Math-7B + DAPO	74.5 $\pm$ 1.0	80.8 $\pm$ 0.9	84.8 $\pm$ 0.8	89.5 $\pm$ 0.7	93.0 $\pm$ 0.7	94.6 $\pm$ 0.6
Qwen2.5-Math-7B + ACE-GRPO	74.2 $\pm$ 0.7	80.9 $\pm$ 0.7	84.5 $\pm$ 0.6	88.9 $\pm$ 0.5	92.6 $\pm$ 0.5	94.3 $\pm$ 0.4
Qwen2.5-Math-7B + ACE-DAPO	75.1 $\pm$ 0.8	82.4 $\pm$ 0.8	86.2 $\pm$ 0.6	91.2 $\pm$ 0.5	94.7 $\pm$ 0.5	96.1 $\pm$ 0.5
Qwen3-8B-Base	60.2	72.5	78.8	83.9	87.2	90.6

Table 1: Pass@

k

(%) on MATH-500. We report mean

\pm

95% confidence interval over 5 independent training runs. Bold = best within each model group; underline = second best.

Model	1	2	4	8	16	32
Qwen2.5-Math-7B	6.3	9.9	13.8	17.5	21.9	26.7
Qwen2.5-Math-7B + GRPO	10.5	14.9	19.7	23.9	28.6	33.7
Qwen2.5-Math-7B + DAPO	11.5	16.7	22.5	27.5	31.8	37.1
Qwen2.5-Math-7B + ACE-GRPO	11.2	16.0	21.2	26.1	30.6	36.4
Qwen2.5-Math-7B + ACE-DAPO	11.7	17.4	23.8	28.5	33.1	38.6
Qwen3-8B-Base	5.1	9.2	11.6	14.2	17.0	19.6

Table 2: Pass@

k

(%) on AIME 2025. AIME 2025 contains 30 problems; we report point estimates as confidence intervals are dominated by test-set size rather than training variance. Bold = best within each model group; underline = second best.

Figure 2: Performance Comparison across Benchmarks. Pass@ $k$ curves for all five methods on MATH-500 (left column) and AIME 2025 (right column) across three model families: Qwen2.5-Math-7B (top row), Qwen3-8B-Base (middle row), and Llama-3.1-8B-Instruct (bottom row). ACE-GRPO and ACE-DAPO consistently outperform their respective baselines (GRPO and DAPO) across all sampling budgets, model families, and benchmarks, with larger gains at higher $k$ values. ACE-DAPO achieves the best overall performance, confirming that ACE’s rollout-level correction composes with DAPO’s token-level diversity preservation and generalizes across model families.

Figure 3: Overconfident Error Dynamics. Left: Overconfident error fraction (OEF) over training. Right: Mean overconfidence magnitude for $c_{i}>0$ errors. ACE-GRPO effectively suppresses both metrics compared to standard GRPO.

Figure 4: Entropy Dynamics. Token-level entropy over the first 20 training steps. Left: On Qwen2.5-Math-7B, ACE-GRPO retains substantially more entropy than standard GRPO, which suffers rapid entropy collapse. Right: On Qwen3-8B-Base, ACE-GRPO maintains more stable entropy, demonstrating consistency across architectures. We report entropy dynamics for the two Qwen models only; Llama-3.1-8B-Instruct is excluded because its lower baseline accuracy makes the entropy signal less directly comparable (see § 5 for discussion).

Method	@1	@2	@4	@8	@16	@32
GRPO (baseline)	73.4	79.5	83.2	86.2	89.7	91.3
ACE-ReLU	73.2	80.3	83.9	87.6	91.2	93.1
ACE-Softplus (ours)	74.2	80.9	84.5	88.9	92.6	94.3

Table 3: Ablation: modulation function on MATH-500 (Qwen2.5-Math-7B,

\alpha=1.0

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

$\alpha$	Pass@1 (%)	Pass@32 (%)
0.0 (GRPO)	73.4	91.3
0.1	73.5	91.9
0.5	73.8	93.2
1.0 (default)	74.2	94.3
2.0	73.5	93.5
5.0	72.4	92.0

Appendix

Table 4: Sensitivity to

\alpha

on MATH-500 (Qwen2.5-Math-7B).

\alpha{=}1.0

achieves optimal Pass@32 while preserving Pass@1.

Hyperparameter	Qwen2.5-Math-7B	Qwen3-8B-Base	Llama-3.1-8B-Instruct
Total epochs	10	10	10
Training batch size	2048	1024	1024
Mini-batch size	1024	1024	1024
Micro-batch size per GPU	16	16	16
Learning rate	$1\times 10^{-5}$	$5\times 10^{-7}$	$1\times 10^{-6}$
Optimizer	AdamW	AdamW	AdamW

Appendix

Table 5: Training hyperparameters for ACE-GRPO experiments.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Adaptive Negative Reinforcement for LLM Reasoning:Dynamically Balancing Correction and Diversity in RLVR
May 8, 2026Yash Ingle, Jaival Chauhan, Ankit Yadav +1Reinforcement Learning RL for Language Model Reasoning
Sardar Vallabhbhai National Institute of Technology (SVNIT), Surat, India
ConSteer-RL: Steering Reasoning Capabilities in Large Language Models via Confidence-Aware Reinforcement Learning
Jun 6, 2026Qing Miao, Yiming Zhao, Jing Yang +5Reward Shaping RL for Language Model Reasoning
Xi’an Jiaotong University · University of Science and Technology of China
Process Supervision of Confidence Margin for Calibrated LLM Reasoning
Apr 25, 2026Liaoyaqi Wang, Chunsheng Zuo, William Jurayj +2Confidence Estimation in Language Models RL for Language Model Reasoning
Johns Hopkins University

May 8, 2026J/K move · Enter open · S save

Adaptive Negative Reinforcement for LLM Reasoning:Dynamically Balancing Correction and Diversity in RLVR

Yash Ingle, Jaival Chauhan, Ankit Yadav, Sudhakar Mishra

Sardar Vallabhbhai National Institute of Technology (SVNIT), Surat, India

Reinforcement Learning RL for Language Model Reasoning Mathematical Reasoning Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has become a highly effective method for improving the reasoning abilities of Large Language Models (LLMs). Recent research shows that Negative Sample Reinforcement (NSR) -- which focuses on penalizing incorrect steps rather than simply rewarding correct ones -- can match or even exceed the performance of more complex frameworks like PPO and GRPO across the entire Pass@k spectrum. However, current NSR techniques usually apply a fixed penalty throughout the training process and treat every incorrect response with the same weight. To address these limitations, we propose two extensions to the NSR framework: Adaptive Negative Sample Reinforcement. Rather than using a fixed update rule, A-NSR uses time-dependent scheduling functions. In the initial training phases, the system focuses heavily on correcting errors to stabilize the model. As training continues, it shifts toward more subtle and controlled updates. We also introduce Confidence-Weighted Negative Reinforcement, which operates on the principle that different mistakes carry different levels of importance. CW-NSR assigns specific penalty weights based on the model's normalized sequence likelihood. If the model is highly confident in a wrong path, it receives a larger penalty and for uncertain errors -- where the model is effectively exploring -- are penalized less strictly. Our formal analysis shows how these mechanisms govern token-level updates, allowing the model to leverage prior-guided probability redistribution while providing a natural defense against overfitting. We evaluated these methods on difficult reasoning datasets, including MATH, AIME 2025, and AMC23, using the Qwen2.5-Math-1.5B architecture.

Abstract

RL for Language Model Reasoning

Explore similar work

Adaptive Negative Reinforcement for LLM Reasoning:Dynamically Balancing Correction and Diversity in RLVR

ConSteer-RL: Steering Reasoning Capabilities in Large Language Models via Confidence-Aware Reinforcement Learning

Process Supervision of Confidence Margin for Calibrated LLM Reasoning