Reinforcement Learning with Verifiable Rewards (RLVR) has become the leading paradigm for enhancing reasoning in Large Language Models (LLMs). However, standard RLVR algorithms suffer from a well-documented pathology: while they improve Pass@1 accuracy through sharpened sampling, they simultaneously narrow the model's reasoning boundary and reduce generation diversity. We identify a root cause that existing methods overlook: the uniform penalization of errors. Current approaches -- whether data-filtering methods that select prompts by difficulty, or advantage normalization schemes -- treat all incorrect rollouts within a group identically. We show that this uniformity allows overconfident errors (incorrect reasoning paths that the RL process has spuriously reinforced) to persist and monopolize probability mass, ultimately suppressing valid exploratory trajectories. To address this, we propose the Asymmetric Confidence-aware Error Penalty (ACE). ACE introduces a per-rollout confidence shift metric, c_i = log(pi_theta(y_i|x) / pi_ref(y_i|x)), to dynamically modulate negative advantages. Theoretically, we demonstrate that ACE's gradient can be decomposed into the gradient of a selective regularizer restricted to overconfident errors, plus a well-characterized residual that partially moderates the regularizer's strength. We conduct extensive experiments fine-tuning Qwen2.5-Math-7B, Qwen3-8B-Base, and Llama-3.1-8B-Instruct on the DAPO-Math-17K dataset using GRPO and DAPO within the VERL framework. Evaluated on MATH-500 and AIME 2025, ACE composes seamlessly with existing methods and consistently improves the full Pass@k spectrum across all three model families and benchmarks.
Figures & tables
Figure 1: ACE Method Overview. Top: Incorrect rollouts fall into three regimes based on the confidence shift ci=log(πθ(yi∣x)/πref(yi∣x)) . Bottom-left: Standard GRPO assigns a uniform penalty ∣A^−∣ to all errors regardless of regime. Bottom-right: ACE modulates the penalty via Softplus(ci) , strongly penalizing overconfident errors while leaving self-correcting errors nearly untouched.
Model
@1
@2
@4
@8
@16
@32
Qwen2.5-Math-7B
63.0
76.3
83.2
88.1
91.2
93.5
Qwen2.5-Math-7B + GRPO
73.4 ± 0.8
79.5 ± 0.7
83.2 ± 0.7
86.2 ± 0.5
89.7 ± 0.5
91.3 ± 0.5
Qwen2.5-Math-7B + DAPO
74.5 ± 1.0
80.8 ± 0.9
84.8 ± 0.8
89.5 ± 0.7
93.0 ± 0.7
94.6 ± 0.6
Qwen2.5-Math-7B + ACE-GRPO
74.2 ± 0.7
80.9 ± 0.7
84.5 ± 0.6
88.9 ± 0.5
92.6 ± 0.5
94.3 ± 0.4
Qwen2.5-Math-7B + ACE-DAPO
75.1 ± 0.8
82.4 ± 0.8
86.2 ± 0.6
91.2 ± 0.5
94.7 ± 0.5
96.1 ± 0.5
Qwen3-8B-Base
60.2
72.5
78.8
83.9
87.2
90.6
Table 1: Pass@ k (%) on MATH-500. We report mean ± 95% confidence interval over 5 independent training runs. Bold = best within each model group; underline = second best.
Model
1
2
4
8
16
32
Qwen2.5-Math-7B
6.3
9.9
13.8
17.5
21.9
26.7
Qwen2.5-Math-7B + GRPO
10.5
14.9
19.7
23.9
28.6
33.7
Qwen2.5-Math-7B + DAPO
11.5
16.7
22.5
27.5
31.8
37.1
Qwen2.5-Math-7B + ACE-GRPO
11.2
16.0
21.2
26.1
30.6
36.4
Qwen2.5-Math-7B + ACE-DAPO
11.7
17.4
23.8
28.5
33.1
38.6
Qwen3-8B-Base
5.1
9.2
11.6
14.2
17.0
19.6
Table 2: Pass@ k (%) on AIME 2025. AIME 2025 contains 30 problems; we report point estimates as confidence intervals are dominated by test-set size rather than training variance. Bold = best within each model group; underline = second best.
Figure 2: Performance Comparison across Benchmarks. Pass@ k curves for all five methods on MATH-500 (left column) and AIME 2025 (right column) across three model families: Qwen2.5-Math-7B (top row), Qwen3-8B-Base (middle row), and Llama-3.1-8B-Instruct (bottom row). ACE-GRPO and ACE-DAPO consistently outperform their respective baselines (GRPO and DAPO) across all sampling budgets, model families, and benchmarks, with larger gains at higher k values. ACE-DAPO achieves the best overall performance, confirming that ACE’s rollout-level correction composes with DAPO’s token-level diversity preservation and generalizes across model families.
Figure 3: Overconfident Error Dynamics. Left: Overconfident error fraction (OEF) over training. Right: Mean overconfidence magnitude for ci>0 errors. ACE-GRPO effectively suppresses both metrics compared to standard GRPO.
Figure 4: Entropy Dynamics. Token-level entropy over the first 20 training steps. Left: On Qwen2.5-Math-7B, ACE-GRPO retains substantially more entropy than standard GRPO, which suffers rapid entropy collapse. Right: On Qwen3-8B-Base, ACE-GRPO maintains more stable entropy, demonstrating consistency across architectures. We report entropy dynamics for the two Qwen models only; Llama-3.1-8B-Instruct is excluded because its lower baseline accuracy makes the entropy signal less directly comparable (see § 5 for discussion).
Method
@1
@2
@4
@8
@16
@32
GRPO (baseline)
73.4
79.5
83.2
86.2
89.7
91.3
ACE-ReLU
73.2
80.3
83.9
87.6
91.2
93.1
ACE-Softplus (ours)
74.2
80.9
84.5
88.9
92.6
94.3
Table 3: Ablation: modulation function on MATH-500 (Qwen2.5-Math-7B, α=1.0 ).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
α
Pass@1 (%)
Pass@32 (%)
0.0 (GRPO)
73.4
91.3
0.1
73.5
91.9
0.5
73.8
93.2
1.0 (default)
74.2
94.3
2.0
73.5
93.5
5.0
72.4
92.0
Appendix
Table 4: Sensitivity to α on MATH-500 (Qwen2.5-Math-7B). α=1.0 achieves optimal Pass@32 while preserving Pass@1.
Hyperparameter
Qwen2.5-Math-7B
Qwen3-8B-Base
Llama-3.1-8B-Instruct
Total epochs
10
10
10
Training batch size
2048
1024
1024
Mini-batch size
1024
1024
1024
Micro-batch size per GPU
16
16
16
Learning rate
1×10−5
5×10−7
1×10−6
Optimizer
AdamW
AdamW
AdamW
Appendix
Table 5: Training hyperparameters for ACE-GRPO experiments.
Reinforcement learning with verifiable rewards (RLVR) has become a highly effective method for improving the reasoning abilities of Large Language Models (LLMs). Recent research shows that Negative Sample Reinforcement (NSR) -- which focuses on penalizing incorrect steps rather than simply rewarding correct ones -- can match or even exceed the performance of more complex frameworks like PPO and GRPO across the entire Pass@k spectrum. However, current NSR techniques usually apply a fixed penalty throughout the training process and treat every incorrect response with the same weight. To address these limitations, we propose two extensions to the NSR framework: Adaptive Negative Sample Reinforcement. Rather than using a fixed update rule, A-NSR uses time-dependent scheduling functions. In the initial training phases, the system focuses heavily on correcting errors to stabilize the model. As training continues, it shifts toward more subtle and controlled updates. We also introduce Confidence-Weighted Negative Reinforcement, which operates on the principle that different mistakes carry different levels of importance. CW-NSR assigns specific penalty weights based on the model's normalized sequence likelihood. If the model is highly confident in a wrong path, it receives a larger penalty and for uncertain errors -- where the model is effectively exploring -- are penalized less strictly. Our formal analysis shows how these mechanisms govern token-level updates, allowing the model to leverage prior-guided probability redistribution while providing a natural defense against overfitting. We evaluated these methods on difficult reasoning datasets, including MATH, AIME 2025, and AMC23, using the Qwen2.5-Math-1.5B architecture.
Yash Ingle, Jaival Chauhan, Ankit Yadav +1
Sardar Vallabhbhai National Institute of Technology (SVNIT), Surat, India
Reinforcement Learning from Verifiable Rewards (RLVR) has recently become a key paradigm for improving the reasoning abilities of Large Language Models (LLMs), yet it remains limited by sparse binary rewards and its ignorance of model-internal uncertainty. In this paper, we propose ConSteer-RL, a simple yet effective framework that integrates token-level confidence signals derived from model log-probabilities into RLVR training. Specifically, building upon the Group Relative Policy Optimization (GRPO) framework, we construct a confidence-aware reward by aggregating per-token probabilities into a scalar confidence score and incorporating it into an awareness-based reward shaping mechanism that penalizes overconfident errors while reinforcing correct and confident reasoning. Experimental results demonstrate that ConSteer-RL consistently outperforms strong GRPO baselines, achieving average improvements of 2.3%-4.0% across different model scales.
Qing Miao, Yiming Zhao, Jing Yang +5
Xi’an Jiaotong University · University of Science and Technology of China
Scaling test-time computation with reinforcement learning (RL) has emerged as a reliable path to improve large language models (LLM) reasoning ability. Yet, outcome-based reward often incentivizes models to be overconfident, leading to hallucinations, unreliable confidence-based control, and unnecessary compute allocation. We introduce Reinforcement Learning with Confidence Margin (RLCM), a calibration-aware RL framework that jointly optimizes correctness and confidence reliability via a margin-enhanced process reward over intermediate-budget completions. Rather than aligning confidence to correctness likelihoods, RLCM encourages to widen the confidence margin between correct and incorrect steps within a single reasoning trajectory. Across mathematical, code, logic and science benchmarks, our method substantially improves calibration while maintaining or improving accuracy. We further show that, with calibrated confidence signals, the resulting models enable more efficient conformal risk control and effective confidence-weighted aggregation.