Flat-RLVR: Sharpness-Aware Reinforcement Learning with Verifiable Rewards for Reasoning Capacity in LLMs

Yeonbum Kim; Hyun Kyu Lee; Sung Whan Yoon

cs.LGOct 21, 2026

Flat-RLVR: Sharpness-Aware Reinforcement Learning with Verifiable Rewards for Reasoning Capacity in LLMs

Authors: Yeonbum Kim, Hyun Kyu Lee, Sung Whan Yoon

Organizations: Graduate School of Artificial Intelligence, Ulsan National Institute of Science and Technology, Ulsan, Republic of Korea · Department of Electrical Engineering, Ulsan National Institute of Science and Technology (UNIST), Ulsan, Republic of Korea.

Abstract

Reinforcement learning with verifiable rewards (RLVR), particularly through group-relative policy optimization (GRPO), has emerged as a central paradigm for the post-training of large language models (LLMs). While RLVR is highly effective at enhancing the reasoning capabilities of LLMs, it commonly induces distribution sharpening, whereby the policy distribution becomes excessively peaked and rare-yet-correct trajectories are progressively suppressed, thus diminishing sample diversity. To this end, we promote flat-reward optima in RLVR to smooth the policy distribution. We theoretically link flat reward RLVR to reduced distribution sharpening, and empirically show improved sample diversity while maintaining single-attempt accuracy. Our method, called Flat-RLVR, searches for flat reward regions via sharpness-aware minimization during LLM post-training, combining this with GRPO variants to achieve better trade-offs between sampling efficiency and reasoning capacity.

Figures & tables

Figure 1: RLVR and Flat -RLVR on LLM reasoning capacity. Gray nodes indicate paths that are unlikely to be sampled by the model, black nodes indicate paths that are likely to be sampled, i.e., each path consists of a sequence of outputs, later formulated by $o_{j}$ , where $j$ is the path index. Green nodes indicate correct solutions, and red nodes indicate incorrect.

Figure 2: Illustration of the loss landscape and reasoning capacity of RLVR and Flat -RLVR. Flat -RLVR converges to a flatter minimum in the loss landscape by jointly optimizing for loss reduction and sharpness minimization, whereas RLVR is drawn to a sharper minimum driven solely by reward maximization. As shown in the trees (see Figure 1 ), Flat -RLVR explores a broader reasoning tree, while RLVR collapses to a narrow set of paths.

\hat{A}_{i}=\frac{r_{i}-\mathrm{mean}(\{r_{j}\})}{\mathrm{std}(\{r_{j}\})}

Algorithm 1 Flat -RLVR

Panel of Figure 3 — Figure 3: Pass@K curves of BASE model ((a), (c), and (d) Qwen2.5-Math-1.5B and (b) Llama-3.2-1B-Instruct), GRPO, and Flat -GRPO (ours) on in-domain mathematical reasoning benchmarks, traced from $K=1$ to $K=256$ . All Pass@ $K$ curves per-BASE/benchmark are provided in Figure 8 .

Base	RL-Tuned	GRPO	Flat -GRPO
$\checkmark$	$\checkmark$	$57.7$	$61.2$
$\checkmark$	$\times$	$6.3$	$2.8$
$\times$	$\checkmark$	$5.4$	$6.3$
$\times$	$\times$	$30.6$	$29.7$
Total		$100.0$	$100.0$

Table 1: Solvable-problem coverage (%) averaged over all in-domain benchmarks on Qwen2.5-Math-1.5B. Each row is a category defined by whether the base and the RL-tuned model solve a problem (

\checkmark

) or not (

\times

Model	Method	Avg.	MATH500	AMC23	AIME24	AIME25	AIME26
Qwen2.5	GRPO	15.6 $\pm 1.0$ /64.8 $\pm 3.7$	34.6 $\pm 0.0$ / 72.7 $\pm 0.7$	30.2 $\pm 2.8$ /95.8 $\pm 7.1$	7.0 $\pm 1.2$ /62.2 $\pm 4.7$	2.9 $\pm 0.1$ /48.9 $\pm 12.6$	3.7 $\pm 0.8$ /44.4 $\pm 4.7$
	Flat -GRPO	15.2 $\pm 0.6$ / 67.3 $\pm 3.0$	34.0 $\pm 0.0$ /72.5 $\pm 0.5$	28.9 $\pm 2.4$ / 96.7 $\pm 7.1$	6.5 $\pm 1.5$ / 64.4 $\pm 9.5$	3.3 $\pm 2.4$ / 51.1 $\pm 4.7$	3.8 $\pm 0.7$ / 52.2 $\pm 4.7$
	DAPO	16.3 $\pm 0.2$ /65.2 $\pm 5.8$	35.7 $\pm 0.1$ /72.1 $\pm 2.0$	32.0 $\pm 0.7$ / 98.3 $\pm 3.5$	6.9 $\pm 1.0$ /63.3 $\pm 8.2$	3.1 $\pm 0.8$ /51.1 $\pm 9.5$	3.9 $\pm 0.2$ /41.1 $\pm 12.6$
	Flat -DAPO	16.3 $\pm 0.1$ / 68.6 $\pm 0.5$	35.7 $\pm 0.2$ / 72.4 $\pm 0.4$	32.1 $\pm 0.2$ /97.5 $\pm 6.2$	6.9 $\pm 0.6$ 66.7 $\pm 14.3$	3.1 $\pm 0.2$ / 57.8 $\pm 4.7$	3.9 $\pm 0.5$ / 48.9 $\pm 12.6$
	CISPO	15.7 $\pm 0.3$ /65.5 $\pm 2.6$	34.7 $\pm 0.1$ / 72.8 $\pm 1.4$	30.6 $\pm 0.9$ /98.3 $\pm 7.1$	6.9 $\pm 0.8$ /63.3 $\pm 8.2$	2.9 $\pm 0.4$ /50.0 $\pm 14.3$	3.9 $\pm 0.4$ /43.3 $\pm 0.0$
	Flat -CISPO	15.6 $\pm 0.1$ / 68.5 $\pm 3.0$	33.8 $\pm 0.2$ /72.6 $\pm 0.4$	30.7 $\pm 0.8$ / 99.2 $\pm 3.5$	6.5 $\pm 0.8$ / 67.8 $\pm 4.7$	3.1 $\pm 0.4$ / 55.6 $\pm 17.2$	4.2 $\pm 0.2$ / 47.8 $\pm 4.7$

Table 2: Pass@

1

/Pass@256 (

\%

) values across two base models with varying RLVR methods at

G=6

group size, reported as mean

\pm 95\%

CI over 3 seeds. Avg. : average Pass@

1

/Pass@

256

. Bold : better within baseline/ Flat pair.

Benchmarks	GRPO	F-GRPO	Flat -GRPO
MATH500	34.6 $\pm 0.0$ / 72.7 $\pm 0.3$	33.2 $\pm 0.0$ / 72.1 $\pm 0.6$	34.0 $\pm 0.0$ / 72.5 $\pm 0.2$
AMC23	30.2 $\pm 1.1$ / 95.8 $\pm 2.9$	28.4 $\pm 1.8$ / 98.3 $\pm 1.4$	28.9 $\pm 1.0$ / 96.7 $\pm 2.9$
AIME24	7.0 $\pm 0.5$ / 62.2 $\pm 1.9$	6.4 $\pm 0.7$ / 66.7 $\pm 6.7$	6.5 $\pm 0.6$ / 64.4 $\pm 3.8$
AIME25	2.9 $\pm 0.1$ / 48.9 $\pm 5.1$	2.6 $\pm 0.2$ / 47.8 $\pm 1.9$	3.3 $\pm 1.0$ / 51.1 $\pm 1.9$
AIME26	3.7 $\pm 0.3$ / 44.4 $\pm 1.9$	3.5 $\pm 0.1$ / 41.1 $\pm 1.9$	3.8 $\pm 0.3$ / 52.2 $\pm 1.9$
Avg.	15.6 $\pm 0.45$ / 64.8 $\pm 1.7$	14.8 $\pm 0.7$ / 65.2 $\pm 2.3$	15.2 $\pm 0.4$ / 67.3 $\pm 1.3$

Table 3: Pass@1/Pass@256 (

\%

) for GRPO, F-GRPO, and Flat -GRPO (ours), with Qwen2.5-Math-1.5B (top) and Llama-3.2-1B-Instruct (bottom) as the backbone, reported as mean

\pm 95\%

CI over 3 seeds.

Method	Avg.	MATH500	AMC23	AIME24	AIME25	AIME26
BASE	16.7 $\pm 0.2$ /60.8 $\pm 1.3$	44.5 $\pm 0.1$ /70.5 $\pm 0.3$	31.1 $\pm 0.8$ /98.3 $\pm 7.2$	4.5 $\pm 0.2$ /47.8 $\pm 9.6$	1.6 $\pm 0.1$ /50.0 $\pm 8.3$	2.0 $\pm 0.3$ /37.8 $\pm 9.6$
GRPO	18.5 $\pm 0.2$ /59.4 $\pm 0.7$	46.9 $\pm 0.1$ / 70.3 $\pm 0.6$	36.9 $\pm 0.9$ /95.8 $\pm 3.6$	5.2 $\pm 0.1$ /42.2 $\pm 12.6$	1.6 $\pm 0.5$ /46.7 $\pm 8.3$	2.3 $\pm 0.3$ /42.2 $\pm 4.8$
Flat -GRPO	18.2 $\pm 0.2$ / 61.4 $\pm 1.6$	46.7 $\pm 0.1$ /69.5 $\pm 0.3$	35.9 $\pm 0.7$ / 97.5 $\pm 0.0$	4.7 $\pm 0.5$ / 48.9 $\pm 9.6$	1.6 $\pm 0.2$ / 47.8 $\pm 4.8$	2.2 $\pm 0.3$ / 43.3 $\pm 0.0$

Table 4: Pass@1/Pass@256 (

\%

) results on a larger model (3B) with mean ±95

\%

CI over 3 seeds. Flat -RLVR consistently improves the average Pass@

256

while preserving the average Pass@

1

at this larger scale.

Method	MBPP
BASE	42.3 $\pm 0.1$ / 91.5 $\pm 1.1$
GRPO	56.7 $\pm 0.0$ / 85.6 $\pm 0.9$
Flat -GRPO	54.5 $\pm 0.1$ / 89.6 $\pm 1.6$

Table 5: Pass@1/Pass@256 (

\%

) with Qwen2.5-Coder-1.5B-Instruct on code generation test (MBPP), reported as mean ±95

\%

CI over 3 seeds. Flat -GRPO achieves higher Pass@256 than GRPO at comparable Pass@1.

Panel of Figure 4 — Figure 4: Correct-set log-probability gap $D_{h,l}$ on a fixed probe set (a) and trajectory of $\log\pi_{\theta}(o_{\ell})$ , the log-probability of the lowest-probability correct response (b) during training.

Method	LPF $\downarrow$
GRPO	0.4434
Flat -GRPO	0.0598

Table 6: Low-Pass Filter (LPF) of the model fine-tunded by GRPO and Flat -GRPO algorithms at perturbation scale

\sigma=0.001

Panel of Figure 5 — Figure 5: Pass@K curves on (a) AMC23 and (b) AIME25 test datasets (from $K=1$ to $K=256$ ) comparing GRPO with Flat -GRPO under two SAM periods, $n=5$ and $n=10$ .

Method	Total (h)	allocated( $\%$ )
Flat -GRPO ( $n=5$ )	$17.00$	$90.82$
Flat -GRPO ( $n=10$ )	$16.87$	$88.42$
GRPO	$16.91$	$90.96$

Table 7: Wall-clock training time and averaged per-GPU memory allocation for Flat -GRPO with

n

-step

	per-step wall clock time (s)
ordinary step	90.17
SAM step	90.56 (+0.43 $\%$ )

Table 8: Average time for each step in Flat -GRPO. We compare SAM steps with ordinary steps to estimate the computational overhead stemming from the additional gradient of SAM.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Training entropy curves on GSM8K with Qwen2.5-0.5B-Instruct: SAM with Gradient Centralization (GC) applied at both the ascent and descent steps (GC2+aSAM+GRPO) versus only at the ascent step (GC+aSAM+GRPO). Applying GC before the descent step substantially mitigates entropy collapse compared to the ascent-only variant.

Figure 7: Training reward curves on GSM8K with Qwen2.5-0.5B-Instruct: SAM with cosine annealing (aSAM + GRPO) versus without (SAM + GRPO) (i.e., periodic-SAM without cosine annealing). Without annealing, the training reward collapses to zero, halting effective learning.

Method	Avg.	MATH500	AMC23	AIME24	AIME25	AIME26
GRPO	15.2/63.1	34.6/73.0	28.8/92.5	6.4/63.3	2.8/43.3	3.3/43.3
GRPO + SAM	9.8 $\downarrow 5.4$ /64.6 $\uparrow 1.5$	22.5/73.0	18.8/100	4.2/63.3	1.6/46.7	2.1/40.0
GRPO + SAM + Anne.	13.8 $\downarrow 1.4$ /65.5 $\uparrow 2.4$	30.7/72.4	26.7/95.0	5.8/60.0	2.4/53.3	3.3/46.7
GRPO + SAM + GC	11.6 $\downarrow 3.6$ /64.7 $\uparrow 1.6$	25.8/72.8	22.5/97.5	4.4/56.7	2.3/50.0	3.1/46.7
GRPO + SAM + F	15.5 $\uparrow 0.3$ /65.8 $\uparrow 2.7$	35.1/71.6	28.8/97.5	7.2/66.7	2.6/46.7	3.9/46.7
Flat -GRPO	15.0 $\downarrow 0.2$ /67.5 $\uparrow 4.4$	33.9/72.4	27.7/95.0	7.1/66.7	2.8/50.0	3.4/53.3

Appendix

Table 9: Additive Ablation Study on mathemathic benchmarks. We fix periodic calculation (SAM) and add each strategy separately; Cosine Annealing (Anne.), Gradient Centralization (GC), and Focal Weight (F), reporting Pass@1/Pass@256 at each stage. Avg.: average Pass@1/Pass@256.

\uparrow

\&

\downarrow

: the change Pass@1/Pass@256 relative to the naive GRPO.

Qwen2.5-Math-1.5B (Base vs. GRPO)
Backbone	Base	RL	MATH500	AMC23	AIME24	AIME25	AIME26	Avg.
Qwen2.5	$\checkmark$	$\checkmark$	$71.2$	$87.5$	$53.3$	$43.3$	$33.3$	$57.7$
	$\checkmark$	$\times$	$0.6$	$7.5$	$6.7$	$10.0$	$6.7$	$6.3$
	$\times$	$\checkmark$	$1.8$	$5.0$	$10.0$	$0.0$	$10.0$	$5.4$
	$\times$	$\times$	$26.4$	$0.0$	$30.0$	$46.7$	$50.0$	$30.6$
Qwen2.5-Math-1.5B (Base vs. Flat -GRPO)

Appendix

Table 10: Full per-benchmark solvable-problem coverage (%) for the base–RL pairs on both backbones. Each row is a category defined by whether the base model and the RL-tuned model solve a problem (

\checkmark

) or not (

\times

): (

\checkmark

\times

) counts problems the RL stage forgets , and (

\times

\checkmark

) counts problems it newly solves . Averages correspond to Table 1 in the main text.

Method	Parameter	Value
SAM- $n$	period $n$	$10$
Cosine Annealing	$\rho_{\text{init}}$	$0.01$
	$\rho_{\text{min}}$	$0.002$
Focal Weight	$\gamma$	$1.0$
	$\rho^{\text{focal}}_{\text{min}}$	$0.001$

Appendix

Table 11: Training hyperparameters for Flat -RLVR.

Parameter	Qwen2.5-Math-1.5B	Llama-3.2-1B-Instruct	Qwen2.5-0.5B-Instruct	Qwen2.5-3B-Instruct	Qwen2.5-Coder-1.5B-Instruct
Optimizer	AdamW
$(\beta_{1},\beta_{2})$	$(0.9,0.999)$
Weight decay	$0.01$
Learning rate	$1\times 10^{-6}$
LR scheduler	Linear
Num. generations ( $G$ )	$6$

Appendix

Table 12: Full training hyperparameters for all models and methods.

Figure 8: Pass@K curves on all in-domain benchmarks (MATH500, AMC23, AIME24, AIME25, AIME26), comparing the base model, GRPO, and Flat -GRPO (ours) from

K=1

K=256

. Top row: Qwen2.5-Math-1.5B; bottom row: Llama-3.2-1B-Instruct.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,103 papers

Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening
Jan 29, 2026Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer +1Efficient Language Model Inference RL for Language Model Reasoning
Huawei Noah’s Ark Lab · UCL Centre for Artificial Intelligence
A Critical Look at Power Sampling when Test Time Sharpening LLMs
Oct 9, 2026Abiel Malkani Talwar, Christopher Nemeth, Henry MossRL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Lancaster University Lancaster, United Kingdom
ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning
May 1, 2026Zihan Lin, Xiaohan Wang, Jie Cao +6RL for Language Models Reinforcement Learning
MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Meituan, Beijing, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China.

Jan 29, 2026J/K move · Enter open · S save

Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening

Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer, Haitham Bou Ammar

Huawei Noah’s Ark Lab · UCL Centre for Artificial Intelligence

Efficient Language Model Inference RL for Language Model Reasoning

Open paper AskPDF arXiv

Reinforcement learning (RL) post-training is a dominant approach for improving the reasoning performance of large language models (LLMs), yet growing evidence suggests that its gains arise primarily from distribution sharpening rather than the acquisition of new capabilities. Recent work has shown that sampling from the power distribution of LLMs using Markov chain Monte Carlo (MCMC) can recover performance comparable to RL post-training without relying on external rewards; however, the high computational cost of MCMC makes such approaches impractical for widespread adoption. In this work, we propose a theoretically grounded alternative that eliminates the need for iterative MCMC. We derive a token-level decomposition of the autoregressive power distribution, showing that the global power distribution can be approximated by a token-level scaled low-temperature one, where the scaling factor captures future trajectory quality. Leveraging this insight, we introduce a training-free and verifier-free algorithm that sharpens the base model's generative distribution autoregressively. Empirically, we evaluate our method on math, QA, and code tasks, and show that our method matches or surpasses one-shot GRPO without relying on any external rewards, while reducing inference latency by over

10\times

compared to MCMC-based sampling.

Abstract

RL for Language Model Reasoning

Explore similar work

Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening

A Critical Look at Power Sampling when Test Time Sharpening LLMs

ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning