LLM post-training via reinforcement learning has shown consistent improvements in accuracy at the cost of reducing behavioral coverage - a phenomenon known as sharpening that can stall progress on difficult tasks. Repeated sampling objectives such as pass@k and max@k are known to mitigate sharpening, but the mechanism behind it remains poorly understood. Moreover, existing approaches for @k optimization use dedicated algorithms, and no general framework exists for adapting established policy optimization methods from @1 to @k. We derive the optimal policies for KL-regularized pass@k and max@k objectives in closed form. Our explicit solutions reveal that @k optimization naturally redistributes learning pressure based on prompt difficulty, directing it toward harder problems. Crucially, we show that standard @1 objectives achieve these exact same optimal policies through prompt-adaptive regularization or reward transformation. Ultimately, this yields an optimizer-independent explanation for why @k objectives reduce sharpening on easy tasks while prioritizing success on harder ones. Building on these insights, we introduce @k Adaptive Pressure for Policy Optimization (KAPPO). Following a one-time reference calibration, KAPPO seamlessly adapts existing @1 methods to @k via prompt-adaptive regularization or reward transformation, without otherwise altering their training procedures. Experiments with several RL methods on mathematical reasoning and code generation demonstrate that KAPPO provides direct control over sharpening. Crucially, we demonstrate that @k pressure can be fully realized with single completion training, matching the performance of multi-completion @k optimizers at a fraction of the cost.
Figures & tables
Figure 1: Adaptive pressure induced by @k objectives. (a) For binary rewards, increasing k strengthens regularization on easy tasks while retaining greater optimization pressure on difficult tasks. (b–c) For continuous rewards, we illustrate the same effect at k=4 using synthetic “easy” Beta(8,2) and “difficult” Beta(2,8) reference reward distributions on [0,1] . The easy task receives stronger effective regularization and smaller transformed rewards than the difficult task. All panels use β=0.5 .
Figure 2: pass@keval on AIME 2026 as the sampling budget at test time ( keval ) increases. Setting k>1 during training consistently reduces sharpening. In Figure (a) , we plot standard GRPO and @k - A⋆ PO trained with k∈{1,2,4} . In Figure (b) , we plot @k -DPO trained with k∈{1,2} .
Figure 3: MATH Level 5 validation performance of the starting checkpoints, @k - A⋆ PO checkpoints, and PKPO checkpoints. @k - A⋆ PO matches the performance of PKPO at a fraction of the on-policy training rollouts, requiring only a single-time reference calibration at ≤5% training time overhead.
Figure 4: max@keval on the hard subset of LiveCodeBench v6 for Qwen3-4B. Win-rate-based @k - A⋆ PO (left) and @k -GRPO (right) mitigate excessive sharpening for k>1 .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Exact optimal success probability as a function of the true reference success probability, for β=0.1 . Markers show the attainable empirical estimates j/M for M=30 ; the dotted line marks the first positive estimate 1/M . The map is continuous, but its steep rise near zero is compressed on the linear horizontal scale.
Figure 6: Exact finite-bank noise (left) and signed bias (right) for M=30 and β=0.1 , plotted against the true reference success probability. Panels (a) and (b) use q=N/M ; panel (c) uses Eq. 153 . Coefficient errors are divided by the exact βeff(q) ; the endpoint q=1 , where this coefficient diverges for k>1 , is omitted. Noise uses logarithmic axes, while signed bias uses symmetric-logarithmic axes that are linear within [−10−6,10−6] . Zero noise at deterministic endpoints is omitted from the logarithmic axes.
Figure 7: Win-rate transformation and exact finite-bank noise and bias for M=30 and β=0.1 . To compare different k , values are normalized by ΔT=T(1)−T(0) ; this normalization is only for display, and the transformation panel additionally subtracts T(0) . The horizontal axis is the true reference quantile of a fixed raw reward. Noise and bias use the same axis conventions as Figure 6 .
Figure 8: AIME 2026 test performance of the starting checkpoints and the selected @k - A⋆ PO and PKPO checkpoints across evaluation sampling budgets. Each subfigure reports Qwen3-4B (left) and OLMo-3-7B (right).
Model
Method
pass@1
pass@2
pass@4
pass@8
pass@16
OLMo-3-7B
Starting checkpoint
37.38
50.94
63.16
73.29
81.80
@2 - A⋆ PO
65.34
76.98
84.38
89.11
92.22
PKPO ( ktrain=2 )
66.08
77.38
84.73
89.46
92.52
@4 - A⋆ PO
64.46
76.51
84.39
89.33
92.45
PKPO ( ktrain=4 )
64.31
76.18
83.92
89.04
92.45
Qwen3-4B
Starting checkpoint
23.93
38.32
54.32
68.16
77.79
Appendix
Table 1: MATH Level 5 validation performance (in %) of the starting checkpoints and selected @k - A⋆ PO and PKPO checkpoints, corresponding to Figure 3 .
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
An emerging hypothesis about reinforcement learning (RL) post-training of large language models (LLMs) is that it merely sharpens existing behaviors of a base model, improving single-shot accuracy at the cost of solution coverage. Although this trade-off has been observed in math and coding tasks, it need not extend to agentic tasks, where multi-turn tool use and interaction may require capabilities newly acquired during post-training. Our surprising finding is that pre-trained LLMs, equipped with a light inference harness, can serve as capable agents. Despite far lower accuracy (pass@1), they often surpass their post-trained counterparts in solution coverage (pass@K) given a sufficient test-time budget. We further analyze the underlying mechanism and show that post-training pushes tasks toward two extremes, always solved or never solved, and thereby improves sampling efficiency and consistency at the cost of solution coverage. To measure this cost, we propose Sharpening Tax, a diagnostic metric that quantifies the loss in test-time scalability after post-training. Across 14 base/post-trained model pairs from four families and three agentic benchmarks (42 cases in total), the tax is prevalent in most settings, can be estimated from a few rollouts, and correlates well with other metrics. Finally, we present posterior-tempered group sampling (PTGS), a simple plug-and-play Bayesian sampler that adapts the sampling temperature per prompt to its estimated difficulty. Applied during RL training in two agentic environments, PTGS pays a smaller tax than the fixed-temperature baseline, solving more tasks under repeated sampling while also improving single-shot accuracy.
Changdae Oh, Qi Zeng, Qi Qi +7
Meta Superintelligence Labs · University of Wisconsin–Madison · Work done at Meta +2
Reinforcement learning (RL) post-training is a dominant approach for improving the reasoning performance of large language models (LLMs), yet growing evidence suggests that its gains arise primarily from distribution sharpening rather than the acquisition of new capabilities. Recent work has shown that sampling from the power distribution of LLMs using Markov chain Monte Carlo (MCMC) can recover performance comparable to RL post-training without relying on external rewards; however, the high computational cost of MCMC makes such approaches impractical for widespread adoption. In this work, we propose a theoretically grounded alternative that eliminates the need for iterative MCMC. We derive a token-level decomposition of the autoregressive power distribution, showing that the global power distribution can be approximated by a token-level scaled low-temperature one, where the scaling factor captures future trajectory quality. Leveraging this insight, we introduce a training-free and verifier-free algorithm that sharpens the base model's generative distribution autoregressively. Empirically, we evaluate our method on math, QA, and code tasks, and show that our method matches or surpasses one-shot GRPO without relying on any external rewards, while reducing inference latency by over 10× compared to MCMC-based sampling.
Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer +1
Huawei Noah’s Ark Lab · UCL Centre for Artificial Intelligence
RL with verifiable rewards can substantially improve LLM reasoning, yet standard GRPO-style training often treats easy, hard, and learnable questions alike through uniform sampling and weighting, leading to inefficient compute allocation. We study GRPO by tracking token log-probabilities, group-normalized advantages, and the induced token-level update weights. This reveals three recurring dynamics as training proceeds: (1) confidence inflation, (2) advantage contraction, and (3) hierarchical convergence. These findings suggest that the utility of each update depends strongly on both question difficulty and the model's current competence. Motivated by this, we propose Confidence and Difficulty-adaptive Policy Optimization (CoDaPO), which assigns each question a bounded value from rollout confidence and empirical difficulty. CoDaPO then uses this value to reweight policy updates and resample high-value learnable questions within mini-batches, thereby increasing discovery within the learnable band under a fixed compute budget. Across twelve benchmarks, CoDaPO consistently improves accuracy over existing RL methods. Our code is publicly available at https://github.com/tmlr-group/CoDaPO.
Zhanke Zhou, Xiangyu Lu, Chentao Cao +4
1TMLR Group, Department of Computer Science, Hong Kong Baptist University · 2Stanford University · 3Sydney AI Centre, The University of Sydney