cs.LGNov 2, 2026

Controlling Sharpening in LLM Post-Training

Authors: Simon Matrenok, Nicolas Menet, Nicolas Flammarion, Andreas Krause

Organizations: School of Computer and Communication Sciences (IC), EPFL · Department of Computer Science, ETH Zürich

Abstract

LLM post-training via reinforcement learning has shown consistent improvements in accuracy at the cost of reducing behavioral coverage - a phenomenon known as sharpening that can stall progress on difficult tasks. Repeated sampling objectives such as pass@k\mathrm{pass@k} and max@k\mathrm{max@k} are known to mitigate sharpening, but the mechanism behind it remains poorly understood. Moreover, existing approaches for @k\mathrm{@k} optimization use dedicated algorithms, and no general framework exists for adapting established policy optimization methods from @1\mathrm{@}1 to @k\mathrm{@k}. We derive the optimal policies for KL-regularized pass@k\mathrm{pass@k} and max@k\mathrm{max@k} objectives in closed form. Our explicit solutions reveal that @k\mathrm{@k} optimization naturally redistributes learning pressure based on prompt difficulty, directing it toward harder problems. Crucially, we show that standard @1\mathrm{@}1 objectives achieve these exact same optimal policies through prompt-adaptive regularization or reward transformation. Ultimately, this yields an optimizer-independent explanation for why @k\mathrm{@k} objectives reduce sharpening on easy tasks while prioritizing success on harder ones. Building on these insights, we introduce @k\mathrm{@k} Adaptive Pressure for Policy Optimization (KAPPO). Following a one-time reference calibration, KAPPO seamlessly adapts existing @1\mathrm{@}1 methods to @k\mathrm{@k} via prompt-adaptive regularization or reward transformation, without otherwise altering their training procedures. Experiments with several RL methods on mathematical reasoning and code generation demonstrate that KAPPO provides direct control over sharpening. Crucially, we demonstrate that @k\mathrm{@k} pressure can be fully realized with single completion training, matching the performance of multi-completion @k\mathrm{@k} optimizers at a fraction of the cost.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Sharpening Tax in Post-Training

    Oct 1, 2026Changdae Oh, Qi Zeng, Qi Qi +7Agentic RLTest-Time Scaling

  2. Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening

    Jan 29, 2026Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer +1Efficient Language Model InferenceRL for Language Model Reasoning

  3. The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning

    Jun 6, 2026Zhanke Zhou, Xiangyu Lu, Chentao Cao +4RL for Language Model ReasoningPolicy Optimization