cs.LGFeb 22, 2026

Smooth Gate Functions for Soft Advantage Policy Optimization

Authors: Egor Denisov, Svetlana Glazyrina, Maksim Kryzhanovskiy, Roman Ischenko

Organizations: Lomonosov Moscow State University, Moscow, Russia · Institute for Artificial Intelligence, Lomonosov Moscow State University, Moscow, Russia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Group Relative Policy Optimization (GRPO) has significantly advanced the training of large language models and enhanced their reasoning capabilities, while it remains susceptible to instability due to the use of hard clipping. Soft Adaptive Policy Optimization (SAPO) addresses this limitation by replacing clipping with a smooth sigmoid-based gate function, which leads to more stable updates. We have decided to push this theory further and investigate the impact of different gate functions on both training stability and final model performance. We formalize the key properties that admissible gates should satisfy and identify several families of such functions for empirical evaluation. This paper presents an analysis of our findings based on experiments conducted with the Qwen2.5-7B-Instruct model on mathematical reasoning tasks. These results provide practical guidance for designing smoother and more robust policy optimization objectives for large language model training.

Figures & tables

Figure 1: Temperature-dependent behavior of the considered gate functions (top row) and their gradients (bottom row) for τ∈{1,5,10}\tau\in\{1,5,10\} . All functions are normalized to be positive on [0;+∞)[0;+\infty) and to pass through the point (1,1)(1,1) . Increasing the temperature sharpens the transition around x=1x=1 , leading to more localized and higher gradient peaks for smooth gates, while the clipped variant exhibits piecewise-linear behavior with constant gradients in its active region.
Figure 2
Figure 4: Policy entropy (left) and effective update ratio (right) over training for different gate functions and GRPO. For all SAPO-like methods τpos\tau_{pos} is set to 1010 and τneg\tau_{neg} is set to 1212 .
Figure 5: Policy entropy (left) and effective update ratio (right) over training for ferff_{\mathrm{erf}} with different temperature configurations.
ModelGSM8KMATH500AIME
Qwen2.5-7B-Instruct80.356.63.3
+ GRPO80.658.68.3
+ SAPO (sigmoid baseline)82.257.88.3
+ SAPO (erf)82.756.210.0
+ SAPO (arctan)82.355.85.0
+ SAPO (softsign)82.857.25.0
Table 1: Accuracy (%) of different model configurations on mathematical reasoning benchmarks.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

CategoryParameterValue
GenerationMAX_PROMPT_LENGTH1536
MAX_COMPLETION_LENGTH512
NUM_GENERATIONS8
BatchingPER_DEVICE_TRAIN_BATCH_SIZE1
GRAD_ACCUM16
NUM_ITERATIONS2
Appendix
Table 2: Training and generation hyperparameters used across all experiments.

Explore similar work

CardsList

  1. Soft Sequence Policy Optimization

    Feb 22, 2026Svetlana Glazyrina, Maksim Kryzhanovskiy, Roman IschenkoLLM AlignmentRL Post-Training

    Lomonosov Moscow State University, Moscow, Russia · Institute for Artificial Intelligence, Lomonosov Moscow State University, Moscow, Russia

  2. GRPO Training Dynamics for Small Language Models

    Sep 30, 2026Rajat Ghosh, Vaishnavi Bhargava, Henry Wong +2RL for Language Model ReasoningGroup Relative Policy Optimization

    Nutanix

  3. Holder Policy Optimisation

    May 12, 2026Yuxiang Chen, Dingli Liang, Yihang Chen +8RL for Language Model ReasoningGroup Relative Policy Optimization

    University College London, London, United Kingdom. · 3The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · 2Shanghai Jiao Tong University, Shanghai, China.