cs.LGNov 7, 2026

BRAT: Bounded Residual-Aligned Training for Stable Multi-Pass LLM Post-Training

Authors: Xunlan Zhou, Yuzhe Zhao, Xuhong Huang, Feng Chen, Xuanlin Chen, Lang Feng, Bo An

Organizations: Nanjing University · Shanghai Jiao Tong University · The University of Hong Kong · Nanyang Technological University

Abstract

Reinforcement learning for large language model post-training often reuses each batch of sampled rollouts for several gradient passes to amortize the cost of generation. Under such multi-pass reuse, existing objectives fail in different ways: unclipped updates collapse as the policy drifts away from the sampling policy, adding a negative baseline avoids collapse but reinforces every sampled response, and PPO-style ratio clipping cuts off gradients abruptly. We first ask why reuse is fragile. Measuring how well an update estimated from a small rollout group aligns with the KL-regularized target on held-out rollouts of the same prompts, we find that this alignment drops by nearly an order of magnitude from in-sample to held-out rollouts across three base models, suggesting that repeated passes largely overfit the sampled group. Motivated by this finding, we propose Bounded Residual-Aligned Training (BRAT), motivated by a proximal update toward a bounded reweighting of the sampling policy. BRAT assigns each token a gradient weight that decreases linearly with its probability ratio and vanishes at a trust-region boundary, and normalizes group-centered rewards by their maximum magnitude, keeping advantages zero-mean and bounded without extra hyperparameters. With four reuse passes, BRAT avoids training collapse, limits policy drift, and achieves higher average accuracy with lower seed variance on most benchmarks.

Explore similar work

CardsList
  1. EasyPPO: Stabilizing the Critic Is Key

    Sep 29, 2026Xuanyi Zhou, Qiuyang Mang, Huanzhi Mao +7RL for Language ModelsProximal Policy Optimization

  2. Towards Better Training Signal: Advantage Clipped Policy Optimization

    Sep 29, 2026Ruichuan Huang, Jinghan Liu, Congliang ChenPolicy OptimizationRL for Language Model Reasoning