Asynchronous reinforcement learning increases training throughput, but data from stale policies and numerical mismatch between training and inference can destabilize learning. To stabilize off-policy learning, score centering (SC) removes the effect of reward baselines on the expected update. Our gradient diagnostics show less extreme gradients with SC than with importance sampling (IS), but SC can still fail when the learner moves far from the behavior policy that generated the data. To limit this policy mismatch, we introduce Regularized Score Centering (RSC), which adds a reverse Kullback-Leibler (KL) penalty to encourage the learner's next-token distribution to stay close to the behavior policy's next-token distribution. RSC works with or without truncated importance sampling (TIS), which caps token importance weights. Our theoretical analysis supports this approach, showing that KL regularization improves convergence guarantees. We evaluate dense and mixture-of-experts models on mathematical reasoning and agentic tasks, including online shopping, tool use, games, and fully asynchronous software engineering. RSC with TIS substantially improves sample efficiency across all agentic benchmarks, attaining higher training rewards with the same number of sampled responses. In final evaluations, it achieves top performance across all six agentic tasks, improving by 14.3% on AppWorld and reaching 9.9 times the strongest baseline's score on BabyAI.
Figures & tables
Figure 1: Regularization strengthens score centering. (a) RSC adds reverse KL to SC or SC+TIS, comparing distributions over next actions under the learner πθ and behavior policy μ at the same history. (b–d) Training curves show mean ± sample standard deviation across three seeds (settings in Appendix A.2 ). Stronger methods are colored, and others are gray.
Figure 2: Weighting and KL shape logit gradients. Curves show the derivative coefficient after factoring out 1−πθ(a∣h) , with A=1 , c=2 and β=0.05 for RSC. The expressions on the right give ∇zaw for fixed μ , A=Aμ(h,a) and standard, uncapped reverse KL.
Figure 3: Weighting, centering and KL shape the update. We hold the learner and behavior checkpoints fixed and reuse the sampled actions, response advantages, and 128 auxiliary Monte Carlo draws across methods. The panels therefore compare update rules on the same data, before any optimizer step. Both rows use behavior probability μ(a∣h) : panels (a–e) show logit gradient norms, and panels (f–j) show parameter gradient norms. Black lines show the 99th percentile within each probability bin, and the top row reports this percentile for μ<0.01 . Appendix A.1 gives the diagnostic protocol.
Figure 4: Monte Carlo centering improves on Top-128. Rows show training (top) and evaluation (bottom). Countdown uses three seeds, with bands indicating sample standard deviation. The math experiments use one seed per method and rollout precision.
Figure 5: Learning on math tasks and Crafter at the same refresh interval. The left pair shows math tasks with a dense model and the right pair shows Crafter, each with training performance and signed mean logπθ−logμ . Curves and bands show mean ± sample standard deviation across three seeds. The policy-gap metric for IS-PPOClip is available only for Crafter.
Figure 6: Score centering under increasing staleness. Columns show Countdown reward and signed logμ−logπθ , then math training accuracy with the MoE model and ∣logπθ−logμ∣ . Sampler refresh intervals increase down the rows. Curves and bands show mean ± sample standard deviation. Countdown uses three seeds. MoE uses two seeds at the two shorter intervals and three at the longest. Countdown log-probability panels cover the first three refresh cycles.
Figure 7: Training reward across six interaction tasks. Curves show mean ± sample standard deviation across three seeds. Crafter and Craftax use total episode rewards. Appendix A.2 specifies training settings.
Figure 8: Fully asynchronous SWE training. Each method and configuration uses one seed. The left pair shows configuration A and the right pair configuration B, each with training reward followed by the absolute log-probability difference. Axes use restricted ranges. Appendix A.4 specifies the configurations and smoothing.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Action limit
Tokens/action
Response tokens
Context tokens
WebShop
15
64
8,192
12,288
AppWorld
30
400
12,288
16,384
BabyAI
Varies by task
16
24,576
32,768
TextWorld
50
64
12,288
16,384
Crafter
2,000
16
24,576
32,768
Craftax
100,000
16
24,576
32,768
Appendix
Table 3: Agentic episode limits. Token limits for responses and contexts include environment observations.
Configuration
Trajectory budget
Learning rate
Max model calls/episode
A
128K
10−6
200
B
128K
2×10−6
200
Appendix
Table 4: Asynchronous SWE configurations. Trajectory budgets include tool observations.
Figure 9: Clipping the KL penalty on BabyAI. Mean ± sample standard deviation across three seeds.
Figure 10: Sensitivity to group-size correction. Solid and dashed curves compare each RSC variant with and without the group-size correction at fixed group size. Curves and bands show mean training reward ± sample standard deviation across three seeds.
Table 5: Surrogate integrands and logit derivatives. Here ρ=πθ(a∣h)/μ(a∣h) , A=Aμ(h,a) and c≥1 . The last column factors out 1−πθ(a∣h) from the logit derivative, with μ , A and the other logits fixed. RSC uses β>0 and standard, uncapped reverse KL.
Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe. Because the correction is additive, score centering also composes with importance sampling -- their composition outperforms pure importance-sampling baselines in our staleness experiments.
Reinforcement learning has become a widely used approach to post-training large language models. In a typical implementation, learning is off-policy: The policy generating the responses differs from the policy being trained because of training-inference mismatch or policy staleness in asynchronous training. This discrepancy is commonly addressed by importance sampling, whose ratio is usually regularized in practice through truncation, masking, or clipping. Under regularized importance sampling, the importance-weighted policy gradient is no longer invariant to constant shifts in the reward. This invariance is recovered by using the covariance between the reward and the importance-weighted score as the estimand, which coincides with the policy gradient when the ratio is not regularized. This note develops this covariance perspective for both sequence-level and token-level importance sampling. The covariance perspective reveals the roles of the group-mean baseline and score centering in policy gradient estimation. Both perform centering in covariance estimation, but they differ in the mean being subtracted: The group-mean baseline uses a sample mean, whereas score centering uses a population mean. The two lead to distinct unbiased covariance estimators with different variances. Whether both forms of centering should be used simultaneously depends on whether importance sampling is applied at the sequence level or at the token level. With sequence-level importance sampling, score centering adds no benefit once the group-mean baseline is used; with token-level importance sampling, using the group-mean baseline without score centering generally introduces bias when the importance ratio is regularized.
Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization, but the resulting stale, off-policy data destabilizes optimization and can cause policy collapse. Existing methods gate tokens by ratio magnitude alone, applying one threshold at every position. We show that the ratio's natural scale is set by token entropy, so deviations from mid-trajectory weight updates stay within this scale and carry genuine exploration. We further identify an overlooked low-entropy regime that breaks this scaling, where a near-zero probability amplifies train--inference mismatch into noise far beyond what the local entropy admits. A magnitude threshold admits this noise and discards the exploration. We therefore propose the Entropy-Normalized Trust Region (ENTR). Across long-horizon agentic tasks and mathematical reasoning benchmarks, ENTR outperforms existing asynchronous methods. It improves avg@1 on BrowseComp-Plus by 6.9% over the strongest baseline, trains stably up to 30 policy versions of staleness, and matches synchronous GRPO at a 2.6× speedup.
Guanqun Zhao, Zijun Xie, Binbin Zheng +5
Beijing University of Posts and Telecommunications · Baidu Inc. · Peking University +2