Large language model (LLM) reinforcement learning (RL) pipelines are often bottlenecked by rollout generation, making end-to-end training slow. Recent work mitigates this by running rollouts with quantization to accelerate decoding, which is the most expensive stage of the RL loop. However, these setups destabilize optimization by amplifying the training-inference gap: rollouts are operated at low precision, while learning updates are computed at full precision. To address this challenge, we propose QaRL (Rollout Alignment Quantization-Aware RL), which aligns training-side forward with the quantized rollout to minimize mismatch. We further identify a failure mode in quantized rollouts: long-form responses tend to produce repetitive, garbled tokens (error tokens). To mitigate these problems, we introduce TBPO (Trust-Band Policy Optimization), a sequence-level objective with dual clipping for negative samples, aimed at keeping updates within the trust region. On Qwen3-30B-A3B MoE for math problems, QaRL outperforms quantized-rollout training by +5.5 while improving stability and preserving low-bit throughput benefits.
Figures & tables
Figure 1: (a) Quantized rollout alone introduces mismatch. (b) Aligned QaRL alleviates it, improving performance across model sizes, including MoE.
Figure 2: Overview of the QaRL pipeline in a hybrid RL system. ❶ The quantized rollout engine θlowbit generates samples. ❷ The training engine maintains θBF16 master weights and performs rollout-aligned low-bit GEMM to compute current logprob. ❸ Policy gradients are computed using replay buffer data to update the model via STE. ❹ The updated low-bit weights Wlowbit are synchronized to the rollout engine.
Figure 3: Token level policy clipping regions. Axes represent token probabilities under the old and current policies, with the slope defining the rprox=probcurrent/probold , with arrows indicating the direction of the policy update.
Figure 4: Distribution of token level rprox and wmismatch . Lines indicate clipping boundaries.
Figure 5: Relationship between rprox and old policy probability. Low-probability negative tokens are more prone to extreme rprox .
Figure 6: A mid response error propagates to future tokens. Although the initial garbled tokens are clipped, the repetitive tokens induced by this error are not clipped by token-level objectives.
Figure 7: Training reward curves across different models. Our QaRL TBPO demonstrates stability over quantized rollout training, and converging to reward levels nearly identical to the full-precision BF16 baseline.
Model
In-Distribution Performance
Out-of-Distribution Performance
AIME 24/25
AMC
MATH-500
Minerva
Olympiad
Avg.
ARC-c
GPQA
MMLU-Pro
LiveCodeBench
Avg.
Qwen2.5-1.5B-Math
4.5/2.8
26.5
50.8
21.6
20.3
21.0
11.7
12.4
10.4
2.7
9.3
BF16 GRPO
12.5/9.2
43.5
71.4
36.2
34.5
34.5
58.9
27.8
26.7
11.9
31.3
w4a16 rollout GRPO
7.9/6.4
38.1
65.3
30.2
28.9
29.3
50.1
19.8
21.2
4.3
23.8
w4a16 QaRL TBPO
12.5/10.4
46.6
69.8
31.9
32.6
33.9
57.6
21.1
25.3
8.1
28.0
Qwen2.5-7B-Math
15.0/6.4
46.2
67.4
32.9
23.9
33.4
62.6
28.5
32.1
8.0
32.8
Table 1: Main results on in-distribution math and out-of-distribution benchmarks. LiveCodeBench results are reported in pass@4 , while all other metrics use pass@1 .
Figure 8: Training dynamics (Reward/KL) of Qwen2.5-Math 1.5B (a-b) and 7B (c-d) across different optimization objectives.
Figure 9: (a) Reward curves under different quant scheme. (b) Per-step training time speedup ratio.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter Name
Value
trainer.nnodes
1
trainer.n_gpu_per_node
8
data.train_batch_size
512
data.max_prompt_length
2048
data.max_response_length
16384
rollout.n
8
Appendix
Table 2: Hyperparameters for Experiment.
Qwen3-8B-Base
Method
MATH-500
AIME25
AMC
Avg.
BF16 GRPO
88.1
19.3
64.3
57.2
BF16 TBPO
86.3
20.0
62.4
56.2
w4a16 GRPO
82.2
12.5
53.0
49.2
w4a16 TBPO
84.9
13.3
58.3
52.1
Appendix
Table 3: Performance comparison between GRPO and TBPO under full-precision (BF16) and quantized (w4a16) rollout training settings.
Figure 10: Comparison of RL training entropy
Figure 11: Different quant scheme speed on Qwen3-30B-A3B MoE model.
Table 6: Comparison of clipping and weighting strategies. Gray highlights clipped tokens, while different colors represent varying weighting magnitudes (darker shades indicate weights further from 1). GRPO employs token-level clipping and weighting, whereas TBPO utilizes sequence-level granularity. In GRPO, the garbled tokens (marked by 1) are not clipped, while the exploration tokens (marked by 2) are clipped. In contrast, TBPO’s sequence-level approach correctly clips the garbled tokens without erroneously clipping the exploration tokens.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Reinforcement learning (RL) for post-training large language models (LLMs) incurs substantial computation and memory overhead during rollout generation, which motivates low-precision rollout for efficient RL training. However, existing FP4 RL methods suffer from a key limitation: they primarily optimize quantization accuracy on the training and rollout paths independently rather than directly reducing the discrepancy between the two quantized execution paths. In this work, we propose TRACE (Train-Rollout Quantization Alignment via Compact GuidancE), an FP4 quantization framework for RL training of Mixture-of-Experts (MoE) language models that addresses the limitation of existing FP4 RL methods. TRACE incorporates rollout-guided quantization-aware training that uses rollout-side quantization outcomes to guide training-side FP4 rounding decisions, directly reducing train-rollout discrepancy. Moreover, TRACE adopts an efficient quantization-information caching scheme that selectively retains mantissa and scale information from deeper layers to reduce the storage and communication overhead introduced by rollout guidance. We evaluate TRACE on four large-scale MoE language models across reasoning, coding, and long-horizon RL tasks. Our results demonstrate that TRACE enables joint FP4 weight/activation and FP4 KV-cache rollout with RL performance comparable to BF16 rollout, while achieving up to 5.4xrollout speedup and strong final FP4 performance compared with post-hoc FP4 quantization of BF16-trained policies.
Xin Wang, Hao Yu, Zhengyang Zhuge +9
Alibaba Token Hub, Alibaba Group · Ohio State University
Reinforcement learning is crucial for improving large language models' reasoning and generalization. It relies on massive rollouts whose lengths become increasingly long-tailed as context windows grow. In on-policy training, these long-tail rollouts can result in GPU bubbles, reducing system utilization and limiting RL scalability. Asynchronous or partial-rollout methods improve throughput by relaxing synchronization, but inevitably introduce stale off-policy samples (trajectories) that may hurt final accuracy. Existing approaches mainly mitigate this off-policy issue by reweighting off-policy samples during training, yet they can still leave a performance gap compared to fully on-policy training. In this work, rather than passively reweighting samples during training, we propose RollVerify, a lightweight RL framework built on partial rollout that actively verifies and repairs samples before they enter training. Specifically, it introduces an off-policy shift metric OPS, to quantify the off-policy deviation of partially generated trajectories. Guided by the OPS constraint, RollVerify performs both sequence-level and token-level verification to identify and truncate invalid suffixes of trajectories. This yields high-quality samples that protect the models' accuracy while preserving the efficiency gains of partial rollout. Experiments on mathematical and tool-assisted mathematical reasoning show that RollVerify achieves accuracy comparable to on-policy training while reducing training cost. Additional code-generation results provide preliminary evidence beyond mathematics.
Yongqiang Yao, Jinru Tan, Kaihuan Liang +5
Shanghai Jiao Tong University · Central South University · SenseTime Research +2
Reinforcement Learning (RL) training for Large Language Models (LLMs) often suffers from instability due to the discrepancy between training and inference. This training-inference discrepancy stems from two primary factors: an architectural separation between training and inference engines, and the use of low-precision quantization in inference versus higher-precision computation in training. To address training instability issues caused by high training-inference discrepancy, we present the principles and methods for its adaptive control. We propose Adaptive Control Reinforcement Learning (ACRL), which adaptively maintains the training-inference discrepancy within a reasonable range to ensure stable RL training. Beyond stabilization, ACRL inherently increases policy entropy, thereby enhancing exploration and improving accuracy. The experimental results show that when the inference engine utilizes FP8 quantization, ACRL consistently maintains the training-inference discrepancy within a reasonable range and stabilizes RL training. Furthermore, ACRL not only matches the accuracy of the BF16 baseline but also outperforms importance sampling (IS) fixes.