QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch

Hao Gu; Hao Wang; Jiacheng Liu; Lujun Li; Qiyuan Zhu; Bei Liu; Binxing Xu; Lei Wang

cs.LGApr 9, 2026

QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch

Authors: Hao Gu, Hao Wang, Jiacheng Liu, Lujun Li, Qiyuan Zhu, Bei Liu, Binxing Xu, Lei Wang,

Organizations: The Hong Kong University of Science and Technology · City University of Hong Kong · Zhejiang University

Abstract

Large language model (LLM) reinforcement learning (RL) pipelines are often bottlenecked by rollout generation, making end-to-end training slow. Recent work mitigates this by running rollouts with quantization to accelerate decoding, which is the most expensive stage of the RL loop. However, these setups destabilize optimization by amplifying the training-inference gap: rollouts are operated at low precision, while learning updates are computed at full precision. To address this challenge, we propose QaRL (Rollout Alignment Quantization-Aware RL), which aligns training-side forward with the quantized rollout to minimize mismatch. We further identify a failure mode in quantized rollouts: long-form responses tend to produce repetitive, garbled tokens (error tokens). To mitigate these problems, we introduce TBPO (Trust-Band Policy Optimization), a sequence-level objective with dual clipping for negative samples, aimed at keeping updates within the trust region. On Qwen3-30B-A3B MoE for math problems, QaRL outperforms quantized-rollout training by +5.5 while improving stability and preserving low-bit throughput benefits.

Figures & tables

Figure 1: (a) Quantized rollout alone introduces mismatch. (b) Aligned QaRL alleviates it, improving performance across model sizes, including MoE.

Figure 2: Overview of the QaRL pipeline in a hybrid RL system. ❶ The quantized rollout engine

\theta_{\text{lowbit}}

generates samples. ❷ The training engine maintains

\theta_{\text{BF16}}

master weights and performs rollout-aligned low-bit GEMM to compute current logprob. ❸ Policy gradients are computed using replay buffer data to update the model via STE. ❹ The updated low-bit weights

W_{\text{lowbit}}

are synchronized to the rollout engine.

Figure 3: Token level policy clipping regions. Axes represent token probabilities under the old and current policies, with the slope defining the $r_{\text{prox}}=\text{prob}_{\text{current}}/\text{prob}_{\text{old}}$ , with arrows indicating the direction of the policy update.

Figure 4: Distribution of token level $r_{\text{prox}}$ and $w_{\text{mismatch}}$ . Lines indicate clipping boundaries.

Figure 5: Relationship between $r_{\text{prox}}$ and old policy probability. Low-probability negative tokens are more prone to extreme $r_{\text{prox}}$ .

Figure 6: A mid response error propagates to future tokens. Although the initial garbled tokens are clipped, the repetitive tokens induced by this error are not clipped by token-level objectives.

Figure 7: Training reward curves across different models. Our QaRL TBPO demonstrates stability over quantized rollout training, and converging to reward levels nearly identical to the full-precision BF16 baseline.

Model	In-Distribution Performance						Out-of-Distribution Performance
Model	AIME 24/25	AMC	MATH-500	Minerva	Olympiad	Avg.	ARC-c	GPQA	MMLU-Pro	LiveCodeBench	Avg.
Qwen2.5-1.5B-Math	4.5/2.8	26.5	50.8	21.6	20.3	21.0	11.7	12.4	10.4	2.7	9.3
BF16 GRPO	12.5/9.2	43.5	71.4	36.2	34.5	34.5	58.9	27.8	26.7	11.9	31.3
w4a16 rollout GRPO	7.9/6.4	38.1	65.3	30.2	28.9	29.3	50.1	19.8	21.2	4.3	23.8
w4a16 QaRL TBPO	12.5/10.4	46.6	69.8	31.9	32.6	33.9	57.6	21.1	25.3	8.1	28.0
Qwen2.5-7B-Math	15.0/6.4	46.2	67.4	32.9	23.9	33.4	62.6	28.5	32.1	8.0	32.8

Table 1: Main results on in-distribution math and out-of-distribution benchmarks. LiveCodeBench results are reported in

pass@4

, while all other metrics use

pass@1

Figure 8: Training dynamics (Reward/KL) of Qwen2.5-Math 1.5B (a-b) and 7B (c-d) across different optimization objectives.

Figure 9: (a) Reward curves under different quant scheme. (b) Per-step training time speedup ratio.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter Name	Value
trainer.nnodes	1
trainer.n_gpu_per_node	8
data.train_batch_size	512
data.max_prompt_length	2048
data.max_response_length	16384
rollout.n	8

Appendix

Table 2: Hyperparameters for Experiment.

Qwen3-8B-Base
Method	MATH-500	AIME25	AMC	Avg.
BF16 GRPO	88.1	19.3	64.3	57.2
BF16 TBPO	86.3	20.0	62.4	56.2
w4a16 GRPO	82.2	12.5	53.0	49.2
w4a16 TBPO	84.9	13.3	58.3	52.1

Appendix

Table 3: Performance comparison between GRPO and TBPO under full-precision (BF16) and quantized (w4a16) rollout training settings.

Figure 10: Comparison of RL training entropy

Figure 11: Different quant scheme speed on Qwen3-30B-A3B MoE model.

		$\displaystyle r_{\mathrm{seq\text{-}prox}}(\theta)\triangleq\frac{\pi_{\mathrm{current\ learner}}(o\mid q)}{\pi_{\mathrm{old\ learner}}(o\mid q)}=\exp\!\left(\frac{1}{L}\sum_{t=1}^{L}(\log\pi_{\mathrm{current\ learner}}-\log\pi_{\mathrm{old\ learner}})(o_{t}\mid q,o_{<t})\right),$
		$\displaystyle w_{\mathrm{seq\text{-}mismatch}}(\theta)\triangleq\frac{\pi_{\mathrm{old\ learner}}(o\mid q)}{\pi_{\mathrm{old\ sampler}}(o\mid q)}=\exp\!\left(\frac{1}{L}\sum_{t=1}^{L}(\log\pi_{\mathrm{old\ learner}}-\log\pi_{\mathrm{old\ sampler}})(o_{t}\mid q,o_{<t})\right).$

Appendix

Figure 13: Sequence ratio and weight definition.

Panel of Table 6 — Table 6: Comparison of clipping and weighting strategies. Gray highlights clipped tokens, while different colors represent varying weighting magnitudes (darker shades indicate weights further from 1). GRPO employs token-level clipping and weighting, whereas TBPO utilizes sequence-level granularity. In GRPO, the garbled tokens (marked by 1) are not clipped, while the exploration tokens (marked by 2) are clipped. In contrast, TBPO’s sequence-level approach correctly clips the garbled tokens without erroneously clipping the exploration tokens.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,661 papers

TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models
Oct 6, 2026Xin Wang, Hao Yu, Zhengyang Zhuge +9LLM Quantization Quantization-Aware Training
Alibaba Token Hub, Alibaba Group · Ohio State University
RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning
Oct 7, 2026Yongqiang Yao, Jinru Tan, Kaihuan Liang +5RL for Language Models Reinforcement Learning
Shanghai Jiao Tong University · Central South University · SenseTime Research +2
ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning
Jul 27, 2026Wenwu Fan, Qihong Lin, Zhijie Xia +4RL Control RL for Language Models
Computing Product Line, Huawei

Oct 6, 2026J/K move · Enter open · S save

TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models

Xin Wang, Hao Yu, Zhengyang Zhuge, Bochao Mao+8

Alibaba Token Hub, Alibaba Group · Ohio State University

LLM Quantization Quantization-Aware Training LLM Inference Acceleration Mixture-of-Experts Quantization RL Post-Training

Open paper AskPDF arXiv

Reinforcement learning (RL) for post-training large language models (LLMs) incurs substantial computation and memory overhead during rollout generation, which motivates low-precision rollout for efficient RL training. However, existing FP4 RL methods suffer from a key limitation: they primarily optimize quantization accuracy on the training and rollout paths independently rather than directly reducing the discrepancy between the two quantized execution paths. In this work, we propose TRACE (Train-Rollout Quantization Alignment via Compact GuidancE), an FP4 quantization framework for RL training of Mixture-of-Experts (MoE) language models that addresses the limitation of existing FP4 RL methods. TRACE incorporates rollout-guided quantization-aware training that uses rollout-side quantization outcomes to guide training-side FP4 rounding decisions, directly reducing train-rollout discrepancy. Moreover, TRACE adopts an efficient quantization-information caching scheme that selectively retains mantissa and scale information from deeper layers to reduce the storage and communication overhead introduced by rollout guidance. We evaluate TRACE on four large-scale MoE language models across reasoning, coding, and long-horizon RL tasks. Our results demonstrate that TRACE enables joint FP4 weight/activation and FP4 KV-cache rollout with RL performance comparable to BF16 rollout, while achieving up to 5.4xrollout speedup and strong final FP4 performance compared with post-hoc FP4 quantization of BF16-trained policies.

Abstract

Reinforcement Learning

Explore similar work

TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models

RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning

ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning