InterQ-RL: Interaction-Aware Correction of Execution Mismatch in Quantized Agentic RL

Yingbo Hao; Song Liu; Guanghua Yu; Zhaojing Zhou; Jianchen Zhu; Yi Zou

cs.LGOct 26, 2026

InterQ-RL: Interaction-Aware Correction of Execution Mismatch in Quantized Agentic RL

Authors: Yingbo Hao, Song Liu, Guanghua Yu, Zhaojing Zhou, Jianchen Zhu, Yi Zou

Organizations: South China University of Technology · Tencent

Abstract

Multi-turn agent reinforcement learning (RL) relies on long closed-loop rollouts, making generation and KV-cache costs a major bottleneck. Quantized rollout reduces these costs, but its low-precision executor also serves as the behavior policy and can induce execution mismatch with the learner. Existing quantized-RL methods are designed primarily for single-turn generation. They align the learner forward with quantized rollout or correct the current-token rollout-learner ratio, without modeling the policy history that produces each agent state. Even after quantization alignment, residual rollout-learner mismatch accumulates along the state-reaching policy history, so a current action may look locally aligned despite a mismatched history. We introduce InterQ-RL (Interaction-aware Quantized Reinforcement Learning) to correct this post-alignment mismatch. It applies bounded execution-prefix correction over the state-reaching policy history, allocates correction strength across action segments by effective support under a fixed displacement budget, and requires no additional model forward pass. We evaluate dense and mixture-of-experts (MoE) backbones from 4B to 30B on ALFWorld, WebShop, and ScienceWorld, with all quantized arms sharing the same rollout with 8-bit weights, activations, and KV cache (W8A8-KV8). Across four environment-backbone settings, InterQ-RL achieves the strongest quantized endpoint result, matching BF16 to within 0.7 points and exceeding it in two settings. It matches or exceeds the learning speed of alignment alone while avoiding its late performance regression.

Figures & tables

Panel of Figure 1 — Figure 1: (a) ALFWorld valid_unseen success, evaluated every 10 steps. W8A8-KV8 is the unaligned run and collapses; W8A8-KV8+QAT is aligned but drops 10.4 points from step 260 to 380 and only partially recovers by step 400. Flash-RL destabilizes late, while InterQ-RL remains stable and approaches BF16 at step 400. (b) For one step-400 task, the two arms diverge after sharing their first three actions: W8A8-KV8+QAT exhausts the 30-action budget, whereas InterQ-RL finishes in 16. Across all 134 tasks, $10.45\%$ of W8A8-KV8+QAT episodes and $4.48\%$ of InterQ-RL episodes fail after exhausting the action budget.

Figure 2: InterQ-RL aligns quantization, corrects the state-reaching policy history, and allocates correction strength by effective support under a fixed displacement budget. Grey: the learner forward matches rollout W/A and KV quantization at a shared checkpoint. Orange: per-token rollout–learner log-ratios form the prefix correction; feedback changes later contexts but adds no policy-likelihood term. Green: segment-level effective support

\rho_{s}

determines retention

\alpha_{s}

under this budget.

	ALFWorld	WebShop		ScienceWorld (8B)		ScienceWorld (A3B)
Method	Success $\uparrow$	Score $\uparrow$	Success $\uparrow$	Macro $\uparrow$	Solved $\uparrow$	Macro $\uparrow$	Solved $\uparrow$
BF16	97.1	78.0	54.0	66.9	74.1	67.0	74.3
Flash-RL	64.1	72.8	44.4	63.2	69.2	65.8	71.5
AIS	92.3	76.5	51.5	59.6	64.2	65.0	70.6
QuRL	Coll. †	75.3	51.7	57.1	63.9	64.0	69.6
QAT	90.4	74.9	49.7	65.5	75.8	69.3	76.6

Table 1: End-to-end comparison of complete methods under a common training budget. ALFWorld and WebShop report the mean over the final six checkpoint evaluations; ScienceWorld reports the full official test at a fixed endpoint checkpoint. ALFWorld and WebShop use Qwen3-4B; ScienceWorld headers give the backbone size. Bold marks the best quantized result.

\dagger

QuRL collapsed at step 278 and has no common-horizon endpoint.

Figure 5: On ALFWorld, InterQ-RL remains stable while QAT rolls back, Flash-RL destabilizes, QuRL collapses, and AIS learns more slowly. Mean turns in the middle panel are averaged over all training trajectories. For trajectory length and reward, smoothed trends use 10-step block medians followed by light EMA.

	ALFWorld		WebShop		ScienceWorld (8B)		ScienceWorld (A3B)
Method	AULC $\uparrow$	Turns $\downarrow$	AULC $\uparrow$	Turns $\downarrow$	AULC $\uparrow$	Turns $\downarrow$	AULC $\uparrow$	Turns $\downarrow$
BF16	82.1	10.98	75.2	5.23	56.4	20.45	54.1	19.04
Flash-RL	64.9	20.34	72.5	5.09	47.6	31.93	52.1	20.31
AIS	72.5	12.43	72.8	6.61	45.4	22.32	49.6	18.07
QuRL	—	—	72.2	6.23	50.8	21.19	45.5	32.32
QAT	82.3	12.72	72.3	6.51	54.4	22.38	54.1	20.20

Table 2: Learning speed and the interaction cost of successful trajectories. ALFWorld and WebShop use Qwen3-4B; ScienceWorld headers give the backbone size. Turns average over successful episodes only; bold marks the best quantized AULC. QuRL collapsed on ALFWorld at step 278 and has no result on the common horizon.

Figure 6: During continued optimization, K3 grows under QAT and remains stable under InterQ-RL. K3 is computed on the trajectories of each arm and shown as smoothed trends. The QuRL curve on ALFWorld ends at step 278.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Two action segments with indistinguishable

C_{s}

and different

\rho_{s}

. Each block is one token, shaded by that token’s share of squared displacement within its segment. The two values of

C_{s}

differ by

7\times 10^{-6}

Figure 8: Range of the allocated

\alpha_{s}

within each batch over one complete 400-step ALFWorld run: the band spans the smallest to the largest

\alpha_{s}

in that batch. Across the 400 batches the median range is 0.717 to 0.942 with median width 0.215; over the whole run

\alpha_{s}

reaches 0.372 and 0.989.

Environment	Backbone	Steps	AULC set	Endpoint
ALFWorld	Qwen3-4B	400	valid_unseen (10-step)	valid_unseen (final-six)
WebShop	Qwen3-4B	400	official test (10-step)	official test (final-six)
ScienceWorld	Qwen3-8B	400	259 monitor (10-step)	official test (step 400)
	Qwen3-30B-A3B	400	259 monitor (10-step)	official test (step 400)

Appendix

Table 3: Environments, backbones, training steps, and evaluation sets.

Environment	Turn cap	Context	Prompt	Response
ALFWorld	30	24,576	2,048	20,480
WebShop	15	24,576	4,096	20,480
ScienceWorld / 8B	100	40,960	1,024	39,936
ScienceWorld / 30B-A3B	100	50,176	1,024	49,152

Appendix

Table 4: Sequence budgets. All entries are token counts except the turn cap. Context is the rollout engine’s window, Prompt caps the first-turn prompt, and Response caps the actions and observations appended after it; when appending the next observation would exceed this budget, the episode stops at the response boundary.

Hyperparameter	Value
Advantage estimator	GRPO
Optimizer	AdamW
Adam settings	$\beta=(0.9,0.999)$ ; weight decay $0.01$ ; clip norm $1.0$
LR / schedule	$1\times 10^{-6}$ ; constant; no warmup
PPO clip	$\epsilon_{\mathrm{PPO}}=0.2$ ; dual-clip $c=10$ (except QuRL)
KL	penalty in the loss, coefficient 0.001

Appendix

Table 5: GRPO hyperparameters.

Figure 9: (a) ALFWorld valid_unseen success. (b) Smoothed rollout–learner K3; arrows mark off-panel peaks. All quantized arms share the W8A8-KV8+QAT substrate.

Variant	Scope	Retention	Success $\uparrow$	AULC $\uparrow$	Turns $\downarrow$	Cap. $\downarrow$
BF16	—	—	97.14	82.07	10.98	2.24
QAT	none	none	90.42	82.29	12.72	10.45
QAT+TIS	current token	soft clip $C=2$	90.55	70.59	9.86	5.22
QAT+AIS	current token	clip $C=5$ + gate	92.66	76.27	12.46	6.72
InterQ-RL	segment prefix	$\alpha_{s}(\rho_{s})$	96.39	81.96	10.06	4.48

Appendix

Table 6: Token-local corrections with learner alignment on ALFWorld; all quantized arms use the W8A8-KV8 rollout and W/A+KV-QAT learner.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,742 papers

QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch
Apr 9, 2026Hao Gu, Hao Wang, Jiacheng Liu +9Reinforcement Learning RL Fine-Tuning
The Hong Kong University of Science and Technology · City University of Hong Kong · Zhejiang University
KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization
Oct 10, 2026Sihwan Park, Sung-Yub Kim, Doohyuk Jang +1KV-Cache Quantization RL Post-Training
KAIST · NAVER Cloud · AITRICS
HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models
Jul 29, 2026Hei Yi Mak, Shadan Golestan, Hoang Le +10LLM Quantization 4-Bit Quantization
Huawei

Apr 9, 2026J/K move · Enter open · S save

QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch

Hao Gu, Hao Wang, Jiacheng Liu, Lujun Li+8

The Hong Kong University of Science and Technology · City University of Hong Kong · Zhejiang University

Reinforcement Learning RL Fine-Tuning RL for Language Model Reasoning Quantization-Aware Training

Open paper AskPDF arXiv

Large language model (LLM) reinforcement learning (RL) pipelines are often bottlenecked by rollout generation, making end-to-end training slow. Recent work mitigates this by running rollouts with quantization to accelerate decoding, which is the most expensive stage of the RL loop. However, these setups destabilize optimization by amplifying the training-inference gap: rollouts are operated at low precision, while learning updates are computed at full precision. To address this challenge, we propose QaRL (Rollout Alignment Quantization-Aware RL), which aligns training-side forward with the quantized rollout to minimize mismatch. We further identify a failure mode in quantized rollouts: long-form responses tend to produce repetitive, garbled tokens (error tokens). To mitigate these problems, we introduce TBPO (Trust-Band Policy Optimization), a sequence-level objective with dual clipping for negative samples, aimed at keeping updates within the trust region. On Qwen3-30B-A3B MoE for math problems, QaRL outperforms quantized-rollout training by +5.5 while improving stability and preserving low-bit throughput benefits.

Abstract

Reinforcement Learning

Explore similar work

QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch

KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization

HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models