InterQ-RL: Interaction-Aware Correction of Execution Mismatch in Quantized Agentic RL

Yingbo Hao; Song Liu; Guanghua Yu; Zhaojing Zhou; Jianchen Zhu; Yi Zou

cs.LGOct 26, 2026

InterQ-RL: Interaction-Aware Correction of Execution Mismatch in Quantized Agentic RL

Authors: Yingbo Hao, Song Liu, Guanghua Yu, Zhaojing Zhou, Jianchen Zhu, Yi Zou

Organizations: South China University of Technology · Tencent

Abstract

Multi-turn agent reinforcement learning (RL) relies on long closed-loop rollouts, making generation and KV-cache costs a major bottleneck. Quantized rollout reduces these costs, but its low-precision executor also serves as the behavior policy and can induce execution mismatch with the learner. Existing quantized-RL methods are designed primarily for single-turn generation. They align the learner forward with quantized rollout or correct the current-token rollout-learner ratio, without modeling the policy history that produces each agent state. Even after quantization alignment, residual rollout-learner mismatch accumulates along the state-reaching policy history, so a current action may look locally aligned despite a mismatched history. We introduce InterQ-RL (Interaction-aware Quantized Reinforcement Learning) to correct this post-alignment mismatch. It applies bounded execution-prefix correction over the state-reaching policy history, allocates correction strength across action segments by effective support under a fixed displacement budget, and requires no additional model forward pass. We evaluate dense and mixture-of-experts (MoE) backbones from 4B to 30B on ALFWorld, WebShop, and ScienceWorld, with all quantized arms sharing the same rollout with 8-bit weights, activations, and KV cache (W8A8-KV8). Across four environment-backbone settings, InterQ-RL achieves the strongest quantized endpoint result, matching BF16 to within 0.7 points and exceeding it in two settings. It matches or exceeds the learning speed of alignment alone while avoiding its late performance regression.

Figures & tables

Panel of Figure 1 — Figure 1: (a) ALFWorld valid_unseen success, evaluated every 10 steps. W8A8-KV8 is the unaligned run and collapses; W8A8-KV8+QAT is aligned but drops 10.4 points from step 260 to 380 and only partially recovers by step 400. Flash-RL destabilizes late, while InterQ-RL remains stable and approaches BF16 at step 400. (b) For one step-400 task, the two arms diverge after sharing their first three actions: W8A8-KV8+QAT exhausts the 30-action budget, whereas InterQ-RL finishes in 16. Across all 134 tasks, $10.45\%$ of W8A8-KV8+QAT episodes and $4.48\%$ of InterQ-RL episodes fail after exhausting the action budget.

Figure 2: InterQ-RL aligns quantization, corrects the state-reaching policy history, and allocates correction strength by effective support under a fixed displacement budget. Grey: the learner forward matches rollout W/A and KV quantization at a shared checkpoint. Orange: per-token rollout–learner log-ratios form the prefix correction; feedback changes later contexts but adds no policy-likelihood term. Green: segment-level effective support

\rho_{s}

determines retention

\alpha_{s}

under this budget.

	ALFWorld	WebShop		ScienceWorld (8B)		ScienceWorld (A3B)
Method	Success $\uparrow$	Score $\uparrow$	Success $\uparrow$	Macro $\uparrow$	Solved $\uparrow$	Macro $\uparrow$	Solved $\uparrow$
BF16	97.1	78.0	54.0	66.9	74.1	67.0	74.3
Flash-RL	64.1	72.8	44.4	63.2	69.2	65.8	71.5
AIS	92.3	76.5	51.5	59.6	64.2	65.0	70.6
QuRL	Coll. †	75.3	51.7	57.1	63.9	64.0	69.6
QAT	90.4	74.9	49.7	65.5	75.8	69.3	76.6

Table 1: End-to-end comparison of complete methods under a common training budget. ALFWorld and WebShop report the mean over the final six checkpoint evaluations; ScienceWorld reports the full official test at a fixed endpoint checkpoint. ALFWorld and WebShop use Qwen3-4B; ScienceWorld headers give the backbone size. Bold marks the best quantized result.

\dagger

QuRL collapsed at step 278 and has no common-horizon endpoint.

Figure 5: On ALFWorld, InterQ-RL remains stable while QAT rolls back, Flash-RL destabilizes, QuRL collapses, and AIS learns more slowly. Mean turns in the middle panel are averaged over all training trajectories. For trajectory length and reward, smoothed trends use 10-step block medians followed by light EMA.

	ALFWorld		WebShop		ScienceWorld (8B)		ScienceWorld (A3B)
Method	AULC $\uparrow$	Turns $\downarrow$	AULC $\uparrow$	Turns $\downarrow$	AULC $\uparrow$	Turns $\downarrow$	AULC $\uparrow$	Turns $\downarrow$
BF16	82.1	10.98	75.2	5.23	56.4	20.45	54.1	19.04
Flash-RL	64.9	20.34	72.5	5.09	47.6	31.93	52.1	20.31
AIS	72.5	12.43	72.8	6.61	45.4	22.32	49.6	18.07
QuRL	—	—	72.2	6.23	50.8	21.19	45.5	32.32
QAT	82.3	12.72	72.3	6.51	54.4	22.38	54.1	20.20

Table 2: Learning speed and the interaction cost of successful trajectories. ALFWorld and WebShop use Qwen3-4B; ScienceWorld headers give the backbone size. Turns average over successful episodes only; bold marks the best quantized AULC. QuRL collapsed on ALFWorld at step 278 and has no result on the common horizon.

Figure 6: During continued optimization, K3 grows under QAT and remains stable under InterQ-RL. K3 is computed on the trajectories of each arm and shown as smoothed trends. The QuRL curve on ALFWorld ends at step 278.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Two action segments with indistinguishable

C_{s}

and different

\rho_{s}

. Each block is one token, shaded by that token’s share of squared displacement within its segment. The two values of

C_{s}

differ by

7\times 10^{-6}

Figure 8: Range of the allocated

\alpha_{s}

within each batch over one complete 400-step ALFWorld run: the band spans the smallest to the largest

\alpha_{s}

in that batch. Across the 400 batches the median range is 0.717 to 0.942 with median width 0.215; over the whole run

\alpha_{s}

reaches 0.372 and 0.989.

Environment	Backbone	Steps	AULC set	Endpoint
ALFWorld	Qwen3-4B	400	valid_unseen (10-step)	valid_unseen (final-six)
WebShop	Qwen3-4B	400	official test (10-step)	official test (final-six)
ScienceWorld	Qwen3-8B	400	259 monitor (10-step)	official test (step 400)
	Qwen3-30B-A3B	400	259 monitor (10-step)	official test (step 400)

Appendix

Table 3: Environments, backbones, training steps, and evaluation sets.

Environment	Turn cap	Context	Prompt	Response
ALFWorld	30	24,576	2,048	20,480
WebShop	15	24,576	4,096	20,480
ScienceWorld / 8B	100	40,960	1,024	39,936
ScienceWorld / 30B-A3B	100	50,176	1,024	49,152

Appendix

Table 4: Sequence budgets. All entries are token counts except the turn cap. Context is the rollout engine’s window, Prompt caps the first-turn prompt, and Response caps the actions and observations appended after it; when appending the next observation would exceed this budget, the episode stops at the response boundary.

Hyperparameter	Value
Advantage estimator	GRPO
Optimizer	AdamW
Adam settings	$\beta=(0.9,0.999)$ ; weight decay $0.01$ ; clip norm $1.0$
LR / schedule	$1\times 10^{-6}$ ; constant; no warmup
PPO clip	$\epsilon_{\mathrm{PPO}}=0.2$ ; dual-clip $c=10$ (except QuRL)
KL	penalty in the loss, coefficient 0.001

Appendix

Table 5: GRPO hyperparameters.

Figure 9: (a) ALFWorld valid_unseen success. (b) Smoothed rollout–learner K3; arrows mark off-panel peaks. All quantized arms share the W8A8-KV8+QAT substrate.

Variant	Scope	Retention	Success $\uparrow$	AULC $\uparrow$	Turns $\downarrow$	Cap. $\downarrow$
BF16	—	—	97.14	82.07	10.98	2.24
QAT	none	none	90.42	82.29	12.72	10.45
QAT+TIS	current token	soft clip $C=2$	90.55	70.59	9.86	5.22
QAT+AIS	current token	clip $C=5$ + gate	92.66	76.27	12.46	6.72
InterQ-RL	segment prefix	$\alpha_{s}(\rho_{s})$	96.39	81.96	10.06	4.48

Appendix

Table 6: Token-local corrections with learner alignment on ALFWorld; all quantized arms use the W8A8-KV8 rollout and W/A+KV-QAT learner.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,742 papers

Apr 9, 2026

QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch

Large language model (LLM) reinforcement learning (RL) pipelines are often bottlenecked by rollout generation, making end-to-end training slow. Recent work mitigates this by running rollouts with quantization to accelerate decoding, which is the most expensive stage of the RL loop. However, these setups destabilize optimization by amplifying the training-inference gap: rollouts are operated at low precision, while learning updates are computed at full precision. To address this challenge, we propose QaRL (Rollout Alignment Quantization-Aware RL), which aligns training-side forward with the quantized rollout to minimize mismatch. We further identify a failure mode in quantized rollouts: long-form responses tend to produce repetitive, garbled tokens (error tokens). To mitigate these problems, we introduce TBPO (Trust-Band Policy Optimization), a sequence-level objective with dual clipping for negative samples, aimed at keeping updates within the trust region. On Qwen3-30B-A3B MoE for math problems, QaRL outperforms quantized-rollout training by +5.5 while improving stability and preserving low-bit throughput benefits.

Hao Gu, Hao Wang, Jiacheng Liu +9

The Hong Kong University of Science and Technology · City University of Hong Kong · Zhejiang University

Reinforcement Learning RL Fine-Tuning RL for Language Model Reasoning Quantization-Aware Training

Ask PDF

Oct 10, 2026

KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization

Asynchronous rollout improves the efficiency of reinforcement learning with verifiable rewards (RLVR) by reducing under-utilization during long-tail generation. Sustained concurrency also concentrates context processing into fewer decode steps, increasing the KV workload per step. We characterize this KV concentration and find that KV cache quantization delivers substantially greater acceleration than weight-only and weight-activation quantization in our matched-workload comparisons. However, INT4 KV rollout can degrade learning even with token-level truncated importance sampling (TIS). We propose KV4-RL, which combines calibration-free INT4 KV quantization with selective INT8 protection of attention sinks and recent context. Protecting a bounded number of positions reduces sampler mismatch while retaining most of the speedup from INT4. Across 400 iterations of Qwen3-8B-Base GRPO training, KV4-RL achieves

1.80\times

cumulative rollout speedup and

1.27\times

cumulative full-iteration speedup over BF16 rollout, reaching

1.97\times

rollout speedup over the final 50 iterations. Under a common BF16 evaluator, its final six-benchmark mean accuracy is within 0.5 percentage points of BF16-rollout training for both Qwen3-4B-Base and Qwen3-8B-Base.

Sihwan Park, Sung-Yub Kim, Doohyuk Jang +1

KAIST · NAVER Cloud · AITRICS

KV-Cache Quantization RL Post-Training Reinforcement Learning with Verifiable Rewards Asynchronous RL Mixed-Precision Quantization

Ask PDF

Jul 29, 2026

HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models

We present, to our knowledge, the first end-to-end FP4 RL post-training, in which both the rollout and training policies, including their forward and backward passes, operate at 4-bit precision. A systematic study reveals that the dominant source of degradation in FP4 RL is not training-side quantization error but rollout activation quantization: outliers stretch the dynamic range so far that a large number of activation values underflow to zero under FP4. Counterintuitively, restoring the training policy to higher precision while keeping the rollout in FP4 makes accuracy worse than full FP4 baseline, exposing rollout-training mismatch as the principal failure mode and ruling out standard pretraining-style fixes. We address this with Rollout Residual Quantization (Rollout-ResQ): a single residual correction term constrained to a hardware-friendly sparsity pattern, added only to the FP4 rollout matmul -- a lightweight correction that recovers most of the precision lost to outlier-driven underflow without inflating the rollout's compute footprint. On Qwen2.5-3B and Qwen2.5-Math-7B, Rollout-ResQ paired with the HiFloat4 (HiF4) format -- whose three-level hierarchical scaling preserves resolution under FP4's tight 4-bit budget -- closes the accuracy gap to BF16 from 4.9% to 1.1%, bringing fully quantized FP4 RL within striking distance of full precision. Applied to the open-standard MXFP4, the same recipe narrows the gap from 13.6% to 5.3%, revealing that FP4 format choice is a key factor that determines the ceiling on recoverable accuracy. Together, these results establish HiF4 as the enabling format for end-to-end FP4 RL post-training, and Rollout-ResQ as the activation-side mechanism that makes the gap to BF16 closable.

Hei Yi Mak, Shadan Golestan, Hoang Le +10

Huawei

LLM Quantization 4-Bit Quantization Activation Quantization RL Post-Training

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch

KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization

HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models