KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization

Sihwan Park; Sung-Yub Kim; Doohyuk Jang; Eunho Yang

cs.LGOct 10, 2026

KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization

Authors: Sihwan Park, Sung-Yub Kim, Doohyuk Jang, Eunho Yang

Organizations: KAIST · NAVER Cloud · AITRICS

Abstract

Asynchronous rollout improves the efficiency of reinforcement learning with verifiable rewards (RLVR) by reducing under-utilization during long-tail generation. Sustained concurrency also concentrates context processing into fewer decode steps, increasing the KV workload per step. We characterize this KV concentration and find that KV cache quantization delivers substantially greater acceleration than weight-only and weight-activation quantization in our matched-workload comparisons. However, INT4 KV rollout can degrade learning even with token-level truncated importance sampling (TIS). We propose KV4-RL, which combines calibration-free INT4 KV quantization with selective INT8 protection of attention sinks and recent context. Protecting a bounded number of positions reduces sampler mismatch while retaining most of the speedup from INT4. Across 400 iterations of Qwen3-8B-Base GRPO training, KV4-RL achieves $1.80\times$ cumulative rollout speedup and $1.27\times$ cumulative full-iteration speedup over BF16 rollout, reaching $1.97\times$ rollout speedup over the final 50 iterations. Under a common BF16 evaluator, its final six-benchmark mean accuracy is within 0.5 percentage points of BF16-rollout training for both Qwen3-4B-Base and Qwen3-8B-Base.

Figures & tables

Figure 1: Illustration of synchronous and quota-based partial rollout with

T=4

prompt groups,

G=2

responses per prompt, and four logical decode slots. Each row represents one position in the active decode batch. A–D denote distinct prompts, and A1/A2 denote the two responses from prompt A. Colors distinguish the update cohorts shown in the schematic but do not indicate assignments fixed in advance. Training, weight loading, and re-prefill boundaries are marked explicitly.

Figure 2: Synchronous and asynchronous rollout traces for Qwen3-4B-Base at mid-training (iteration 300) on H200 (DP2

\times

TP2), with matched KV budgets and concurrency limits. (a) Active batch size and mean context length. (b) Nominal resident KV volume per GPU. (c) Decode-step latency. Brackets identify individual collection intervals in the displayed replica, while dashed horizontal lines denote means computed from unsmoothed decode calls across both replicas over twelve measured rounds. Nominal resident volume measures active logical contexts rather than reserved memory or physical HBM traffic. Appendix B.3 specifies the trace and timing protocol.

Figure 3: Workload characteristics and quantized decode performance for asynchronous rollout at early, middle, and late stages of Qwen3-4B-Base training (iterations 100, 300, and 500) on H200 with TP2. (a) Nominal per-GPU memory and compute demands, with the axes normalized using peak HBM bandwidth and dense BF16 compute throughput so that equal heights represent equal idealized service times. (b) Decode-forward speedup over BF16 on five fixed, uniform-context workload shapes from each checkpoint. W8A8 denotes FP8 weight-activation execution in this panel. Appendix B.4 specifies the workload construction, numerical implementations, and baseline-time weighting.

Figure 4: Sampler fidelity, learning, and execution measurements for selective INT8 protection. (a) Qwen3-8B-Base learning curves under BF16 evaluation, with shading denoting standard deviation over three evaluation seeds. (b) Shared-prefix conditional KL for different allocations of a fixed 144-position protection budget, with KV4 and KV8 included as references. (c) Shared-prefix conditional KL and decode-forward speedup over BF16 with sink width

s=16

and varying recent width

w

. Panels (b) and (c) use the mid-training Qwen3-4B-Base checkpoint (iteration 300) and a heterogeneous-state replay suite distinct from Fig. 3 . KL axes are logarithmic, and error bars show 95% response-cluster bootstrap intervals. Protected positions use INT8 KV storage with BF16 PV arithmetic.

Figure 5: Overview of KV4-RL framework and its attention execution. (a) Quota-based partial rollout transfers completed prompt groups to the BF16 learner and carries unfinished responses across policy updates. (b) The complete history is stored in paged INT4 format, while an INT8 side cache stores the protected sink and recent positions. Attention processes the INT4 and protected regions as disjoint sets and combines their outputs using regional softmax statistics.

Figure 6: End-to-end acceleration and learning progress for Qwen3-8B-Base over 400 training iterations. (a) Rollout collection time per iteration, with 20-iteration means over faint raw traces. (b) Ratios of cumulative BF16 time to cumulative method time for rollout collection and full iterations. KV4 is omitted because its generated responses are substantially shorter. (c) Equally weighted mean accuracy over MATH-500 and AIME 2024/2025 versus cumulative full-iteration time, with error bars denoting standard deviation over three evaluation seeds. Cumulative time excludes initialization, restart downtime, evaluation, and checkpoint saving.

		avg@ $8$			avg@ $32$
Model	Rollout	MATH-500	OlympiadBench	MinervaMath	AIME24	AIME25	AMC23	Mean
Qwen3-8B-Base	BF16	$88.2_{\pm 0.6}$	$56.3_{\pm 0.2}$	$51.9_{\pm 0.4}$	$29.5_{\pm 1.9}$	$20.7_{\pm 0.4}$	$67.9_{\pm 1.2}$	$52.4_{\pm 0.5}$
	KV8	$87.6_{\pm 0.4}$	$56.6_{\pm 0.4}$	$52.0_{\pm 0.6}$	$29.4_{\pm 0.6}$	$21.3_{\pm 0.5}$	$69.9_{\pm 1.1}$	$52.8_{\pm 0.4}$
	KV4	$82.9_{\pm 0.2}$	$50.1_{\pm 0.2}$	$51.2_{\pm 0.8}$	$18.5_{\pm 0.7}$	$15.6_{\pm 0.3}$	$68.4_{\pm 1.4}$	$47.8_{\pm 0.1}$
[4pt/2pt]	KV4-RL	$87.7_{\pm 0.3}$	$56.8_{\pm 0.4}$	$51.8_{\pm 1.4}$	$29.7_{\pm 0.6}$	$22.1_{\pm 0.7}$	$68.9_{\pm 1.1}$	$52.8_{\pm 0.1}$
Qwen3-4B-Base	BF16	$83.0_{\pm 0.2}$	$50.1_{\pm 0.2}$	$48.0_{\pm 0.7}$	$19.1_{\pm 0.2}$	$16.5_{\pm 0.4}$	$65.2_{\pm 1.0}$	$47.0_{\pm 0.1}$

Table 1: Final Qwen3-Base benchmark accuracy (%) under BF16 evaluation. The 4B and 8B checkpoints are evaluated after 600 and 400 training iterations, respectively. MATH-500, OlympiadBench, and MinervaMath use avg@

8

, while AIME 2024/2025 and AMC 2023 use avg@

32

. Subscripts denote standard deviation over three evaluation seeds. Mean assigns equal weight to the six tasks within each seed before computing the reported mean and standard deviation. All values are rounded to one decimal place.

Figure 7: Training trajectories for Qwen3-8B-Base. (a) Mean correctness reward. (b) Mean completed-response length. (c) Logged reference-KL penalty computed with the

k_{3}

estimator against the initial policy. This statistic is distinct from the shared-prefix sampler KL in Eq. 5.2 . Thick curves show 20-iteration means, and faint curves show the corresponding raw traces.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Hardware per training run	4 $\times$ NVIDIA H200 GPUs
Rollout parallelism	Two DP replicas with TP2 each
Learner	FSDP2 with BF16 compute and FP32 master weights
Prompt groups / responses per group	$T=128$ / $G=8$
Prompt / response token limits	1,024 / 16,384
Sampling temperature / top- $p$	$1.0$ / $1.0$ with top- $k$ disabled

Appendix

Table A1: Shared training configuration with ten learning-rate warmup updates. Frozen trace and replay experiments use explicit KV budgets instead of automatic training-pool sizing.

Task	Questions	$k$	Periodic	Dataset source
MATH-500	500	8	Yes	HuggingFaceH4/MATH-500
AIME 2024	30	32	Yes	Maxwell-Jia/AIME_2024
AIME 2025	30	32	Yes	yentinglin/aime_2025
AMC 2023	40	32	No	math-ai/amc23
OlympiadBench	674	8	No	math-ai/olympiadbench
MinervaMath	272	8	No	math-ai/minervamath

Appendix

Table A2: Evaluation tasks. Every task is included at the endpoint. OlympiadBench refers to the English text-only open-ended mathematics competition subset. MinervaMath is the OCWCourses university-level STEM set.

Checkpoint	Phase 1	Phase 2	Phase 3	Phase 4	Phase 5
4B–100	608/621	576/798	448/1,033	304/1,384	168/1,963
4B–300	640/874	576/1,170	464/1,506	336/1,968	200/2,627
4B–500	640/1,463	544/1,895	400/2,568	272/3,518	176/4,767
8B–100	624/547	592/676	496/879	352/1,134	192/1,529
8B–250	656/1,863	384/3,271	320/3,908	272/4,435	208/5,305
8B–400	368/3,461	256/4,712	224/5,529	200/6,196	176/6,972

Appendix

Table A3: Representative replay shapes

(B,\ell)

in generation-phase order. Context lengths are in tokens. The five phases have equal weight in baseline decode time. The first 8B-late context was reduced from the sampled 3,463 to 3,461 tokens to fit the pinned page budget.

Measurement	Included work
Decode replay	Captured forward, TP communication, and argmax at fixed states
Decode throughput	Committed tokens per summed GPU decode-service time
Rollout collection	Collection including prefill/re-prefill and reward scoring
Training iteration	Collection and learning with weight synchronization and phase switches

Appendix

Table A4: Timing scopes. Quantization can change forward cost, feasible resident concurrency, and the amount of work completed in a collection round. These measurements therefore answer different performance questions.

Figure A1 : Workload concentration across training for Qwen3-4B-Base and Qwen3-8B-Base. Each point reports the ratio of asynchronous to synchronous collection at a frozen BF16 checkpoint. Active batch size and nominal resident KV volume are averaged over pure-decode calls. Decode-step latency is measured with CUDA events, and decode-service throughput is the number of committed decode tokens divided by summed decode time. Both TP2 replicas are pooled after three warmup rounds, with twelve measured rounds at 4B and six at 8B. The dashed line denotes parity between the two schedulers.

Figure A2 : Fixed-workload decode-forward speedup for Qwen3-8B-Base on H200 with TP2. Early, Mid, and Late correspond to iterations 100, 250, and 400. Each result aggregates five matched batch–context shapes using equal BF16-time weights, and the dashed line denotes BF16 parity. W4A16 uses vLLM Machete, FP8 W8A8 uses our adaptation of the vLLM CUTLASS GEMM, FP8 KV8 uses FlashInfer FA2, and integer KV8 and KV4 use our paged-attention kernels. KV4 denotes the unprotected INT4 cache. Appendix B.4 specifies the execution paths and timing boundaries.

Figure A3 : Fidelity and decode cost of selective protection. Panels (a,c) compare allocations of the same 144 INT8-protected positions at 4B and 8B. Panels (b,d) report BF16-relative decode speedup against shared-prefix

D_{\mathrm{KL}}(p_{\mathrm{BF16}}|q)

. The connected INT8 series fixes the sink width at 16 and varies the recent width

w

, with the enlarged marker denoting the default

w=128

. BF16 protection uses the same 16/128 positions, while KV8 stores the full cache in INT8. KL axes are logarithmic, and KL error bars are 95% response-cluster bootstrap intervals over 32 responses. Speedups use matched heterogeneous decode states. The 4B measurements use the iteration-300 policy and traces. The 8B fidelity measurements use the iteration-200 policy, with timing states matched to iteration-250 trace statistics.

Figure A4 : Cache-representation fidelity ablations for the quantized KV representation. (a) Rotation choices for affine INT4 with group size 128. (b) Quantization group size versus storage cost at 4B under K-only rotation. (c) Affine INT4, NVFP4, and MXFP4 representations with the rotation applied to each format indicated below the corresponding result. All probes reconstruct the stored cache and use BF16 attention operands, thereby excluding the integer query and probability arithmetic of the deployed kernels. Storage cost includes codes and metadata. Panels (a,c) use logarithmic KL axes, and error bars are 95% response-cluster bootstrap intervals over 32 histories.

Figure A5 : Periodic BF16 evaluation of Qwen3-8B-Base policies trained with four rollout precisions. (a) MATH-500 avg@

8

. (b) AIME 2024 avg@

32

. (c) AIME 2025 avg@

32

. Each rollout precision uses one training run, and every checkpoint is evaluated with seeds 42, 43, and 44. Curves report the mean across these evaluations, with shading denoting one sample standard deviation.

		pass@ $8$			pass@ $32$
Model	Rollout	MATH-500	OlympiadBench	MinervaMath	AIME24	AIME25	AMC23	Mean
Qwen3-8B-Base	BF16	$96.5_{\pm 0.3}$	$73.8_{\pm 0.5}$	$70.7_{\pm 2.4}$	$66.7_{\pm 6.7}$	$52.2_{\pm 5.1}$	$97.5_{\pm 0.0}$	$76.2_{\pm 1.9}$
	KV8	$96.5_{\pm 0.1}$	$73.6_{\pm 0.3}$	$69.6_{\pm 1.5}$	$64.4_{\pm 3.8}$	$55.6_{\pm 1.9}$	$96.7_{\pm 1.4}$	$76.1_{\pm 0.6}$
	KV4	$92.3_{\pm 0.6}$	$66.3_{\pm 0.7}$	$69.0_{\pm 0.8}$	$56.7_{\pm 3.3}$	$37.8_{\pm 1.9}$	$93.3_{\pm 3.8}$	$69.2_{\pm 0.4}$
[4pt/2pt]	KV4-RL	$96.2_{\pm 0.2}$	$74.2_{\pm 0.7}$	$71.3_{\pm 2.9}$	$64.4_{\pm 1.9}$	$57.8_{\pm 1.9}$	$96.7_{\pm 1.4}$	$76.8_{\pm 1.1}$
Qwen3-4B-Base	BF16	$92.5_{\pm 0.6}$	$66.8_{\pm 1.1}$	$63.8_{\pm 0.2}$	$45.6_{\pm 3.8}$	$42.2_{\pm 3.8}$	$95.0_{\pm 2.5}$	$67.7_{\pm 1.2}$

Appendix

Table A5 : Final Qwen3-Base pass@

k

accuracy (%) under BF16 evaluation. The 4B and 8B checkpoints are evaluated after 600 and 400 training iterations, respectively. MATH-500, OlympiadBench, and MinervaMath use pass@

8

, while AIME 2024/2025 and AMC 2023 use pass@

32

Figure A6 : End-to-end timing for Qwen3-4B-Base over 600 training iterations. (a) Rollout collection time per iteration, with 20-iteration means computed separately within each uninterrupted logging segment. (b) Ratios of cumulative BF16 time to cumulative method time for rollout collection and full iterations. Cumulative ratios are computed from summed timers over the same iteration interval. KV4 is omitted from panel (b) because its generated responses are substantially shorter. Timing excludes initialization, restart downtime, evaluation, and checkpoint saving.

Figure A7 : Training trajectories for Qwen3-4B-Base over 600 iterations. (a) Mean correctness reward. (b) Mean completed-response length. (c) Logged reference-KL penalty computed with the

k_{3}

Figure A8 : Iteration-time composition for BF16 and KV4-RL on Qwen3-8B-Base. Policy and reference log-probability computation are separate learner forward passes, while Other is the residual between the logged total and the displayed components. Curves show 20-iteration means, and the annotated percentages report mean component shares over the final 50 iterations.

KV-Cache Quantization

Key-value cache quantization compresses transformer attention state to reduce memory use and bandwidth during long-context inference.

See all 102 papers

Mar 25, 2026

Prune as You Generate: Online Rollout Pruning for Faster and Better RLVR

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs). However, methods such as GRPO and DAPO suffer from substantial computational cost, since they rely on sampling many rollouts for each prompt. Moreover, in RLVR the relative advantage is often sparse: many samples become nearly all-correct or all-incorrect, yielding low within-group reward variance and thus weak learning signals. In this paper, we introduce arrol (Accelerating RLVR via online Rollout Pruning), an online rollout pruning method that prunes rollouts during generation while explicitly steering the surviving ones more correctness-balanced to enhance learning signals. Specifically, arrol trains a lightweight quality head on-the-fly to predict the success probability of partial rollouts and uses it to make early pruning decisions. The learned quality head can further weigh candidates to improve inference accuracy during test-time scaling. To improve efficiency, we present a system design that prunes rollouts inside the inference engine and re-batches the remaining ones for log-probability computation and policy updates. Across GRPO and DAPO on Qwen-3 and LLaMA-3.2 models (1B-8B), arrol improves average accuracy by +2.30 to +2.99 while achieving up to 1.7x training speedup, and yielding up to +8.33 additional gains in average accuracy in test-time scaling. The code is available at https://github.com/Hsu1023/ARRoL.

Haobo Xu, Sirui Chen, Ruizhong Qiu +5

University of Illinois at Urbana-Champaign · Amazon

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards LLM Inference Acceleration

Ask PDF

Apr 9, 2026

QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch

Large language model (LLM) reinforcement learning (RL) pipelines are often bottlenecked by rollout generation, making end-to-end training slow. Recent work mitigates this by running rollouts with quantization to accelerate decoding, which is the most expensive stage of the RL loop. However, these setups destabilize optimization by amplifying the training-inference gap: rollouts are operated at low precision, while learning updates are computed at full precision. To address this challenge, we propose QaRL (Rollout Alignment Quantization-Aware RL), which aligns training-side forward with the quantized rollout to minimize mismatch. We further identify a failure mode in quantized rollouts: long-form responses tend to produce repetitive, garbled tokens (error tokens). To mitigate these problems, we introduce TBPO (Trust-Band Policy Optimization), a sequence-level objective with dual clipping for negative samples, aimed at keeping updates within the trust region. On Qwen3-30B-A3B MoE for math problems, QaRL outperforms quantized-rollout training by +5.5 while improving stability and preserving low-bit throughput benefits.

Hao Gu, Hao Wang, Jiacheng Liu +9

The Hong Kong University of Science and Technology · City University of Hong Kong · Zhejiang University

Reinforcement Learning RL Fine-Tuning RL for Language Model Reasoning Quantization-Aware Training

Ask PDF

Jul 29, 2026

HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models

We present, to our knowledge, the first end-to-end FP4 RL post-training, in which both the rollout and training policies, including their forward and backward passes, operate at 4-bit precision. A systematic study reveals that the dominant source of degradation in FP4 RL is not training-side quantization error but rollout activation quantization: outliers stretch the dynamic range so far that a large number of activation values underflow to zero under FP4. Counterintuitively, restoring the training policy to higher precision while keeping the rollout in FP4 makes accuracy worse than full FP4 baseline, exposing rollout-training mismatch as the principal failure mode and ruling out standard pretraining-style fixes. We address this with Rollout Residual Quantization (Rollout-ResQ): a single residual correction term constrained to a hardware-friendly sparsity pattern, added only to the FP4 rollout matmul -- a lightweight correction that recovers most of the precision lost to outlier-driven underflow without inflating the rollout's compute footprint. On Qwen2.5-3B and Qwen2.5-Math-7B, Rollout-ResQ paired with the HiFloat4 (HiF4) format -- whose three-level hierarchical scaling preserves resolution under FP4's tight 4-bit budget -- closes the accuracy gap to BF16 from 4.9% to 1.1%, bringing fully quantized FP4 RL within striking distance of full precision. Applied to the open-standard MXFP4, the same recipe narrows the gap from 13.6% to 5.3%, revealing that FP4 format choice is a key factor that determines the ceiling on recoverable accuracy. Together, these results establish HiF4 as the enabling format for end-to-end FP4 RL post-training, and Rollout-ResQ as the activation-side mechanism that makes the gap to BF16 closable.

Hei Yi Mak, Shadan Golestan, Hoang Le +10

Huawei

LLM Quantization 4-Bit Quantization Activation Quantization RL Post-Training

Ask PDF

Abstract

KV-Cache Quantization

Explore similar work

Prune as You Generate: Online Rollout Pruning for Faster and Better RLVR

QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch

HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models