HiLoRe: What to Store, Compress, or Recompute for Efficient GRPO Training

Xinrui Chen; Mengyang Li; Ou Wu; Ji Zhang

cs.LGSep 27, 2026

HiLoRe: What to Store, Compress, or Recompute for Efficient GRPO Training

Authors: Xinrui Chen, Mengyang Li, Ou Wu, Ji Zhang

Organizations: Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China · Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin, China · University of Southern Queensland

Abstract

Group-relative policy optimization (GRPO) makes learner-side activations a major memory-computation bottleneck: gradient checkpointing reduces activation memory through recomputation, but fixed schedules can leave roughly 18 GB unused on a 48-GB GPU despite substantial recomputation overhead. Existing activation-management methods set state fidelity from execution cost, tensor properties, or generic compression sensitivity, without explicitly incorporating GRPO's analytic update structure into state-fidelity allocation. We formalize this dependence as policy-update exposure, linking the current GRPO loss coefficients to state-level approximation sensitivity. These coefficients are available before backward without an additional backward pass. We introduce HiLoRe, which allocates graph-attributed recovery units among high-precision storage, low-precision compression, and deterministic recomputation using measured recovery utility and update-conditioned approximation risk. It combines high-precision storage and deterministic recomputation with low-precision recovery under a calibrated risk budget. Across five model-task settings with 2K responses and memory < 1.10 times GC's per-GPU actor-update peak, HiLoRe's actor-update throughput gains reach 13.5% over GC and 7.9% over the fastest evaluated baseline, with paired mean downstream-score differences below 0.6 percentage points.

Figures & tables

Figure 1: Motivation of update-conditioned recovery. Fixed recovery underutilizes memory headroom, while GRPO exposes update-dependent state fidelity before backward.

Figure 2: HiLoRe maps GRPO-native signals and utility to fidelity-constrained H/L/R scheduling.

Panel of Figure 3 — Algorithm 1 HiLoRe Recovery Allocation

Model	Method	Efficiency			Gradient Fidelity	Training Quality
Model	Method	Peak MiB $\downarrow$	Tok./s $\uparrow$	Gain (%) $\uparrow$	Grad. err. $\downarrow$	MATH500 $\uparrow$	GSM8K $\uparrow$
Qwen2.5-3B	GC / All-R \venuebox arXiv’16	28,874	2,455.72	–	$\mathbf{0.0000}$	$60.20_{\pm 1.00}$	$77.33_{\pm 0.15}$
	Rockmate \venuebox ICML’23	29,992	2,066.73	$-15.84_{\pm 0.79}$	$0.0031$	$60.20_{\pm 0.80}$	$76.90_{\pm 0.04}$
	ALAM \venuebox ICLR’24	$\mathbf{28{,}595}$	2,381.56	$-3.02_{\pm 0.57}$	0.0131	$59.13_{\pm 0.50}$	$76.57_{\pm 0.23}$
	Adacc \venuebox arXiv’25	31,400	$2{,}579.00$	$+5.02_{\pm 0.49}$	0.0107	$60.00_{\pm 0.40}$	$77.08_{\pm 0.12}$
	INSTANT \venuebox ICLR’26	29,236	2,515.39	$+2.43_{\pm 0.47}$	0.0105	$59.80_{\pm 0.40}$	$77.00_{\pm 0.12}$

Table 1: Actor-update efficiency, gradient fidelity, and downstream quality on DeepMath10K with 2K responses. Red and orange indicate the

\mathbf{best}

and second-best values within each model.

Figure 5: Memory–throughput frontier, update fidelity, and recovery allocation.

Table 8

Figure 6: Update sparsity, risk prediction, and component ablations in post-update diagnostic replays.

Update regime	Norm. exposure	H (%)	L (%)
Low $\|\widehat{A}_{t}\|$	0.48	4.8	18.5
High $\|\widehat{A}_{t}\|$	1.71	14.7	9.4
Clipped	0.27	2.9	21.3
Active unclipped	1.43	12.6	11.2

Table 5: Exposure and recovery allocation.

Figure 7: Sensitivity to the internal fidelity budget $\delta$ , calibration size, and risk-refresh interval.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Setting
Parallelism	Four-rank FSDP FULL_SHARD ; decoder-layer wrapping; TP=PP=SP=1; checkpoint regions inside wrappers
FSDP options	use_orig_params=True for LoRA, False for full-parameter training; no forward prefetch; BACKWARD_PRE ; all-gather limiting
Precision	FP32 persistent parameters, trainable-parameter AdamW moments, gradient reduction/accumulation, and floating-point buffers; BF16 parameter computation
Updates	Gradient reduction after each microbatch backward; gradients accumulate over the complete rollout batch before a single optimizer step
Checkpoint	Non-reentrant checkpointing with RNG preservation and early stopping; TF32 and strict deterministic mode disabled; separate model/method RNG streams
Residency	GPU actor; CPU-offloaded reference; rollout weights and KV caches released before actor updates

Appendix

Table 6: Shared learner execution settings; LoRA defaults apply unless specified.

Workload	Train	Validation	Construction
DeepMath10K ( He et al., 2025 )	10,000	2,048	Disjoint DeepMath-103K subsets; topic/difficulty stratification
TACO-Verified ( Li et al., 2023 )	11,874	1,024	Source/difficulty stratification; remaining eligible problems form training set
K&K ( Xie et al., 2025 )	4,000	1,000	Published three- to seven-person training partitions; 800/200 train/validation split per 1,000-problem partition
DSR-sub	1,209	—	Existing training subset; frozen recovery-hyperparameter transfer

Appendix

Table 7: Training and validation splits.

Method	Search space	Configs
GC / No-GC	Fixed All-R / All-H	2 fixed
SAC	Retain GEMM, attention, or both; decoder-layer or attention/MLP regions; all or even layers	12
Rockmate	Decoder-layer or attention/MLP partitions; activation-memory allowance fractions $\{0.90,0.95,1.00\}$ ; DP memory units $\{1,4\}$ MiB	12
GACT	Average bits $\{2,3,4,8\}$ ; group size 256; CKPT	8
ALAM	Average bits $\{0.5,2,8\}$ ; group size 256; CKPT; one group-size-128 refinement per precision	9
CompAct	Rank ratio $\{1/4,1/2,7/8\}$ ; CKPT	6

Appendix

Table 8: Method-specific tuning spaces and configuration counts. CKPT denotes decoder-layer checkpointing on/off.

Benchmark	Split/version	Problems	Metric
MATH500 ( Lightman et al., 2023 )	Full test set	500	Final-answer accuracy
GSM8K ( Cobbe et al., 2021 )	main test	1,319	Numerical-answer accuracy
LiveCodeBench ( Jain et al., 2025 )	release_v6 ; code generation; full tests	1,055	Greedy pass@1
MBPP+ ( Liu et al., 2023 )	v0.2.0	378	Greedy pass@1
K&K ( Xie et al., 2025 )	Three- to seven-person test; 100 problems per size	500	Macro-averaged complete-assignment accuracy
K&K transfer	Eight-person test	100	Complete-assignment accuracy

Appendix

Table 9: Test sets and evaluation metrics.

Model	Method	Efficiency			Gradient Fidelity	Training Quality
Model	Method	Peak MiB $\downarrow$	Tok./s $\uparrow$	Gain (%) $\uparrow$	Grad. err. $\downarrow$	MATH500 $\uparrow$	GSM8K $\uparrow$
Qwen2.5-3B	GACT \venuebox ICML’22	28,329	2,397.52	$-2.37_{\pm 0.39}$	0.0146	$59.27_{\pm 0.70}$	$76.67_{\pm 0.31}$
	SAC \venuebox PyTorch’25	30,680	2,336.37	$-4.86_{\pm 0.82}$	0.0029	$60.20_{\pm 0.60}$	$77.38_{\pm 0.19}$
	CompAct \venuebox NAACL’25	29,058	2,346.19	$-4.46_{\pm 0.47}$	0.0124	$59.73_{\pm 0.50}$	$76.90_{\pm 0.31}$
Phi-3.5-mini	GACT \venuebox ICML’22	17,338	2,695.95	$+3.28_{\pm 0.29}$	0.0142	$43.33_{\pm 0.70}$	$85.27_{\pm 0.27}$
	SAC \venuebox PyTorch’25	18,720	2,533.33	$-2.95_{\pm 0.72}$	0.0029	$44.07_{\pm 0.50}$	$85.95_{\pm 0.19}$

Appendix

Table 10: Additional baselines on DeepMath10K with 2K responses.

	DeepMath10K			TACO-Verified			Logic-RL
Signal	$\rho$	Recall	AUROC	$\rho$	Recall	AUROC	$\rho$	Recall	AUROC
Reconstruction error	0.28	0.37	0.63	0.24	0.35	0.61	0.26	0.36	0.62
Static type/layer	0.32	0.42	0.66	0.29	0.39	0.64	0.30	0.40	0.65
GACT-style sensitivity	0.41	0.54	0.72	0.39	0.52	0.70	0.40	0.53	0.71
ALAM-style sensitivity	0.44	0.56	0.73	0.40	0.54	0.72	0.42	0.55	0.72
Advantage only	0.47	0.60	0.76	0.45	0.58	0.74	0.46	0.59	0.75

Appendix

Table 11: Risk prediction underlying Fig. 6 (b), averaged over held-out replays.

Variant	$\rho$ $\uparrow$	Top-10% recall $\uparrow$	AUROC $\uparrow$
Full $\psi_{u}$	0.78	0.81	0.89
w/o ratio/clipping	0.65	0.68	0.81
w/o update mask	0.78	0.81	0.89
w/o KL contribution	0.77	0.79	0.88
w/o activation weighting	0.67	0.70	0.82
w/o susceptibility	0.74	0.76	0.86

Appendix

Table 12: Exposure-score ablations.

$B/B_{\mathrm{GC}}$	Budget (MiB)	Method	Peak (MiB)	Tok./s $\uparrow$	Gain $\uparrow$	Mean err. $\downarrow$	Max err. $\downarrow$	H/L/R (%)
1.00	28,874	GC / All-R	28,874	2,455.72	–	0.0000	0.0000	$0/0/100$
		AGoQ	28,776	2,533.81	$+3.18\%$	0.0110	–	–
		Adacc	28,764	2,476.10	$+0.83\%$	0.0135	–	–
		HiLoRe	28,843	2,540.93	$\mathbf{+3.47\%}$	0.0068	0.0074	$0/8.3/91.7$
1.05	30,318	Rockmate	29,992	2,066.73	$-15.84\%$	0.0031	–	–
		AGoQ	28,776	2,533.81	$+3.18\%$	0.0110	–	–

Appendix

Table 13: Recovery configurations under memory ceilings.

Figure 8: Response-length and model transfer. Throughput gains are relative to paired GC, with error bars showing sample SDs. Gradient errors and additional peak memory are expressed as percentages.

Setting	$\Delta$ Metric 1	$\Delta$ Metric 2
TACO-Verified / Qwen2.5-3B	$-0.06_{\pm 0.36}$	$+0.09_{\pm 0.40}$
Logic-RL K&K / Qwen2.5-3B	$+0.07_{\pm 0.31}$	$+0.10_{\pm 0.17}$
Phi-3.5-mini / DeepMath10K	$+0.53_{\pm 0.50}$	$+0.13_{\pm 0.19}$
Llama-3.1-8B / DeepMath10K	$+0.53_{\pm 0.46}$	$+0.23_{\pm 0.23}$

Appendix

Table 14: Paired downstream quality changes relative to GC, in percentage points.

Figure 9: Full-parameter memory–throughput trade-offs. GC (tuned) adjusts microbatch size per budget with decoder-layer checkpointing.

$\epsilon_{g}$	HiLoRe	Adacc	AGoQ	PRAC	INSTANT
1.00%	8.76 / 0.78	4.30 / 0.95	2.55 / 0.93	1.85 / 0.94	1.70 / 0.96
1.25%	10.16 / 1.04	5.02 / 1.07	3.18 / 1.10	2.76 / 1.12	2.43 / 1.05
1.50%	10.16 / 1.04	5.02 / 1.07	3.18 / 1.10	2.76 / 1.12	2.43 / 1.05
1.75%	10.94 / 1.58	7.10 / 1.68	6.20 / 1.73	6.70 / 1.65	7.40 / 1.67
2.00%	11.28 / 1.81	8.10 / 1.93	7.20 / 1.96	8.70 / 1.94	9.70 / 1.91

Appendix

Table 15: Shared gradient-error tolerance sensitivity. Each cell reports throughput gain over GC (%) / mean gradient error (%).

Regime	Norm. $\|\omega_{t}\|$	Norm. exposure	H (%)	L (%)	L drift
Low $\|\widehat{A}_{t}\|$	0.41	0.48	4.8	18.5	0.0062
High $\|\widehat{A}_{t}\|$	1.84	1.71	14.7	9.4	0.0127
Clipped	0.19	0.27	2.9	21.3	0.0051
Active unclipped	1.37	1.43	12.6	11.2	0.0115

Appendix

Table 16: Update-regime statistics averaged over held-out post-update diagnostic replays.

Sweep	Setting	Gain (%) $\uparrow$	Mean err. $\downarrow$	Max err. $\downarrow$	Scheduling overhead (%)	H/L/R (%)
$\delta/\delta^{\star}$	0	$+6.14\pm 0.48$	0.0030	0.0034	0.19	$13.9/0/86.1$
	0.25	$+7.35\pm 0.47$	0.0057	0.0064	0.21	$11.1/5.6/83.3$
	0.5	$+8.76\pm 0.46$	0.0078	0.0088	0.21	$8.3/11.1/80.6$
	1	$+10.16\pm 0.44$	0.0104	0.0114	0.21	$8.3/16.7/75.0$
	2	$+10.94\pm 0.46$	0.0158	0.0176	0.21	$5.6/22.2/72.2$
	4	$+11.28\pm 0.49$	0.0181	0.0206	0.22	$2.8/30.6/66.6$

Appendix

Table 17: Sensitivity to risk budget, calibration size, and refresh interval. Scheduling overhead includes exposure/risk refresh and allocation.

Component	GC	HiLoRe-HR	HiLoRe
Forward	146.72	146.85	146.91
Recomputation	207.38	164.10	118.64
Candidate encoding	–	–	6.37
L reconstruction	–	–	7.88
Exposure refresh	–	0.72	0.74
Allocation	–	0.51	0.58

Appendix

Table 18: Profiled actor-update components in seconds; totals sum the listed components.

Explore similar work

Rollout-Level Advantage-Prioritized Experience Replay for GRPO
Jun 3, 2026Gyeongtae Yoo, Sanghyeok Park, Soohyuk Jang +2Parallel Rollouts Replay
Department of Electrical and Computer Engineering, Seoul National University · Interdisciplinary Program in AI, Seoul National University · AIIS, ASRI, INMC, and ISRC, Seoul National University
LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches
Oct 5, 2026Shaokun Zhang, Yifan Zhang, Jian Hu +5Large Language Model Reinforcement Learning Large Language Model Training
LeanGRPO: Eliminating Redundant Recomputation in Diffusion RL
Sep 3, 2026Sijie Wang, Zhiqiang Tan, Xinrui Yang +1Diffusion-Based Reinforcement Learning Methods Reward Gradients
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen

Jun 3, 2026, cs.LGJ/K move · Enter open · S save

Rollout-Level Advantage-Prioritized Experience Replay for GRPO

Gyeongtae Yoo, Sanghyeok Park, Soohyuk Jang, Ik-hwan Kim+1

Department of Electrical and Computer Engineering, Seoul National University · Interdisciplinary Program in AI, Seoul National University · AIIS, ASRI, INMC, and ISRC, Seoul National University

Parallel Rollouts Replay Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement learning from verifiable rewards with GRPO is a standard approach for post-training reasoning LLMs. It remains sample inefficient. Each rollout is used for a single gradient update and then discarded. Naive replay is not well suited in this setting because LLM policies drift quickly per gradient step. Stored rollouts therefore become stale and can destabilize training. We propose a rollout-level replay buffer for GRPO that stores and samples individual rollouts rather than whole groups. The buffer bounds staleness through age eviction. Any rollout older than tau_max training steps is removed. The buffer also preserves on-policy data via fresh-anchored composition. Each batch keeps its fresh on-policy rollouts and then concatenates replay rollouts drawn separately from the buffer. We prioritize replay by per-rollout advantage magnitude and recycle individual rollouts whose advantages are large. Across three Qwen3-Base scales on five math benchmarks, our method outperforms GRPO and naive replay baselines. Gains are positive at every scale and reach +1.66 pp on the five-benchmark average at 4B. Under an AES metric that jointly measures accuracy and token efficiency, our method is the only condition with a positive margin over GRPO at every scale.

Abstract

Explore similar work

Rollout-Level Advantage-Prioritized Experience Replay for GRPO

LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches

LeanGRPO: Eliminating Redundant Recomputation in Diffusion RL