FSPO: Policy-Consistent Risk and Pareto-Feasible Control for Budgeted LLM RL Post-Training

Miaobo Hu; Shuhao Hu; Xiaobo Guo; Xin Wang; Bokun Wang; Daren Zha

cs.AIOct 2, 2026

FSPO: Policy-Consistent Risk and Pareto-Feasible Control for Budgeted LLM RL Post-Training

Authors: Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Daren Zha

Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China

Abstract

Adaptive LLM reinforcement-learning post-training changes multiple training actuators online, including rollout temperature, group size, clipping, KL regularization, verifier allocation, and update budget. Three coupled issues remain unresolved. A future-risk model trained from behavior trajectories need not estimate the risk induced by the controller that will be deployed; a score calibrated on logged state-action pairs can become miscalibrated after selective action choice; and independent per-resource minimum costs do not in general certify a feasible multi-resource continuation. We introduce FSPO, a feedback-state controller for budgeted LLM RL post-training that addresses these issues jointly. FSPO learns a policy-consistent risk-to-go model whose Bellman target follows the same frozen controller used for future decisions, together with a long-horizon utility model. Decision-conditioned trajectory calibration (DCTC) calibrates risk on cross-fitted trajectories generated by actions selected by provisional controllers. A Pareto resource continuation certificate (PRCC) admits an action only when a non-dominated cumulative reservation remains feasible over the residual horizon. Under a matched GRPO resource envelope, FSPO reaches 66.11% held-out and 59.43% OOD accuracy, compared with 64.47% and 57.03% for PB2, the strongest evaluated adaptive baseline. Three paired training seeds give gains of +2.42 and +3.19 percentage points over the contextual bandit on held-out and OOD evaluation. Under high behavior-deployment mismatch, policy-consistent risk lowers selected-decision ECE from 0.108 to 0.053; DCTC lowers it from 0.039 to 0.022 at matched acceptance; PRCC removes false-feasible admissions on an 18-action catalog ( $0.197\rightarrow0.000$ ); and enabling all three components reduces trajectory failure from 0.181 to 0.083 in a factorial ablation.

Figures & tables

evidence layer	recorded object	measured output	evaluation unit
online trace	192 transitions/policy	action selection, replay, support, ledger accounting	case–interval transition
post-freeze oracle	120 separate oracle rows	descriptive offline action gap	frozen-trace comparison
executed ledger outcomes	4,608 held-out/OOD rows	paired mechanism sensitivity	case-level outcome row
real-trainer surface	6 artifact classes + metric schema	compute, task quality, seeds, outcomes	scheduler-backed run

Table 1: Evidence layers and their measured outputs. Each layer has a fixed object, unit, and interpretation in the FSPO protocol.

Figure 1: FSPO algorithm and controlled comparison. The online loop reads only observed telemetry, applies the fixed action catalog and feasibility checks, logs the committed transition, and freezes the trace before the oracle join. The lower panel shows the matched policy set.

policy	control family	held-out acc. (%)	OOD acc. (%)	failure	GPU-h
uniform	static random	58.72	50.81	0.082	9.44
best-static	static, calibration-selected	60.08	52.43	0.061	9.31
hand rule	threshold rule	61.34	53.58	0.046	9.17
fixed controller	deterministic feedback score	63.02	55.71	0.031	9.23
adaptive-KL	KL-error feedback	61.91	54.64	0.038	9.24
contextual bandit	online linear UCB	63.69	56.24	0.035	9.35

Table 2: Matched GRPO trainer comparison (Qwen2.5-3B-Instruct; three training seeds; 576 executed outcome rows per policy). Accuracy is higher-is-better; held-out interval failure and training GPU-hours per seed are lower-is-better. Best values are bold and second-best values are underlined. PBT/PB2 GPU-hours are per population member; population totals are in Table 41 .

Figure 2: Component evidence under the matched GRPO contract. (a) Selected-decision ECE versus behavior–deployment action mismatch. (b) All-action (gray) and selected-action ECE for five risk estimators; the DCTC variant calibrates on controller-selected decisions. (c) Held-out accuracy (bars, left axis) and trajectory failure (dashed, right axis) for the eight combinations of policy consistency (P), DCTC (D), and PRCC (R).

stream seed	uniform	rule	bandit	controller	bandit violation	rows
13	1.43397	2.82833	1.67437	2.34404	0.07813	768
17	1.59877	1.10599	1.69818	2.10365	0.11979	768
23	1.63560	0.75640	1.87640	2.20217	0.04167	768
29	1.50880	0.83315	1.82104	2.10102	0.08854	768
31	1.60721	1.23114	1.80662	2.19201	0.06771	768
mean	1.55687	1.35100	1.77532	2.18858	0.0792	3,840

Table 3: Progress per ledger unit across five independent fixture/transition streams (controller seed 13; 24 cases and 192 transitions per policy and stream). The best policy in each row is bold; the last column is the bandit violation rate.

Appendix figures & tables60 assets

Supplementary material from the paper’s appendix.

Appendix

action	$T$	$\|G\|$	clip	$\kappa$	verifier calls	$e$	tokens / ledger unit
conservative	0.70	2	0.10	0.30	1	0.25	9 / 0.0058
balanced	0.90	4	0.20	0.15	2	0.50	16 / 0.0096
exploratory	1.20	7	0.30	0.05	3	0.75	25 / 0.0149

Appendix

Table 4: Composite action catalog and declared ledger costs.

\kappa

is an action field distinct from observed

K_{t}

;

e

is verifier escalation; the listed compute values use the declared ledger unit.

component	definition or search domain	selected value
input and action	telemetry, budget, interval, normalized action; catalog output	24 input fields
encoder	two hidden layers; SiLU; width search: $\{64,96,128,160,192\}$	$128$
ensemble	complete-trajectory bootstrap; member count	$M=5$
optimizer	AdamW; learning rate, weight decay, batch size	$3\times 10^{-4},\,10^{-4},\,128$
temporal target	discount, target-network update rate, risk-loss weight	$\gamma=0.95,\ \tau=0.02,\ \lambda_{r}=1.0$
calibration	monotone map, trajectory-blocked split, risk threshold	isotonic regression; coverage target $0.92$ ; $\delta_{\rm risk}=0.06$

Appendix

Table 5: FSPO fitting and selection configuration. All selections use calibration trajectories.

role	stream seeds	cases	selection rule	persisted evidence
primary screen	13	24	fixed configuration and action catalog	per-policy trace, metrics, oracle join, replay digest
independent audit	13, 17, 23, 29, 31	24 per stream	controller seed fixed at 13; paired fixture/transition streams	five stream summaries, 3,840 observed rows, 120 oracle rows
calibration	13, 17, 23	24 per stream	select rule threshold and $\eta_{\mathrm{ref}}$ only	threshold sweep, calibration summary, selection digest
held-out	29, 31	24 per stream	evaluate frozen threshold and reward contracts	per-policy outcomes, matched deltas, bootstrap samples
OOD	37, 41	24 per stream	shift regime mixture and retain catalog	per-policy outcomes, shift summary, sensitivity rows

Appendix

Table 6: Seed roles and split construction. Each row binds a stream family, its decision role, the number of cases, and the artifact that records the resulting trace.

field	range / unit	feature map	protocol use
reward mean	real, ledger-normalized	$\tanh(\cdot)$	recent progress level
reward standard deviation	nonnegative	$\min(1,\cdot)$	dispersion scale
reward dispersion	$[0,1]$	identity	rule threshold and bandit feature
group pass rate	$[0,1]$	identity	verifier and sampling context
policy entropy	nonnegative	$\min(1,\cdot)$	exploration signal
clip saturation	$[0,1]$	identity	trust-region pressure

Appendix

Table 7: Policy-visible state dictionary. All fields are recorded before action selection unless the “post-transition” marker is shown; the oracle score is stored in a separate post-freeze record.

field	record stage	type	validation rule
event id	pre-action	string	unique within the run and sortable by case/interval
stream and policy id	pre-action	string	matches manifest and table row
state digest	pre-action	SHA-256	recomputes from visible telemetry only
action id	decision	enum	one of conservative, balanced, exploratory
action propensity	decision	probability	sums to one over catalog actions
ledger token cost	transition	nonnegative real	matches cost contract

Appendix

Table 8: Serialized event schema. Required fields are present for every mechanism row, and trainer fields use the same event schema.

policy	intervals	action coverage	source tokens	source GPU-s
uniform	150	1.000	70003	973.503
rule	150	0.667	70003	973.503
bandit	150	1.000	70003	973.503
controller	150	0.333	70003	973.503

Appendix

Table 9: Observational external telemetry replay. Source tokens and GPU-seconds are shared across policies; action-type coverage counts the three catalog actions represented in shadow recommendations.

split	intervals	mean reduction	median	next-window tokens
calibration (13,17)	100	0.032566	0.009139	46586
evaluation (19)	50	0.032135	0.025191	23367

Appendix

Table 10: Logged-outcome shadow fields by split. The reduction is signed and computed from the next logged interval; the policy recommendation is not an executed intervention.

component	workload or source handle
trainer and evaluator	GRPO post-training implementation and task evaluator
training data	JSONSchemaBench-derived repair examples
held-out evaluation	BFCL-context structural repair split
OOD evaluation	JSONSchema-test repair split
online telemetry	Per-step progress logs, run metadata, scheduler receipts, and executed-outcome log

Appendix

Table 11: Trainer workloads and telemetry. All artifact handles resolve against the experimental source root recorded in the manifest.

control	shared contract
calibration	Rule thresholds, $\eta_{\mathrm{ref}}$ , and learned-controller hyperparameters use calibration seeds.
seed roles	At least three independent training seeds per method; individual rows and uncertainty accompany the aggregate.
method set	Uniform, hand rule, fixed controller, contextual bandit, and learned controller with a distinct label.
model and data	Same base model, tokenizer, prompt contract, training file, optimizer family, sequence limit, and evaluation cadence.
resource envelope	Same declared training-token, step, and compute budgets; actual consumption is reported per seed.
run records	Dry plan, launch, collect, run metadata, summary metrics, and executed action/outcome log.

Appendix

Table 12: Calibration, seed, and resource controls for the trainer comparison. Selection precedes held-out and OOD scoring.

component	protocol	measured value
RL optimizer	GRPO group-relative advantage and clipped policy objective with KL regularization	AdamW; learning rate $1.5\times 10^{-6}$ ; $\beta=(0.9,0.95)$ ; weight decay $0.10$
base model and tokenizer	one frozen initialization and tokenizer shared by all policies	Qwen2.5-3B-Instruct; shared tokenizer and initialization digest sha256:41d8a6c2b7f1
prompt and reward data	training prompts, verifier contract, and reward normalization	BFCL-context training prompts; structure/schema verifier; frozen calibration $z$ -normalizer
rollout horizon	fixed interval horizon; action-dependent response count	24 groups $\times$ 8 intervals $=192$ decisions per seed; $G_{t}\in\{2,4,7\}$
action catalog	temperature, group size, clip, KL coefficient, verifier calls, and escalation	$(0.70,2,0.10,0.30,1,0.25)$ ; $(0.90,4,0.20,0.15,2,0.50)$ ; $(1.20,7,0.30,0.05,3,0.75)$
resource envelope	maximum training tokens, optimizer steps, callback time, and GPU-hours per seed	12.583M tokens; 1,536 optimizer steps; callback $\leq 0.90$ h; 9.60 GPU-hours per seed

Appendix

Table 13: GRPO trainer protocol and receipt fields. The actuator, resource, and evaluator contracts are fixed before policy comparison.

FSPO field	trainer operation	receipt field	value
temperature $T_{t}$	rollout sampler temperature	sampler configuration and realized distribution	$\{0.70,0.90,1.20\}$ ; 100% requested/realized match
group size $G_{t}$	responses per prompt for group-relative advantage	rollout count and token ledger	$\{2,4,7\}$ responses; exact rollout-count receipt
clip $\epsilon_{t}$	clipped surrogate ratio bound	optimizer configuration	$\{0.10,0.20,0.30\}$
KL coefficient $\kappa_{t}$	KL penalty in the policy objective	loss decomposition and KL telemetry	$\{0.30,0.15,0.05\}$
verifier calls $V_{t}$	repeated reward evaluations for each response	verifier receipt and reward rows	$\{1,2,3\}$ calls per response
escalation $e_{t}$	extra micro-steps or retry allowance	optimizer and scheduler rows	$\{0.25,0.50,0.75\}$ mapped to 1/2/3 retry levels

Appendix

Table 14: GRPO actuator mapping and required execution fields.

policy	state use	learned state	matched fields
uniform	none	none	catalog, horizon, budget, transition, stream seed
hand rule	dispersion, reliability, KL, progress, entropy	none	catalog, horizon, budget, transition, stream seed
fixed controller	full visible state and action features	none	catalog, horizon, budget, transition, stream seed
contextual bandit	full visible state and action features	per-action $\theta_{a},N_{a}$	catalog, horizon, budget, transition, stream seed
trainer adapter	declared trainer telemetry	optimizer and policy fields as configured	base model, tokenizer, train file, schedule, compute envelope

Appendix

Table 15: Baseline contracts. The “learned state” column identifies which policy parameters change during a run; all other fields are frozen by the manifest.

policy	selection rule	calibration data	acc. (%)	failure	GPU-h
uniform	uniform draw from the fixed catalog	none	58.72/50.81	0.082	9.44
best-static	best single action selected on calibration	calibration only	60.08/52.43	0.061	9.31
hand rule	fixed threshold rule from the mechanism contract	calibration threshold	61.34/53.58	0.046	9.17
fixed controller	deterministic feedback-state score	fixed coefficients	63.02/55.71	0.031	9.23
contextual bandit	online linear contextual update	calibration reward reference	63.69/56.24	0.035	9.35
NeuralUCB	nonlinear mean and uncertainty heads	calibration trajectories	64.21/56.87	0.029	9.44

Appendix

Table 16: GRPO baselines and matched-control contracts. Every row uses the same trainer callback and evaluator; accuracy is held-out/OOD, failure is the held-out interval failure rate, and GPU-h is training GPU-hours per seed (per population member for PBT/PB2). Best values are bold.

quantity	unit	source	aggregation
ledger tokens	ledger tokens	action cost equation	sum over intervals and cases
ledger compute	ledger units	action cost equation	sum over intervals and cases
ledger latency	ledger time	action cost equation	sum and mean per case
optimizer steps	steps	trainer log	per run and per seed
training tokens	tokens	tokenizer counter	per run and per seed
wall-clock	hours	scheduler receipt	elapsed active interval

Appendix

Table 17: Resource accounting fields. Ledger fields use the declared transition contract; trainer fields use scheduler, optimizer, tokenizer, and evaluator records with their specified units.

seed	uniform		rule		bandit		controller
	progress	viol.	progress	viol.	progress	viol.	progress	viol.
13	1.43397	0.16146	2.82833	0.00000	1.67437	0.07813	2.34404	0.00000
17	1.59877	0.11458	1.10599	0.37500	1.69818	0.11979	2.10365	0.04167
23	1.63560	0.11458	0.75640	0.49479	1.87640	0.04167	2.20217	0.00000
29	1.50880	0.19792	0.83315	0.48438	1.82104	0.08854	2.10102	0.04167
31	1.60721	0.11458	1.23114	0.30208	1.80662	0.06771	2.19201	0.00000

Appendix

Table 18: Per-stream mechanism summary. Progress is progress per ledger unit (higher is better), violation is the interval violation rate (lower is better), and each stream contains 24 cases and 192 transitions per policy. The best policy in each row is bold.

partition	policy	ratio	viol.	accepted	conservative%	exploratory%	paired CI
held-out	uniform	1.5580	0.1562	324	34.11%	32.29%	[1.442,1.674]
held-out	rule (default)	1.0321	0.3932	233	57.55%	18.49%	[0.874,1.189]
held-out	rule (frozen 0.40)	1.9911	0.1927	310	72.40%	8.07%	[1.821,2.161]
held-out	controller	2.1465	0.0208	376	44.01%	4.69%	[2.031,2.258]
held-out	bandit (step ratio)	2.8040	0.0260	374	22.14%	31.51%	[2.612,2.991]
held-out	bandit ( $\eta$ aligned)	1.8138	0.0781	354	49.48%	17.19%	[1.669,1.954]

Appendix

Table 19: Regime and threshold ablation. Ratios and violation rates use the partition summaries; action mix and accepted transitions are aggregated from the per-policy outcome logs. Best ratio, violation, and accepted count within each partition are bold; action mix is descriptive.

contract	policy	held-out ratio	OOD ratio	KL change	reported delta	paired CI
step ratio	bandit	2.8040	2.2508	+0.0061	reference	[2.612,2.991]/[2.006,2.501]
$\eta$ aligned	bandit	1.8138	1.2450	-0.0034	-0.9901 / -1.0058	[1.669,1.954]/[1.018,1.474]
clipped $\eta$	bandit	2.2147	1.7026	-0.0011	-0.5893 / -0.5482	[2.061,2.368]/[1.478,1.925]
no learning	controller	2.1465	2.1499	-0.0078	-0.6575 / -0.1009	[2.031,2.258]/[1.913,2.384]
$\kappa=0.0$	matched action	0.0217	0.0189	reference	reference	[0.0211,0.0223]/[0.0181,0.0197]
$\kappa=1.0$	matched action	0.0204	0.0171	-0.04	-0.0013 / -0.0018	[-0.0018,-0.0008]/[-0.0026,-0.0011]

Appendix

Table 20: Reward-contract and KL sensitivity surface. The

\eta

row reports ratio differences and the

\kappa

row reports mean one-step progress differences; slash-separated values are held-out/OOD. KL change is next-step KL at

\kappa=1

minus

\kappa=0

failure layer	trigger	mechanism count	trainer diagnostic
schema	missing or malformed visible field	0	adapter rejects event and writes validation error
action	action outside catalog or propensity sum mismatch	0	policy callback and scheduler command disagree
budget	ledger or token budget exceeded	0	optimizer steps exceed declared envelope
stability	$g(s_{t},a_{t})>0$	531	KL, memory, or service constraint breach
transition	next state fails deterministic equation	0	telemetry record cannot be replayed
evaluation	evaluator version or split mismatch	0	task score cannot be paired with run seed

Appendix

Table 21: Failure taxonomy. A row is assigned to the first failing layer in the order shown; counts are measured per event.

policy	progress mean	progress SD	progress 95% interval	violation mean	violation SD	streams
uniform	1.55687	0.08352	[1.48366, 1.63008]	0.14063	0.03792	5
rule	1.35100	0.84832	[0.60741, 2.09459]	0.33125	0.20167	5
bandit	1.77532	0.08577	[1.70014, 1.85050]	0.07917	0.02862	5
controller	2.18858	0.09905	[2.10176, 2.27540]	0.01667	0.02282	5

Appendix

Table 22: Independent-stream uncertainty for the mechanism estimands. Intervals use

\bar{x}\pm 1.96s/\sqrt{5}

; means and sample standard deviations are computed over complete stream statistics for seeds

\{13,17,23,29,31\}

contrast field	unit	direction	recorded join
task-quality difference	evaluator score	higher is better	held-out and OOD seed, task id, evaluator digest
GPU-hour difference	GPU-hours	lower is better	scheduler receipt, run id, seed
wall-clock difference	hours	lower is better	launch/collect timestamps and run id
stability difference	interval rate	lower is better	policy trace, case, interval, predicate result
action-mix difference	fraction	descriptive	catalog id, propensity, case, interval
paired interval	metric units	two-sided 95%	complete-case resample id and bootstrap seed

Appendix

Table 23: Paired contrast schema. Every comparison records its unit, direction, seed join, and source digest.

Figure 3: Mechanism result matrix in progress per ledger unit. Upper left: primary 24-case screen. Upper right: mean

\pm

sample SD over five fixture/transition streams. Lower panels: held-out and OOD partitions under the frozen threshold and both bandit reward contracts.

policy	progress / ledger unit	stability violation
uniform	1.4340	0.1615
rule	2.8283	0.0000
bandit (eta-aligned)	1.6744	0.0781
controller	2.3440	0.0000

Appendix

Table 24: Single-stream reconstruction example (seed 13). Values follow the released summary and are bound to the source manifest.

policy	progress/token	tokens	verifier calls	rollback
uniform	0.000869	16.48	1.98	0.1615
controller	0.001454	11.81	1.40	0.0000
rule	0.001823	9.00	1.00	0.0000
bandit ( $\eta$ aligned)	0.001020	15.32	1.83	0.0781

Appendix

Table 25: Efficiency and stability fields from the primary ledger. Progress per token, mean per-interval costs, rollback, and budget rejection are mechanism measurements. Best values are bold.

policy	$P_{\pi}^{\mathrm{off}}$	$P_{*}$	$r_{\pi}^{\mathrm{off}}$
uniform	4.3054	5.3504	0.00544
controller	4.5464	5.3504	0.00419
rule	4.2864	5.3504	0.00554
bandit ( $\eta$ aligned)	4.3354	5.3504	0.00529

Appendix

Table 26: Offline oracle audit on the same 192 ledger transitions.

P_{*}

is the per-trace best-action sum; the gap is

(P_{*}-P_{\pi}^{\mathrm{off}})/192

. Oracle labels are joined after trace freeze and remain outside the online state.

partition	policy label	progress / ledger unit	stability violation
held-out	uniform	1.5580	0.1562
held-out	rule (default)	1.0321	0.3932
held-out	rule (frozen 0.40)	1.9911	0.1927
held-out	controller	2.1465	0.0208
held-out	bandit (step ratio)	2.8040	0.0260
held-out	bandit ( $\eta$ aligned)	1.8138	0.0781

Appendix

Table 27: Held-out/OOD mechanism audit. The rule threshold is frozen at 0.40 using selection seeds {13,17,23}; the reward sensitivity uses a frozen

\eta_{\mathrm{ref}}=2.8448

from the same calibration partition. Rows report ledger outcomes under the declared transition contract.

run record	receipt identifier
scheduler dry plan	dry-plan receipt
scheduler launch record	launch receipt
scheduler collect record	collect receipt
run metadata	run-metadata receipt
summary metrics	summary receipt
executed trainer action/outcomes	executed-outcome log

Appendix

Table 28: Trainer run records. Six receipt classes bind the task and resource measurements to the policy trace and manifest.

policy	records	complete support	mean positive support	min propensity
uniform	192	1.0000	3.0000	0.3333
rule	192	0.0000	1.0000	1.0000
bandit	192	0.0052	1.0156	0.3333
controller	192	0.0000	1.0000	1.0000

Appendix

Table 29: Behavior-policy support in the action-logging audit with randomized exact bandit ties. “Complete support” is the fraction of rows with positive probability for all three catalog actions; the minimum is the smallest probability assigned to the selected action.

Figure 4: Reward and regime sensitivity on the held-out and OOD partitions. Left: progress per ledger unit for the two bandit rewards. Right: mean case-progress difference between the fixed controller and the frozen-threshold rule, with paired 95% bootstrap intervals over complete cases.

control	held-out progress	OOD progress	violation difference	paired 95% CI
full feedback	reference	reference	reference	reference
frozen initial telemetry	-0.2475	-0.3314	+0.0365 / +0.0573	[-0.302,-0.193]/[-0.405,-0.258]
one-interval delay	-0.1397	-0.2046	+0.0260 / +0.0391	[-0.184,-0.096]/[-0.265,-0.145]
within-regime shuffle	-0.2862	-0.3741	+0.0443 / +0.0703	[-0.347,-0.226]/[-0.455,-0.294]
KL channel fixed at calibration mean	-0.0824	-0.1315	+0.0130 / +0.0234	[-0.121,-0.044]/[-0.184,-0.079]
verifier channels fixed at calibration means	-0.1736	-0.2588	+0.0313 / +0.0495	[-0.224,-0.123]/[-0.326,-0.191]

Appendix

Table 30: Effect of feedback information on progress and stability. Each cell reports a paired difference from full feedback over complete held-out or OOD cases; positive progress and negative violation differences are favorable.

risk variant	ECE	Brier	coverage	violation	abstain	accuracy
raw risk head	0.086/0.128	0.114/0.158	0.873/0.821	0.041/0.070	0.019/0.034	63.78/56.11
ensemble uncertainty margin	0.067/0.103	0.101/0.143	0.907/0.858	0.031/0.056	0.035/0.051	64.66/57.09
calibrated immediate risk	0.039/0.074	0.088/0.131	0.931/0.884	0.025/0.047	0.047/0.069	64.97/57.66
calibrated risk-to-go	0.022/0.049	0.071/0.108	0.956/0.918	0.014/0.031	0.057/0.083	66.11/59.43

Appendix

Table 31: Risk calibration on held-out and OOD trajectories. Lower ECE, Brier, and violation are better; higher coverage and task accuracy are better.

$\delta_{\rm risk}$	held-out ECE	OOD ECE	held-out violation	OOD violation	abstain	total cost
0.02	0.018	0.043	0.006	0.017	0.142	9.96
0.04	0.020	0.046	0.010	0.024	0.091	10.28
0.06	0.022	0.049	0.014	0.031	0.057	10.56
0.08	0.025	0.053	0.022	0.043	0.032	10.71
0.10	0.030	0.061	0.034	0.058	0.018	10.84

Appendix

Table 32: Risk-threshold sweep for the calibrated risk-to-go gate. Each row fixes

\delta_{\rm risk}

before evaluating the same held-out/OOD trajectories.

mismatch	risk estimator	selected ECE	Brier	traj. failure	abstain	accuracy
low	behavior-policy Monte-Carlo risk-to-go	0.041	0.083	0.104	0.041	64.71
low	policy-consistent risk-to-go	0.036	0.078	0.096	0.044	64.98
medium	behavior-policy Monte-Carlo risk-to-go	0.069	0.102	0.157	0.038	63.42
medium	policy-consistent risk-to-go	0.042	0.082	0.105	0.049	64.69
high	behavior-policy Monte-Carlo risk-to-go	0.108	0.137	0.236	0.034	60.88
high	policy-consistent risk-to-go	0.053	0.091	0.128	0.057	64.17

Appendix

Table 33: Policy-consistency stress test. The comparison separates behavior-policy Monte-Carlo risk from the policy-consistent risk recursion. Lower selected ECE, Brier, trajectory failure, and abstention are favorable; accuracy is in percent. The better estimator within each mismatch bin is bold.

method	all ECE	selected ECE	Brier	traj. fail.	interval fail.	abstain	accuracy
Immediate-risk head	0.081	0.096	0.114	0.197	0.041	0.019	63.78
Behavior-policy Monte-Carlo risk-to-go	0.059	0.074	0.096	0.154	0.030	0.031	64.39
Policy-consistent risk	0.048	0.058	0.086	0.129	0.024	0.037	64.88
Policy-consistent + global calibration	0.027	0.039	0.078	0.101	0.019	0.049	65.47
Policy-consistent + DCTC	0.026	0.022	0.071	0.083	0.014	0.057	66.11

Appendix

Table 34: Risk estimation and calibration after action selection. Selected ECE is computed on actions chosen by the frozen controller; interval and trajectory failure use the same held-out trajectories.

operating point	method	selected ECE	traj. fail.	accuracy	cost
matched acceptance	global calibration	0.039	0.101	65.47	10.54
matched acceptance	DCTC	0.022	0.083	66.11	10.56
matched empirical risk	global calibration	0.034	0.083	65.18	10.39
matched empirical risk	DCTC	0.022	0.083	66.11	10.56

Appendix

Table 35: Calibration comparison at matched operating points. Equal acceptance fixes the selected-decision coverage; equal empirical risk fixes the calibration-role trajectory failure.

catalog	admission rule	false-feasible	completion	abstain	accuracy
3 actions	one-step budget	0.139	0.821	0.023	65.01
3 actions	independent-minimum baseline	0.087	0.889	0.041	65.54
3 actions	PRCC	0.000	0.958	0.057	66.11
9 actions	independent-minimum baseline	0.143	0.846	0.034	65.72
9 actions	PRCC	0.000	0.969	0.051	66.37
18 actions	independent-minimum baseline	0.197	0.793	0.029	65.61

Appendix

Table 36: Multi-resource continuation stress test across action catalogs. Lower false-feasible, false-infeasible, and abstention rates are favorable; completion and accuracy are higher-is-better. Best values within each catalog size are bold.

PC	DCTC	PRCC	held-out acc.	OOD acc.	traj. fail.	total GPU-h
no	no	no	62.94	54.87	0.181	9.78
yes	no	no	63.83	56.02	0.146	10.13
no	yes	no	63.56	55.77	0.139	9.97
no	no	yes	63.41	55.51	0.159	10.08
yes	yes	no	65.01	57.73	0.111	10.36
yes	no	yes	64.83	57.21	0.137	10.43

Appendix

Table 37: Three-component factorial ablation under the matched GRPO contract. Accuracy is higher-is-better; trajectory failure and total development GPU-hours per seed are lower-is-better. Best values are bold and second-best values are underlined.

variant	held-out accuracy	OOD accuracy	violation	abstain	total GPU-h
full FSPO	66.11	59.43	0.014	0.057	10.56
no trajectory calibration	64.83	57.21	0.031	0.044	10.43
immediate risk instead of risk-to-go	64.96	57.58	0.027	0.049	10.41
no uncertainty margin	65.28	58.06	0.024	0.041	10.32
one-step budget check	65.01	57.73	0.029	0.023	10.36
myopic utility target	64.57	57.02	0.017	0.061	10.28

Appendix

Table 38: FSPO component ablation under a matched resource envelope. Accuracy and coverage are higher-is-better; violation, abstention, and total cost are lower-is-better. Best values are bold.

adaptive actuator	held-out accuracy	OOD accuracy	violation	action-change rate	GPU-h
temperature only	61.72	53.50	0.052	0.181	9.20
group size only	62.41	54.31	0.046	0.203	9.33
clip only	61.95	53.72	0.043	0.167	9.18
KL only	62.88	55.24	0.031	0.191	9.16
verifier only	63.16	55.69	0.029	0.224	9.41
escalation only	61.58	53.27	0.049	0.158	9.30

Appendix

Table 39: Actuator disentanglement under matched telemetry and resource envelopes. Each adaptive row changes only the named actuator family. Best accuracy, violation, and GPU-hours are bold; the action-change rate is descriptive.

calibration behavior	min action count	unseen action rate	OOD state distance	$Q$ error	risk error
uniform exploratory	184	0.000	0.68	0.037	0.028
randomized rule mixture	117	0.000	0.74	0.043	0.032
deterministic hand rule	18	0.126	0.93	0.067	0.051
deterministic feedback controller	11	0.168	1.01	0.074	0.058

Appendix

Table 40: Calibration support and extrapolation diagnostics. Counts use complete trajectories; prediction errors are reported on held-out states with the declared utility and risk targets. Best values are bold.

method	train GPU-h	fit GPU-h	calibration GPU-h	search GPU-h	total GPU-h
uniform	9.44	0.00	0.00	0.00	9.44
adaptive-KL	9.24	0.00	0.22	0.30	9.76
NeuralUCB	9.44	0.28	0.24	0.52	10.48
PBT	38.18	0.00	0.28	1.14	39.60
PB2	38.42	0.22	0.31	1.78	40.73
FSPO	9.29	0.47	0.18	0.62	10.56

Appendix

Table 41: Matched cost accounting for learned and adaptive controllers. GPU-hours are reported per seed; PBT and PB2 training includes all four population members; total cost includes fitting, calibration, search, and final training. Lowest values are bold.

model	task	baseline accuracy	FSPO accuracy	violation	seeds	total GPU-h
Qwen2.5-3B-Instruct	structural repair	64.47	66.11	0.014	3	31.68
Qwen2.5-3B-Instruct	verifier-based reasoning	61.52	63.74	0.019	3	32.64
Qwen2.5-7B-Instruct	verifier-based reasoning	66.13	68.21	0.021	3	56.82

Appendix

Table 42: Cross-model and cross-task transfer. Each row uses a matched resource envelope within its model/task block and at least the declared independent seed count.

source $\rightarrow$ target	mode	method	accuracy	traj. fail.	selected ECE	GPU-h
3B repair $\rightarrow$ 3B reasoning	refit	FSPO	63.74	0.118	0.025	32.64
3B repair $\rightarrow$ 3B reasoning	recalibrate only	FSPO	63.09	0.134	0.033	29.31
3B repair $\rightarrow$ 3B reasoning	fully frozen	FSPO	62.14	0.187	0.068	28.77
3B reasoning $\rightarrow$ 7B reasoning	refit	FSPO	68.21	0.126	0.028	56.82
3B reasoning $\rightarrow$ 7B reasoning	recalibrate only	FSPO	67.46	0.146	0.037	53.34
3B reasoning $\rightarrow$ 7B reasoning	fully frozen	FSPO	66.39	0.204	0.079	52.71

Appendix

Table 43: Cross-setting transfer modes. Refit repeats utility/risk fitting and DCTC in the target block; recalibrate-only freezes the learned utility/risk models and changes only calibration; fully frozen transfers the complete controller. Accuracy, trajectory failure, selected ECE, and GPU-hours are reported per target block; best values within each block are bold.

comparison	evaluation slice	accuracy difference (pp)	GPU-hour difference	failure difference (pp)
controller vs. rule	held-out, fixed budget	+1.68	+0.06	-1.50
bandit η vs. rule	held-out, fixed budget	+2.35	+0.18	-1.10
controller vs. rule	OOD, fixed budget	+2.13	+0.06	-1.70
bandit η vs. rule	OOD, fixed budget	+2.66	+0.18	-1.20
controller vs. rule	fixed quality	+0.06	-0.73	-1.34
bandit η vs. rule	fixed quality	+0.04	-0.41	-0.96

Appendix

Table 44: Matched-budget and matched-quality trainer contrasts. Differences compare the fixed controller and

\eta

-aligned bandit against the frozen hand rule using the same training seeds; the better of the two contrasts in each slice is bold.

method	fit GPU-h	calibration GPU-h	search GPU-h	decision ms	total GPU-h
Adaptive-KL	0.00	0.22	0.30	0.07	9.76
NeuralUCB	0.28	0.24	0.52	0.46	10.48
Safety-Critic Gate	0.43	0.17	0.41	0.59	10.31
FSPO, 3 actions	0.47	0.18	0.62	0.73	10.56
FSPO, 9 actions	0.51	0.20	0.84	1.27	10.84
FSPO, 18 actions	0.56	0.22	1.12	2.06	11.19

Appendix

Table 45: Controller and calibration overhead. Decision latency is measured per training decision and is separated from rollout and optimizer time. Lowest values are bold.

reported quantity	uniform	hand rule	fixed controller	contextual bandit	FSPO
optimizer steps	1508	1521	1527	1524	1531
training tokens	12.46M	12.03M	12.19M	12.39M	12.16M
total wall-clock hours	9.44	9.17	9.23	9.35	9.29
total GPU-hours	9.44	9.17	9.23	9.35	9.29
peak memory (GiB)	52.4	49.8	51.1	54.6	52.7
mean throughput (tokens / s)	366.6	364.4	366.9	368.1	363.6

Appendix

Table 46: Matched-training budget and systems surface. GPU-hours, throughput, memory, and outcome rows are reported per policy and seed aggregation.

held-out metric	uniform	hand rule	fixed controller	contextual bandit	FSPO
downstream task return	0.642	0.671	0.690	0.698	0.732
downstream task accuracy	58.72%	61.34%	63.02%	63.69%	66.11%
trainer progress	0.214	0.247	0.269	0.279	0.316
violation / failure rate	0.082	0.046	0.031	0.035	0.014
action mix (conservative / balanced / exploratory)	33.5/33.1/33.4%	61.3/29.8/8.9%	45.1/43.2/11.7%	39.7/42.5/17.8%	48.8/42.6/8.6%
uncertainty (seed count + CI)	3; [57.93,59.51]	3; [60.61,62.07]	3; [62.28,63.76]	3; [62.90,64.48]	3; [65.42,66.80]

Appendix

Table 47: Held-out BFCL-context GRPO evaluation; accuracy intervals are marginal. Metrics are reported per policy with three independent seeds.

OOD metric	uniform	hand rule	fixed controller	contextual bandit	FSPO
downstream task return	0.562	0.594	0.617	0.624	0.662
downstream task accuracy	50.81%	53.58%	55.71%	56.24%	59.43%
trainer progress	0.178	0.203	0.226	0.232	0.267
violation / failure rate	0.104	0.069	0.052	0.057	0.031
action mix (conservative / balanced / exploratory)	33.4/33.2/33.4%	68.2/25.4/6.4%	52.7/39.1/8.2%	45.6/41.4/13.0%	56.4/38.7/4.9%
uncertainty (seed count + CI)	3; [49.93,51.69]	3; [52.61,54.55]	3; [54.82,56.60]	3; [55.30,57.18]	3; [58.47,60.39]

Appendix

Table 48: OOD JSONSchema-test GRPO evaluation; accuracy intervals are marginal. Metrics are reported per policy with three independent seeds.

policy	split	seeds	GPU-hours	task accuracy	failure rate	outcome rows
uniform	held-out	3	9.44	58.72%	0.082	576
rule	held-out	3	9.17	61.34%	0.046	576
controller	held-out	3	9.23	63.02%	0.031	576
bandit η	held-out	3	9.35	63.69%	0.035	576
FSPO	held-out	3	9.29	66.11%	0.014	576
uniform	OOD	3	9.44	50.81%	0.104	576

Appendix

Table 49: Matched GRPO trainer study. Every policy uses the same base model, tokenizer, training file, optimizer family, sequence limit, evaluation cadence, and declared compute envelope. Best accuracy, failure, and GPU-hour values are bold within each split.

cases	intervals	regime mix	policy	progress	ratio	violation	paired CI
24	8	current mix	uniform	2.7503	1.4340	0.1615	[1.286,1.582]
24	8	current mix	rule	3.1496	2.8283	0.0000	[2.431,3.207]
24	8	current mix	controller	3.2962	2.3440	0.0000	[2.211,2.477]
24	8	current mix	bandit η	3.0006	1.6744	0.0781	[1.542,1.807]
96	8	matched mix	uniform	11.0437	1.4567	0.1458	[1.398,1.515]
96	8	matched mix	rule	12.7364	2.6914	0.0339	[2.553,2.830]

Appendix

Table 50: Scale and horizon study with all four policies at each setting. The first block uses the primary run; expanded settings retain the regime proportions, catalog, and calibration contract. Progress and ratios are higher-is-better; violations are lower-is-better. Best values within each block are bold.

contract	policy	held-out ratio	OOD ratio	held-out violation	OOD violation	paired CI
step ratio	bandit	2.8040	2.2508	0.0260	0.1432	[2.612,2.991]/[2.006,2.501]
eta aligned	bandit	1.8138	1.2450	0.0781	0.2708	[1.669,1.954]/[1.018,1.474]
clipped eta	bandit	2.2147	1.7026	0.0521	0.2135	[2.061,2.368]/[1.478,1.925]
no learning	controller	2.1465	2.1499	0.0208	0.1406	[2.031,2.258]/[1.913,2.384]

Appendix

Table 51: Reward-contract ablation on the held-out/OOD partitions. Ratios are higher-is-better and violations lower-is-better; best values are bold.

baseline	split	$\bm{d_{1}}$	$\bm{d_{2}}$	$\bm{d_{3}}$	mean (pp)	95% CI (pp)
fixed controller	held-out	+3.22	+2.96	+3.09	+3.09	[+2.77,+3.41]
contextual bandit	held-out	+2.55	+2.31	+2.40	+2.42	[+2.12,+2.72]
hand rule	held-out	+4.89	+4.63	+4.79	+4.77	[+4.44,+5.10]
fixed controller	OOD	+3.86	+3.57	+3.73	+3.72	[+3.36,+4.08]
contextual bandit	OOD	+3.33	+3.06	+3.18	+3.19	[+2.85,+3.53]
hand rule	OOD	+5.99	+5.68	+5.88	+5.85	[+5.46,+6.24]

Appendix

Table 52: Paired GRPO accuracy contrasts. Each cell uses the same initialization, data order, evaluator, and resource envelope for both policies; positive differences favor FSPO.

policy	held-out acc.	OOD acc.	return	failure	GPU-hours	outcome rows
uniform	58.72	50.81	0.642	0.082	9.44	576
best-static	60.08	52.43	0.659	0.061	9.31	576
hand rule	61.34	53.58	0.671	0.046	9.17	576
fixed controller	63.02	55.71	0.690	0.031	9.23	576
contextual bandit	63.69	56.24	0.698	0.035	9.35	576
adaptive-KL	61.91	54.64	0.681	0.038	9.24	576

Appendix

Table 53: Matched GRPO results under the declared resource envelope. Accuracy and return are higher-is-better; failure rate and GPU-hours per seed are lower-is-better. Accuracy is in percent; failure is per attempted interval.

comparison	held-out $\Delta$ acc.	OOD $\Delta$ acc.	failure $\Delta$	held-out 95% CI	OOD 95% CI	seeds
FSPO vs. fixed controller	+3.09	+3.72	-0.017	[+2.77,+3.41]	[+3.36,+4.08]	3
FSPO vs. contextual bandit	+2.42	+3.19	-0.021	[+2.12,+2.72]	[+2.85,+3.53]	3
FSPO vs. hand rule	+4.77	+5.85	-0.032	[+4.44,+5.10]	[+5.46,+6.24]	3
FSPO vs. best-static	+6.03	+7.00	-0.047	[+5.65,+6.41]	[+6.57,+7.43]	3
FSPO vs. adaptive-KL	+4.20	+4.79	-0.024	[+3.86,+4.54]	[+4.41,+5.17]	3

Appendix

Table 54: Paired GRPO contrasts for the learned controller. The interval is a two-sided 95% Student-

t

interval over independent training-seed differences, reported separately for held-out and OOD accuracy.

control	held-out acc.	OOD acc.	failure	action change	paired CI	outcome rows
full feedback	66.11	59.43	0.014	0.412	reference	576
frozen initial telemetry	62.77	54.98	0.043	0.167	[-3.72,-2.96]/[-4.91,-3.99]	576
one-interval delay	64.39	57.02	0.026	0.286	[-2.09,-1.35]/[-2.86,-1.96]	576
within-interval shuffle	62.31	54.47	0.049	0.334	[-4.18,-3.42]/[-5.43,-4.49]	576
best-static action	60.08	52.43	0.061	0.000	[-6.41,-5.65]/[-7.43,-6.57]	576

Appendix

Table 55: GRPO feedback-state ablation. Full feedback is the reference; every control preserves the rollout and optimizer budget.

catalog	construction	held-out acc.	OOD acc.	failure	GPU-hours
3 actions	reference catalog	66.11	59.43	0.014	9.29
expanded (9 actions)	independently calibrated combinations	66.83	60.06	0.013	9.37
compositional (18 actions)	expanded combinations with escalation	67.02	60.21	0.014	9.49

Appendix

Table 56: Action-space scaling under the GRPO trainer contract. Accuracy is higher-is-better; failure and GPU-hours per seed are lower-is-better. Best values are bold.

family	calibration relation	ledger ratio	progress	violation	paired CI
declared family	calibration and evaluation share contract	2.4386 / 2.1187	13.3928 / 12.1843	0.0117 / 0.0547	[+0.176,+0.464]
shifted dispersion	coefficients and regime mix changed after calibration	2.2817 / 1.9674	12.7285 / 11.8462	0.0365 / 0.0820	[+0.167,+0.462]
shifted reliability	verifier reliability process changed after calibration	2.1076 / 1.8459	11.9731 / 11.1194	0.0677 / 0.1250	[+0.119,+0.405]

Appendix

Table 57: Independent transition-family generalization. Coefficients are frozen independently of controller selection. Each cell reports controller / best-static mechanism values; the paired CI is for their ledger-ratio difference. The better value in each pair is bold.

variant	held-out acc.	OOD acc.	failure	GPU-hours	paired CI (pp)
full FSPO	66.11	59.43	0.014	9.29	reference
without risk filter	65.02	57.88	0.052	9.24	[-1.46,-0.72]/[-1.96,-1.14]
without uncertainty margins	65.28	58.06	0.024	9.21	[-1.20,-0.46]/[-1.79,-0.95]
without prospective budget mask	65.01	57.73	0.029	9.31	[-1.47,-0.73]/[-2.16,-1.24]
myopic utility target	64.57	57.02	0.017	9.20	[-1.89,-1.19]/[-2.83,-1.99]
without telemetry features	62.77	54.98	0.043	9.18	[-3.72,-2.96]/[-4.91,-3.99]

Appendix

Table 58: FSPO component effects on GRPO. Accuracy is reported in percent, failures as a fraction of attempted intervals, and cost as GPU-hours per seed. Best values are bold.

artifact	role	manuscript use	validation
public summary	normalized summary	Tables 1 – 2	schema and numeric cross-check
source manifest	provenance record	implementation and audit text	digest and source-state check
observed trace	action decisions	online and propensity tables	row coverage and state replay
oracle labels	post-freeze scores	offline oracle table	action-vector join after freeze
held-out/OOD bundle	partition results	held-out/OOD and sensitivity tables	per-policy row counts and digests
trainer receipt set	scheduler gate	trainer surfaces	six required receipts

Appendix

Table 59: Artifact map and table dependencies. Each entry names an artifact role, manuscript use, and deterministic validation check for the release.

check	mechanism	trainer record	evidence file
source commit and dirty flag	verified	recorded	source manifest
configuration digest	verified	recorded	config digest
exact action catalog	verified	recorded	action schema
visible-state digest	verified	recorded	observed trace
transition replay	verified	recorded	replay digest
oracle join after freeze	verified	recorded	oracle labels

Appendix

Table 60: Release and replay checklist. The mechanism column records the replay check; the trainer column records the corresponding receipt.

stage	operation	required receipt	reported fields
1	freeze source, config, model, tokenizer, and data identifiers	manifest and source digest	commit, dirty flag, config hash
2	generate scheduler dry plan for all policies and seeds	dry-plan JSON	GPU type, token budget, step budget, evaluator version
3	launch matched runs with policy callback enabled	launch receipt	run id, policy id, seed, start time
4	collect telemetry and executed action/outcome rows	collect receipt and JSONL log	visible state, action, propensity, outcome, cost
5	evaluate held-out and OOD tasks	evaluator receipt	task return, accuracy, failures, confidence interval
6	join scheduler, trainer, and evaluator records	summary manifest	wall-clock, GPU-hours, tokens, memory, action mix

Appendix

Table 61: Trainer execution plan and receipts. The stages are ordered by dependency and each quantitative field is bound to the corresponding scheduler or evaluator output.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Jun 4, 2026

Cross-Epoch Adaptive Rollout Optimization for RL Post-Training

LLM post-training often relies on reinforcement learning methods that sample multiple rollouts per prompt, yet most existing approaches use a fixed rollout budget for every prompt, despite large differences in the training signal different prompts provide. In this paper, we study adaptive rollout allocation under a fixed global budget and formulate the problem as online resource allocation with prompt-level diminishing returns. Our method, CERO, maintains a Beta posterior over each prompt's success probability and uses the posterior expected Bernoulli variance as a Bayesian estimate of the value of additional rollouts. We use this estimate to construct a concave, saturating utility over cumulative allocations, yielding an objective in which decisions across prompts and epochs are coupled by the global budget. Since the resulting objective is temporally nonseparable, we derive a Fenchel-dual reformulation and update both prompt-level and budget-level dual variables via projected online gradient descent. Under fixed prompt utilities, we prove an

O(\sqrt{K})

regret bound against the offline allocation benchmark. Experiments on mathematical-reasoning problems show that CERO consistently outperforms GRPO across multiple open-weight LLMs and benchmarks, demonstrating that adaptive rollout budgeting can improve sample efficiency.

Yiming Zong, Yige Wang, Jiashuo Jiang

Department of Industrial Engineering & Decision Analytics, Hong Kong University of Science and Technology

Sequential Decision Making RL for Language Model Reasoning RL Post-Training Online Resource Allocation Sample-Efficient RL

Ask PDF

Feb 4, 2026

QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning

GRPO-style reinforcement learning (RL)-based LLM fine-tuning algorithms have recently gained popularity. Relying on heuristic trust-region approximations, however, they can lead to brittle optimization behavior, as global importance-ratio clipping and group-wise normalization fail to regulate samples whose importance ratios fall outside the clipping range. We propose Query-Adaptive Trust-Region policy Optimization (QUATRO), which directly enforces trust-region constraints through a principled optimization. This yields a clear and interpretable objective that enables explicit control over policy updates and stable, entropy-controlled optimization, with a stabilizer terms arising intrinsically from the exact trust-region formulation. Empirically verified on diverse mathematical reasoning benchmarks, QUATRO shows stable training under increased policy staleness and aggressive learning rates, maintaining well-controlled entropy throughout training.

Doyeon Lee, Eunyi Lyou, Hyunsoo Cho +3

Seoul National University · Department of Artificial Intelligence / Institute for Multiscale Matter and Systems, Ewha Woman University · Geogia Institute of Technology

Trust-Region Optimization Policy Learning Policy Optimization RL for Language Model Reasoning Trust Region Policy Optimization RL Fine-Tuning+1

Ask PDF

Jun 2, 2026

RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training

The standard LLM training pipeline applies reinforcement learning (RL) only after pre-training and supervised fine-tuning (SFT). We question this status quo by training a LLM from scratch and applying RL, SFT, and SFT followed by RL directly to intermediate pre-training checkpoints. We find that RL is effective very early, and often matches the full SFT

\to

RL pipeline early as well. Through experiments on harder problems, we find that targeted pre-training data composition is a strong lever for RL effectiveness, even more so than model scale. Beyond reasoning accuracy, applying RL directly to base checkpoints expands the model's distribution; the sharpening effect reported in recent work arises only when RL follows SFT. The general capabilities of the model remain essentially unchanged by RL, while they degrade following SFT. Finally, we merge RL and SFT objectives by parallel averaging, which outperforms across all other training methods discussed, across metrics, while preserving general capabilities. Together, these results suggest that LLM training might benefit from an expanded use of RL.

Rachit Bansal, Clara Mohri, Tian Qin +2

Harvard University

Language Model Pretraining Supervised Fine-Tuning RL for Language Models Reinforcement Learning RL Post-Training

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

Cross-Epoch Adaptive Rollout Optimization for RL Post-Training

QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning

RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training