FAER: Auditable Utility-Aligned Trajectory Replay for Language Model Post-Training

Miaobo Hu; Shuhao Hu; Xiaobo Guo; Xin Wang; Bokun Wang; Tianshu Fu; Daren Zha; Jun Xiao

cs.LGSep 30, 2026

FAER: Auditable Utility-Aligned Trajectory Replay for Language Model Post-Training

Authors: Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Tianshu Fu, Daren Zha, Jun Xiao

Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China

Abstract

Replay selectors often rank cached trajectories by format feedback, confidence, freshness, or response length, although cache-level correctness and downstream learner utility are distinct objectives. We formalize this selection-to-learning gap and introduce FAER as an auditable full-trajectory replay framework. Its training-free fixed selector is a protocol baseline; FAER-UTILITY is the learner-aware selector fitted on disjoint calibration blocks. The normalized gradient alignment is reported as a baseline, while a disposable optimizer-aware virtual update supplies a magnitude-aware utility surface. The audit contract freezes observed fields and replay traces before evaluation labels are joined. On GSM8K with Qwen2.5-1.5B-Instruct, the matched learner study reports quality 0.6329 for the fixed selector, compared with 0.5482 for uniform and 0.6037 for format-feedback under 128 updates. Metadata-only cross-fitted calibration reaches $0.6476\!\pm\!0.0139$ over eight seeds (median 0.6481; paired 95% interval $[+0.079,+0.122]$ ) at 63,276 target-run tokens; its recorded full cost is 189,642 tokens and 3.48 GPU-hours including calibration. The completed FAER-UTILITY row reaches 0.6624 at 62,844 target-run tokens and 4.26 GPU-hours. Format-feedback selects records with correctness 0.6953, compared with 0.3594 for the fixed selector, despite the different downstream ranking. The completed comparison surfaces report the learner-aware ablation, same-seed gap, policy-optimization rows, and strict zero-shot transfer.

Figures & tables

Figure 1: FAER algorithm and audit contract. The online selector sees the frozen trajectory cache and observed fields, scores and samples 32 records without replacement, and commits an ordered trace digest. The evaluator and three audit lanes join hidden labels and provenance only after the freeze boundary.

Stage	Population / budget	Replication
Fixed cache A	64 train / 128 test; 32 selected	8 selection seeds
Fixed cache B	256 train / 1,319 test; 32 selected	8 selection seeds
Measured age	192 family-A + 1,575 family-B records	independent producers
Matched learner	256 / 128 / 256 train/cal/test tasks; 640 rows per seed	3 independent seeds; 1,920 rows

Table 1: Experimental populations and replication units. Each stage has its own cache binding and denominator.

Selector	Quality	Target tokens	Total tokens	GPU-h	95% CI
Uniform	0.5482	65,214	65,214	0.99	reference
Format-feedback	0.6037	64,982	64,982	0.98	$[+0.031,+0.080]$
FAER-Fixed	0.6329	63,771	63,771	0.96	$[+0.058,+0.112]$
Token-efficiency	0.6186	63,428	63,428	0.95	$[+0.043,+0.098]$
FAER-Utility-Meta	0.6476	63,276	189,642	3.48	$[+0.079,+0.122]$
FAER-Utility	0.6624	62,844	189,210	4.26	$[+0.094,+0.134]$

Table 2: Matched learner on the sealed GSM8K split. Target tokens count only the target replay run; total tokens and GPU-hours include selector calibration. Quality is higher-is-better and cost is lower-is-better. The core rows use three independent seeds; the metadata-only calibrated row is the eight-seed aggregate.

Figure 2: Matched learner outcomes. Left: held-out quality versus target-run generated tokens; right: paired quality differences from uniform with 95% intervals. The interval bars belong to the differences, and the quality points show reported means.

Arm set	Blocks	$\bm{\tau}_{\mathrm{SL}}$	$\bm{\rho}_{\mathrm{SL}}$	$\bm{I}_{\mathrm{SL}}^{\mathrm{norm}}$	$\bm{R}_{\mathrm{SL}}$
Four arms	8	0.1250	0.4500	0.4375	0.0294
95% block CI	8	$[-0.083,+0.333]$	$[+0.180,+0.650]$	$[0.333,0.542]$	$[0.018,0.041]$

Table 3: Same-seed selection-to-learning analysis. Each block uses one producer cache, one learner initialization, and the same sealed evaluation IDs across arms. Rank statistics and regret are summarized over independent blocks.

Selector	Quality	Target tokens	Paired 95% CI
FAER-Fixed	0.6329	63,771	$[-0.045,-0.014]$
FAER-Utility-Meta	0.6476	63,276	$[+0.079,+0.122]$
Alignment-only	0.6418	63,086	$[-0.032,-0.009]$
FAER-Utility	0.6624	62,844	reference
Shuffled alignment	0.6484	63,171	$[-0.025,-0.004]$
Random calibration gradient	0.6447	63,263	$[-0.030,-0.006]$

Table 4: Learner-aware utility ablation under the common replay and update budget. Quality is higher-is-better; target tokens count the target replay run.

Appendix figures & tables47 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Verifier-factor audit for the fixed product. The factor

g(\rho)=\rho(1.5-\rho)

peaks at

\rho=0.75

; the companion panel reports the eight-draw selected-record correctness on the reference cache.

Field	Role	Audit treatment
Trajectory ID	sampling and label join key	included in trace; duplicate IDs rejected
Format status $r_{i}$	observed feedback priority	serialized and taint-tested
Verifier confidence $\rho_{i}$	observed reliability priority	producer key recorded; generation lineage separate
Model confidence $c_{i}$	observed confidence/surprise	serialized and taint-tested
Token count $L_{i}$	length-normalized priority and telemetry	producer field and units recorded
Target answer	offline GSM8K reference	excluded from selector projection

Appendix

Table 5: Field contract for the primary cache. Selector fields are read before trace freezing; the oracle label is joined afterward.

Field	Recorded value
Producer	build_cache.py::produce ; source prefix 4c7a1e9d
Inputs	prompt/model-output; 0 pre-freeze target accesses
Order	generate $\rightarrow$ extract $\rightarrow$ verify $\rightarrow$ serialize $\rightarrow$ freeze $\rightarrow$ oracle join
Runtime	Qwen2.5-1.5B-Instruct; verifier-v1.2; environment prefix b91d63af
Lineage	192/192 records; 0/192 process-isolation violations
Selector	0 oracle-key hits; 1,344/1,344 invariant mutations

Appendix

Table 6: Producer lineage in the current release package. Hashes shown here are eight-character prefixes; full digest fields are indexed in Appendix E.3 .

Field	Setting / result
Model	Qwen2.5-1.5B-Instruct
Optimizer / precision	AdamW / bf16
Learning rate	$2\times 10^{-5}$
Update / token / verifier caps	128 / 65,536 generated tokens / 4,096 calls
Stopping rule	128 updates or generated-token cap
Independent seeds	13, 17, 23

Appendix

Table 7: Shared matched learner configuration and run acceptance.

Policy	Quality	Tokens	p50/p95 (s)	95% interval
Reliability	0.5948	64,284	0.821/1.276	$[+0.023,+0.070]$
Uncertainty-PER	0.5817	64,736	0.835/1.309	$[+0.011,+0.056]$

Appendix

Table 8: Additional learner arms under the same configuration. Quality, generated tokens, and synchronized p50/p95 latency use the sealed evaluation split; 95% paired intervals compare quality with uniform.

Package	Cache / task identity	Comparison axis
(a) Learner update	family A; Qwen/GSM8K	replay policy at shared training budget
(b) Freshness + family	families A and B; Qwen/GSM8K	measured age and producer family
(c) End-to-end	second family release; Qwen2.5-1.5B-Instruct; GSM8K	task quality and generated-token cost

Appendix

Table 9: Experiment identities for the extended learner analysis. All packages use seeds 13, 17, and 23; selection size is 32 per split.

Work	Replay unit / priority	Freshness handling
FreshPER	stored experience with PER priority multiplied by exponential age decay	explicit freshness/age decay
TBA	central replay-buffer data sampled by reward or recency	explicit recency prioritization
RLEP	verified successful trajectories mixed with newly generated rollouts	two-phase replay; no explicit per-row age priority
DOTS/RR	difficulty-targeted online selection plus reuse of recent rollouts	recent-rollout replay
Prioritized Replay for RL Post-training	problem-level priority from empirical success statistics	periodic retesting
Rollout-Level Advantage-PER	individual rollouts prioritized by advantage magnitude	age eviction and fresh-anchored composition

Appendix

Table 10: Replay units and age handling in the cited primary-source records. The six neighboring methods perform learner updates; FAER reports both selection and matched learner stages.

Work	Update treatment	Verifier / target timing
FreshPER	PER replay with exponential age decay; no separate importance-ratio correction	reward/task signal enters the RL update; no FAER-style post-freeze target join
TBA	Trajectory Balance objective on off-policy data	reward enters the TB update; no post-freeze selector/evaluator boundary
RLEP	fresh rollouts mixed with replayed verified successes at each update	verification determines replay eligibility before optimization
DOTS/RR	recent-rollout reuse under GRPO-style policy updates	reward/difficulty statistics are training-time signals; no post-freeze oracle split
Prioritized Replay for RL Post-training	adaptive problem resampling under GRPO; no trajectory-level importance correction	empirical success statistics are training-time priority signals
Rollout-Level Advantage-PER	fresh-anchored GRPO batches plus age-bounded replay	verifiable reward/advantage enters priority before replay update

Appendix

Table 11: Update treatment and verifier timing in the source-level comparison. Neighboring methods are summarized from the listed primary sources; reconstruction counts refer to FAER.

field	type and role	acceptance record
trajectory ID	stable string; cache join key	unique within split; train/test disjoint
trajectory text	generated sequence; replay unit	digest-bound byte sequence
format status	binary observed feedback	producer key and value domain
verifier confidence	$[0,1]$ observed score	producer key, version, and units
model confidence	$[0,1]$ observed score	producer key, version, and units
token count	nonnegative integer telemetry	tokenizer identity and count rule

Appendix

Table 12: Row-level audit schema. Required selector fields are serialized before trace freezing; producer and freshness fields are retained with their source digest. The schema binds each field to its row or run identity.

gate	check	artifact field
identity	source, model, data, split, and row-count hashes	manifest digest
schema	types, required keys, unique IDs, and disjoint splits	validation report
taint	target-field mutation preserves projection and priorities	1,344/1,344 result
trace	seed, ordered IDs, priorities, and replay digest agree	trace manifest
oracle	post-freeze answer join and denominator are complete	evaluator digest
learner	matched model, optimizer, budget, and task IDs	18/18 arm-seed runs accepted; checkpoint/metrics digests complete

Appendix

Table 13: Run acceptance sequence for the fixed-cache audit and matched learner extension. A gate is accepted when its artifact field equals the recorded value; the acceptance record is attached to the corresponding run.

Policy	Train	Test	$\Delta$ test–train
Uniform	13/32 (0.4062)	12/32 (0.3750)	$-0.0312$
Format-feedback	15/32 (0.4688)	22/32 (0.6875)	$+0.2188$
Reliability	14/32 (0.4375)	22/32 (0.6875)	$+0.2500$
Surprise	1/32 (0.0312)	5/32 (0.1562)	$+0.1250$
Uncertainty	6/32 (0.1875)	4/32 (0.1250)	$-0.0625$
Feedback-aware	12/32 (0.3750)	14/32 (0.4375)	$+0.0625$

Appendix

Table 14: Seed-13 selected-record correctness for all seven policies. Each entry is correct selected records divided by 32; labels are joined after trace freeze.

Contrast	$\Delta$ correct	$\Delta$ tokens	95% interval
Format-feedback $-$ uniform	$+0.3086$	$-18.37$	$[+0.242,+0.374]$
Feedback-aware $-$ format-feedback	$-0.3321$	$+20.48$	$[-0.401,-0.263]$
Token-efficiency $-$ format-feedback	$-0.0617$	$+7.74$	$[-0.116,-0.008]$
Reliability $-$ format-feedback	$-0.2016$	$-0.29$	$[-0.287,-0.117]$

Appendix

Table 15: Paired selection estimates on family A. All rows have 128/128 marginal and 8,128/8,128 joint inclusion coverage; intervals are 95% intervals for correctness differences. Tokens are mean cached response lengths.

Family/policy	Coverage	Correct	Age (h)	Digest
A / format-feedback	192/192 (100%)	0.7016	23.7 h	a7c91e42
A / age-weighted	192/192 (100%)	0.7312	9.6 h	5f2bd8c1
B / format-feedback	1,575/1,575 (100%)	0.5849	31.4 h	c83e14aa
B / age-weighted	1,575/1,575 (100%)	0.6617	11.8 h	2de709f6

Appendix

Table 16: Measured-age selection package. Coverage uses all producer rows, correctness is selected-record correctness, and age is the mean over selected records. Digests are prefixes.

Policy	Quality	Tokens	p50/p95 (s)
Uniform	0.5482	65,214	0.842/1.327 s
Format-feedback	0.6037	64,982	0.836/1.305 s
Feedback-aware	0.6329	63,771	0.829/1.289 s
Token-efficiency	0.6186	63,428	0.806/1.251 s

Appendix

Table 17: Complete learner quality and cost rows. Quality is evaluated on the sealed split; p50/p95 latency is in seconds. Lower tokens and latency are better.

Policy	95% interval	Checkpoint
Uniform	reference	step-128@61c8a94e
Format-feedback	$[+0.031,+0.080]$	step-128@73ea20bd
Feedback-aware	$[+0.058,+0.112]$	step-128@c2f47a81
Token-efficiency	$[+0.043,+0.098]$	step-128@95b16f3c

Appendix

Table 18: Mature learner checkpoint handles and paired quality intervals relative to uniform. Checkpoint hashes are prefixes.

Arm	Tokens	p50/p95 (s)	Digest
FAER-product	755	1.24/1.61	8d31f2c4
Uniform	751	1.21/1.58	54b92a17
Format-feedback	744	1.19/1.55	c7e6409b
Reliability	746	1.20/1.56	9a4cf6e2
Token-efficiency	739	1.18/1.53	f13e8b75

Appendix

Table 19: Five-arm runtime preflight. Every arm has age coverage 0/192 and held-out correctness 0/16 (0.0000). Tokens are totals; p50/p95 latencies are seconds. Run digests are prefixes.

Check	Result
Answer extraction	191/192 (99.48%); 1 unresolved
Adjudicated agreement	95/96 (98.96%); 1 changed label
Numeric forms	47/48 exact; 48/48 tolerant-equivalent
Cache reconstruction	192/192 row digests match
Trace reconstruction	56/56 ordered traces match
Table regeneration	176/176 numeric cells match

Appendix

Table 20: Evaluator validation and reconstruction outcomes in the current package. Adjudication and numeric-form checks use their stated sample denominators.

setting	policy	decay	correct	SD	Jaccard
no decay	freshness	0.0000	0.3047	0.0534	0.1377
no decay	feedback-aware	0.0000	0.3359	0.0644	0.1976
no decay	format-feedback	0.0000	0.7031	0.0180	0.6998
mild recent	freshness	0.0200	0.2891	0.0534	0.2312
mild recent	feedback-aware	0.0200	0.3125	0.0807	0.2399
mild recent	format-feedback	0.0200	0.6562	0.0000	0.7457

Appendix

Table 21: Freshness/age stress grid on the frozen test cache. Each cell is the mean selected-record correctness over four selection seeds; SD is seed variation and Jaccard is mean selected-set overlap. Age uses the declared index fallback; measured per-row age is reported in the independent-producer package.

Figure 4: Index-age sensitivity over four settings and four selection seeds. Error bars are sample SD of selected-record correctness; all policies use 32-record draws.

Policy	Seed 13	Mean $\pm$ SD	Min–max	Jaccard
Uniform	12/32 (0.3750)	0.2891 $\pm$ 0.0779	0.1875–0.4062	0.1368
Format-feedback	22/32 (0.6875)	0.6953 $\pm$ 0.0277	0.6562–0.7500	0.6917
Reliability	22/32 (0.6875)	0.4922 $\pm$ 0.1053	0.3750–0.6875	0.2220
Surprise	5/32 (0.1562)	0.0781 $\pm$ 0.0472	0.0312–0.1562	0.2210
Uncertainty	4/32 (0.1250)	0.2070 $\pm$ 0.0471	0.1250–0.2812	0.1771
Feedback-aware	14/32 (0.4375)	0.3594 $\pm$ 0.0625	0.2812–0.4375	0.2162

Appendix

Table 22: Primary fixed-cache selection results. Values are seed-13 count/rate, eight-seed mean

\pm

sample SD, observed range, and mean pairwise Jaccard.

Policy	Family A	Family B
Uniform	0.2891	0.3516
Format-feedback	0.6953	0.5781
Reliability	0.4922	0.4258
Uncertainty	0.2070	0.2344
Feedback-aware	0.3594	0.3594
Token-efficiency	0.6328	0.6523

Appendix

Table 23: Eight-seed selected-record correctness on two frozen cache families.

Figure 5: Descriptive selection-to-learning gap across the six arms with both endpoints. The x-axis is selected-record correctness and the y-axis is held-out learner quality; same-seed block statistics are reported separately.

priority	mean	sample SD	min	max
uniform	0.2891	0.0779	0.1875	0.4062
format-feedback	0.6953	0.0277	0.6562	0.7500
reliability	0.4922	0.1053	0.3750	0.6875
surprise	0.0781	0.0472	0.0312	0.1562
uncertainty	0.2070	0.0471	0.1250	0.2812
feedback-aware	0.3594	0.0625	0.2812	0.4375

Appendix

Table 24: Eight fixed-cache selection seeds on the test split. Values are mean, sample SD, minimum, and maximum of selected-record correctness over seeds 13, 17, 23, 29, 31, 37, 41, and 43. Display rounding is half-even. The table summarizes fixed-cache draw sensitivity; independent model/training replication is reported in the paired learner package.

priority	train	test	$\Delta$ test–train
uniform	0.4062	0.3750	$-0.0312$
format-feedback	0.4688	0.6875	$+0.2188$
reliability	0.4375	0.6875	$+0.2500$
surprise	0.0312	0.1562	$+0.1250$
uncertainty	0.1875	0.1250	$-0.0625$
feedback-aware	0.3750	0.4375	$+0.0625$

Appendix

Table 25: Selected-record oracle-correctness fraction by split for every evaluated priority. The oracle is joined after replay;

\Delta

is test minus train for fixed 32-record slices.

priority	$\Delta$ correct	$\Delta$ format	$\Delta$ verifier	$\Delta$ tokens
uniform	$+0.0000$	$+0.0000$	$+0.0000$	$+0.00$
format-feedback	$+0.3125$	$+0.6563$	$+0.4266$	$-18.25$
reliability	$+0.3125$	$+0.3125$	$+0.2031$	$-18.50$
surprise	$-0.2188$	$-0.1875$	$-0.1219$	$+12.97$
uncertainty	$-0.2500$	$-0.3125$	$-0.2031$	$+19.13$
feedback-aware	$+0.0625$	$+0.1250$	$+0.0813$	$+2.31$

Appendix

Table 26: Test-slice deltas relative to uniform selection. Correctness differences are selected-record diagnostic deltas; format validity and verifier confidence are observed; token differences are mean generated-token telemetry. Values use unrounded source values before four-decimal round-half-even display rounding.

Figure 6: Fixed-cache seed-13 telemetry: selected-record correctness and observed format-ok rate versus mean cached response tokens for seven policies. The two panels align outcomes with selector-visible metadata.

Package	Selected-set quality	Learner loss	Task quality
(a) Learner	$0.3594\pm 0.0625$	$0.842\pm 0.018$	$0.6329\pm 0.0108$
(b) Age/family	$0.6617\pm 0.0249$	$0.811\pm 0.017$	$0.6418\pm 0.0097$
(c) End-to-end	$0.6486\pm 0.0217$	$0.804\pm 0.015$	$0.6497\pm 0.0091$

Appendix

Table 27: Extended package outcomes. Quality is higher-is-better; loss is recorded at 128/128 updates. The selected-set entry in (a) uses eight-draw SD; the other

\pm

entries are sample SD across the three independent seeds.

Package	Generated tokens	p50/p95 (s)	Missing age
(a) Learner	$63,771\pm 416$	0.829/1.289	0/1,920
(b) Age/family	$63,204\pm 389$	0.812/1.263	0/1,767
(c) End-to-end	$62,948\pm 361$	0.791/1.221	0/1,920

Appendix

Table 28: Extended cost and age records. Latency is synchronized p50/p95 in seconds; tokens retain the supplied

\pm

summary. Missing-age counts use the package population.

Priority	Task quality	Tokens	Paired 95% interval
Full product	0.6329	63,771	reference
Remove uncertainty	0.6164	64,093	$[-0.029,-0.004]$
Remove surprise	0.6088	64,261	$[-0.037,-0.011]$
Monotone verifier factor	0.6205	63,984	$[-0.023,-0.002]$
Calibrated verifier	0.6261	63,856	$[-0.015,+0.002]$
Remove format feedback	0.6109	64,182	$[-0.034,-0.010]$

Appendix

Table 29: Effect of feedback components under the mature learner contract. All arms use three independent seeds, 128 updates, and the shared generated-token cap. The interval compares quality with the full product.

Axis	Setting	Task quality	Generated tokens
Update horizon	32 updates	0.5787	16,117
Update horizon	64 updates	0.6089	31,986
Update horizon	128 updates, FAER	0.6329	63,771
Measured-age decay	$\lambda=0$	0.6214	63,486
Measured-age decay	selected $\lambda=0.04$	0.6418	63,204
Measured-age decay	$2\lambda$	0.6306	62,917

Appendix

Table 30: Sensitivity of learner quality to update horizon and measured-age decay. The full 128-update FAER endpoint is the reference; other values follow the same independent-seed protocol.

Control	Trace Jaccard	Task quality	Paired interval
Original fields	1.0000	0.6329	reference
Verifier permutation	0.3614	0.5946	$[-0.055,-0.022]$
Age permutation within run	1.0000	0.6329	$[0.000,0.000]$
Format-status permutation	0.4278	0.6072	$[-0.039,-0.012]$

Appendix

Table 31: Counterfactual controls for selector fields. Jaccard compares the controlled and original selected sets; learner quality uses the matched checkpoint protocol.

Selector	Final quality	Reward	Generated tokens	KL
Uniform	0.5594	0.5217	65,188	0.0379
Format-feedback replay	0.6031	0.5758	64,736	0.0436
FreshPER	0.6226	0.6019	64,127	0.0418
FAER-Fixed	0.6387	0.6194	63,692	0.0427
FAER-Utility	0.6575	0.6426	62,961	0.0443

Appendix

Table 32: Replay selection under a matched policy-optimization learner. All values are aggregated over the declared independent seeds; final quality and reward are higher-is-better, while generated tokens and KL are protocol diagnostics.

Seeds	Policy	Mean $\pm$ SD	Median	95% CI	Tokens
5	uniform	$0.5513\pm 0.0168$	0.5508	reference	65,102
5	feedback-aware	$0.6316\pm 0.0129$	0.6332	$[+0.060,+0.101]$	63,846
5	utility-calibrated	$0.6489\pm 0.0114$	0.6497	$[+0.077,+0.119]$	63,318
8	uniform	$0.5467\pm 0.0193$	0.5486	reference	65,241
8	feedback-aware	$0.6311\pm 0.0148$	0.6305	$[+0.063,+0.103]$	63,802
8	utility-calibrated	$0.6476\pm 0.0139$	0.6481	$[+0.079,+0.122]$	63,276

Appendix

Table 33: Independent producer–learner replication. Each arm uses the matched 128-update budget. Quality reports mean and sample SD, median, and a 95% paired difference interval against uniform; tokens are mean generated tokens per run.

Policy	Replay unit	Priority / fitting	Quality	Tokens
FreshPER adaptation	complete trajectory	PER with age decay	0.6242	63,688
Uniform	complete trajectory	equal weight	0.5482	65,214
Format-feedback	complete trajectory	observed format signal	0.6037	64,982
Fixed FAER	complete trajectory	fixed product	0.6329	63,771
Surprise-only	complete trajectory	$1-c$	0.5893	65,407
Age-only	complete trajectory	$\exp(-\lambda a)$	0.6418	63,204

Appendix

Table 34: Existing replay baseline and matched internal controls. The comparison records the external method’s replay unit and matches the cache, update, and token contracts. Quality is higher-is-better and generated tokens lower-is-better.

Task	Model	Fixed FAER	Utility-calibrated FAER
GSM8K	Qwen2.5-1.5B-Instruct	0.6329 / 63,771	0.6489 / 62,914
MATH-500	Qwen2.5-1.5B-Instruct	0.3816 / 65,318	0.3954 / 65,104
SVAMP	Qwen2.5-1.5B-Instruct	0.7137 / 58,664	0.7284 / 58,203
GSM8K	Qwen2.5-7B-Instruct	0.7043 / 62,087	0.7168 / 61,642

Appendix

Table 35: Cross-task and cross-model transfer surface. Selector parameters are fitted on GSM8K/Qwen2.5-1.5B-Instruct, frozen before every target split, and never refit on the target task or model. Each cell reports held-out quality and generated-token cost under the common evaluator contract.

Target task	Target model	Fixed	Utility-Meta	Utility
MATH-500	Qwen2.5-1.5B-Instruct	0.3816	0.3954	0.4047
SVAMP	Qwen2.5-1.5B-Instruct	0.7137	0.7284	0.7369
GSM8K	Qwen2.5-7B-Instruct	0.7043	0.7168	0.7256
MBPP	Qwen2.5-1.5B-Instruct	0.4128	0.4271	0.4369

Appendix

Table 36: Zero-shot transfer from GSM8K/Qwen2.5-1.5B-Instruct. Target rows use frozen source-domain selector parameters and no target-domain refitting.

Treatment	Transform	Brier	Correct	Jaccard
Fixed product	$\rho(1.5-\rho)$	0.1728	0.3594	1.0000
Remove uncertainty	$\rho$	0.1609	0.4219	0.4386
Reported monotone	$\rho^{0.85}$	0.1587	0.4531	0.4019
Monotone quadratic	$\rho(2-\rho)$	0.1616	0.4375	0.3864
Calibrated verifier	$\operatorname{cal}(\rho)$	0.1472	0.4062	0.5173

Appendix

Table 37: Verifier-shape and calibration diagnostics on the matched cache. Brier score is lower-is-better; correctness uses selected records and Jaccard compares each draw with the fixed product under the same sampling seed.

Selector	Quality	Tokens	GPU-h	95% CI
FAER-Utility-Meta (cross-fitted control)	0.6494	189,642	3.48	$[+0.006,+0.027]$
Random weights	0.5893	251,208	3.32	$[-0.061,-0.028]$
Permuted fitted weights	0.6048	250,734	3.36	$[-0.044,-0.014]$
Entropy-matched fixed FAER	0.6336	191,485	2.72	$[-0.010,+0.011]$
Correctness-calibrated selector	0.6124	188,906	2.68	$[-0.031,-0.009]$

Appendix

Table 38: Controls for cross-fitted utility calibration. Quality is measured on sealed target blocks; tokens and GPU-hours include calibration and target execution. The paired 95% interval compares target quality with fixed FAER.

Selector	Diversity	Duplicates	Quality	Tokens
Format-feedback	0.4127	0.1562	0.6037	64,982
Fixed FAER	0.5274	0.0625	0.6329	63,771
Diversity-matched format-feedback	0.5196	0.0703	0.6174	64,336

Appendix

Table 39: Replay diversity and redundancy under the common learner budget. Diversity is mean pairwise embedding distance; duplicates are the fraction of selected records sharing a preregistered near-duplicate fingerprint.

Estimator	Spearman $\rho$	Kendall $\tau$	Sign acc.	Top-32 regret
Gradient cosine	0.4176	0.2918	0.7064	0.0473
Raw gradient dot product	0.5062	0.3617	0.7441	0.0386
Virtual update, first-order	0.6827	0.4984	0.8019	0.0214
Virtual update + norm correction	0.7489	0.5681	0.8376	0.0127
Shuffled virtual utility	0.0216	0.0149	0.5078	0.0946
Random optimizer-state control	0.0693	0.0481	0.5297	0.0868

Appendix

Table 40: Utility-estimator fidelity on disjoint calibration examples. Correlations compare the predicted score with measured one-step calibration-loss improvement; sign accuracy is the fraction of correct improvement/degradation signs and top-

k

regret is lower-is-better.

Utility estimator	Quality	Target tokens	Paired 95% CI
Cosine alignment	0.6418	63,086	$[-0.032,-0.009]$
Raw gradient dot product	0.6473	63,014	$[-0.025,-0.006]$
Virtual SGD update	0.6511	62,967	$[-0.020,-0.003]$
Virtual AdamW update, first-order	0.6572	62,901	$[-0.010,-0.001]$
Virtual AdamW + norm correction	0.6624	62,844	reference
Shuffled virtual utility	0.6484	63,171	$[-0.025,-0.004]$

Appendix

Table 41: Optimizer-awareness ablation under a common replay and update budget. The downstream learner remains AdamW; only the virtual score used to select trajectories changes. Quality is higher-is-better and the paired interval compares with the optimizer-aware row.

Selector	Target-matched quality	Total-matched quality	Total tokens	GPU-h
Uniform	0.5482	0.6173	189,176	4.25
Format-feedback	0.6037	0.6358	189,094	4.24
FreshPER	0.6242	0.6446	189,128	4.23
FAER-Fixed	0.6329	0.6505	189,163	4.24
FAER-Utility	0.6624	0.6624	189,210	4.26

Appendix

Table 42: Target-budget and total-compute matching. Target-matched quality fixes only the target replay execution; total-matched quality includes selector fitting and utility construction.

Selected $k$	Passes	Uniform	Format-feedback	FAER-UTILITY
16	4	0.5239	0.5796	0.6371
32	1	0.4981	0.5492	0.6031
32	2	0.5276	0.5834	0.6365
32	4	0.5482	0.6037	0.6624
64	2	0.5563	0.6111	0.6581
128	1	0.5604	0.6066	0.6518

Appendix

Table 43: Sensitivity to selected-set size and replay reuse. Each cell is held-out learner quality under the same model, evaluator, and per-condition budget definition.

Experiment	Recorded outcome
Fixed-cache selection	family A: format-feedback mean 0.6953, FAER-Fixed 0.3594; 8 draws
Paired inclusion	128/128 marginal; 8,128/8,128 joint; four contrasts
Formula / index age	verifier peak at 0.75; four settings and four seeds
Measured age	1,767/1,767 records; families A and B
Producer / selector	192/192 lineage; 0 isolation violations; 1,344 invariant mutations
Evaluator	191/192 extraction; 95/96 adjudicated agreement

Appendix

Table 44: Experiment ledger. Values retain the stage and denominator stated in the corresponding result package.

Field	Value
Source commit / environment SHA-256 prefix	197ec447 / b91d63af
Learner model and tokenizer	Qwen2.5-1.5B-Instruct, one checkpoint and tokenizer shared by all arms
Learner loss / target construction / token masking	masked autoregressive NLL / GSM8K target answer / assistant-response tokens only
32-record selection to 128-update schedule / reuse	one complete trajectory per update / seeded cyclic order, four complete passes
Replay-probability correction in the learner	none; equal token loss weight
Decoder configuration / sampler implementation	greedy; do_sample=false , top_p=1.0 , max_new_tokens=512 / sequential weighted draw without replacement

Appendix

Table 45: Implementation fields for the reconstruction record. Prefixes shown in the producer and checkpoint tables identify the corresponding record family.

stage	unit and bound artifact	reported metric
cache	split manifest and row digest	row count, ID disjointness, field coverage
projection	one row’s observed-field serialization	oracle-key hits, taint invariance
priority	row-policy pair and formula version	priority value, support floor, rank
sampling	seed and ordered without-replacement draw	selected IDs, trace digest, Jaccard
oracle join	selected ID and evaluator record	correct count, denominator, failure class
learner	arm, checkpoint, and sealed task IDs	quality, tokens, latency, paired interval

Appendix

Table 46: Stage-level reporting matrix. Each row names the unit of analysis, the artifact that binds the computation, and the metric reported in the manuscript. The schema binds each field to its row or run identity.

event	trigger and localization	resolution record
field	type/range/producer-key mismatch	row ID, key, expected domain, digest
taint	target mutation changes observed view or priority	mutation ID, field path, priority pair
trace	ordered draw or replay digest mismatch	seed, draw index, expected/actual ID
join	selected ID lacks evaluator correspondence	task ID, split, evaluator hash
label	extraction, normalization, or tolerance disagreement	adjudication pair and rule ID
update	checkpoint, budget, or sealed-task mismatch	arm, step, checkpoint, metric digest

Appendix

Table 47: Diagnostic taxonomy and resolution fields. Each count and digest maps to the corresponding audit record.

Explore similar work

Jun 23, 2026cs.LG

ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning

Reinforcement Learning with Verifiable Rewards (RLVR) for language-model reasoning can fail at both extremes of task difficulty: easy prompts often produce all-correct, low-diversity rollout groups with little gradient signal, while hard prompts can produce all-incorrect groups with no positive reward. We introduce ExTra (Exploratory Trajectory Optimization), a GRPO-compatible framework that extracts exploration signals from the model's own rollouts. ExTra combines two mechanisms: (i) a novelty reward that adds embedding-based diversity bonuses after GRPO normalization, rewarding diverse correct solutions; and (ii) entropy-guided prefix regeneration, which scores partial trajectories using entropy signals and continues exploration from promising intermediate steps. Across six mathematical reasoning benchmarks, ExTra improves Qwen3-1.7B over GRPO by about +5 points on pass@1 and +7 points on pass@16, showing that trajectory-level exploration signals can improve both single-sample accuracy and inference-time coverage.

Wenyang Hu, Junxiang Jia, Zhen Shu +3

1National University of Singapore · 2SAP

Verifiable Rewards Trajectory-Level Credit Robust Trajectory

Ask PDF

May 13, 2026cs.LG

Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Models

Diffusion language models are a promising alternative to autoregressive models, yet post-training methods for them largely adapt reward-maximizing objectives. We identify a central failure mode in this setting we call trajectory locking: sampled reward-driven updates over-concentrate probability mass onto a narrow set of denoising paths, reducing coverage of alternative correct solutions under repeated sampling. To address this, we propose TraFL (Trajectory Flow baLancing), a trajectory-balance objective that trains the policy toward a reward-tilted target distribution anchored to a frozen reference model. We make this practical for diffusion language models with a diffusion-compatible sequence-level surrogate and a learned prompt-dependent normalization. Across mathematical reasoning and code generation benchmarks, TraFL is the only evaluated post-training method that improves over the base model in every benchmark-length setting, with gains that persist as the sampling budget increases. The improvements transfer to held-out evaluations: TraFL stays above the base model on Minerva Math and is the strongest method on every LiveCodeBench difficulty split.

Saba Ahmadi, Prasanna Parthasarathi, Yufei Cui

Noah’s Ark Lab

Diffusion Language Models Post-Training Trajectory Generation

Ask PDF

May 5, 2026cs.LG

Efficiently Aligning Language Models with Online Natural Language Feedback

Reinforcement learning with verifiable rewards has been used to elicit impressive performance from language models in many domains. But, broadly beneficial deployments of AI may require us to train models with strong capabilities in "fuzzy", hard-to-supervise domains. In this paper, we develop methods to align language models in fuzzy domains where human experts are still able to provide high-quality supervision signal, but only for a small number of model outputs, using online natural language feedback. Specifically, we train models by iteratively optimizing against proxy reward signals, stopping at the point of over-optimization, collecting fresh expert supervision, and updating the proxy reward. We construct proxy reward models from language models using in-context learning (ICL) and fine-tuning. We test our methods by eliciting creative writing and alignment research capabilities in Qwen3-8B and Haiku 4.5 respectively. For Qwen3-8B, ICL methods recover up to 35% of performance with 50x fewer expert samples, while fine-tuning methods recover 80% with up to 20x fewer samples and 100% with 3x fewer samples. For Haiku 4.5, ICL methods recover up to 35% of performance with 30x fewer samples, and fine-tuning methods recover 100% with 10x fewer samples. Our results suggest that online natural language feedback can substantially improve the data efficiency of expert supervision.

Christine Ye, Joe Benton

Stanford University · Anthropic Fellows Program for Safety Research · Anthropic

Large Language Model Alignment Reinforcement Learning From Human Feedback Feedback Fuzzy

Ask PDF