What Pretraining and Midtraining Make Learnable from Rewards?

Chiwun Yang; Xiaoyu Li

cs.LGSep 29, 2026

What Pretraining and Midtraining Make Learnable from Rewards?

Authors: Chiwun Yang, Xiaoyu Li

Organizations: City University of Hong Kong · University of New South Wales

Abstract

A reward can identify a correct answer while leaving the computation needed for new inputs undetermined. We study how pretraining and midtraining supply the information and computation that make reward adaptation effective. In sequential state computation and contextual memory, we characterize mechanisms that agree on every training reward yet demand different held-out answers. Task-independent source observations resolve this ambiguity. We construct finite sampled Adam paths from specified random initializations through source prediction and reward adaptation in the same parameters, proving how prediction acquires execution or retrieval and rewards learn their task-specific use. Experiments with pretrained Qwen2.5 checkpoints test this division of labor. Across eight worlds, Sequential models trained with correct source and first-operation supervision reach 82.61% success, versus 44.15% for a private-random source control. Memory replay preserves retrieval during reward adaptation, and an independent eight-world confirmation achieves 75.32% task success versus 49.86% after matched alternative-retrieval training. GSM8K and HotpotQA separate accuracy at reward entry, subsequent gain and final performance. Together, these results connect information acquisition, executable computation and reward-guided task learning.

Figures & tables

Panel of Figure 1 — Figure 1 : Source prediction prepares a shared computation for reward adaptation. States $s_{i}$ and operations $u_{i}$ illustrate composition; keys $k_{i}$ , values $v_{i}$ and query $q$ illustrate retrieval. Rewards select task-specific use.

Figure 3 : Acquiring and retaining source computation. (a) Strict trajectories on 512 source probes per world with first operation supplied, using the action objective. (b) Correct first field on 256 Memory records. Points and gray lines show paired worlds; diamonds give means and 95% intervals. (c) Both answers correct on 256 original/value-flip pairs during reward training, for correct/wrong source with pure RL or replay. A/B branches are averaged within each of eight worlds; bands are untruncated 95% intervals. Replay adds 131,072 source targets and 128 CE updates per branch.

Figure 4 : Source preparation and reward learning. (a) Sequential action-objective routes, worlds 500–507. (b) Independent Memory confirmation, worlds 508–515: selected versus alternative retrieval after a shared correct-source start. Success uses 1,024 fixed held-out records; A/B are averaged within each world, with 95% intervals across eight worlds. Source costs differ between panels ( Sections A.1 and A.2 ).

Table 4

Appendix figures & tables52 assets

Supplementary material from the paper’s appendix.

Appendix

Module	Effect	95% CI	Exact $p$	Holm $p$
Sequential	67.57	[37.04, 98.10]	0.007812	0.015625
Memory	0.67	[-1.22, 2.57]	0.640625	0.640625

Appendix

Table 3 : Predeclared source-control effects on task selectivity (percentage points), eight paired worlds per module. Two-sided exact sign-flip tests; Holm adjustment over both primary tests.

Module	Comparison	Metric	Effect	95% CI	Exact $p$
Seq.	Correct: action–state	Select.	43.55	[1.47, 85.64]	0.08594
Seq.	Correct: action–state	Success	54.62	[23.75, 85.50]	0.01562
Seq.	Control: action–state	Select.	0.96	[-0.09, 2.02]	0.04688
Seq.	Control: action–state	Success	31.74	[13.15, 50.32]	0.01562
Seq.	Source interaction	Select.	42.59	[0.82, 84.36]	0.08594
Seq.	Source interaction	Success	22.88	[-1.44, 47.20]	0.08594

Appendix

Table 4 : All predeclared secondary contrasts at 8,192 queries (percentage points). Eight paired world differences; 95% Student-

t

intervals and unadjusted exact sign-flip tests.

Module	Route	Entry	Gain	Final
Sequential	Correct/state	$0.20\,{\pm 0.46}$	$27.80\,{\pm 30.37}$	$27.99\,{\pm 30.48}$
Sequential	Control/state	$0.00\,{\pm 0.00}$	$12.41\,{\pm 15.57}$	$12.41\,{\pm 15.57}$
Sequential	Correct/action	$4.83\,{\pm 6.72}$	$77.78\,{\pm 17.72}$	$82.61\,{\pm 16.80}$
Sequential	Control/action	$2.32\,{\pm 3.39}$	$41.83\,{\pm 11.12}$	$44.15\,{\pm 9.71}$
Memory	Correct/pure	$49.70\,{\pm 0.71}$	$-1.54\,{\pm 3.80}$	$48.16\,{\pm 3.61}$
Memory	Control/pure	$41.41\,{\pm 11.93}$	$2.35\,{\pm 9.53}$	$43.76\,{\pm 14.75}$

Appendix

Table 5 : Original eight-world task success: reward entry, within-world reward gain, and final score at 8,192 queries. A/B branches are averaged within each world; means and 95% Student-

t

half-widths over eight worlds. Entry and final are percentages; gain is percentage points.

Figure 5 : Correct and matched-control task selectivity at 8,192 reward queries: (a) Sequential action objective; (b) Memory source replay. Small dots are eight worlds, gray lines pair worlds, and diamonds give route means with untruncated 95% intervals. Primary inference uses paired differences, not overlap of route intervals.

Figure 6 : Sequential learning curves for (a) task A and (b) task B, with correct/private-random source crossed with state/action-plus-state targets. Each point averages eight worlds on 1,024 fixed evaluation items per world; bands are untruncated 95% Student-

t

intervals.

Figure 7 : Memory learning curves for (a) task A and (b) task B. Correct/wrong retrieval is crossed with pure reward training/source replay; the same ancestors and evaluation items are paired. Means and untruncated 95% intervals over eight worlds.

Figure 8 : (a) Legal first-operation output and (b) complete syntax on the fixed held-out reward inputs. Four source routes, A/B averaged within world, with 95% intervals across eight worlds. Correct syntax and task correctness are separate metrics.

Figure 9 : (a) Strict complete source trajectory, (b) required state-prefix content and (c) termination, on 512 fixed source probes per world with the first operation supplied. A/B are averaged within world; bands are 95% intervals over eight worlds.

Figure 10 : (a) Correct first field on 256 source records and (b) joint correctness on 256 original/relevant-flip pairs. A/B are averaged within each of eight worlds; bands are untruncated 95% intervals. Pure/replay branches share ancestors; replay contributes 131,072 additional source targets.

Figure 11 : Sequential training diagnostics: (a) positive training rewards, (b) nonzero RL updates and (c) nonzero source CE updates (zero for these routes). Cumulative counts average A/B within world, then eight worlds, with 95% intervals. Logged calls are distinguished from measured nonzero updates.

Figure 12 : Memory training diagnostics: (a) positive training rewards, (b) nonzero RL updates and (c) nonzero source CE updates. A/B are averaged within world, with 95% intervals over eight worlds. CE calls do not increment the verifier-query count.

Figure 13 : Complete original Sequential comparison. (a) State-only and action-plus-state source objectives, correct and private-random source, on 1,024 fixed held-out records. (b) Action-objective source-selectivity effects for all eight worlds, with their mean and 95% interval. A/B are averaged within world. World-level results correspond to Tables 3 and 4 .

Figure 14 : Independent Memory confirmation. (a) Separate task-A and task-B success curves for selected and other retrieval; means and 95% Student-

t

bands over eight worlds, using 1,024 held-out records per branch. (b) Each point is a world’s selected-minus-other task-selectivity effect after averaging A/B; black diamond and error bar give the mean and 95% interval. All fixed evaluation points and all worlds are retained.

World	Selected A	Selected B	Other A	Other B	Accuracy $\Delta$	Selectivity $\Delta$
508	73.34	75.10	52.44	50.68	22.66	45.31
509	83.69	91.31	49.32	49.32	38.18	76.37
510	80.66	81.84	49.80	49.41	31.64	63.28
511	70.90	73.05	47.56	48.63	23.88	47.75
512	73.54	71.09	50.10	50.59	21.97	43.95
513	73.83	72.27	48.83	50.00	23.63	47.27

Appendix

Table 6: Independent Memory confirmation: all 32 branch endpoints after 8,192 reward queries. Success is a percentage of 1,024 fixed held-out records per branch. Differences first average A/B within world; all eight worlds are included.

Figure 15 : Actual reward learning and projection fidelity in world 508. Left: original-checkpoint success on 1,024 fixed records; bands are conditional 95% item-bootstrap intervals. Right: original versus reconstructed checkpoint success on the same 256 slice-evaluation inputs. Squares mark source checkpoints and circles mark RL checkpoints; overlapping points are retained. The diagonal denotes equal scores. The source routes receive

2^{22}

additional tokens before RL; direct RL skips this stage. All three routes share format preparation and use 8,192 reward queries.

Figure 16 : Historical source acquisition, worlds 300–307. (a) Sequential complete trajectories on 512 probes with first operation supplied; (b) Memory first-field correctness and (c) original/value-flip pair correctness, each on 256 records or pairs. Correct source and matched structural controls are paired within world; points are worlds, diamonds means with untruncated 95% Student-

t

intervals. This protocol is separate from worlds 500–507 in Section A.1 .

Module	Effect	95% interval	$p$	$p_{H}$
Sequential	$5.96$	$[-8.92,20.85]$	$1.000$	$1.000$
Memory	$-0.87$	$[-2.10,0.35]$	$0.125$	$0.250$

Appendix

Table 7: Correct-source minus matched-control task selectivity after 8,192 reward queries, eight worlds per module. Effects and 95% paired Student-

t

intervals are in percentage points;

p

is the exact two-sided sign-flip value and

p_{H}

applies Holm correction to the two comparisons.

Figure 17 : Held-out success during reward adaptation, eight worlds per route. Colors distinguish correct source and the matched control (random private states for Sequential; wrong retrieval for Memory); solid and dashed lines denote tasks A and B. Markers are world means on 1,024 prompts per world at each query budget, with 95% Student-

t

bands. Training samples at temperature one; evaluation is greedy. Intervals are shown without clipping to the success-rate range.

Figure 18 : Correct-source minus matched-control task selectivity at 8,192 queries. Each dot is one complete world-level A/B contrast; worlds 1–8 correspond to seeds 300–307. The blue line is their mean, the shaded band its 95% Student-

t

interval, and the gray line marks zero. Units are percentage points. Both comparisons include all eight worlds; vertical scales differ between panels.

Figure 19 : Source-computation probes through reward adaptation, using the same metrics and matched controls as Figure 16 . Lines join the five measured query budgets; shaded bands are 95% Student-

t

intervals over eight worlds after averaging A/B within each world. Shared ancestors are not counted as additional independent samples. Sequential probes supply the first operation; Memory pairs require both the original and relevant-value-flipped bit to be correct.

Figure 20 : Historical Sequential execution and private-task adaptation, using

2^{20}

source prediction tokens. (a) Greedy success on 1,024 held-out prompts per world: eight worlds for correct source, wrong source and direct RL, and three for format control. (b) Final private-bit accuracy among outputs with valid public execution and the correct public endpoint, computed within each world before averaging; direct and format routes have no eligible outputs. (c) Shared-source A/B branches in three worlds: paired task selectivity

\Delta_{\rm task}

in percentage points (pp). Small translucent dots show individual worlds; connected markers in (a) and large markers in (b,c) show world means. Shading and error bars are 95% Student-

t

intervals across worlds; horizontal jitter separates overlapping points. Dotted lines mark 50% private accuracy in (b) and zero task selectivity in (c).

Comparison	Independent seeds and scope
Direct RL, correct source, paired wrong source	8 sequential and 5 contextual, 0.5B
Random initialization and 1.5B transfer	3 sequential seeds per three-arm panel
Source dose	5 sequential seeds, 0.5B, shared source
Alternative learning routes	3 sequential LM seeds
Alternate downstream task choice	3 sequential seeds, shared source states
Format-only source then RL	3 sequential seeds

Appendix

Table 8: Experimental comparisons. Source-only measurements reuse source checkpoints; the fixed external base model contributes one checkpoint.

Task / model / initialization	Source rate	Reward rate
Contextual memory / 0.5B / pretrained	$3\times 10^{-5}$	$10^{-5}$
Sequential / 0.5B / pretrained	$3\times 10^{-5}$	$10^{-6}$
Sequential / 0.5B / random	$10^{-4}$	$10^{-6}$
Sequential / 1.5B / pretrained	$3\times 10^{-5}$	$10^{-6}$
GSM8K / 1.5B / pretrained	$10^{-5}$	$10^{-5}$
HotpotQA / 1.5B / pretrained	$3\times 10^{-5}$	$10^{-5}$

Appendix

Table 9: Adam learning rates used in the historical controlled and public-task comparisons, selected on pilot worlds by source validation cross-entropy and training-support validation feedback. Source and reward stages reset optimizer moments.

Measurement	Run A	Run B
Two-candidate completion accuracy	46/96	49/96
Original records correct	14/32	17/32
Prediction reverses on relevant-value flip	1/32	0/32
Original and relevant-flip pair both correct	0/32	0/32
Query-field CE, before	0.2045	0.2114
Query-field CE, after	0.0430	0.0397

Appendix

Table 10: Source-rule diagnostics from two matched memory source runs. Intervention triples share an original record. Fractions report correct cases over the displayed denominator; CE denotes cross-entropy.

Figure 21 : Public-task starting scores and reward gains, five paired training seeds. (a) Open and filled markers show mean greedy success before and after RL; segments connect the same route. (b) Within-seed reward gain minus direct RL’s gain, in percentage points: small translucent dots are seeds, large markers are means, and the dotted line is zero. Horizontal bars are 95% Student-

t

intervals. Each seed and route uses the same full endpoint sets: 1,319 GSM8K and 1,024 HotpotQA prompts. Direct starting scores evaluate one fixed pretrained checkpoint.

Task	Method	Decoding	Metric	Mean and 95% CI
GSM8K	Direct RL	greedy	numeric correct	$62.3\,{\textstyle\pm 4.1}$
GSM8K	Direct RL	greedy	Format error	$0.5\,{\textstyle\pm 0.2}$
GSM8K	Direct RL	greedy	Length (tokens)	$108.1\,{\textstyle\pm 8.8}$
GSM8K	Direct RL	greedy	truncated	$0.3\,{\textstyle\pm 0.1}$
GSM8K	Correct source	greedy	numeric correct	$62.0\,{\textstyle\pm 2.9}$
GSM8K	Correct source	greedy	Format error	$2.2\,{\textstyle\pm 4.2}$

Appendix

Table 11: Public-task secondary greedy measurements. Percentages apply to correctness, F1 and error rates; generated length is tokens. RL uses five seeds and answer-only SFT three. F1 is answer token-overlap F1; format and truncation rates use all generated answers. Entries are seed means with 95% Student-

t

half-widths.

Method	Decoding	Type	Metric	Mean and 95% CI
Direct RL	greedy	bridge	Answer F1	$60.5\,{\textstyle\pm 1.1}$
Direct RL	greedy	bridge	success	$46.2\,{\textstyle\pm 0.4}$
Direct RL	greedy	comparison	Answer F1	$60.2\,{\textstyle\pm 3.5}$
Direct RL	greedy	comparison	success	$52.9\,{\textstyle\pm 3.7}$
Correct source	greedy	bridge	Answer F1	$58.3\,{\textstyle\pm 1.1}$
Correct source	greedy	bridge	success	$44.2\,{\textstyle\pm 0.9}$

Appendix

Table 12: HotpotQA greedy results within each indicated question-type subset. Success is normalized exact match; rates are percentages with seed-level 95% Student-

t

half-widths.

Training	Method	Evaluation	Checkpoint	Seeds	Success (%)
GSM8K	Base/Bo16	GSM8K	base	1	$44.5\,{\textstyle[38.3,50.8]}^{\dagger}$
GSM8K	Correct source/Bo16	GSM8K	source	5	$22.1\,{\textstyle\pm 4.9}$
GSM8K	Direct RL	BBH logic	feedback	5	$40.1\,{\textstyle\pm 1.6}$
GSM8K	Direct RL	GSM-Symbolic	feedback	5	$54.5\,{\textstyle\pm 3.2}$
GSM8K	Direct RL	BBH tracking	feedback	5	$11.4\,{\textstyle\pm 17.6}$
GSM8K	Correct source	BBH logic	feedback	5	$30.9\,{\textstyle\pm 21.0}$

Appendix

Table 13: External-transfer and verification-selection results. Bo16 denotes best-of-16 selection with test-time verification.

\pm

gives training-seed 95% Student-

t

half-widths; brackets give item/template-bootstrap intervals for a fixed model. BBH logic/tracking are three-object logical deduction/object tracking.

Training route	Seeds	Before (%)	After (%)	Gain (pp)
Sequential
Correct source	8	$3.30\,{\textstyle\pm 3.27}$	$41.67\,{\textstyle\pm 5.56}$	$38.38\,{\textstyle\pm 5.21}$
Direct RL	8	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$
Wrong source	8	$3.94\,{\textstyle\pm 3.48}$	$38.15\,{\textstyle\pm 10.68}$	$34.20\,{\textstyle\pm 9.87}$
Format source	3	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$
Memory

Appendix

Table 14: Greedy evaluation on identical before/after prompts. Start and end are success percentages; gain is percentage points. Entries are means with 95% Student-

t

half-widths across paired seeds. Start is the feedback-entry evaluation on the same prompts; direct public-task starts evaluate one fixed base checkpoint.

Source prediction tokens	Worlds	Final success (%)
$2^{18}$	5	$0.00\,{\scriptstyle\pm 0.00}$
$2^{19}$	5	$0.00\,{\scriptstyle\pm 0.00}$
$2^{20}$	5	$41.99\,{\scriptstyle\pm 8.92}$

Appendix

Table 15: Sequential source dose on five matched worlds. Greedy success after 8,192 reward queries; values are percentages with 95% Student-

t

half-widths. Source checkpoints share their training trajectories.

Setting	Route	Worlds	Success (%)
0.5B, pretrained, $2^{20}$ source tokens	Balanced source	3	$39.71\,{\textstyle\pm 32.80}$
	Interleaved source updates	3	$38.57\,{\textstyle\pm 24.05}$
	Classical table learner	3	$100.00\,{\textstyle\pm 0.00}$
0.5B, random initialization, $2^{20}$ source tokens	Correct source	3	$0.00\,{\textstyle\pm 0.00}$
	Wrong source	3	$0.00\,{\textstyle\pm 0.00}$
	Direct RL	3	$0.00\,{\textstyle\pm 0.00}$

Appendix

Table 16: Greedy sequential endpoints for initialization, scale and alternative-route controls after 8,192 reward queries. Entries are seed means with 95% Student-

t

half-widths; the classical learner uses the same source records and permitted training feedback.

Route	Seeds	Public path (%)	Correct public (%)	Private $\mid$ public (%)
Correct source	8	$96.22\,{\textstyle\pm 8.56}$	$83.25\,{\textstyle\pm 11.38}$	$50.12\,{\textstyle\pm 1.81}$
Wrong source	8	$98.13\,{\textstyle\pm 2.87}$	$76.77\,{\textstyle\pm 21.79}$	$49.87\,{\textstyle\pm 1.09}$
Direct RL	8	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$	–
Format source	3	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$	–

Appendix

Table 17: Sequential greedy endpoints. Public path requires valid syntax and a consistent public trajectory. Correct public additionally requires the target terminal public coordinate. Private accuracy conditions on this latter event; – denotes a zero denominator in every seed. The first two rates use all prompts; the last is the within-seed conditional ratio. Entries are seed means and 95% Student-

t

half-widths.

Route	Seed	$N$	$P$	$C$	$K$
Correct source	0	1024	1012	695	345
Correct source	1	1024	1024	1021	515
Correct source	2	1024	1024	824	381
Correct source	3	1024	1024	773	412
Correct source	4	1024	1024	1017	497
Correct source	5	1024	1024	771	405

Appendix

Table 18: Exact denominators for the sequential greedy decomposition.

N

is all prompts,

P

valid public paths,

C

correct public endpoints, and

K

correct private bits among

C

. The conditional accuracy is

K/C

separately in each seed;

C=0

is missing.

Route	Reward task	Test A (%)	Test B (%)	A $\mid$ A or B (%)
Correct source	A	$42.77\,{\textstyle\pm 12.42}$	$40.59\,{\textstyle\pm 21.93}$	$51.57\,{\textstyle\pm 5.88}$
Correct source	B	$41.83\,{\textstyle\pm 14.49}$	$39.45\,{\textstyle\pm 22.79}$	$51.73\,{\textstyle\pm 6.34}$
Direct RL	A	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$	–
Direct RL	B	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$	–

Appendix

Table 19: Greedy cross-scoring of the same saved outputs, three paired worlds. A/B success sets are disjoint. The conditional A score uses the union of the two sets as denominator; it is undefined for direct runs with no eligible outputs. Entries are seed means with 95% Student-

t

half-widths.

Route	Seed	$N$	$C$	$N_{A}$	$N_{B}$
Correct source	3	1024	773	412	361
Correct source	4	1024	1017	497	520
Correct source	5	1024	771	405	366
Source / task B	3	1024	824	439	385
Source / task B	4	1024	988	482	506
Source / task B	5	1024	685	364	321

Appendix

Table 20: Exact cross-task greedy counts.

C=N_{A}+N_{B}

is the common eligible denominator: valid public execution with the correct public endpoint. Each world uses 1,024 prompts shared across its A/B branches and scorers; task B changes only the required private bit.

Route	Seeds	Valid (%)	Correct (%)	Correct $\mid$ valid (%)	Constant runs
Correct source	5	$100.00\,{\textstyle\pm 0.00}$	$49.18\,{\textstyle\pm 1.43}$	$49.18\,{\textstyle\pm 1.43}$	5
Wrong source	5	$100.00\,{\textstyle\pm 0.00}$	$49.18\,{\textstyle\pm 1.43}$	$49.18\,{\textstyle\pm 1.43}$	5
Direct RL	5	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$	–	0

Appendix

Table 21: Contextual greedy endpoints. Correctness conditional on a valid bit is undefined if no valid answer is produced. Constant counts runs producing a single valid action on all 1,024 prompts. A balanced input-independent bit rule has 50% population accuracy. Entries are seed means with 95% Student-

t

half-widths.

Route	Malformed	Path error	Public error	Private error	Success
Correct source	$3.72\,{\textstyle\pm 8.58}$	$0.06\,{\textstyle\pm 0.14}$	$12.96\,{\textstyle\pm 11.00}$	$41.58\,{\textstyle\pm 6.16}$	$41.67\,{\textstyle\pm 5.56}$
Wrong source	$1.81\,{\textstyle\pm 2.83}$	$0.06\,{\textstyle\pm 0.10}$	$21.36\,{\textstyle\pm 22.37}$	$38.62\,{\textstyle\pm 11.18}$	$38.15\,{\textstyle\pm 10.68}$
Direct RL	$100.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$
Format source	$2.25\,{\textstyle\pm 5.25}$	$97.75\,{\textstyle\pm 5.25}$	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$

Appendix

Table 22: Sequential greedy endpoint error partition (percent of all prompts). Malformed, invalid public path, wrong public endpoint, wrong private bit and success are mutually exclusive and exhaustive. Private-bit failures here use all prompts as denominator; conditional accuracy uses only correct-public outputs. Entries are seed means with 95% Student-

t

half-widths.

Task	Evaluation	Comparator	Seeds	Difference (pp)
sequential	greedy	Direct RL	8	$38.38\,{\textstyle\pm 5.21}$
sequential	greedy	Wrong source	8	$4.17\,{\textstyle\pm 8.95}$
sequential	greedy	Format source	3	$41.96\,{\textstyle\pm 10.16}$
contextual	greedy	Direct RL	5	$-1.84\,{\textstyle\pm 3.15}$
contextual	greedy	Wrong source	5	$-1.11\,{\textstyle\pm 2.06}$
GSM8K	greedy	Direct RL	5	$5.97\,{\textstyle\pm 3.66}$

Appendix

Table 23: Paired differences of reward-stage gains: correct source minus the indicated route. Only shared seeds enter each contrast; entries are seed-paired mean differences in percentage points with 95% Student-

t

half-widths.

Training route	Seeds	Before (%)	After (%)	Gain (pp)
Sequential
Correct source	8	$5.18\,{\textstyle\pm 1.71}$	$5.48\,{\textstyle\pm 1.68}$	$0.31\,{\textstyle\pm 0.55}$
Direct RL	8	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$
Wrong source	8	$5.16\,{\textstyle\pm 1.59}$	$5.73\,{\textstyle\pm 1.72}$	$0.56\,{\textstyle\pm 0.77}$
Format source	3	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$	$0.00\,{\textstyle\pm 0.00}$
Memory

Appendix

Table 24: Sampled-decoding robustness check on the same checkpoints and prompts as the greedy results (one sampled answer per prompt at temperature one). Start and end are success percentages; gain is percentage points; entries are seed means with 95% Student-

t

half-widths. The two sequential endpoint contrasts use exact two-sided sign-flip tests; adjusted

p

-values use a fixed three-hypothesis Holm family, of which the two measured endpoint contrasts are shown and the third hypothesis is entered conservatively at

p=1

Figure 22 : Sampled evaluation of (a) Sequential, eight worlds, and (b) Memory, five worlds. Each checkpoint uses one answer per prompt at temperature one on 1,024 prompts per world. Curves and large markers show world means, small translucent dots individual worlds, and shaded bands 95% Student-

t

intervals. The dotted 50% line in (b) marks binary chance. Figures 20 , 14 and 17 report greedy evaluation of these historical checkpoints.

Figure 23 : Greedy learning curves for (a) GSM8K and (b) HotpotQA on the same 512 prompts at every checkpoint, five training seeds. Curves and large markers are seed means, small translucent dots are individual seeds, and bands are 95% Student-

t

intervals. The full endpoint comparison in Figures 21 and 2 uses the larger matched evaluation sets.

Figure 24 : Two effects of source observations on distinct horizontal scales. (a) Remaining contextual ambiguity versus the number

N

of source records in eight worlds; the dashed curve is the analytic expectation

\tfrac{1}{4}(\tfrac{3}{4})^{N}

. (b) Greedy Sequential success after 8,192 reward queries versus source prediction tokens, five matched worlds. Solid curves and large markers are world means; small translucent dots are individual worlds. Shading in (a) and error bars in (b) are 95% Student-

t

intervals.

Route	Seeds	Source tokens	Answer labels	Success (%)
Source only, all worlds	8	1,048,576	0	$3.30\,{\textstyle\pm 3.27}$
Source only, paired worlds	3	1,048,576	0	$3.55\,{\textstyle\pm 3.76}$
Continue source prediction	3	2,097,152	0	$1.40\,{\textstyle\pm 3.40}$
Answer-only adaptation	3	1,048,576	8,192	$0.00\,{\textstyle\pm 0.00}$

Appendix

Table 25: Greedy sequential source-only and alternative-adaptation endpoints.

Candidates	Seeds	Prompts	Test queries/seed	Success (%)
1	3	1,024	1,024	$4.95\,{\textstyle\pm 6.29}$
2	3	1,024	2,048	$8.72\,{\textstyle\pm 9.86}$
4	3	1,024	4,096	$15.92\,{\textstyle\pm 16.01}$
8	3	1,024	8,192	$26.73\,{\textstyle\pm 24.57}$

Appendix

Table 26: Verification-selected sampling from the fixed sequential source checkpoint. Uncertainty is the 95% Student-

t

half-width across training seeds.

Checkpoint	Independent units	Prompts	Success (%)
Logical deduction (250 prompts)
Base checkpoint	1 fixed model	250	$30.40\,[24.80,36.00]$
After source prediction	3 training seeds	250	$34.40\,{\textstyle\pm 2.63}$
After source + reward	3 training seeds	250	$34.27\,{\textstyle\pm 3.19}$
Object tracking (250 prompts)
Base checkpoint	1 fixed model	250	$21.20\,[16.40,26.40]$

Appendix

Table 27: Greedy external transfer after controlled sequential training of Qwen2.5-0.5B.

Task	Seeds	Prompts/seed	Answer labels/seed	Success (%)
GSM8K	3	1,319	8,192	$12.36\,{\textstyle\pm 0.38}$
HotpotQA	3	1,024	8,192	$53.74\,{\textstyle\pm 1.38}$

Appendix

Table 28: Greedy answer-only supervised adaptation on public tasks.

Source tokens	Before RL (%)	After RL (%)	Gain (pp)
GSM8K; 8,192 reward queries
None (direct)	$8.57\,\text{\footnotesize$ \pm 0.00 $}$	$62.32\,\text{\footnotesize$ \pm 4.14 $}$	$53.75\,\text{\footnotesize$ \pm 4.14 $}$
$2^{20}$	$3.11\,\text{\footnotesize$ \pm 0.57 $}$	$61.76\,\text{\footnotesize$ \pm 3.15 $}$	$58.65\,\text{\footnotesize$ \pm 3.67 $}$
$2^{22}$	$2.32\,\text{\footnotesize$ \pm 0.39 $}$	$62.05\,\text{\footnotesize$ \pm 2.93 $}$	$59.73\,\text{\footnotesize$ \pm 3.16 $}$
HotpotQA; 8,192 reward queries
None (direct)	$23.93\,\text{\footnotesize$ \pm 0.00 $}$	$47.60\,\text{\footnotesize$ \pm 1.02 $}$	$23.67\,\text{\footnotesize$ \pm 1.02 $}$

Appendix

Table 29: Public source-dose endpoints. Five paired training seeds; the same 1,319 GSM8K or 1,024 HotpotQA prompts at both endpoints and across routes. Rates are percentages; gains are percentage points. Uncertainty is a seed-level 95% Student-

t

half-width. Direct starting scores evaluate one fixed pretrained checkpoint.

Contrast	Endpoint difference (pp)	Gain difference (pp)
GSM8K
$2^{20}$ minus direct	$-0.56\,\text{\footnotesize$ \pm 4.87 $}$	$4.90\,\text{\footnotesize$ \pm 5.28 $}$
$2^{22}$ minus direct	$-0.27\,\text{\footnotesize$ \pm 3.63 $}$	$5.97\,\text{\footnotesize$ \pm 3.66 $}$
$2^{20}$ minus $2^{22}$	$-0.29\,\text{\footnotesize$ \pm 5.64 $}$	$-1.08\,\text{\footnotesize$ \pm 6.38 $}$
HotpotQA
$2^{20}$ minus direct	$-1.37\,\text{\footnotesize$ \pm 0.37 $}$	$-12.99\,\text{\footnotesize$ \pm 1.67 $}$

Appendix

Table 30: Seed-paired public source-dose contrasts, in percentage points (pp). Each effect is calculated within seed before forming the five-seed mean and 95% Student-

t

half-width.

Figure 25 : Acquired computation before and after the answer-format stage, two worlds at

2^{24}

source tokens. (a) Sequential, original representation with state targets: correct first-

H

state prefixes and correct END placement, out of 512 probes with the first operation supplied. (b) Memory, discrete representation with the all-token objective: correct first-bit answers on original records and on both members of record/value-flip pairs, each out of 256. Circles and squares are the two worlds, slightly offset where they coincide. Colors identify metrics; lines join the same world and metric across stages.

Figure 26 : Two-world acquisition probes before and after the format stage at

2^{24}

source tokens. Left: exact Sequential state prefixes, excluding termination, on 512 probes. Right: both members correct in each of 256 Memory original/relevant-flip pairs. Open blue markers are source endpoints; filled purple markers follow the format stage. Circles and squares denote the two worlds, and links pair the same world and representation. Legend abbreviations: “Orig.” is the original representation and “Course” the Sequential curriculum representation; “state” uses state-token targets and “weighted” upweights private state targets; “all” and “bit” are the Memory all-token and future-bit objectives. The Sequential diagnostic uses source-validation support with the first operation given. Figure 25 shows the original-representation Sequential state-target and discrete Memory all-token conditions, adding termination and first-bit accuracy to the two diagnostics shown here.

Representation	Objective	World	Content before	Content after	Pair before/after
Sequential; content /512
Original	State	1	512	11	–
Original	State	2	512	12	–
Original	Weighted	1	512	6	–
Original	Weighted	2	512	14	–
Curriculum	State	1	512	6	–

Appendix

Table 31: Acquisition-probe counts at

2^{24}

source tokens. Sequential columns count correct first-

H

state prefixes out of 512; Memory columns count correct original bits and relevant-flip pairs out of 256. Each cell shows before/after format. Before the format stage no Sequential model emits END; after it, every model places END correctly on all 512 prompts.

Route and remaining reward problem	Observed source tokens	Training verifier queries
Information resources: identify a mechanism and its task binding
Few source observations; reward queries recover the mechanism class	$O(\log q)$	$\Theta(q^{2})$
Logarithmic $H$ ; $a=1-c/H$ with fixed $c$ ; Theorem C.2 .
Learn a transition table first; rewards identify only the initial operation	$O(q^{2}\log q+H)$	At most $q-1$
Neural training: acquire execution, then adapt the same parameters
Source Adam and two sampled reward updates	At most $NB_{s}T_{s}$	$2B_{R}$

Appendix

Table 32: What source learning leaves for reward queries. The first two routes compare information resources at fixed confidence and the stated noise regimes; the last gives sufficient observation and query budgets for a finite Adam learning path. Here

q

is the number of states and controls and

H

is the execution horizon.

Explore similar work

Sep 30, 2026cs.LG

Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining

Continued pretraining enables language models to adapt to new domains and knowledge, but often at the cost of forgetting previously acquired capabilities. Replay can mitigate this trade-off, but fixed replay mixtures allocate training independently of the model's actual retention needs. We introduce Replay on Demand (RoD), which instead derives the replay allocation from the model's learning dynamics. RoD jointly prioritizes adaptation samples by their remaining learning potential and replay samples by their observed forgetting. Their competition for a shared training budget yields an online curriculum that determines what to train on at each step. Across models, scales, and adaptation domains, RoD reaches or improves upon the adaptation-forgetting frontier of tuned fixed-replay baselines and model merging without prescribing a replay allocation in advance. Replay concentrates on sources that are more vulnerable to forgetting and dynamically increases and redistributes as forgetting emerges during training. Together, our results show that replay can be allocated online from the model's evolving state, targeting what is needed, when it is needed.

Lukas Thede, Shengzhuang Chen, Stefan Winzeck +3

University of Tübingen, Tübingen AI Center · Helmholtz Munich · Munich Center for Machine Learning (MCML) +3

Replay Pretraining Rapid Adaptation Forgetting Curriculum

Ask PDF

Jul 27, 2026cs.LG

What do Reward Models Memorize?

This paper studies what discriminatively trained reward models (RMs) memorize by measuring counterfactual memorization on two human preference datasets. We show that RMs 1) misallocate memorization to easy, high margin preference pairs, 2) memorize dataset-specific shortcuts (e.g., model identity, user sampling strategy), and 3) overgeneralize simple heuristic correlates of human preference (e.g., length, compliance) when confronted with unseen preference pairs. Overall, our findings indicate that discriminative training of RMs from human preference data results in biased RMs not yet capable of judging response quality in context-dependent scenarios.

Ivo Verhoeven, Pushkar Mishra, Ekaterina Shutova

ILLC, University of Amsterdam, The Netherlands · Google DeepMind, London, United Kingdom

Progress Reward Modeling Preference Datasets Memorization Discriminative Counterfactual Learning Heuristics

Ask PDF

Sep 28, 2026cs.AI

Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling

Reinforcement learning from human feedback (RLHF) aligns large language models (LLMs) with human preferences, yet most pipelines learn a single reward model that overlooks individual differences in preferences. Personalized reward models (PRMs) address this by conditioning rewards on user-specific feedback, most commonly through in-context learning (ICL), where a user's historical comparisons are supplied as contextual preference pairs. However, we identify a key limitation of ICL-based PRMs: they fail to capture the preference relations conveyed by contextual pairs. To address this, we propose Preference-Aligned Test-Time Training (P-TTT), which explicitly encodes these relations into user-specific fast weights for personalized reward prediction. P-TTT introduces sequence-level update and apply operations to match the response-level granularity of preference feedback, together with a preference-aligned objective that directly uses pairwise preference relations to guide fast-weight adaptation. Notably, P-TTT is simple to implement and computationally efficient, updating fast weights within a single forward pass without inference-time backpropagation. Extensive experiments show that P-TTT more effectively captures historical preference relations and outperforms state-of-the-art methods by a large margin.

Bohao Wang, Xiaoyan Zhao, Yang Zhang +4

Zhejiang University · National University of Singapore

Large Language Model Personalization Preference Alignment Learning

Ask PDF