Can a search over innate drives find intrinsic rewards that help a reinforcement-learning agent learn, transfer to a task it never saw, and resist an exploit that was absent during the search? Each candidate reward combines four generic events (a new view, a pickup, a drop and a toggle), each with a weight, a relief and a refill rate. We ran three pre-registered stages; the search protocol was locked in a public, timestamped commit before any search result existed. CMA-ES and an equal-budget random search used 1,560 PPO training runs, and the final evaluation and tamper test used 340. On the held-out task, KeyCorridorS3R2, the CMA-ES-selected reward (E) beat task-only PPO, episodic novelty and the random-search-selected reward (R) in all 3 registered comparisons, all of them meaningful by the registered size rule. Among these generic rewards E transferred best; the earlier paper's satiation reward, which on this task is given hand-wired key and locked-door events, did better (20 of 20 seeds solved against 4 of 20; descriptive). On UnlockPickup, where that reward uses generic events, E's mean area exceeded it by 0.276 (descriptive). On the training tasks there was no registered difference between E and R: both E-over-R tests missed. A free reset action, absent during the search, collapsed E's learning on DoorKey-8x8 (mean area 0.829 without it, 0.136 with it; registered) to below task-only PPO (descriptive). The searched drives helped on every task tested, but a reward given task-specific events did better on the held-out task, and selection gave E no protection against an exploit absent from the search.
Figures & tables
Figure 1: Three pre-registered stages. Stage 1 fixed the detector and selected tasks. In Stage 2, every candidate was trained on both training tasks with 3 seeds each; the top 3 candidates per method were re-evaluated on fresh seeds and the best re-evaluated candidate per method was frozen as E or R. The Stage 2 lock also registered every Stage 3 test. Stage 3 evaluated E, R, task-only PPO (T) and episodic novelty (N) on the two training tasks and on the held-out KeyCorridorS3R2, with the earlier paper’s satiation reward (S) as a descriptive arm, and retrained E with a free reset action. Seed sets never overlap across roles.
Task
Candidate role
Budget
Best baseline area
Outcome
Reason
DoorKey-8x8
training
1M
0.195
kept
–
KeyCorridorS3R1
training
1M
0.855
rejected
too easy (area >0.85 )
UnlockPickup
training
1M
0.006
kept
–
KeyCorridorS3R2
held out
2M
0.053
kept
–
BlockedUnlockPickup
held out
2M
0.008
rejected
too hard (area <0.05 )
DoorKey-16x16
held out
2M
0.045
rejected
too hard (area <0.05 )
Table 1: Stage 1 task selection ( registered rule, applied mechanically). Best baseline mean training-curve area over task-only PPO and episodic novelty, 5 calibration seeds per arm, at the budget the rule uses. A training candidate also needed positive task success by 2M in some baseline seed; UnlockPickup met that condition. Produced by make table1 ; data: results/table1_calibration.csv .
Figure 2: Both searches found candidates that learned DoorKey-8x8; CMA-ES reached higher search scores on UnlockPickup ( 3 -seed search scores; not confirmed on re-evaluation, Table 2 ). Search fitness of all 240 candidates ( 3 seeds per task each) by CMA-ES generation or random-search batch, with each method’s best score so far. Candidates in one generation or batch shared a seed panel; the random-search batch index has no optimization meaning. Search scores are the selection signal, not independent estimates ( pre-specified descriptive data, exploratory/post hoc display). Produced by make figure2 ; data: results/figure2_search.csv .
Method
Candidate
Search
Re-eval.
Change
DK area
DK solved
UP area
UP solved
CMA-ES
92
0.707
0.587
− 0.120
0.839
10/10
0.336
5/10
CMA-ES
113 ( E )
0.692
0.596
− 0.096
0.818
10/10
0.373
3/10
CMA-ES
94
0.676
0.497
− 0.179
0.766
9/10
0.227
2/10
Random search
57 ( R )
0.610
0.632
+0.022
0.875
10/10
0.389
4/10
Random search
116
0.584
0.626
+0.042
0.911
10/10
0.340
4/10
Random search
103
0.510
0.619
+0.108
0.870
10/10
0.368
5/10
Table 2: Every CMA-ES finalist fell on re-evaluation; every random-search finalist rose. Search fitness ( 3 seeds per training task) and re-evaluated fitness ( 10 fresh seeds per training task) of the 3 finalists per method; the re-evaluated score selects the winner, so it is also a selection-panel score. Values pre-specified descriptive ; the change column is an exploratory/post hoc display. Produced by make table2 ; data: results/table2_finalists.csv .
Figure 3: Search scores overstated the CMA-ES finalists. Finalist search fitness against re-evaluated fitness; filled markers are the frozen winners E and R. Points below the dashed line scored lower on fresh seeds. pre-specified descriptive values; the winner’s-curse reading in the text is exploratory/post hoc . Produced by make figure3 ; data: results/figure3_finalists.csv .
Method
Mean area (SD)
Solved
95% CI (solved)
Median first solve
KeyCorridorS3R2 (held out)
E (CMA-ES-selected)
0.457 (0.139)
4/20
[0.06, 0.44]
2M (censored)
R (random-search-selected)
0.298 (0.102)
0/20
[0.00, 0.17]
2M (censored)
T (task-only)
0.004 (0.002)
0/20
[0.00, 0.17]
2M (censored)
N (episodic novelty)
0.064 (0.035)
0/20
[0.00, 0.17]
2M (censored)
S (satiation) †
0.768 (0.046)
20/20
[0.83, 1.00]
0.63M
Table 3: Final evaluation, 20 seeds per arm and task ( pre-specified descriptive descriptives; the registered tests are in Table 4 ). Mean training-curve area with its standard deviation across seeds, solved seeds with Clopper–Pearson 95% intervals, and median first-solved step, censored at the nominal budget. † S is descriptive by design. On KeyCorridor tasks its grasp event counts only key pickups and its open event counts only the opening of the door locked at the start of the episode, and its door need refills; on DoorKey-8x8 and UnlockPickup it uses the generic pickup and toggle events, and on UnlockPickup E’s mean area exceeded S’s. Produced by make table3 ; data: results/table3_final.csv .
Task
Test
Mean gap
U
Raw p
Holm p
Rank test
Meaningful
Fisher p
Primary family, held-out task (Holm, m =3)
KeyCorridorS3R2
E > T
+0.452
400.0
5×10−6
1×10−5
hit
yes
0.053
KeyCorridorS3R2
E > N
+0.393
400.0
7.3×10−12
2.2×10−11
hit
yes
0.053
KeyCorridorS3R2
E > R
+0.159
329.0
0.00015
0.00015
hit
yes
0.053
Secondary family, training tasks (Holm, m =6)
DoorKey-8x8
E > T
+0.551
389.0
1.4×10−9
8.5×10−9
hit
yes
2.3×10−5
Table 4: Every planned comparison. One-sided Mann–Whitney tests of E against each comparator on mean training-curve area ( registered ); Holm within each family. “Meaningful” additionally requires an observed mean gap of at least 0.05 . A permutation p of 5×10−6 is the floor for 200,000 resamples. The last column is the protocol’s exploratory one-sided Fisher test on solve counts ( pre-specified descriptive ), which supplies no hits. Produced by make table4 ; data: results/table4_comparisons.csv .
Figure 4: E learned the held-out task better than T, N and R, and S, whose KeyCorridor events are hand-wired, learned it best; a free reset action collapsed E’s learning on DoorKey-8x8. Training success (rolling success over the last 100 completed training episodes) for every seed (thin) and the seed mean (thick), 20 seeds per line. (a) KeyCorridorS3R2, never used during the search. (b) E retrained with and without the reset action. The registered endpoints are the curve areas (Table 4 ); on UnlockPickup, where S uses generic events, E’s mean area exceeded S’s (Table 3 ). † On KeyCorridor tasks S’s events are hand-wired (key pickups only; only the opening of the initially locked door) and its door need refills. The registered Stage 3 figure redrawn from the same data, with relabelled legend entries, a new tamper-on colour, a fuller panel (b) label and a shorter height. Produced by make figure4 ; data: results/figure4_heldout_tamper.csv .
Figure 5: Per-seed areas show the same ordering as the means, with wide spread for E on the held-out task. Training-curve area of every seed; filled markers are solved seeds, hollow markers unsolved; bars are seed means, 20 seeds per arm. (a)–(c) Final evaluation; S on KeyCorridorS3R2 has hand-wired events, while on UnlockPickup, where S uses generic events, E’s areas were higher. (d) E retrained on DoorKey-8x8 with the reset action off and on. pre-specified descriptive . Produced by make figure5 ; data: results/figure5_per_seed.csv .
Endpoint
Level
Tamper off
Tamper on
Signed start payoff P0 (fixed before data)
REG
0.6396 >0 : collapse branch
Mean curve area
REG
0.829
0.136
Off minus on; one-sided U , p
REG
0.693; U =393.0, p = 5×10−6
Solved seeds
PSD
20/20
4/20
Presses per step, all training
PSD
1.4%
10.3%
Presses per step, final 10%
PSD
0.004%
7.9%
Table 5: A free reset action collapsed E’s learning, and the tamper-on agents collected nearly the whole episode ceiling. E retrained on DoorKey-8x8, 20 seeds per condition. Press rates and returns are means over seeds. The final 10% of training is the last tenth of environment steps. Level: registered or pre-specified descriptive . Produced by make table5 ; data: results/table5_tamper.csv .
Figure 6: Tamper-on agents kept pressing and kept high policy entropy; tamper-off agents stopped pressing and became decisive. (a) Share of environment steps on the reset action and (b) PPO policy entropy over training, mean over 20 seeds per condition (all seeds in every logging window) with 95% intervals. Dashed lines are arithmetic references for a uniform policy over the 7 actions ( derived reference ), not measured policies. Without the tamper flag the action is a no-op, but its presses are still counted. pre-specified descriptive . Produced by make figure6 ; data: results/figure6_tamper_dynamics.csv .
Reward
Event
Weight
Relief
Growth per step
Steps per full refill
Spent by one event
E
new_view
0.554
0.928
0.0039
258
no
pickup
0.632
0.633
0
–
yes
drop
0.966
0.564
0
–
yes
toggle
0.406
0.690
0
–
yes
R
new_view
0.368
0.803
0.0086
117
no
pickup
0.039
0.483
0
–
no
Table 6: The frozen genomes. Effective float32 parameters. “Spent by one event”: a single occurrence empties the need (relief at least h_{0}=\hrefhttps://github.com/chasemetoyer/found-by-search/blob/v1.1/results/numbers.json#L28 ) and the need never refills, so only the first such event in an episode pays. A record of the frozen rewards; “steps per full refill” is 1/g ( derived reference ). Produced by make table6 ; data: results/table6_genomes.csv .
Figure 7: E pays its first pickup, drop and toggle once per episode; R slightly penalizes drops. (a) Weights, (b) reliefs (dashed: h0 ) and (c) refill rates of the frozen E and R genomes, per event. Hatched bars mark events whose single occurrence empties the need and whose need never refills. R’s drop penalty is at most about 0.003 of intrinsic reward per drop ( derived reference ). A record of the frozen rewards. Produced by make figure7 ; data: results/figure7_genomes.csv .
Table 7: Registered rules and their outcomes. Stage 1 rules come from the locked Stage 1 protocol; Stage 2 and Stage 3 rules from the locked Stage 2 protocol. The Stage 1 separation test was registered as a two-sided, descriptive test with no prediction. Produced by make table7 ; data: results/table7_outcomes.csv .
Amendment and scope
Created (UTC)
Commit
First public push (UTC)
Scientific change
S1: infrastructure
28 Oct, 03:20:36
8f22e05
03:20:38 (feed)
false
S1: infrastructure
28 Oct, 03:24:33
b34b2c4
03:24:34 (feed)
false
S1: setup repairs
28 Oct, 03:30:52
196c525
03:30:53 (feed)
false
S1: report only
28 Oct, 03:45:10
373ce9f
04:23:26 (feed)
false
S1: report only
28 Oct, 04:36:22
69f50f3
04:36:45 (feed)
false
S2: read-only dashboard
28 Oct, 16:19:47
b48e68b
16:19:50 (feed)
false
Appendix
Table 8: Every amendment. Creation times are recorded inside each file. “First public push” is the earliest push of a commit containing the amendment, from GitHub’s public event feed (feed) or, where the feed did not reach, from the public-commit receipts verified before the Stage 3 launch (receipt). The last column prints each file’s scientific_changes field. Produced by make table8 ; data: results/table8_amendments.csv .
Stage
Cap
Guard
Measured compute
Provider billing
Stage 1
$15
$2.88
–
$1.52
Stage 2
35\to50
$29.41
$22.37
$22.07
Stage 3
10\to15
$7.04
$4.81
$5.08
Appendix
Table 9: Spending by stage, in three quantities that overlap and must not be added. The guard is a conservative reservation total, not a charge. The measured compute estimate multiplies requested resources by measured wall time. Provider billing covers complete billing intervals only and is not a final invoice. Produced by make table9 ; data: results/table9_costs.csv .
Event
Feature
Groups
All candidates: means (n)
Random search: means (n)
toggle
weight sign
positive vs negative
0.351 vs 0.075 (160/80)
0.204 vs 0.068 (61/59)
refill
zero vs nonzero
0.307 vs 0.170 (156/84)
0.160 vs 0.110 (66/54)
Spearman ρ , weight
0.53
0.21
pickup
weight sign
positive vs negative
0.320 vs 0.106 (172/68)
0.183 vs 0.071 (71/49)
refill
zero vs nonzero
0.338 vs 0.156 (136/104)
0.176 vs 0.107 (53/67)
Spearman ρ , weight
0.37
0.33
Appendix
Table 10: Genome features and combined search fitness ( exploratory/post hoc ). Mean combined fitness by weight sign and by zero versus nonzero refill, for all 240 candidates and for the 120 random-search candidates alone, with group sizes; Spearman correlation of weight with combined fitness. No p -values were computed. Full tables, including per-task outcomes and every feature, are in the Stage 3 report archive. Produced by make table10 ; data: results/table10_exploratory.csv .
Setting
Value
Parallel environments
16
Rollout length per environment
128
Learning rate (linearly annealed)
0.00025
Discount γ
0.99
GAE λ
0.95
Minibatches per update
4
Appendix
Table 11: PPO settings (the earlier paper’s) and software versions. From the archived trainer configuration and the locked environment. Produced by make table11 ; data: results/table11_hyper.csv .
Search agents enable Large Language Models (LLMs) to iteratively retrieve and use information for complex multi-hop questions. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising approach for post-training such agents, but its reliance on sparse, outcome-based supervision can make credit assignment difficult and limit learning efficiency. In this paper, we systematically investigate how intermediate supervision can improve reinforcement learning for search agents. We study a range of reward-shaping and credit-assignment strategies that provide learning signals from intermediate retrieval steps. Building on these insights, we develop a training framework that combines intermediate signals with final outcome rewards to improve learning from multi-step search trajectories. Experiments across multiple benchmarks under matched training conditions demonstrate improvements in aggregate search-agent performance and show that both the choice of intermediate signal and where its credit is assigned affect training behaviour. These findings show that reward design and credit assignment are important design dimensions for training effective search agents.
Wenyu Huang, Xinyu Hou, Pavlos Vougiouklis +2
University of Edinburgh, UK · Huawei Technologies Research & Development (UK) Limited
Intrinsic rewards are designed to guide exploration in reinforcement learning by assigning value to an agent's experience, for example through prediction error or learning progress. However, maximizing these rewards need not produce the most informative experience available. We propose a formal criterion for exploration that compares policies by the counterfactual information they acquire: how well their histories can substitute for experience under alternative policies. We construct a single, simple environment in which specified count-based, prediction-error, empowerment, and information-gain objectives have maximizing policies that are Pareto-suboptimal at acquiring counterfactual information. We explain these failures and establish conditions under which existing intrinsic rewards successfully encourage optimal exploration. We also construct an objective that assigns a higher value whenever exploration strictly improves under our criterion.
Scott W. Viteri, Laura Gomezjurado Gonzalez, Clark Barrett
Search-augmented language agents should retrieve external information only when necessary and ground their answers in retrieved evidence. Yet terminal correctness assigns equal credit to direct answers from parametric knowledge, evidence-grounded answers, and answers recalled after redundant search. Process supervision and LLM judges provide richer feedback but can require high-cost annotation or inference; low-cost, non-contrastive policy confidence signals do not directly identify whether retrieved evidence changes an answer. We propose Search-G1, a reward framework that couples retrieval necessity with evidence reliance through two intervention-calibrated hidden-state readouts for selective grounded search. A prompt readout predicts closed-book sufficiency, whose complement proxies policy-relative retrieval necessity; an answer-commit readout predicts evidence reliance as sensitivity to evidence deletion conditional on the realized reasoning prefix. A correctness-first reward gates evidence-reliance credit by retrieval necessity, rewards sufficient direct answers, and penalizes repeated search. After calibration, scoring needs neither process annotations nor LLM-judge inference. As policy-relative targets, trajectory distributions, and representations may drift, Search-G1 periodically regenerates targets and refits its readouts. Across four QA benchmarks and two scales, Search-G1 achieves competitive task accuracy, with stronger counterfactual evidence dependence and fewer searches in the main comparisons, complemented by independent human audits of answer support.
Ruoxi Cheng, Haoxuan Ma, Hongyi Zhang +7
Fudan University · Tencent · Nanjing University +2