Found by Search, Broken by a Shortcut: Transfer and Tampering of Searched Intrinsic Rewards
Organizations: Northeastern University, Boston, MA
Abstract
Can a search over innate drives find intrinsic rewards that help a reinforcement-learning agent learn, transfer to a task it never saw, and resist an exploit that was absent during the search? Each candidate reward combines four generic events (a new view, a pickup, a drop and a toggle), each with a weight, a relief and a refill rate. We ran three pre-registered stages; the search protocol was locked in a public, timestamped commit before any search result existed. CMA-ES and an equal-budget random search used 1,560 PPO training runs, and the final evaluation and tamper test used 340. On the held-out task, KeyCorridorS3R2, the CMA-ES-selected reward (E) beat task-only PPO, episodic novelty and the random-search-selected reward (R) in all 3 registered comparisons, all of them meaningful by the registered size rule. Among these generic rewards E transferred best; the earlier paper's satiation reward, which on this task is given hand-wired key and locked-door events, did better (20 of 20 seeds solved against 4 of 20; descriptive). On UnlockPickup, where that reward uses generic events, E's mean area exceeded it by 0.276 (descriptive). On the training tasks there was no registered difference between E and R: both E-over-R tests missed. A free reset action, absent during the search, collapsed E's learning on DoorKey-8x8 (mean area 0.829 without it, 0.136 with it; registered) to below task-only PPO (descriptive). The searched drives helped on every task tested, but a reward given task-specific events did better on the held-out task, and selection gave E no protection against an exploit absent from the search.
Figures & tables
| Task | Candidate role | Budget | Best baseline area | Outcome | Reason |
|---|---|---|---|---|---|
| DoorKey-8x8 | training | 1M | 0.195 | kept | – |
| KeyCorridorS3R1 | training | 1M | 0.855 | rejected | too easy (area ) |
| UnlockPickup | training | 1M | 0.006 | kept | – |
| KeyCorridorS3R2 | held out | 2M | 0.053 | kept | – |
| BlockedUnlockPickup | held out | 2M | 0.008 | rejected | too hard (area ) |
| DoorKey-16x16 | held out | 2M | 0.045 | rejected | too hard (area ) |
| Method | Candidate | Search | Re-eval. | Change | DK area | DK solved | UP area | UP solved |
|---|---|---|---|---|---|---|---|---|
| CMA-ES | 92 | 0.707 | 0.587 | 0.120 | 0.839 | 10/10 | 0.336 | 5/10 |
| CMA-ES | 113 ( E ) | 0.692 | 0.596 | 0.096 | 0.818 | 10/10 | 0.373 | 3/10 |
| CMA-ES | 94 | 0.676 | 0.497 | 0.179 | 0.766 | 9/10 | 0.227 | 2/10 |
| Random search | 57 ( R ) | 0.610 | 0.632 | +0.022 | 0.875 | 10/10 | 0.389 | 4/10 |
| Random search | 116 | 0.584 | 0.626 | +0.042 | 0.911 | 10/10 | 0.340 | 4/10 |
| Random search | 103 | 0.510 | 0.619 | +0.108 | 0.870 | 10/10 | 0.368 | 5/10 |
| Method | Mean area (SD) | Solved | 95% CI (solved) | Median first solve |
|---|---|---|---|---|
| KeyCorridorS3R2 (held out) | ||||
| E (CMA-ES-selected) | 0.457 (0.139) | 4/20 | [0.06, 0.44] | 2M (censored) |
| R (random-search-selected) | 0.298 (0.102) | 0/20 | [0.00, 0.17] | 2M (censored) |
| T (task-only) | 0.004 (0.002) | 0/20 | [0.00, 0.17] | 2M (censored) |
| N (episodic novelty) | 0.064 (0.035) | 0/20 | [0.00, 0.17] | 2M (censored) |
| S (satiation) † | 0.768 (0.046) | 20/20 | [0.83, 1.00] | 0.63M |
| Task | Test | Mean gap | Raw | Holm | Rank test | Meaningful | Fisher | |
|---|---|---|---|---|---|---|---|---|
| Primary family, held-out task (Holm, =3) | ||||||||
| KeyCorridorS3R2 | E T | +0.452 | 400.0 | hit | yes | 0.053 | ||
| KeyCorridorS3R2 | E N | +0.393 | 400.0 | hit | yes | 0.053 | ||
| KeyCorridorS3R2 | E R | +0.159 | 329.0 | 0.00015 | 0.00015 | hit | yes | 0.053 |
| Secondary family, training tasks (Holm, =6) | ||||||||
| DoorKey-8x8 | E T | +0.551 | 389.0 | hit | yes | |||
| Endpoint | Level | Tamper off | Tamper on |
|---|---|---|---|
| Signed start payoff (fixed before data) | REG | 0.6396 : collapse branch | |
| Mean curve area | REG | 0.829 | 0.136 |
| Off minus on; one-sided , | REG | 0.693; =393.0, = | |
| Solved seeds | PSD | 20/20 | 4/20 |
| Presses per step, all training | PSD | 1.4% | 10.3% |
| Presses per step, final 10% | PSD | 0.004% | 7.9% |
| Reward | Event | Weight | Relief | Growth per step | Steps per full refill | Spent by one event |
|---|---|---|---|---|---|---|
| E | new_view | 0.554 | 0.928 | 0.0039 | 258 | no |
| pickup | 0.632 | 0.633 | 0 | – | yes | |
| drop | 0.966 | 0.564 | 0 | – | yes | |
| toggle | 0.406 | 0.690 | 0 | – | yes | |
| R | new_view | 0.368 | 0.803 | 0.0086 | 117 | no |
| pickup | 0.039 | 0.483 | 0 | – | no |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Registered rule or test | Result | Outcome |
| Stage 1 (locked 28 October 2026, 03:12:23 UTC) | ||
| Validation gate before training | 53 tests, 0 failures | passed |
| Determinism check (report, no tuning) | same seed, separate containers | not bit-identical |
| Task-selection rule | kept: DoorKey-8x8, UnlockPickup, KeyCorridorS3R2 | applied mechanically |
| rejected: KeyCorridorS3R1, BlockedUnlockPickup, DoorKey-16x16, MultiRoom-N2-S4 | ||
| 1M baseline separation (two-sided, Holm =3) | descriptive test, no prediction | not separated in 3 of 3 |
| Amendment and scope | Created (UTC) | Commit | First public push (UTC) | Scientific change |
|---|---|---|---|---|
| S1: infrastructure | 28 Oct, 03:20:36 | 8f22e05 | 03:20:38 (feed) | false |
| S1: infrastructure | 28 Oct, 03:24:33 | b34b2c4 | 03:24:34 (feed) | false |
| S1: setup repairs | 28 Oct, 03:30:52 | 196c525 | 03:30:53 (feed) | false |
| S1: report only | 28 Oct, 03:45:10 | 373ce9f | 04:23:26 (feed) | false |
| S1: report only | 28 Oct, 04:36:22 | 69f50f3 | 04:36:45 (feed) | false |
| S2: read-only dashboard | 28 Oct, 16:19:47 | b48e68b | 16:19:50 (feed) | false |
| Stage | Cap | Guard | Measured compute | Provider billing |
|---|---|---|---|---|
| Stage 1 | $15 | $2.88 | – | $1.52 |
| Stage 2 | \to50 | $29.41 | $22.37 | $22.07 |
| Stage 3 | \to15 | $7.04 | $4.81 | $5.08 |
| Event | Feature | Groups | All candidates: means (n) | Random search: means (n) |
|---|---|---|---|---|
| toggle | weight sign | positive vs negative | 0.351 vs 0.075 (160/80) | 0.204 vs 0.068 (61/59) |
| refill | zero vs nonzero | 0.307 vs 0.170 (156/84) | 0.160 vs 0.110 (66/54) | |
| Spearman , weight | 0.53 | 0.21 | ||
| pickup | weight sign | positive vs negative | 0.320 vs 0.106 (172/68) | 0.183 vs 0.071 (71/49) |
| refill | zero vs nonzero | 0.338 vs 0.156 (136/104) | 0.176 vs 0.107 (53/67) | |
| Spearman , weight | 0.37 | 0.33 |
| Setting | Value |
|---|---|
| Parallel environments | 16 |
| Rollout length per environment | 128 |
| Learning rate (linearly annealed) | 0.00025 |
| Discount | 0.99 |
| GAE | 0.95 |
| Minibatches per update | 4 |
Intrinsic Reward Methods
Intrinsic reward methods give agents internally generated learning signals to guide exploration or improvement when external rewards are sparse.
See all 32 papers