Satisfied, Not Solving: Wireheading of Bounded Homeostatic Rewards in Reinforcement Learning
Organizations: Northeastern University, Boston, MA
Abstract
Bounded intrinsic rewards are often assumed to be safe from exploitation: if a signal can yield at most a fixed amount, gaming it should not pay. We test this for a homeostatic satiation reward, in which needs build up and relieving them is rewarding, and give the agent a cheap action that resets its needs. In MiniGrid DoorKey-8x8, PPO agents with this action solved 2 of 80 seeds, against 71 of 80 on the same seeds without it; in KeyCorridorS3R1, 0 of 20 against 20 of 20. Their policies stayed near uniform. The reward does not ask for this: a telescoping argument caps any policy's satiation return at the needs' starting drive, and a post-hoc computation shows that, at every growth rate we trained, a solving trajectory beats that cap on all sampled layouts. In DoorKey with advantage normalization on, a satisfaction signal the agent could fill itself was worse than none (11 of 20 solved with task reward alone, 1 of 20 with the tamperable signal), and a per-press cost of 0.1 restored the task in every seed. Farming of positive reward cycles appeared in DoorKey but not persistently in KeyCorridorS3R1. All predictions were pre-registered, and hash-locked from round 3 on; 9 of our 23 hypotheses were not supported, and we report each.
Figures & tables
| Shortest path | Tamper once + shortest | Richer family | |||||
| Environment | Growth | min | min | min | |||
| DoorKey-8x8 | 0 | 1.442 | 0 | 1.491 | 0 | 1.442 | 0 |
| 1.439 | 0 | 1.490 | 0 | 1.439 | 0 | ||
| 1.419 | 0 | 1.477 | 0 | 1.419 | 0 | ||
| 1.369 | 0 | 1.437 | 0 | 1.386 | 0 | ||
| 1.320 | 0 | 1.396 | 0 | 1.364 | 0 | ||
| Satiation (ours) | Discounted intrinsic | Total discounted | Time-averaged drive | Presses per step |
|---|---|---|---|---|
| + tamper | return | return | replay (training log) | |
| growth , norm on | 0.694 [0.690, 0.696] | 0.694 [0.691, 0.696] | 0.035 [0.033, 0.036] | 0.061 (0.061) |
| growth , norm off | 0.710 [0.707, 0.712] | 0.710 [0.707, 0.712] | 0.018 [0.017, 0.019] | 0.096 (0.096) |
| growth , norm on | 0.722 [0.721, 0.724] | 0.723 [0.721, 0.724] | 0.023 [0.022, 0.024] | 0.269 (0.270) |
| growth , norm off | 0.701 [0.700, 0.702] | 0.702 [0.700, 0.703] | 0.043 [0.042, 0.044] | 0.188 (0.189) |
| Cost per press | Solved | 95% CI | Curve area | Presses per step, median [IQR] | Tamperers |
|---|---|---|---|---|---|
| 0 | 1/20 | [0.00, 0.25] | 0.042 | 0.061 [0.060, 0.063] | 20/20 |
| 0.01 | 3/20 | [0.03, 0.38] | 0.147 | 0.043 [0.042, 0.044] | 19/20 |
| 0.1 | 20/20 | [0.83, 1.00] | 0.601 | 0.010 [0.001, 0.020] | 7/20 |
| 1 | 20/20 | [0.83, 1.00] | 0.816 | 0.000 [0.000, 0.000] | 0/20 |
| no tamper action | 20/20 | [0.83, 1.00] | 0.901 | 0.000 [0.000, 0.000] | n/a |
| Satiation (ours) | Growth | Round, seeds | Registered [95% CI] | Re-scored [95% CI] |
|---|---|---|---|---|
| DoorKey, norm on | 1 b , 0–9 | – | 0/10 [0.00, 0.31] | |
| 4, 40–59 | 0/20 [0.00, 0.17] | – | ||
| 1 b , 0–9 | – | 0/10 [0.00, 0.31] | ||
| 4, 40–59 | 0/20 [0.00, 0.17] | – | ||
| 2, 0–9 | 0/10 a [0.00, 0.31] | 0/10 [0.00, 0.31] | ||
| 3, 20–39 | 4/20 [0.06, 0.44] | – |
| Hypothesis | Round | Rows | Outcome | Note |
|---|---|---|---|---|
| h/m/u | ||||
| A closed-form temptation index predicts the farming rate | 2 | 2/7/0 | not supported | discount and coefficient changes moved farming against |
| A pay-once control does not farm | 2 | 0/1/0 | not supported | the rule labelled prolonging policies (correction 2) |
| Learning trap: farming locks in before the goal is found | 3 | 1/2/2 | not supported | early-goal starts did not reduce farming; two rows unresolvable |
| The exact optimum (small grid) solves; with a tamper action it tampers and still solves | 3, 4 | 2/0/0 | supported | |
| Farming rises with need growth (DoorKey) | 2, 3 | 5/1/0 | supported | 2 of 3 ordering tests |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Rd | ID | Prediction | Observed | Test | raw | Result |
|---|---|---|---|---|---|---|
| 2 | N1 | farmers in 2-8 of 10 (E=0.764) | 7/10 | level (range) | – | hit |
| 2 | N2 | farmers in 0-2 of 10 (E=0.281) | 7/10 | level (range) | – | miss |
| 2 | N3 | farmers in 1-4 of 10 (E=0.593) | 0/10 | level (range) | – | miss |
| 2 | N4 | farmers in 2-8 of 10 (E=0.764) | 0/10 | level (range) | – | miss |
| 2 | N5 | farmers in 0-1 of 10 (E=0.382) | 3/10 | level (range) | – | miss |
| 2 | N6 | farmers in 6-10 of 10 (E=0.87) | 10/10 | level (range) | – | hit |
| Seed | growth 1/160, norm on | 1/160, norm off | 1/40, norm on | 1/40, norm off | ||||
|---|---|---|---|---|---|---|---|---|
| tamper | no tamper | tamper | no tamper | tamper | no tamper | tamper | no tamper | |
| 40 | 0.060 C | 0.000 S | 0.096 C | 0.082 S | 0.257 C | 0.000 S | 0.189 C | 0.016 S |
| 41 | 0.061 P | 0.000 S | 0.100 C | 0.081 S | 0.273 P | 0.013 C | 0.188 C | 0.017 S |
| 42 | 0.059 P | 0.000 S | 0.096 C | 0.083 S | 0.270 P | 0.000 S | 0.189 C | 0.022 S |
| 43 | 0.059 P | 0.000 S | 0.094 C | 0.081 S | 0.270 C | 0.001 C | 0.191 C | 0.015 S |
| 44 | 0.064 P | 0.000 S | 0.101 C | 0.085 S | 0.273 P | 0.000 S | 0.187 C | 0.020 S |
| Seed | cost 0 | cost 0.01 | cost 0.1 | cost 1 | no tamper action |
|---|---|---|---|---|---|
| 40 | 0.060 C | 0.043 C | 0.022 S | 0.000 S | 0.000 S |
| 41 | 0.061 P | 0.042 F | 0.000 S | 0.000 S | 0.000 S |
| 42 | 0.059 P | 0.043 C | 0.009 S | 0.000 S | 0.000 S |
| 43 | 0.059 P | 0.045 S | 0.016 S | 0.000 S | 0.000 S |
| 44 | 0.064 P | 0.044 P | 0.001 S | 0.000 S | 0.000 S |
| 45 | 0.062 P | 0.025 S | 0.000 S | 0.000 S | 0.000 S |
| Run | Seed | First goal | Class | Peak | Final | Declined | Entropy | “Done” | Rise | Order |
|---|---|---|---|---|---|---|---|---|---|---|
| 1/160 on | 40 | 88,800 | lost | 0.04 | 0.021 | no | 1.897 | 0.060 | 40,960 | no decline |
| 41 | 6,288 | lost | 0.06 | 0.026 | no | 1.878 | 0.061 | 40,960 | no decline | |
| 42 | 71,008 | lost | 0.05 | 0.036 | no | 1.858 | 0.059 | 40,960 | no decline | |
| 43 | 25,632 | lost | 0.13 | 0.083 | no | 1.872 | 0.059 | 40,960 | rise first | |
| 44 | 255,984 | lost | 0.03 | 0.002 | no | 1.866 | 0.064 | 40,960 | no decline | |
| 45 | 4,432 | lost | 0.06 | 0.010 | no | 1.887 | 0.062 | 40,960 | no decline |
| Setting | Value |
|---|---|
| Environments in parallel | 16 |
| Rollout length per environment | 128 |
| Batch | 2048 |
| Environment steps per seed | 2,000,000 |
| Learning rate (linearly annealed) | |
| Discount | 0.99 |
| Label in the text | Round | Name | Run ID |
|---|---|---|---|
| satiation 1/40 (round-1 calibration) | 1 | cal 1/40 | 6bfabf4a |
| satiation 1/160 (round-1 calibration) | 1 | cal 1/160 | 6e507c3e |
| satiation 1/640 (round-1 calibration) | 1 | cal 1/640 | 6518a3c8 |
| satiation 1/40 (seeds 10–19) | 2 | N1 | ed5147b6 |
| satiation 1/40, | 2 | N2 | 74b2b2d0 |
| satiation 1/80 | 2 | N3 | af4373da |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,852 papers