cs.LGOct 1, 2026
SaveWhen Do Intrinsic Rewards Lead to Exploration?
Organizations: Stanford University
Abstract
Intrinsic rewards are designed to guide exploration in reinforcement learning by assigning value to an agent's experience, for example through prediction error or learning progress. However, maximizing these rewards need not produce the most informative experience available. We propose a formal criterion for exploration that compares policies by the counterfactual information they acquire: how well their histories can substitute for experience under alternative policies. We construct a single, simple environment in which specified count-based, prediction-error, empowerment, and information-gain objectives have maximizing policies that are Pareto-suboptimal at acquiring counterfactual information. We explain these failures and establish conditions under which existing intrinsic rewards successfully encourage optimal exploration. We also construct an objective that assigns a higher value whenever exploration strictly improves under our criterion.
Figures & tables
Figure 1: The alarm environment. Each digit–color pair encodes one observation; the displayed values are examples, and coins denote independent fair draws. In inspection mode, the startup digit reports whether the alarm will ever ring. The color distribution is known and identical in every possible world. After startup, display observations alternate with alarm observations, which report whether the alarm rings at that tick regardless of the agent’s actions.
| Alarm: inspection target | Four bits: FULL target | |||
| Objective | Complete return | Complete return | ||
| Information | ||||
| Brier | ||||
| Hellinger | ||||
| Absolute | ||||
| (eventual reference) | ||||
Table 1: How well do objective optima reproduce native experiments? Entries give all eventual target deficiencies attained by global maximizers within the stated policy families. Zero means sufficiency here; positive values mean strict domination by a sufficient policy. All later collection is allowed. MOP, state-visitation entropy, and DIAYN use worst-world values; other rewards use their stated priors. Errors from different examples are not a performance ranking. The eventual reference (Section 5 ) is undiscounted and has in every example.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Comparator | Wins | Ties | Losses | Mixed |
|---|---|---|---|---|---|
| Native mean | Ordinary Brier | 18 | 2 | 2 | 0 |
| Native maximum | Ordinary Brier | 19 | 1 | 2 | 0 |
| Native mean | 5%-Brier control | 16 | 0 | 6 | 0 |
| Native maximum | 5%-Brier control | 18 | 1 | 2 | 1 |
| Fixed native mean | Its reward-matched control | 6 | 1 | 15 | 0 |
Table 2: Worst four-step target error after three collected interactions. Counts favor the method named in the first column; every row retains all 22 classes. “Mixed” means signs differ across saved representatives. The last row uses one fixed paired reference per class and cannot be read as changing only the return floor in the preceding envelope comparison. The 5%-Brier and matched controls minimize the native depth-three maximum; they are not ordinary Brier optimizers.
Figure 2: Fixed-budget simulation errors and returns. Three-step policies on all 22 classes: (a) hardest four-step target error, (b) 5%-Brier control minus native audit, and (c) Brier return forgone as a fraction of its attainable range. The control minimizes the native maximum subject to retaining Brier return within 5% of its attainable range below the maximum. Bars enclose saved representatives and numerical bounds, not all optima or sampling uncertainty. Positive differences in (b) favor the native method. The reward-matched comparison in Figure 3 instead uses one fixed reference per class.
Figure 3: Retaining the reference’s posterior return changes the comparison. (a) Each fixed native-mean reference versus native maximum constrained to retain its Brier return, with allowance . Positive control-minus-reference audit differences favor the reference: six wins, one numerical tie, and fifteen losses. (b) Actual Brier sacrifice of the same paired policies, alongside the original unconstrained native-maximum envelope. The return floor may be exceeded. Both paired audits retain and replay every four-step target witness. These post-design scalar results do not imply dominance: 21 pairs of experiments are numerically incomparable and one is equivalent within tolerance.
| Error tolerance | Five steps | ||||
|---|---|---|---|---|---|
| Objective | Three steps | Four steps | Five steps | ||
| Information | 1/22 | 5/22 | 8/22 | 6/22 | 13/22 |
| Brier | 1/22 | 6/22 | 10/22 | 9/22 | 14/22 |
| Categorical pseudo-count | 1/22 | 3/22 | 5/22 | 3/22 | 12/22 |
| Uniform | 1/22 | 1/22 | 4/22 | 3/22 | 12/22 |
| Native mean | 1/22 | 4/22 | 14/22 | 9/22 | 18/22 |
Table 3: Classes reproducing all six fixed references. Counts are verified successes divided by checked classes, with all 22 classes checked for every method. Thresholds concern worst-world history simulation error; they are not world-identification error rates.
Figure 4: Return cost of reproducing a reference. (a) Brier to its Brier reference in Delay . (b) Information to the Brier reference in Irreversible . Stars show separate reference-aware alternatives: costs are of maximum Brier gain and a numerically zero loss of information gain, respectively. These policies are selected afresh at each budget; the lines do not track one continuing policy.