Reinforcement learning (RL) trains representations on data selected by the agent's policy, which then uses the resulting returns to guide its next choices. We show that this loop can sustain a lower-return policy even when representation fitting is globally optimal on those data. In a self-confirming superposition trap, every optimal code assigns overlapping directions to features that rarely occur together under the current policy. An alternative action brings them together, causing interference that lowers its return and reinforces avoidance, although refitting to that action would yield more return at the same capacity. We characterize the dimensions admitting a trap in a tied two-step model and show separately that equal feature frequencies, continued visitation, and independent controller learning need not prevent it. Because fitting weights errors by visitation, an avoided action can lose its return advantage at little cost to the objective. In a finite-action model, we bound this distortion and derive a replay condition: sufficient training weight on the best separately adapted action preserves its ranking despite residual error. Neural PPO experiments show how the feedback develops during learning: agents initialized toward different actions develop different interference patterns, opposite mean return rankings, and different final policies at the same capacity. We therefore test whether retaining access to neglected states can improve control. Keeping these states in training reduces measured interference and improves sequential return, with gains even when the encoder is frozen. Related interventions on state access, replay weights, and feature overlap improve control on MiniGrid and DMControl. For agents that learn through a world model, protected fitting improves DreamerV3--Crafter's cumulative training scores at unchanged capacity.
Figures & tables
Figure 1: Optimal feature sharing can favor the worse policy.
Figure 2: Globally optimal fitting can sustain the worse policy. (a) Optimal overlaps. (b) Adapted return at δ=0.5 . Arrows show updates, and filled/open points mark stable/unstable policies.
Figure 3: Different co-activation patterns lead the same initial policy toward opposite branches.
Figure 4: Initial preference changes cross-talk allocation and mean branch-return ordering.
Figure 5: Temporary stage access improves rank-two return, including with frozen encoders.
Figure 6: Stage mixing improves DoorKey success more consistently than factor mixing.
Table 1: Returns after continuing TD-MPC2 and SAC with each intervention.
Figure 7: Dreamer training and matched-input forecasts. Labels give pixel MSE ×1,000 .
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Return increases with rank, while the linear controller gives higher means than the MLP. Small points show five seeds; large markers give means with sample SD. Conditions are offset horizontally, and panels use separate return scales.
Figure 9: Initial preference changes the allocation of cross-talk and control error under rank-two PPO. Means use all 100 seeds per condition; cross-talk measures absolute Jacobian responses.
Rank
Initial p(E)
Final p(E)<0.5
Mean E−S gap
Return [95% CI]
2
0.095
96/100
−0.313
−0.2511[−0.2557,−0.2453]
2
0.86
0/100
0.651
−0.1205[−0.1215,−0.1194]
64
0.095
0/100
0.153
−0.0260[−0.0262,−0.0257]
64
0.86
0/100
0.154
−0.0281[−0.0283,−0.0278]
Appendix
Table 2: Low-start rank-two PPO learns lower-return behavior. Each condition uses 100 seeds; brackets give 95% seed-bootstrap intervals.
Figure 10: Rank-two agents retain lower returns, stronger partner responses, and negative advance–exit rollout gaps. Training at p0=0.1 (a–c) and endpoints across starts (d) use 20 seeds per condition and 95% intervals.
Rank
Encoder
Hub-only fork
Uniform, then Hub
Paired gain
2
Trainable
0.468[0.430,0.501]
1.234[1.209,1.255]
0.766[0.716,0.815]
2
Frozen
−0.326[−0.369,−0.282]
1.066[1.033,1.098]
1.392[1.325,1.457]
64
Trainable
1.673[1.663,1.681]
1.676[1.668,1.684]
0.004[−0.004,0.011]
Appendix
Table 3: Temporary exposure changes improve narrow return after Hub training resumes. Gains are paired differences; brackets give 95% intervals over 20 training seeds.
Figure 11: Policy basins under three interventions in the tied task. Schematic phase lines assume exact adaptation and 0<δ<1 ; endpoint policies remain fixed. Eliminating collisions can precede basin removal, while strong entropy leaves an interior attractor.
Figure 12: Estimated low-policy basin width under orthogonality regularization at κ=80 , with penalty normalized by the analytical λcrit(δ) . The curve is the post-hoc δ=0.3 diagnostic; open markers are follow-up estimates for δ=0.3,0.6 . The separate lower panel marks the registered G6 escape thresholds from one fixed initialization. G6 failed its registered criterion; these penalty estimates do not measure basin width. The dashed line marks the theoretical basin-removal threshold.
Figure 13: Numerical untied-controller branch-loss differences for η∈{0.2,1,5,20} , divided by the exact endpoint magnitude (1+η)−2 . A deterministic angle search gives a monotone crossing of zero at all four noise levels. These finite-noise calculations do not give a value for the asymptotic threshold η0 in Proposition 20 .
Figure 14: Replay restores return ordering with residual representation error. Both panels use the same policy and finite parameter class, with 40% uniform reference replay added in (b). Percentages denote a1 ’s weight. Hollow markers use separate fits, whereas solid markers use one shared fit (construction in Section G.4 ).
Figure 15: Finite-sample fits recover the predicted geometry and preference reversal at low p ; orthogonal capacity removes the reversal. Curves in (a) are analytic predictions; markers and bars in (a–b) show medians and 10–90% ranges over 50 datasets per value of p .
Figure 16: E1 recovers the solved chain from the action distribution to deployed control. Panel A shows the finite-sample squared Gram entries, with circles marking the medians over 50 datasets. The medians lie on the three theory curves, and the collision moves from pair {1,2} to pair {2,3} across the golden-ratio boundaries p± , as Proposition 8 predicts. Panel B then reads the same fits through the order parameter D(p) , which crosses the task preference δ=0.5 at the separator pδ , so the deployed preference is reversed on the shaded side. Panel C closes the chain by comparing the Gram-predicted gap with fresh forced-rollout measurements. The scarce-rank fits at p=0.2 and p=0.8 lie on the identity line at the reversed and restored gaps, while the central-phase fits spread along the line. The orthogonal k=3 control stays at gap δ , and the median absolute bridge error for rank-two codes is 3.9×10−3 .
Figure 17: Finite-sample recovery of the solved chain: (a) collision transfer, (b) gap reversal at low p and its removal by orthogonal capacity, and (c) agreement between Gram prediction and fresh rollout. Marks and bars are medians and 10–90% ranges over 50 datasets; panel (c) shows every dataset. The actor is not trained.
N
p=p−
p=1/2
p=p+
256
0.000 / 0.485
0.177 / 0.507
0.000 / 0.356
1024
0.002 / 0.254
0.102 / 0.236
0.000 / 0.218
4096
0.000 / 0.106
0.039 / 0.074
0.011 / 0.143
16384
0.000 / 0.065
0.020 / 0.043
0.011 / 0.077
Appendix
Table 4: Sampling variation in the empirical squared-Gram optimum. Entries are the median / 90th percentile of the maximum absolute difference between empirical-optimum and population-optimum squared Gram entries over 100 independent datasets per cell. The table shows the two phase thresholds and the central value of p ; the released data also include four values away from the thresholds.
Experiment
Quantity
Bound
Measured
E1
fits within 10−8 of the exact empirical optimum
≥0.99
1,148 of 1,150
E1
median forced-rollout gap error, rank two
—
3.9×10−3
E2
basin agreement with sign(p0−pδ)
≥0.99
746 of 746
E2
separator error over 19 preferences, κ=80
≤0.01
0.0044
E2
floor separator error, 4 points
≤0.01
6.3×10−4
E2
replay transition error, 2 references
≤0.02
0.0123
Appendix
Table 5: Registered quantities of the three experiments. Bounds are the frozen pass criteria, and every verdict is reported, including the two failures.
Figure 18: All 100 seeds per arm. Boxes, center lines, and whiskers show interquartile ranges, medians, and full ranges. Outlined ranges marked 4 identify low-start rank-two runs ending at high E .
Figure 19: Branch competence and absolute cross-talk for the same 400 PPO runs as Figure 9 . In each row, upper filled bands and lower open bands show (a) forced-branch reconstruction MSE on S and E , respectively, and (b) cross-talk x12 and x23 , on the pairs exclusive to E and S , respectively. Cross-talk is a symmetrized absolute Jacobian response, not an encoder angle. Dotted and dashed MSE guides mark the Bayes floor ( 0.165 ) and an achievable rank-two reference ( 0.235 ); the latter is not a proved neural-class optimum. Boxes show interquartile ranges, center lines medians, and whiskers full ranges over all 100 training seeds per arm, including the four alternate-endpoint runs. Reconstruction uses the mean terminal control.
Figure 20: Closed-loop dynamics in the solved class. Panel A draws the two basins with the analytic separator on the κ=80 grid as an illustration, while the registered basin gate runs on the κ=20 grid ( Appendix J ). Panel B shows the separator error decreasing roughly as 1/κ over the tested update counts, with bands over ten representation initializations.
Figure 21: Fresh-seed cohort of the minimal learned-representation bridge. Representation constraints shift the basin boundary toward each class’s population prediction, with the capacity control at 0 of 11,700 trapped runs (A). The visitation-floor intervention moves the boundary in the predicted direction (B), where the dashed equality line is a secondary reference and the registered gate tests direction.
Gate
Registered estimand
Bound
Measured
Verdict
G1
trapped runs, three-dimensional control
0
0 of 779
pass
G2
basin agreement with sign(p0−pδ)
≥0.99
746 of 746
pass
G3
separator error over 19 preferences, κ=80
≤0.02
0.0044
pass
G4
critical temperature inside the computed bracket
3 of 3
3 of 3
pass
G5
floor-mapped separator error, four points
≤0.02
<0.00063
pass
G6
fixed-start escape point against λcrit
≤0.1
0.169 , 0.101
fail
Appendix
Table 6: Registered gates for the solved-class loop E2. All separator and threshold errors are absolute deviations from the computed values of Sections 2 and 5 .
Gate
Registered estimand and bound
Measured
Verdict
B1
full-rank control trap rate ≤0.005
0 of 11,700
pass
B2
constrained boundary error CI within ±0.02 , six cells
6 of 6 , widest ±0.007
pass
B3
unconstrained boundary error CI within ±0.03 , six cells
6 of 6 , all within ±0.023
pass
B4
floor shift CI lower bound >0 , six pairs
6 of 6
pass
B5
collision-pair order ≥0.95 per side
1.000 / 0.999
pass
B6
forced-gap sign ≥0.95 per side
0.992 / 1.000
pass
Appendix
Table 7: Registered gates for the sampled bridge, with 35,100 records from 50 fresh seeds. Per-side entries read trapped/escaped and pool the two-dimensional arms. Boundary intervals resample seeds jointly across matched arms and floors and use the 2.5th and 97.5th percentiles. The return-gate resampling defect is described below.
Table 8: Registered gates for the final PPO cohort, with 100 fresh seeds per main arm. Lower bounds are the lower endpoints of exact two-sided 95% binomial intervals. Return contrasts use 95% seed-bootstrap intervals. The two endpoint-conditioned entries in F1, F5 and F6 concern 96 trapped low-start runs and 100 escaped high-start runs.
Seed
r
MC gap
Analytic gap
Final pE
Final J
Max Gram error
0
0.4
-0.492603
-0.499766
0.05
-0.205030
0.000787
0
0.6
1.487152
1.499311
0.95
0.244970
0.001044
1
0.4
-0.487099
-0.498816
0.05
-0.205024
0.002418
1
0.6
1.492229
1.498647
0.95
0.244970
0.001726
2
0.4
-0.501893
-0.499512
0.05
-0.205016
0.000907
2
0.6
1.497788
1.498831
0.95
0.244974
0.001812
Appendix
Table 9: Matching study by dataset seed and exposure. Gap, policy and return entries average the four nested initializations; the last column is the largest squared-Gram error among those four fits. The Monte Carlo gap is measured before continuation, and the analytic gap uses the same fitted codes. No initialization or evaluation block is treated as an independent dataset seed.
Features → dimensions
Maximum fitting excess
Initial E−SS -fitted
Initial E−SE -fitted
Final return E -fitted −S -fitted
16→2
9.25×10−5
−0.21873
0.65623
0.21875
16→4
2.30×10−4
−0.06246
0.18746
0.06250
32→2
7.50×10−5
−0.11718
0.35155
0.11719
32→4
1.98×10−4
−0.03643
0.10935
0.03646
Appendix
Table 10: Nearly optimal fits yield opposite return rankings across four dimension settings. Final return differences compare the common-start continuations after 4,096 updates.
Figure 22: The branch preferences in Figure 3 b persist throughout continued learning. Curves and bands summarize all 40 continuations per fitting condition in each setting.
Figure 23: Complete trajectories for Figure 10 , including initialization. Both ranks use p0=0.1 and 20 training seeds. Means and pointwise 95% seed-bootstrap intervals use the same seeds and saved checkpoints throughout. Initial returns are strongly negative before either controller learns the task.
Initial pE
Rank 2
Rank 64
0.1
0.606[0.600,0.612]
1.673[1.666,1.679]
0.5
0.605[0.600,0.610]
1.676[1.669,1.684]
0.9
0.606[0.600,0.612]
1.675[1.666,1.683]
Appendix
Table 11: Ordinary training from Hub, 20 seeds per cell and 16,777,216 phases per run. Entries are stochastic original-Hub return with pointwise 95% seed-bootstrap intervals. This cohort has a different training history and budget from Table 3 .
Figure 24: Paired final return differences for Table 3 . Points and bars show means and pointwise 95% bootstrap intervals over 20 complete training-seed blocks.
Study
Observed result
Interpretation
Late fixed-mask exposure
Five forks/control runs return 0.615 – 0.628 ; no observed high exits.
Mask12/13 exposure with trainable/frozen encoders did not recover this mature narrow checkpoint.
Late balanced-E exposure
Five forks/control runs return 0.612 – 0.636 ; no observed high exits.
Balanced masks likewise did not produce late recovery in the tested budget.
Fresh balanced-E pilot
Hub follow-up returns 0.445 , with root E MSE 0.620 and no observed high exits.
Useful auxiliary learning alone did not maintain downstream behavior.
Appendix
Table 12: Retained sequential development failures, all using seed 1. Late interventions start from the 8,388,608-phase ordinary-training pilot checkpoint. Fixed-mask forks receive 4,194,304 additional phases; balanced-E forks receive 8,388,608. The fresh pilot receives 4,194,304 pretraining and 4,194,304 Hub-only phases. These results do not establish impossibility.
Code
σ
Controller
ΔqO
ΔqB
JO
JB
Exact S
0.1
Tied
−0.4990
−0.4990
−0.0699
−0.0699
OLS
−0.3216
−0.0261
−0.0593
−0.1070
Pop. linear
−0.3207
−0.0259
−0.0593
−0.1071
MLP
−0.0812
0.0098
−0.0378
−0.0484
Bayes
−0.0797
0.0106
−0.0374
−0.0481
Exact S
0.3
Tied
−0.4992
−0.4992
−0.2300
−0.2300
Appendix
Table 13: Independent controllers and fitting exposure change action values through frozen codes. Means use twenty seed blocks at δ=0.5 ; O/B denote original/balanced fitting, with returns at the original policy.
n
k
Controller
p0
Final p(E)
DS
DE
Δq
Jsampled
16
8
Linear
0.1
0.363
0.345
0.414
0.0028±0.0204
−0.3579±0.0119
16
8
Linear
0.9
0.949
0.416
0.321
0.1666±0.0084
−0.2723±0.0090
16
8
MLP
0.1
0.500
0.363
0.422
0.0124±0.0330
−0.3675±0.0120
16
8
MLP
0.9
0.949
0.424
0.347
0.1484±0.0109
−0.3008±0.0076
16
16
Linear
0.1
0.883
0.175
0.174
0.0726±0.0005
−0.1206±0.0002
16
16
Linear
0.9
0.949
0.176
0.174
0.0734±0.0014
−0.1158±0.0003
Appendix
Table 14: Capacity improves return, while negative-gap estimates occur only at low-start n=16,k=8 . All 28 conditions report five-seed means; gap and sampled return include sample SD.
Success by training seed (%)
Mean
Mean
Method
k
0
1
2
success (%)
return
PPO
4
0.0
0.0
0.0
0.0
0.000
PPO + entropy
4
0.0
3.0
0.5
1.2
0.004
PPO
16
0.5
0.0
0.5
0.3
0.002
PPO + entropy
16
39.0
26.0
12.0
25.7
0.161
PPO
64
0.0
0.0
0.0
0.0
0.000
Appendix
Table 15: DoorKey 8×8 : mean success with entropy regularization is nonmonotone in actor rank. Final success (%) and original-task return for all three training seeds per condition; the last row is the auxiliary PFO comparison.
Success by training seed (%)
Mean
Mean
k
0
1
2
success (%)
return
16
100.0
100.0
100.0
100.0
0.937
128
99.5
100.0
49.5
83.0
0.742
Appendix
Table 16: The smaller DoorKey task is learnable at rank 16. Final success (%) and return on 5×5 , three training seeds per rank.
Rank
Encoder
Native
Stage mixture
Factor mixture
16
Trainable
22.5/0.1357
30.5/0.2034
23.2/0.1464
16
Frozen
35.3/0.2444
40.7/0.2789
28.0/0.1748
128
Trainable
12.5/0.0780
20.8/0.1294
20.5/0.1259
128
Frozen
20.2/0.1356
25.5/0.1690
19.5/0.1170
Appendix
Table 17: DoorKey continuation outcomes at fixed capacity. Entries give mean native-start success (%) / original return over three parents. Every arm has 262,144 additional learner transitions. Native and broad stage controls are shared with the subsequent factor study.
Trainable
Frozen
Rank
Seed
Native
Stage
Factor
Native
Stage
Factor
16
0
43.0
51.5
18.0
54.5
52.5
34.0
16
1
7.0
20.5
14.5
28.5
43.5
33.0
16
2
17.5
19.5
37.0
23.0
26.0
17.0
128
0
20.5
31.0
27.5
31.0
35.0
20.0
128
1
9.5
12.5
10.5
18.0
26.5
17.5
Appendix
Table 18: Every parent-level native-start success rate (%). T/F denotes trainable/frozen sensory encoder; the actor head and critic learn in both conditions. No parent is excluded.
Rank
Encoder
Optimal-action probability (%)
Excess steps
Sensitivity (pp)
16
Trainable
22.0→40.3
1.119→0.839
8.5→19.9
16
Frozen
29.0→39.5
1.012→0.782
11.6→9.6
128
Trainable
29.3→39.2
0.994→0.819
9.4→10.2
128
Frozen
27.0→35.4
0.993→0.859
5.1→11.9
Appendix
Table 19: Factor-mixture versus native continuation on the fixed panel. Each arrow gives the native and factor means over three parents. Sensitivity is an absolute probability difference in percentage points; excess steps assume optimal continuation after one sampled action.
Exposure
No pickup
Pickup, no opening
Opened, no goal
Success
Native
907
818
132
543
Stage mixture
893
706
96
705
Factor mixture
1,019
722
112
547
Appendix
Table 20: Failure-stage decomposition of all 7,200 final native-start episodes, with 2,400 per exposure across twelve endpoints. Counts are evaluation outcomes, not independent training replications.
Learner
Intervention
Finger-turn-hard
Hopper-hop
TD-MPC2
Baseline
694.9±15.2
97.2±85.5
Phase replay
780.6±41.6
133.2±74.2
Overlap λ=0.003
759.0±62.6
108.5±78.0
Overlap λ=0.03
814.8±43.9
60.2±47.4
Prior entropy ×2
738.4±64.3
114.7±57.6
Prior entropy ×5
799.9±75.5
105.3±105.2
Appendix
Table 21: Original continuation results across all three training seeds. Entries are mean ± sample SD of the three checkpoint means, each estimated from 50 episodes. Unlike Table 1 , this table summarizes variation across training runs.
Continuation
Finger-turn-hard
Hopper-hop
Entropy off throughout
244.5±41.1
49.8±1.3
Entropy restored
171.0±51.3
29.5±1.5
Appendix
Table 22: Entropy restoration from SAC parents trained with entropy disabled. Each selected checkpoint uses the five-evaluation protocol of Table 1 . These parents are separate from standard SAC.
Seed/statistic
Uniform
Protected
Difference
0
7.968
8.229
0.261
1
7.578
8.862
1.284
2
7.943
8.201
0.258
Mean
7.830
8.431
0.601
SD
0.218
0.374
0.592
Appendix
Table 23: Cumulative training Crafter scores at one million steps. Differences are protected minus uniform. SD is the sample standard deviation across three paired seeds.
We show that a threshold in decision capacity determines whether self-play reinforcement learning agents collapse under asymmetric rule perturbations. Across poker variants, matrix games, a dice game, and multiple learning algorithms, eliminating all positive-reach contingent decisions causes rapid convergence to a deterministic exploitation attractor, a fixed point at near-maximal loss. Preserving even a single positive-reach contingent decision point prevents this collapse. A frozen baseline and fixed-opponent control confirm that the mechanism is co-adaptation under constraint, not the perturbation itself. The phenomenon is timing-invariant, fully reversible upon action restoration, and intensifies under function approximation. These results establish a sharp threshold at zero reach-weighted contingent action capacity, with severity scaling continuously via reach-weighted capacity in the tested domains.
Safety in reinforcement learning (RL) is typically enforced through objective shaping while keeping environment dynamics stationary with respect to observable state-action pairs. Under delayed harm, this can lead to replay: after a washout period, reintroducing the same stimulus under matched observable conditions reproduces a similar harmful cascade. We introduce the Replay Suppression Diagnostic (RSD), a controlled exposure-decay-replay protocol that isolates this failure mode under frozen-policy evaluation. We show that, under stationary observable transition kernels, replay cannot be structurally suppressed without inducing a persistent shift in replay-time action distributions. Motivated by platform-mediated systems, we propose Regret-Aware Policy Optimization (RAPO), which augments the environment with persistent harm-trace and scar fields and applies a bounded, mass-preserving transition reweighting to reduce reachability of historically harmful regions. On graph diffusion tasks (50-1000 nodes), RAPO suppresses replay, reducing re-amplification gain (RAG) from 0.98 to 0.33 on 250-node graphs while retaining 82% of task return. Disabling transition deformation only during replay restores re-amplification (RAG 0.91), isolating environment-level deformation as the causal mechanism.
Prakul Sunil Hiremath
Department of Computer Science and Engineering, VTU, Belagavi, India · Aliens on Earth (AoE) Autonomous Research Group, Belagavi, India
Reinforcement learning with verifiable rewards is typically performed on-policy, keeping training data close to the current policy but limiting learning to trajectories that the policy can discover itself. Off-policy methods such as supervised fine-tuning, on the other hand, can leverage external knowledge beyond the base model's capabilities, but may suffer from large distribution shift. The key challenge is thus to expand exploration without sacrificing learnability. In this work, we introduce Minimal Intervention Reinforcement Learning (MInTRL), which expands the exploration frontier through sparse, local interventions in otherwise on-policy rollouts. During generation, a judge-intervention policy periodically reviews the current policy's output, replaces erroneous suffixes with short corrections, and immediately returns control to the policy. During training, MInTRL adopts a sequence-level advantage-regression objective that eliminates the need for importance sampling. We show that sparse, local interventions can substantially improve coverage beyond finite-budget on-policy sampling while preserving the overall on-policy nature of the resulting trajectories. Across math and code benchmarks, MInTRL consistently outperforms standard on-policy and off-policy baselines. Ablations show that MInTRL remains effective with self-intervention and across different judge policies, while performance peaks at moderate intervention intensity, highlighting the importance of intervening minimally. These results establish minimal intervention as an effective paradigm for enhancing on-policy RL.