Self-Confirming Superposition Traps in Reinforcement Learning

Dai Shi; Andi Han; Feng Chen; Yiqun Duan; Junbin Gao; José Miguel Hernández-Lobato

cs.LGSep 26, 2026

Self-Confirming Superposition Traps in Reinforcement Learning

Authors: Dai Shi, Andi Han, Feng Chen, Yiqun Duan, Junbin Gao, José Miguel Hernández-Lobato

Organizations: University of Cambridge · University of Sydney · University of Adelaide · Facebook

Abstract

Reinforcement learning (RL) trains representations on data selected by the agent's policy, which then uses the resulting returns to guide its next choices. We show that this loop can sustain a lower-return policy even when representation fitting is globally optimal on those data. In a self-confirming superposition trap, every optimal code assigns overlapping directions to features that rarely occur together under the current policy. An alternative action brings them together, causing interference that lowers its return and reinforces avoidance, although refitting to that action would yield more return at the same capacity. We characterize the dimensions admitting a trap in a tied two-step model and show separately that equal feature frequencies, continued visitation, and independent controller learning need not prevent it. Because fitting weights errors by visitation, an avoided action can lose its return advantage at little cost to the objective. In a finite-action model, we bound this distortion and derive a replay condition: sufficient training weight on the best separately adapted action preserves its ranking despite residual error. Neural PPO experiments show how the feedback develops during learning: agents initialized toward different actions develop different interference patterns, opposite mean return rankings, and different final policies at the same capacity. We therefore test whether retaining access to neglected states can improve control. Keeping these states in training reduces measured interference and improves sequential return, with gains even when the encoder is frozen. Related interventions on state access, replay weights, and feature overlap improve control on MiniGrid and DMControl. For agents that learn through a world model, protected fitting improves DreamerV3--Crafter's cumulative training scores at unchanged capacity.

Figures & tables

Figure 1: Optimal feature sharing can favor the worse policy.

Figure 2: Globally optimal fitting can sustain the worse policy. (a) Optimal overlaps. (b) Adapted return at

\delta=0.5

. Arrows show updates, and filled/open points mark stable/unstable policies.

Figure 3: Different co-activation patterns lead the same initial policy toward opposite branches.

Figure 4: Initial preference changes cross-talk allocation and mean branch-return ordering.

Figure 5: Temporary stage access improves rank-two return, including with frozen encoders.

Figure 6: Stage mixing improves DoorKey success more consistently than factor mixing.

Learner	Intervention	Finger-turn-hard	Hopper-hop
TD-MPC2	Baseline	$749.3\,\pm\,35.7$	$177.1\,\pm\,2.9$
	Phase replay	$\color[rgb]{0.1406,0.3203,0.5}\mathbf{863.3}\,\pm\,60.3$	$206.0\,\pm\,2.3$
	Overlap $\lambda=0.003$	$825.6\,\pm\,70.4$	$184.3\,\pm\,3.9$
	Overlap $\lambda=0.03$	$782.6\,\pm\,37.5$	$103.0\,\pm\,4.4$
	Prior entropy $\times 2$	$734.8\,\pm\,25.9$	$172.6\,\pm\,4.1$
	Prior entropy $\times 5$	$797.8\,\pm\,45.4$	$\color[rgb]{0.1406,0.3203,0.5}\mathbf{217.2}\,\pm\,4.1$

Table 1: Returns after continuing TD-MPC2 and SAC with each intervention.

Figure 7: Dreamer training and matched-input forecasts. Labels give pixel MSE $\times 1{,}000$ .

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Return increases with rank, while the linear controller gives higher means than the MLP. Small points show five seeds; large markers give means with sample SD. Conditions are offset horizontally, and panels use separate return scales.

Figure 9: Initial preference changes the allocation of cross-talk and control error under rank-two PPO. Means use all 100 seeds per condition; cross-talk measures absolute Jacobian responses.

Rank	Initial $p(E)$	Final $p(E)<0.5$	Mean $E-S$ gap	Return [95% CI]
2	$0.095$	96/100	$-0.313$	$-0.2511\ [-0.2557,-0.2453]$
2	$0.86$	0/100	$0.651$	$-0.1205\ [-0.1215,-0.1194]$
64	$0.095$	0/100	$0.153$	$-0.0260\ [-0.0262,-0.0257]$
64	$0.86$	0/100	$0.154$	$-0.0281\ [-0.0283,-0.0278]$

Appendix

Table 2: Low-start rank-two PPO learns lower-return behavior. Each condition uses 100 seeds; brackets give 95% seed-bootstrap intervals.

Figure 10: Rank-two agents retain lower returns, stronger partner responses, and negative advance–exit rollout gaps. Training at

p_{0}=0.1

(a–c) and endpoints across starts (d) use 20 seeds per condition and 95% intervals.

Rank	Encoder	Hub-only fork	Uniform, then Hub	Paired gain
2	Trainable	$0.468\ [0.430,0.501]$	$1.234\ [1.209,1.255]$	$0.766\ [0.716,0.815]$
2	Frozen	$-0.326\ [-0.369,-0.282]$	$1.066\ [1.033,1.098]$	$1.392\ [1.325,1.457]$
64	Trainable	$1.673\ [1.663,1.681]$	$1.676\ [1.668,1.684]$	$0.004\ [-0.004,0.011]$

Appendix

Table 3: Temporary exposure changes improve narrow return after Hub training resumes. Gains are paired differences; brackets give 95% intervals over 20 training seeds.

Figure 11: Policy basins under three interventions in the tied task. Schematic phase lines assume exact adaptation and

0<\delta<1

; endpoint policies remain fixed. Eliminating collisions can precede basin removal, while strong entropy leaves an interior attractor.

Figure 12: Estimated low-policy basin width under orthogonality regularization at

\kappa=80

, with penalty normalized by the analytical

\lambda_{\rm crit}(\delta)

. The curve is the post-hoc

\delta=0.3

diagnostic; open markers are follow-up estimates for

\delta=0.3,0.6

. The separate lower panel marks the registered G6 escape thresholds from one fixed initialization. G6 failed its registered criterion; these penalty estimates do not measure basin width. The dashed line marks the theoretical basin-removal threshold.

Panel of Figure 13 — Figure 13: Numerical untied-controller branch-loss differences for $\eta\in\{0.2,1,5,20\}$ , divided by the exact endpoint magnitude $(1+\eta)^{-2}$ . A deterministic angle search gives a monotone crossing of zero at all four noise levels. These finite-noise calculations do not give a value for the asymptotic threshold $\eta_{0}$ in Proposition 20 .

Figure 14: Replay restores return ordering with residual representation error. Both panels use the same policy and finite parameter class, with 40% uniform reference replay added in (b). Percentages denote

a_{1}

’s weight. Hollow markers use separate fits, whereas solid markers use one shared fit (construction in Section G.4 ).

Figure 15: Finite-sample fits recover the predicted geometry and preference reversal at low

p

; orthogonal capacity removes the reversal. Curves in (a) are analytic predictions; markers and bars in (a–b) show medians and 10–90% ranges over 50 datasets per value of

p

Figure 16: E1 recovers the solved chain from the action distribution to deployed control. Panel A shows the finite-sample squared Gram entries, with circles marking the medians over 50 datasets. The medians lie on the three theory curves, and the collision moves from pair

\{1,2\}

to pair

\{2,3\}

across the golden-ratio boundaries

p_{\pm}

, as Proposition 8 predicts. Panel B then reads the same fits through the order parameter

\mathcal{D}(p)

, which crosses the task preference

\delta=0.5

at the separator

p_{\delta}

, so the deployed preference is reversed on the shaded side. Panel C closes the chain by comparing the Gram-predicted gap with fresh forced-rollout measurements. The scarce-rank fits at

p=0.2

and

p=0.8

lie on the identity line at the reversed and restored gaps, while the central-phase fits spread along the line. The orthogonal

k=3

control stays at gap

\delta

, and the median absolute bridge error for rank-two codes is

3.9\times 10^{-3}

Panel of Figure 17 — Figure 17: Finite-sample recovery of the solved chain: (a) collision transfer, (b) gap reversal at low $p$ and its removal by orthogonal capacity, and (c) agreement between Gram prediction and fresh rollout. Marks and bars are medians and 10–90% ranges over 50 datasets; panel (c) shows every dataset. The actor is not trained.

$N$	$p=p_{-}$	$p=1/2$	$p=p_{+}$
256	0.000 / 0.485	0.177 / 0.507	0.000 / 0.356
1024	0.002 / 0.254	0.102 / 0.236	0.000 / 0.218
4096	0.000 / 0.106	0.039 / 0.074	0.011 / 0.143
16384	0.000 / 0.065	0.020 / 0.043	0.011 / 0.077

Appendix

Table 4: Sampling variation in the empirical squared-Gram optimum. Entries are the median / 90th percentile of the maximum absolute difference between empirical-optimum and population-optimum squared Gram entries over 100 independent datasets per cell. The table shows the two phase thresholds and the central value of

p

; the released data also include four values away from the thresholds.

Experiment	Quantity	Bound	Measured
E1	fits within $10^{-8}$ of the exact empirical optimum	$\geq 0.99$	1,148 of 1,150
E1	median forced-rollout gap error, rank two	—	$3.9\times 10^{-3}$
E2	basin agreement with $\operatorname{sign}(p_{0}-p_{\delta})$	$\geq 0.99$	746 of 746
E2	separator error over 19 preferences, $\kappa=80$	$\leq 0.01$	0.0044
E2	floor separator error, 4 points	$\leq 0.01$	$6.3\times 10^{-4}$
E2	replay transition error, 2 references	$\leq 0.02$	0.0123

Appendix

Table 5: Registered quantities of the three experiments. Bounds are the frozen pass criteria, and every verdict is reported, including the two failures.

Figure 18: All 100 seeds per arm. Boxes, center lines, and whiskers show interquartile ranges, medians, and full ranges. Outlined ranges marked 4 identify low-start rank-two runs ending at high

E

Figure 19: Branch competence and absolute cross-talk for the same 400 PPO runs as Figure 9 . In each row, upper filled bands and lower open bands show (a) forced-branch reconstruction MSE on

S

and

E

, respectively, and (b) cross-talk

x_{12}

and

x_{23}

, on the pairs exclusive to

E

and

S

, respectively. Cross-talk is a symmetrized absolute Jacobian response, not an encoder angle. Dotted and dashed MSE guides mark the Bayes floor (

0.165

) and an achievable rank-two reference (

0.235

); the latter is not a proved neural-class optimum. Boxes show interquartile ranges, center lines medians, and whiskers full ranges over all 100 training seeds per arm, including the four alternate-endpoint runs. Reconstruction uses the mean terminal control.

Figure 20: Closed-loop dynamics in the solved class. Panel A draws the two basins with the analytic separator on the $\kappa=80$ grid as an illustration, while the registered basin gate runs on the $\kappa=20$ grid ( Appendix J ). Panel B shows the separator error decreasing roughly as $1/\kappa$ over the tested update counts, with bands over ten representation initializations.

Figure 21: Fresh-seed cohort of the minimal learned-representation bridge. Representation constraints shift the basin boundary toward each class’s population prediction, with the capacity control at 0 of 11,700 trapped runs (A). The visitation-floor intervention moves the boundary in the predicted direction (B), where the dashed equality line is a secondary reference and the registered gate tests direction.

Gate	Registered estimand	Bound	Measured	Verdict
G1	trapped runs, three-dimensional control	$0$	$0$ of $779$	pass
G2	basin agreement with $\operatorname{sign}(p_{0}-p_{\delta})$	$\geq 0.99$	$746$ of $746$	pass
G3	separator error over $19$ preferences, $\kappa=80$	$\leq 0.02$	$0.0044$	pass
G4	critical temperature inside the computed bracket	$3$ of $3$	$3$ of $3$	pass
G5	floor-mapped separator error, four points	$\leq 0.02$	$<0.00063$	pass
G6	fixed-start escape point against $\lambda_{\rm crit}$	$\leq 0.1$	$0.169$ , $0.101$	fail

Appendix

Table 6: Registered gates for the solved-class loop E2. All separator and threshold errors are absolute deviations from the computed values of Sections 2 and 5 .

Gate	Registered estimand and bound	Measured	Verdict
B1	full-rank control trap rate $\leq 0.005$	$0$ of $11{,}700$	pass
B2	constrained boundary error CI within $\pm 0.02$ , six cells	$6$ of $6$ , widest $\pm 0.007$	pass
B3	unconstrained boundary error CI within $\pm 0.03$ , six cells	$6$ of $6$ , all within $\pm 0.023$	pass
B4	floor shift CI lower bound $>0$ , six pairs	$6$ of $6$	pass
B5	collision-pair order $\geq 0.95$ per side	$1.000$ / $0.999$	pass
B6	forced-gap sign $\geq 0.95$ per side	$0.992$ / $1.000$	pass

Appendix

Table 7: Registered gates for the sampled bridge, with 35,100 records from 50 fresh seeds. Per-side entries read trapped/escaped and pool the two-dimensional arms. Boundary intervals resample seeds jointly across matched arms and floors and use the 2.5th and 97.5th percentiles. The return-gate resampling defect is described below.

Gate	Frozen estimand and bound	Measured	Verdict
F1	occupied-branch error $\leq 1.10\times$ achievable rank-two reference; lower bound $>0.8$	$96$ of $96$ , $100$ of $100$	pass
F2	trap rate given low start, lower bound $>0.8$	$0.96$ , lower bound $0.901$	pass
F3	escape rate given high start, lower bound $>0.8$	$1.00$ , lower bound $0.964$	pass
F4	rank-64 trapped runs, at most $1$ of $100$ per arm	$0$ and $0$	pass
F5	cross-talk order on both sides, lower bound $>0.5$	$96$ of $96$ , $100$ of $100$	pass
F6	forced-gap sign on both sides, lower bound $>0.5$	$96$ of $96$ , $100$ of $100$	pass

Appendix

Table 8: Registered gates for the final PPO cohort, with 100 fresh seeds per main arm. Lower bounds are the lower endpoints of exact two-sided 95% binomial intervals. Return contrasts use 95% seed-bootstrap intervals. The two endpoint-conditioned entries in F1, F5 and F6 concern 96 trapped low-start runs and 100 escaped high-start runs.

Seed	$r$	MC gap	Analytic gap	Final $p_{E}$	Final $J$	Max Gram error
0	0.4	-0.492603	-0.499766	0.05	-0.205030	0.000787
0	0.6	1.487152	1.499311	0.95	0.244970	0.001044
1	0.4	-0.487099	-0.498816	0.05	-0.205024	0.002418
1	0.6	1.492229	1.498647	0.95	0.244970	0.001726
2	0.4	-0.501893	-0.499512	0.05	-0.205016	0.000907
2	0.6	1.497788	1.498831	0.95	0.244974	0.001812

Appendix

Table 9: Matching study by dataset seed and exposure. Gap, policy and return entries average the four nested initializations; the last column is the largest squared-Gram error among those four fits. The Monte Carlo gap is measured before continuation, and the analytic gap uses the same fitted codes. No initialization or evaluation block is treated as an independent dataset seed.

Features $\to$ dimensions	Maximum fitting excess	Initial $E-S$ $S$ -fitted	Initial $E-S$ $E$ -fitted	Final return $E$ -fitted $-$ $S$ -fitted
$16\to 2$	$9.25\times 10^{-5}$	$-0.21873$	$0.65623$	$0.21875$
$16\to 4$	$2.30\times 10^{-4}$	$-0.06246$	$0.18746$	$0.06250$
$32\to 2$	$7.50\times 10^{-5}$	$-0.11718$	$0.35155$	$0.11719$
$32\to 4$	$1.98\times 10^{-4}$	$-0.03643$	$0.10935$	$0.03646$

Appendix

Table 10: Nearly optimal fits yield opposite return rankings across four dimension settings. Final return differences compare the common-start continuations after 4,096 updates.

Figure 22: The branch preferences in Figure 3 b persist throughout continued learning. Curves and bands summarize all 40 continuations per fitting condition in each setting.

Figure 23: Complete trajectories for Figure 10 , including initialization. Both ranks use

p_{0}=0.1

and 20 training seeds. Means and pointwise 95% seed-bootstrap intervals use the same seeds and saved checkpoints throughout. Initial returns are strongly negative before either controller learns the task.

Initial $p_{E}$	Rank 2	Rank 64
0.1	$0.606\ [0.600,0.612]$	$1.673\ [1.666,1.679]$
0.5	$0.605\ [0.600,0.610]$	$1.676\ [1.669,1.684]$
0.9	$0.606\ [0.600,0.612]$	$1.675\ [1.666,1.683]$

Appendix

Table 11: Ordinary training from Hub, 20 seeds per cell and 16,777,216 phases per run. Entries are stochastic original-Hub return with pointwise 95% seed-bootstrap intervals. This cohort has a different training history and budget from Table 3 .

Figure 24: Paired final return differences for Table 3 . Points and bars show means and pointwise 95% bootstrap intervals over 20 complete training-seed blocks.

Study	Observed result	Interpretation
Late fixed-mask exposure	Five forks/control runs return $0.615$ – $0.628$ ; no observed high exits.	Mask12/13 exposure with trainable/frozen encoders did not recover this mature narrow checkpoint.
Late balanced-E exposure	Five forks/control runs return $0.612$ – $0.636$ ; no observed high exits.	Balanced masks likewise did not produce late recovery in the tested budget.
Fresh balanced-E pilot	Hub follow-up returns $0.445$ , with root E MSE $0.620$ and no observed high exits.	Useful auxiliary learning alone did not maintain downstream behavior.

Appendix

Table 12: Retained sequential development failures, all using seed 1. Late interventions start from the 8,388,608-phase ordinary-training pilot checkpoint. Fixed-mask forks receive 4,194,304 additional phases; balanced-E forks receive 8,388,608. The fresh pilot receives 4,194,304 pretraining and 4,194,304 Hub-only phases. These results do not establish impossibility.

Code	$\sigma$	Controller	$\Delta q_{\rm O}$	$\Delta q_{\rm B}$	$J_{\rm O}$	$J_{\rm B}$
Exact $S$	0.1	Tied	$-0.4990$	$-0.4990$	$-0.0699$	$-0.0699$
		OLS	$-0.3216$	$-0.0261$	$-0.0593$	$-0.1070$
		Pop. linear	$-0.3207$	$-0.0259$	$-0.0593$	$-0.1071$
		MLP	$-0.0812$	$0.0098$	$-0.0378$	$-0.0484$
		Bayes	$-0.0797$	$0.0106$	$-0.0374$	$-0.0481$
Exact $S$	0.3	Tied	$-0.4992$	$-0.4992$	$-0.2300$	$-0.2300$

Appendix

Table 13: Independent controllers and fitting exposure change action values through frozen codes. Means use twenty seed blocks at

\delta=0.5

; O/B denote original/balanced fitting, with returns at the original policy.

$n$	$k$	Controller	$p_{0}$	Final $p(E)$	$D_{S}$	$D_{E}$	$\Delta q$	$J_{\rm sampled}$
16	8	Linear	$0.1$	$0.363$	$0.345$	$0.414$	$0.0028\pm 0.0204$	$-0.3579\pm 0.0119$
16	8	Linear	$0.9$	$0.949$	$0.416$	$0.321$	$0.1666\pm 0.0084$	$-0.2723\pm 0.0090$
16	8	MLP	$0.1$	$0.500$	$0.363$	$0.422$	$0.0124\pm 0.0330$	$-0.3675\pm 0.0120$
16	8	MLP	$0.9$	$0.949$	$0.424$	$0.347$	$0.1484\pm 0.0109$	$-0.3008\pm 0.0076$
16	16	Linear	$0.1$	$0.883$	$0.175$	$0.174$	$0.0726\pm 0.0005$	$-0.1206\pm 0.0002$
16	16	Linear	$0.9$	$0.949$	$0.176$	$0.174$	$0.0734\pm 0.0014$	$-0.1158\pm 0.0003$

Appendix

Table 14: Capacity improves return, while negative-gap estimates occur only at low-start

n=16,k=8

. All 28 conditions report five-seed means; gap and sampled return include sample SD.

		Success by training seed (%)			Mean	Mean
Method	$k$	0	1	2	success (%)	return
PPO	4	0.0	0.0	0.0	0.0	0.000
PPO + entropy	4	0.0	3.0	0.5	1.2	0.004
PPO	16	0.5	0.0	0.5	0.3	0.002
PPO + entropy	16	39.0	26.0	12.0	25.7	0.161
PPO	64	0.0	0.0	0.0	0.0	0.000

Appendix

Table 15: DoorKey

8\times 8

: mean success with entropy regularization is nonmonotone in actor rank. Final success (%) and original-task return for all three training seeds per condition; the last row is the auxiliary PFO comparison.

$k$	0	1	2	success (%)	return
	Success by training seed (%)			Mean	Mean
16	100.0	100.0	100.0	100.0	0.937
128	99.5	100.0	49.5	83.0	0.742

Appendix

Table 16: The smaller DoorKey task is learnable at rank 16. Final success (%) and return on

5\times 5

, three training seeds per rank.

Rank	Encoder	Native	Stage mixture	Factor mixture
16	Trainable	$22.5\,/\,0.1357$	$30.5\,/\,0.2034$	$23.2\,/\,0.1464$
16	Frozen	$35.3\,/\,0.2444$	$40.7\,/\,0.2789$	$28.0\,/\,0.1748$
128	Trainable	$12.5\,/\,0.0780$	$20.8\,/\,0.1294$	$20.5\,/\,0.1259$
128	Frozen	$20.2\,/\,0.1356$	$25.5\,/\,0.1690$	$19.5\,/\,0.1170$

Appendix

Table 17: DoorKey continuation outcomes at fixed capacity. Entries give mean native-start success (%) / original return over three parents. Every arm has 262,144 additional learner transitions. Native and broad stage controls are shared with the subsequent factor study.

		Trainable			Frozen
Rank	Seed	Native	Stage	Factor	Native	Stage	Factor
16	0	43.0	51.5	18.0	54.5	52.5	34.0
16	1	7.0	20.5	14.5	28.5	43.5	33.0
16	2	17.5	19.5	37.0	23.0	26.0	17.0
128	0	20.5	31.0	27.5	31.0	35.0	20.0
128	1	9.5	12.5	10.5	18.0	26.5	17.5

Appendix

Table 18: Every parent-level native-start success rate (%). T/F denotes trainable/frozen sensory encoder; the actor head and critic learn in both conditions. No parent is excluded.

Rank	Encoder	Optimal-action probability (%)	Excess steps	Sensitivity (pp)
16	Trainable	$22.0\to 40.3$	$1.119\to 0.839$	$8.5\to 19.9$
16	Frozen	$29.0\to 39.5$	$1.012\to 0.782$	$11.6\to 9.6$
128	Trainable	$29.3\to 39.2$	$0.994\to 0.819$	$9.4\to 10.2$
128	Frozen	$27.0\to 35.4$	$0.993\to 0.859$	$5.1\to 11.9$

Appendix

Table 19: Factor-mixture versus native continuation on the fixed panel. Each arrow gives the native and factor means over three parents. Sensitivity is an absolute probability difference in percentage points; excess steps assume optimal continuation after one sampled action.

Exposure	No pickup	Pickup, no opening	Opened, no goal	Success
Native	907	818	132	543
Stage mixture	893	706	96	705
Factor mixture	1,019	722	112	547

Appendix

Table 20: Failure-stage decomposition of all 7,200 final native-start episodes, with 2,400 per exposure across twelve endpoints. Counts are evaluation outcomes, not independent training replications.

Learner	Intervention	Finger-turn-hard	Hopper-hop
TD-MPC2	Baseline	$694.9\,\pm\,15.2$	$97.2\,\pm\,85.5$
	Phase replay	$780.6\,\pm\,41.6$	$\mathbf{133.2}\,\pm\,74.2$
	Overlap $\lambda=0.003$	$759.0\,\pm\,62.6$	$108.5\,\pm\,78.0$
	Overlap $\lambda=0.03$	$\mathbf{814.8}\,\pm\,43.9$	$60.2\,\pm\,47.4$
	Prior entropy $\times 2$	$738.4\,\pm\,64.3$	$114.7\,\pm\,57.6$
	Prior entropy $\times 5$	$799.9\,\pm\,75.5$	$105.3\,\pm\,105.2$

Appendix

Table 21: Original continuation results across all three training seeds. Entries are mean

\pm

sample SD of the three checkpoint means, each estimated from 50 episodes. Unlike Table 1 , this table summarizes variation across training runs.

Continuation	Finger-turn-hard	Hopper-hop
Entropy off throughout	$244.5\,\pm\,41.1$	$49.8\,\pm\,1.3$
Entropy restored	$171.0\,\pm\,51.3$	$29.5\,\pm\,1.5$

Appendix

Table 22: Entropy restoration from SAC parents trained with entropy disabled. Each selected checkpoint uses the five-evaluation protocol of Table 1 . These parents are separate from standard SAC.

Seed/statistic	Uniform	Protected	Difference
0	7.968	8.229	0.261
1	7.578	8.862	1.284
2	7.943	8.201	0.258
Mean	7.830	8.431	0.601
SD	0.218	0.374	0.592

Appendix

Table 23: Cumulative training Crafter scores at one million steps. Differences are protected minus uniform. SD is the sample standard deviation across three paired seeds.

Representation Learning for RL

Representation learning for reinforcement learning builds state, task, and dynamics features that help agents learn, explore, and transfer.

See all 69 papers

A Structural Threshold in Decision Capacity Governs Collapse in Self-Play Reinforcement Learning
May 4, 2026Arahan KujurGame-Playing Agents Self-Play RL
Independent Researcher
Regret-Aware Policy Optimization: Environment-Level Memory for Replay Suppression under Delayed Harm
Apr 8, 2026Prakul Sunil HiremathReinforcement Learning Safe RL
Department of Computer Science and Engineering, VTU, Belagavi, India · Aliens on Earth (AoE) Autonomous Research Group, Belagavi, India
MInTRL: Off-policy Intervention can boost On-policy RL
Sep 11, 2026Mingyu Chen, Yefan Tao, Gerald Friedland +2Reinforcement Learning Reinforcement Learning with Verifiable Rewards
Amazon Web Services · Boston University

May 4, 2026J/K move · Enter open · S save

A Structural Threshold in Decision Capacity Governs Collapse in Self-Play Reinforcement Learning

Arahan Kujur

Independent Researcher

Game-Playing Agents Self-Play RL Game Theory Multi-Agent Reinforcement Learning Game-Theoretic RL

Open paper AskPDF arXiv

We show that a threshold in decision capacity determines whether self-play reinforcement learning agents collapse under asymmetric rule perturbations. Across poker variants, matrix games, a dice game, and multiple learning algorithms, eliminating all positive-reach contingent decisions causes rapid convergence to a deterministic exploitation attractor, a fixed point at near-maximal loss. Preserving even a single positive-reach contingent decision point prevents this collapse. A frozen baseline and fixed-opponent control confirm that the mechanism is co-adaptation under constraint, not the perturbation itself. The phenomenon is timing-invariant, fully reversible upon action restoration, and intensifies under function approximation. These results establish a sharp threshold at zero reach-weighted contingent action capacity, with severity scaling continuously via reach-weighted capacity in the tested domains.

Abstract

Representation Learning for RL

Explore similar work

A Structural Threshold in Decision Capacity Governs Collapse in Self-Play Reinforcement Learning

Regret-Aware Policy Optimization: Environment-Level Memory for Replay Suppression under Delayed Harm

MInTRL: Off-policy Intervention can boost On-policy RL