Self-discovering RL in the Era of Experience: Is Learning History an Asset or a Burden?

Haomin Luo

cs.AISep 27, 2026

Self-discovering RL in the Era of Experience: Is Learning History an Asset or a Burden?

Authors: Haomin Luo

Organizations: University of Cambridge · Models2 AI

Abstract

The pursuit of recursive self-improvement (RSI) toward general intelligence is divided between macro-level language model scaling and the interaction-driven principles of "Era of Experience". Yet, any self-improving architecture ultimately rests upon its underlying optimization engine: if general intelligence requires learning from grounded interaction, the reinforcement learning (RL) update rule itself must be capable of cumulative adaptation. While algorithm self-discovery has produced Disco103 that surpassed PPO to achieve SOTA benchmark performance -- its internal update machinery remains an uninspected black box. We present the first causal mechanistic audit of a self-discovered RL rule, structured directly around the five pillars of the Era of Experience: extended horizon, grounded reward scales, continuing streams, within-lifetime change, and exploration depth. By surgically pinning, freezing, and transplanting recurrent states while holding meta-parameters fixed, we test when learning history acts as an asset or a burden. Three findings organize the audit: (1) Recurrent history actively expands usable reward scales, sustaining a six-decade window versus three under zero-pinning. (2) Decoupling historical content from its maintenance reveals that the penalty of mismatched history stems from perpetual clamping; allowing imported state to evolve naturally attenuates this burden. (3) Under environmental change, controlling replay retention reverses the apparent adaptation advantage over DQN, demonstrating that external data turnover can confound internal plasticity. Validated through capability thresholds and ported to a second rule (OPEN), this work grounds macro-RSI ambitions in micro-level learning dynamics, establishing a foundational audit standard for next-generation, self-evolving RL algorithms.

Figures & tables

Figure 1: From the Era of Experience to a five-axis causal audit. Five demands become operational transformations of the learning loop. Rule weights

\eta

remain fixed while agent parameters and recurrent history

(h,c)

evolve. State interventions, value-interface analysis, and replay controls distinguish internal history, numerical representation, and external memory; boundary-signal and discovery-event controls complete the profile.

Figure 2: Interventions reach from learning history to numerical meaning. (A) Zero-pinning and timed freezing control the availability and acquisition time of history while agent parameters update. (B) A 400-iteration donor supplies hidden and cell state to a fresh 2,000-iteration recipient. Natural transfer initializes once and then evolves; clamping restores the donor pair after every iteration. Matched and cross-scale donors complete the comparison. (C) The detached target is transformed, clipped, and projected onto neighboring atoms; scalar readout applies the inverse transform after expectation. Appendices A and Q provide execution details and derivations.

Axis	Question	Readout	Control
A1 Horizon	How does progress accumulate?	Endpoint; own-target mastery	Rate sweep; freeze time
A2 Reward units	Which magnitudes remain usable?	Contiguous scale window	State/EMA; donor; interface
A3 Streams	What changes without resets?	Continuing reward rate	Masking; synthetic boundaries
A4 Change	How is competence rebuilt?	Levels; AUC; attainment	Change type; replay; state
A5 Exploration	When does experience inform?	Discovery; sustained solution	Depth; entropy; random policy

Table 1: Five axes form a capability profile. Each axis has a distinct question, readout, and explanatory control.

Figure 3: History, reward units, and numerical geometry. (A) Reference-indexed Catch, 12 seeds per cell, integer-trimmed final-20% scores with pointwise 95% intervals. (B) Absolute-capability window widths and pointwise marginal intervals across three environments. (C) Low-scale matched/cross donors under natural transfer or repeated clamping, 12 seeds and pointwise score intervals. The interaction aggregates within-seed difference-of-differences before trimming and uses a family-11 interval (Equation 8 ). (D) Unit-scale interface assay: score (zoomed axis) and effective bins, each with pointwise 95% seed-bootstrap intervals. Full grids, both recipient scales, all eight inheritance conditions, and the analytical interface appear in the appendix.

Figure 4: A capability profile across time, streams, change, and exploration. (A) Reference CatchBig: endpoint and time to each run’s own attained-level target, seven configurations, 12 seeds. (B) CatchStream (

n=12

) and Tracking (

n=18

): normalized reward rate with pointwise 95% seed-bootstrap intervals. (C) Breakout action reversal: pre/final absolute returns, four arms and 18 seeds per arm. (D) Reference DeepSea staircase: four algorithms and a Disco entropy ablation, five depths, 15 seeds per cell (375 runs); solved fractions use final-window threshold 0.495, with the dotted line marking majority solution. Complete tables and controls appear in the atlas.

Appendix figures & tables53 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Lifetime state	Other recipient components
Live	Update	Update
Zero-pinned	Restore zero after each iteration	Update
Frozen at $t_{f}$	Restore own state captured at $t_{f}$	Update
Natural transfer	Initialize with donor; then update	Fresh recipient; then update
Perpetual clamp	Initialize with donor; restore each iteration	Fresh recipient; then update
OPEN reset-once	Zero both GRU carries once; then update	Preserve inherited agent and other optimizer state

Appendix

Table 2: State-operation semantics. “Update” means the component follows its usual learner update. Transplantation transfers the lifetime pair only.

Test family	Outputs	Max. absolute error
Forward outputs	26	$0.002304$
Inner outputs/gradients	16	$4.470\times 10^{-5}$
Updated parameters	12	$2.733\times 10^{-4}$
Value estimator	9	$5.471\times 10^{-14}$
Meta-gradient	42	$4.170\times 10^{-7}$
Float32/float64 control	42	$2.945\times 10^{-7}$

Appendix

Table 3: Cross-framework numerical checks on shared inputs. Errors are maximum absolute discrepancies over the recorded outputs in each family. Updated parameters are measured after one Adam step; the final row compares precisions within PyTorch.

Task	Criterion	Live	Pin	EMA	Both	$\Delta W$	Adjusted CI
Catch	Half-best	6	3	4	3	3	[2, 3]
Catch	$S\geq 0.8$	6	3	4	3	3	[2, 3]
Catchdense	Half-best	6	4	5	4	2	[2, 2]
Catchdense	$S\geq 0.8$	6	3	4	3	3	[2, 3]
Catchlong	Half-best	5	2	3	2	3	[0, 4]
Catchlong	$S\geq 0.8$	3	2	3	2	1	[1, 3]

Appendix

Table 4: All scale-window estimates, 12 seeds per cell. Integer-trimmed and exact-IQM results coincide for these 12-seed summaries. Half-best rows belong to the primary family; absolute rows to its threshold-sensitivity companion. Intervals use the corresponding family-11 tails.

Figure 5: Reference-indexed Catch scale response and window uncertainty. Twelve seeds per condition; curves show final-20% normalized scores and pointwise 95% bootstrap bands. Width intervals are pointwise; the live-minus-pinned difference is three decades with adjusted interval

[2,3]

. The source arrays and complete environment table retain both absolute and half-best criteria.

Condition	$10^{-3}$	95% CI	$10^{3}$	95% CI
live	0.6785	[0.345, 0.905]	0.9961	[0.992, 0.998]
ownfreeze400	0.6735	[0.417, 0.891]	0.9926	[0.990, 0.994]
match clamp	0.9326	[0.700, 0.982]	0.9952	[0.992, 0.997]
match init	0.8971	[0.690, 0.982]	0.9937	[0.990, 0.997]
cross clamp	0.0165	[0.004, 0.068]	0.9954	[0.993, 0.998]
cross init	0.5307	[0.201, 0.868]	0.9957	[0.991, 0.999]

Appendix

Table 5: All eight state conditions at each recipient scale, 12 seeds per cell. Tail scores use the common final-20% grid; intervals are pointwise 95%. Execution settings are specified in Appendix A .

Figure 6: Full inheritance trajectories at both recipient scales. Matched donors use the recipient scale; cross donors use the opposite extreme. Donors train for 400 iterations and recipients for 2,000. Twelve seeds per cell; curves and marginal scores have pointwise 95% intervals. Low-scale interaction

0.39265

[0.07896,0.86094]

and high-scale interaction

0.00236

[-0.00783,0.00889]

use the family-11 procedure, estimated separately within each recipient stratum.

Change / replay	Trim	Adjusted CI	IQM	Adjusted CI
action flip / 100k	0.5161	[0.427, 0.584]	0.5159	[0.428, 0.585]
action flip / 28,672	-0.0898	[-0.175, -0.030]	-0.0897	[-0.176, -0.030]
action flip / clear	-0.3154	[-0.403, -0.253]	-0.3163	[-0.402, -0.252]
reward move / 100k	0.4841	[0.440, 0.516]	0.4856	[0.439, 0.516]
reward move / 28,672	0.0701	[0.030, 0.125]	0.0689	[0.030, 0.128]
reward move / clear	-0.0901	[-0.151, -0.042]	-0.0893	[-0.153, -0.042]

Appendix

Table 6: All six planned Disco-minus-DQN AUC comparisons, 18 seeds, family-11 adjusted intervals. Exact middle-50% IQM is the estimator-sensitivity companion.

Change	System	Threshold	Attained	RM time ( $10^{3}$ )
action flip	disco	0.8	18/18	46.3
action flip	disco	0.9	17/18	60.9
action flip	DQN clear	0.8	18/18	9.6
action flip	DQN clear	0.9	18/18	12.1
action flip	DQN 28,672	0.8	18/18	31.2
action flip	DQN 28,672	0.9	18/18	32.7

Appendix

Table 7: Common-capability attainment, fixed origin at the switch. Restricted mean time includes censored runs at the 87,000-interaction horizon; an observed threshold crossing is an attainment event, not a sustained-return guarantee.

Figure 7: Replay controls and post-switch learning trajectories. Catch

8\times 8

, 18 seeds, switch at 87,000 interactions. Left: normalized training-return streams with pointwise intervals. Right: paired Disco-minus-DQN post-switch AUC, with family-11 adjusted intervals. The capacities are 100,000 and 28,672 transitions; the cleared configuration uses capacity 100,000 and removes its contents at the switch. Update schedules remain those of each system. Fixed-origin threshold analyses use a completed-episode washout.

Contrast	Game	Scale	Estimate	Adjusted CI
Live - zero	breakout	0.1	0.001087	[-0.000839, 0.004019]
Live - zero	breakout	1.0	0.002232	[-0.001190, 0.004171]
Live - zero	breakout	10.0	0.001520	[0.000607, 0.002340]
Live - zero	freeway	0.1	0.000322	[-0.001213, 0.000946]
Live - zero	freeway	1.0	0.000099	[-0.000380, 0.000843]
Live - zero	freeway	10.0	0.000267	[-0.000591, 0.001192]

Appendix

Table 8: OPEN terminal-state contrasts, eight seed blocks per cell. The original family-34 adjustment and 200,000 bootstrap draws are retained. Units are unscaled evaluation reward per interaction.

Contrast	Game	Scale	Estimate	Adjusted CI
Live - initial	breakout	0.1	0.038862	[0.037537, 0.041191]
Live - random	breakout	0.1	0.038857	[0.037670, 0.040970]
Live - initial	breakout	1.0	0.039761	[0.038012, 0.040659]
Live - random	breakout	1.0	0.039755	[0.038452, 0.040359]
Live - initial	breakout	10.0	0.039524	[0.038656, 0.040489]
Live - random	breakout	10.0	0.039518	[0.038597, 0.040272]

Appendix

Table 9: OPEN terminal capability checks, eight seed blocks per cell. Family-34 adjusted intervals use 200,000 bootstrap draws. Units are unscaled evaluation reward per interaction.

Breakout
Scale	OPEN $-$ Adam	Disco $-$ OPEN	Disco $-$ Adam
0.01	-0.00190 [-0.00533, 0.00194]	0.00976 [0.00094, 0.02234]	0.00786 [-0.00139, 0.02124]
0.1	-0.00128 [-0.00931, 0.00161]	0.00518 [0.00056, 0.01644]	0.00389 [-0.00588, 0.01504]
1	-0.00017 [-0.00676, 0.00156]	0.00369 [-0.00057, 0.01234]	0.00352 [-0.00558, 0.01271]
10	-0.00204 [-0.00286, -0.00031]	0.00397 [-0.00038, 0.01603]	0.00194 [-0.00236, 0.01393]
100	-0.00169 [-0.00258, -0.00097]	0.00288 [0.00165, 0.00448]	0.00119 [-0.00014, 0.00234]

Appendix

Table 10: Task-conditional system profiles at equal interaction budgets. All 30 contrasts use eight seeds per cell and the original family-30 adjusted intervals. Each cell gives the difference of marginal final-window trimmed means, followed by its interval in unscaled reward-per-interaction units. Adam denotes PPO with Adam. This cohort is distinct from the OPEN carry interventions.

Depth	Learner	Found	Wilson CI	Tail success	95% CI
12	disco	11/15	[0.480, 0.891]	0.517	[0.152, 0.866]
12	a2c	15/15	[0.796, 1.000]	0.889	[0.445, 1.000]
12	dqn	13/15	[0.621, 0.963]	0.737	[0.570, 0.757]
12	random	100/100	[0.963, 1.000]	0.000	[0.000, 0.000]
14	disco	6/15	[0.198, 0.643]	0.000	[0.000, 0.282]
14	a2c	12/15	[0.548, 0.930]	0.000	[0.000, 0.000]

Appendix

Table 11: DeepSea discovery and final-window success: 15 runs per learner cell and 100 random-policy campaigns per depth. Discovery intervals are Wilson 95%; tail intervals are pointwise seed bootstrap. The horizon is 12,000 episodes at each depth.

Contrast	Scale	Estimate	Pointwise 95%	Family-15 interval
$\Delta_{\mathrm{Default}}$	$10^{-3}$	0.8225	[0.5456, 0.9301]	[0.3536, 0.9768]
$\Delta_{\mathrm{Default}}$	$1$	0.0024	[-0.0069, 0.0109]	[-0.0165, 0.0186]
$\Delta_{\mathrm{Default}}$	$10^{3}$	0.0024	[-0.0024, 0.0073]	[-0.0054, 0.0093]
$\Delta_{\mathrm{S30}}$	$10^{-3}$	0.9615	[0.8220, 0.9828]	[0.6680, 0.9843]
$\Delta_{\mathrm{S30}}$	$1$	0.0035	[-0.0003, 0.0098]	[-0.0041, 0.0140]
$\Delta_{\mathrm{S30}}$	$10^{3}$	-0.0004	[-0.0031, 0.0013]	[-0.0045, 0.0025]

Appendix

Table 12: Complete P1 interface-by-state analysis: all 15 final-20% (tail20) contrasts from 216 runs, with 12 paired seeds per cell and 10,000 seed-bootstrap draws.

\Delta_{i}

is the live-minus-zero state effect for interface

i

; interaction rows compare those per-seed effects before aggregation. Default and S30 use signed-hyperbolic supports 300 and 30; Linear uses linear support 300. The last two columns separately retain pointwise 95% intervals and Bonferroni-adjusted intervals for this secondary exploratory family of 15 contrasts. Integer-trimmed means and exact middle-50% IQM coincide at the displayed precision.

Cell	$n$	Legacy gap [95% CI]	Reference gap [95% CI]	Shift [95% CI]
A1 catchbig, disco-A2C	12	-0.39 [-0.59333, -0.17658]	-0.016667 [-0.063333, 0.013333]	0.34333 [0.10333, 0.59008]
A2 catch x 1e-3, disco-pinned@0	12	0.00166 [0.0015133, 0.00174]	0.0017033 [0.00164, 0.0017533]	$3.3333\!\times\!10^{-5}$ [ $-2.6667\!\times\!10^{-5}$ , 0.00017667]
A2 catch x 1, disco-pinned@0	12	0 [0, 0.0066667]	0 [0, 0.0066667]	0 [-0.01, 0.0066667]
A2 catch x 1e3, disco-pinned@0	12	826.67 [3.3333, 1666.7]	1413.3 [546.67, 1730]	56.667 [-20, 886.67]
A3 masked, disco-A2C	18	0.249 [0.24635, 0.25125]	0.2487 [0.24575, 0.2511]	-0.00015 [-0.00205, 0.0005]
A4 action-flip 8x8, disco-A2C	18	0.95 [0.79595, 1.108]	0.95 [0.802, 1.082]	-0.01 [-0.074, 0.064]

Appendix

Table 13: Paired sensitivity to the Retrace convention. Each row retains its source metric and paired contrast. Scale cells use raw reward units; other cells use their reported normalized score or return metric. The column

n

gives the reported seed/run count.

Assay	Conditions	Seeds	Iterations
E-A4m	Breakout: 2 protocols $\times$ 4 arms	18	8,000
E1	3 tasks $\times$ 7 scales $\times$ 4 state/EMA arms	12	2,000
E2	2 recipient scales $\times$ 8 inheritance conditions	12	2,000
E3	27 grid/change/agent/replay cells	18	3,000
E-A5	5 depths $\times$ 5 arms	15	1,655–3,310
E-LR	3 baselines $\times$ 4 learning rates	12	20,000

Appendix

Table 14: Reference-indexed matrices. Counts are seeds per algorithm–condition cell. Each iteration uses 58 interactions. Transplants additionally train a donor for 400 iterations; DeepSea budgets scale with depth.

Comparator	Endpoint	Mastery	Endpoint difference	Mastery difference
Reference: Disco103 ; endpoint 0.9739, mastery 133.4
Disco $3\!\times\!10^{-4}$	0.9948	500.2	-0.02087 [-0.03652, -0.001739]	-345.6 [-410.4, -264.4]
A2C	0.9704	350.9	0.008696 [-0.01913, 0.02435]	-196.7 [-249.4, -131.9]
PPO	0.9078	245.5	0.06609 [0.04, 0.09043]	-110.2 [-149.8, -56.54]
DQN	0.8835	262	0.08522 [0.05739, 0.1217]	-119.9 [-149.3, -73.47]
Pinned 0	0.7896	275	0.1826 [0.1391, 0.247]	-122.8 [-201.6, -59.93]

Appendix

Table 15: Reference-indexed CatchBig: endpoint and mastery. Groups identify the reference arm: Disco103 at learning rate

10^{-2}

, or the

3\times 10^{-4}

arm. Each row compares that reference with a comparator, using 12 paired seeds. Differences are reference minus comparator, with 95% intervals. Mastery uses each run’s own final-score target, in

10^{3}

environment steps. Component IQMs and paired-difference IQMs are distinct estimands.

Figure 8: Learning-rate and normalization sensitivity. A–C show all baseline rates, 12 seeds per cell; black rings identify measured default settings. D compares A2C and A2C-PopArt at all three scales. Points are tail IQMs with pointwise 95% seed-bootstrap intervals; lines connect tested settings.

Figure 9: Reward-scale response, usable windows, and terminal-state geometry. (A–C) The E1 state/EMA factorial contains four arms at seven scales in each environment, with 12 seeds per cell. Curves use the same final-20% scores and pointwise 95% seed-block intervals as Figure 3 . (D–F) Widest contiguous passing spans at absolute scores 0.7, 0.8, and 0.9, with pointwise window intervals. At 0.8, live/pinned widths are 6/3, 6/3, and 3/2 decades. (G–I) Live hidden/cell-state centroid displacement from unit scale, normalized by the corresponding unit-scale radius; geometry uses terminal states from the same runs.

Environment	$R_{\rm random}$	$R_{\rm optimal}$	Reference span
Catch	-0.75000	1.00000	1.75000
CatchDense	-3.04774	0.75000	3.79774
CatchLong	-0.87500	1.00000	1.87500

Appendix

Table 16: Reward-scale normalization references. Scores are

(R/s-R_{\rm random})/(R_{\rm optimal}-R_{\rm random})

with no clipping;

s

is removed exactly once. CatchDense uses its fixed 40,000-episode reference estimate.

Environment	State	Score $\geq 0.7$	Score $\geq 0.8$	Score $\geq 0.9$
Catch	Live	6 [5, 6]	6 [5, 6]	5 [5, 6]
Catch	Pin 0	3 [3, 3]	3 [3, 3]	3 [3, 3]
Catch	EMA held	4 [4, 4]	4 [4, 4]	4 [4, 4]
Catch	Both	3 [3, 4]	3 [3, 4]	3 [3, 3]
CatchDense	Live	6 [6, 6]	6 [6, 6]	5 [5, 6]
CatchDense	Pin 0	4 [3, 4]	3 [3, 4]	3 [3, 3]

Appendix

Table 17: Absolute usable scale windows in decades. Each value is the widest contiguous passing segment of the seven tested multipliers; bracketed intervals are pointwise 95% seed-block bootstrap intervals. The three thresholds are sensitivity views of the same final-20% scores and the same 1,008 runs, not additional experiments.

Environment	State	Width [95% interval]	Point threshold
Catch	Live	6 [6, 6]	0.498
Catch	Pin 0	3 [3, 4]	0.497
Catch	EMA held	4 [4, 4]	0.500
Catch	Both	3 [3, 4]	0.499
CatchDense	Live	6 [6, 6]	0.498
CatchDense	Pin 0	4 [4, 4]	0.500

Appendix

Table 18: Half-best scale-window sensitivity using the same 1,008 runs. The threshold is half the best IQM score of the arm and is recomputed within each bootstrap draw; this relative criterion is shown alongside the absolute-capability windows, not substituted for them.

Scale	Live	Pin 0	EMA held	Both
$10^{-3}$	0.828 [0.553, 0.931]	0.004 [-0.003, 0.009]	0.005 [-0.003, 0.011]	0.005 [-0.002, 0.010]
$10^{-2}$	0.977 [0.956, 0.992]	0.012 [0.001, 0.021]	0.012 [0.003, 0.296]	0.006 [-0.002, 0.010]
$10^{-1}$	0.976 [0.944, 0.991]	0.176 [0.007, 0.555]	0.996 [0.993, 0.998]	0.419 [0.072, 0.829]
$10^{0}$	0.994 [0.985, 0.999]	0.991 [0.985, 0.995]	0.999 [0.998, 1.000]	0.995 [0.985, 0.998]
$10^{1}$	0.997 [0.995, 0.999]	0.993 [0.987, 0.997]	1.000 [0.999, 1.000]	0.994 [0.983, 0.997]
$10^{2}$	0.997 [0.995, 0.999]	0.988 [0.983, 0.992]	1.000 [0.998, 1.000]	0.991 [0.983, 0.997]

Appendix

Table 19: Catch scale response. Each cell gives the IQM of final-20% normalized scores over 12 seeds and its pointwise 95% seed-bootstrap interval. All 28 conditions use 116,000 interactions and reference indexing.

Scale	Live	Pin 0	EMA held	Both
$10^{-3}$	0.881 [0.838, 0.920]	-0.131 [-0.183, -0.023]	0.007 [-0.039, 0.025]	-0.019 [-0.145, 0.024]
$10^{-2}$	0.957 [0.938, 0.971]	0.126 [-0.027, 0.246]	0.586 [0.244, 0.778]	-0.015 [-0.139, 0.034]
$10^{-1}$	0.972 [0.965, 0.980]	0.781 [0.644, 0.870]	0.801 [0.764, 0.835]	0.584 [0.377, 0.800]
$10^{0}$	0.988 [0.973, 0.994]	0.997 [0.992, 0.999]	0.921 [0.911, 0.930]	0.913 [0.910, 0.915]
$10^{1}$	0.996 [0.992, 0.999]	0.999 [0.995, 1.000]	0.999 [0.998, 1.000]	0.945 [0.944, 0.948]
$10^{2}$	0.997 [0.993, 0.998]	0.999 [0.998, 1.000]	0.999 [0.998, 1.000]	0.952 [0.936, 0.964]

Appendix

Table 20: CatchDense scale response. Each cell gives the IQM of final-20% normalized scores over 12 seeds and its pointwise 95% seed-bootstrap interval. All 28 conditions use 116,000 interactions and reference indexing.

Scale	Live	Pin 0	EMA held	Both
$10^{-3}$	0.144 [0.034, 0.429]	-0.002 [-0.008, 0.004]	0.005 [-0.001, 0.007]	0.003 [-0.002, 0.007]
$10^{-2}$	0.741 [0.457, 0.854]	0.001 [-0.007, 0.011]	0.002 [-0.005, 0.009]	0.003 [-0.002, 0.007]
$10^{-1}$	0.772 [0.470, 0.855]	0.014 [-0.001, 0.034]	0.159 [0.009, 0.575]	0.002 [-0.002, 0.006]
$10^{0}$	0.961 [0.926, 0.983]	0.338 [0.136, 0.538]	0.944 [0.892, 0.986]	0.275 [0.007, 0.682]
$10^{1}$	0.988 [0.986, 0.992]	0.996 [0.994, 0.997]	0.997 [0.994, 0.998]	0.971 [0.940, 0.984]
$10^{2}$	0.987 [0.983, 0.990]	0.998 [0.997, 0.999]	0.995 [0.993, 0.997]	0.992 [0.990, 0.994]

Appendix

Table 21: CatchLong scale response. Each cell gives the IQM of final-20% normalized scores over 12 seeds and its pointwise 95% seed-bootstrap interval. All 28 conditions use 116,000 interactions and reference indexing.

Environment	State	Scale	$h$ : shift / radius	$c$ : shift / radius
Catch	Live	0.001	2.92 [2.22, 4.82]	3.35 [2.55, 5.58]
Catch	Live	0.01	2.55 [1.95, 4.21]	3.26 [2.46, 5.43]
Catch	Live	0.1	2.32 [1.77, 3.67]	2.91 [2.21, 4.77]
Catch	Live	1	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]
Catch	Live	10	4.12 [2.68, 7.62]	8.39 [5.29, 16.29]
Catch	Live	100	10.88 [7.48, 19.13]	22.49 [14.88, 42.55]

Appendix

Table 22: Terminal-state displacement in the scale assay. For each environment and arm, the centroid displacement from unit scale is divided by the mean Euclidean radius of the 12 unit-scale states; bootstrap draws recompute that denominator. All 7 scales and 12 seeds are retained. Brackets are pointwise 95% intervals. The two pinned arms have identically zero states, so their normalized ratios are undefined rather than zero.

Environment	State	Vector	Between-scale fraction [95% interval]	PC1+2 fraction	Unit radius
Catch	Live	$h$	0.974 [0.966, 0.986]	0.970	0.113
Catch	Live	$c$	0.989 [0.983, 0.995]	0.994	1.372
Catch	Live	$hc$	0.989 [0.983, 0.995]	0.994	1.377
Catch	EMA held	$h$	0.941 [0.934, 0.964]	0.943	0.049
Catch	EMA held	$c$	0.991 [0.990, 0.994]	0.990	0.585
Catch	EMA held	$hc$	0.991 [0.990, 0.994]	0.990	0.587

Appendix

Table 23: Descriptive terminal-state geometry from the same scale assay. Each row contains 84 terminal states (7 scales by 12 seeds). Between-scale fraction is centroid sum of squares divided by total centered sum of squares. PCA uses unstandardized centered states. These are descriptive associations; pinned states have zero total variation and undefined PCA fractions.

Algorithm	Reward scale	$n$	Score [95% CI]
a2c	0.001	12	0.007746 [0.0025397, 0.011556]
a2c-popart	0.001	12	0.0054603 [0.00037776, 0.0093968]
a2c	1	12	0.99263 [0.9906, 0.99492]
a2c-popart	1	12	0.9746 [0.97244, 0.97879]
a2c	1000	12	0.0022857 [-0.0044508, 0.0062222]
a2c-popart	1000	12	0.99581 [0.94539, 0.99898]

Appendix

Table 24: E-POP: complete PopArt reward-scale comparison. The column

n

gives the reported seed/run count.

Scale	$n$	Descaled rate	Retention [95% CI]	Passing
Breakout / A2C ; unit-scale rate 0.08442
0.01	8	0.0652	0.7695 [0.7621, 0.7791]	no
0.1	8	0.08611	1.018 [1.003, 1.034]	yes
1	8	0.08442	1 [1, 1]	yes
10	8	0.08328	0.9848 [0.9687, 1.004]	yes
100	8	0.07725	0.916 [0.9023, 0.9316]	yes

Appendix

Table 25: MinAtar reward-scale response. Rate is reward per interaction after removing the multiplier. Retention forms each seed’s ratio to its unit-scale rate before IQM aggregation and the 95% interval. It is reported only for arms with unit-scale rate IQM at least

10^{-3}

; otherwise retention and passing status are N/A and absolute rates remain visible. Passing requires retention at least 0.8.

Environment	Algorithm	Usable scales	Low scale	High scale
minatar_breakout	a2c	4	0.1	100
minatar_breakout	disco	5	0.01	100
minatar_breakout	disco-noema	5	0.01	100
minatar_breakout	disco-zerornn	5	0.01	100
minatar_freeway	a2c	2	1	10
minatar_freeway	disco	3	1	100

Appendix

Table 26: MinAtar usable reward scales. N/A indicates a unit-scale reference below the

10^{-3}

reliability threshold.

Recipient scale	Condition	$n$	Tail score [95% interval]
0.001	Live	12	0.67855 [0.34494, 0.90503]
0.001	Own freeze at 400	12	0.67347 [0.41660, 0.89117]
0.001	Matched / init	12	0.89714 [0.68961, 0.98168]
0.001	Cross / init	12	0.53070 [0.20054, 0.86776]
0.001	Random / init	12	0.68372 [0.31915, 0.91384]
0.001	Matched / clamp	12	0.93261 [0.69950, 0.98213]

Appendix

Table 27: Inherited-state assay: all 192 recipient runs. Each arm uses 12 seeds at each recipient scale; scores are IQMs over final-20% normalized scores and brackets are pointwise 95% intervals. Init transfers donor

h,c

once and then permits evolution; clamp overwrites

h,c

each learner step. These marginal scores are distinct from the within-seed mismatch interaction, which is formed before aggregation.

Figure 10: Analytical and measured views of the value interface. (A) Analytical forward-transform values h(x), shown separately from measured occupancy. (B) An illustrative two-hot projection of transformed target 0.35 onto adjacent atoms 0 and 1; weights 0.65 and 0.35 preserve the first moment. (C–D) Catch interface experiments at scale 1, 12 runs per condition, with pointwise 95% seed-bootstrap intervals for IQM score and effective bins. (E) Executed scalar prediction and target paths: inverse transformation follows the categorical expectation; the target is forward transformed before clipping and two-hot projection. Panel B illustrates the mathematical projection.

Interface	$n$	Score [95% CI]	Effective bins [95% CI]	Score difference
default	12	0.99695 [0.98806, 0.99975]	2.2221 [2.1918, 2.2528]	0
linear	12	0.99949 [0.99517, 1]	1.4692 [1.4151, 1.5199]	0.0025397
support30	12	0.99962 [0.9939, 1]	2.098 [2.0282, 2.1705]	0.0026667

Appendix

Table 28: E-IF: categorical-interface interventions. Effective bins are the exponentiated entropy of the mean categorical distribution. Score differences subtract marginal IQMs. The column

n

gives the reported seed/run count.

Figure 11: Continuing experience and boundary information. A–D show all arms in episodic Catch, masked Catch, Tracking and CatchStream on their task-specific normalized reward-rate scales. A–C use 18 runs per arm; D uses 12. E retains the raw reward-rate unit for Breakout’s paired masking effect. F distinguishes terminal signals and physical resets. Intervals are pointwise 95% seed-bootstrap intervals.

Exposure	Algorithm	$n$	Reward-rate score [95% CI]	Threshold steps [95% CI]	Fraction reaching threshold
episodic	a2c	18	0.99615 [0.9958, 0.99657]	33640 [32712, 34568]	1
episodic	disco	18	0.99757 [0.99641, 0.9981]	22272 [21344, 23200]	1
episodic	dqn	18	0.92842 [0.92598, 0.93111]	64960 [64960, 65656]	1
episodic	ppo	18	0.98472 [0.94737, 0.98701]	13688 [11832, 17864]	1
masked	a2c	18	0.00070186 [-0.0015157, 0.0027543]	—	0
masked	disco	18	0.99651 [0.99433, 0.99761]	28304 [26442, 31320]	1

Appendix

Table 29: Termination-channel and reset-exposure comparisons. A blank threshold time denotes an unavailable finite estimate; fractions retain all evaluated runs. Threshold times use environment steps. The column

n

gives the reported seed/run count.

Exposure	Algorithm	$n$	Reward-rate score [95% CI]	Threshold steps [95% CI]	Fraction reaching threshold
synth100	a2c	18	-0.00027205 [-0.0037605, 0.0027645]	—	0
synth100	disco	18	0.99776 [0.99696, 0.99855]	30160 [28536, 32016]	1
synth29	a2c	18	0.99231 [0.98917, 0.99435]	$1.0347\!\times\!10^{5}$ [94656, $1.29\!\times\!10^{5}$ ]	1
synth29	disco	18	0.99651 [0.99433, 0.99761]	28304 [26442, 31320]	1
synth7	a2c	18	0.99615 [0.9958, 0.99657]	33640 [32712, 34568]	1
synth7	disco	18	0.99757 [0.99641, 0.9981]	22272 [21344, 23200]	1

Appendix

Table 30: Synthetic-terminal intervals in masked Catch. Synthetic terminal intervals are 7, 29, and 100 environment steps; the environment’s underlying reset dynamics follow masked Catch. The column

n

gives the reported seed/run count.

Algorithm	$n$	Reward-rate score [95% CI]	Ever reaches 0.8
a2c	12	0.00622 [-0.00067487, 0.012812]	0
disco	12	0.93896 [0.92218, 0.95135]	1
dqn	12	0.76918 [0.76285, 0.77705]	1

Appendix

Table 31: E-A3c: learning with continuous CatchStream dynamics. The final score averages the last 15% of log rows. The final column records whether a run reaches 0.8 at any logged time. The column

n

gives the reported seed/run count.

Algorithm	$n$	Episodic IQM	Masked IQM	Masked minus episodic [95% CI]
disco	18	0.08749	0.0841	-0.0041612 [-0.0055838, -0.0011324]
a2c	18	0.084701	0.07773	-0.0071963 [-0.0082076, -0.0066663]
ppo	18	0.084262	0.081661	-0.0029975 [-0.0048288, -0.001675]
dqn	18	0.085854	0.083464	-0.002365 [-0.0029163, -0.0018238]

Appendix

Table 32: MinAtar termination masking by algorithm. The column

n

gives the reported seed/run count.

Contrast	$n$	Paired difference [95% CI]
masking penalty:disco -a2c	18	0.0036713 [0.0022662, 0.0067003]
masking penalty:disco -ppo	18	-0.000915 [-0.0033301, 0.0026551]
masking penalty:disco -dqn	18	-0.0014237 [-0.0033904, 0.001704]

Appendix

Table 33: Paired contrasts of MinAtar masking effects. The column

n

gives the reported seed/run count.

Figure 12: Fixed-origin adaptation profiles and auxiliary-loss effects. (A) Post-switch normalized AUC for 27 Catch cells, 18 seeds each. (B) Nine observation/reward-mapping cells, 12 seeds each, in normalized score units. (C) Eight Breakout cells, 18 seeds each, in absolute-return units; its color scale is separate from A–B. Gray cells are unexecuted configurations. All AUCs are divided by the post-switch duration. (D) Paired no-auxiliary-minus-live differences in AUC and final-window score; all 18 seeds and pointwise 95% bootstrap intervals are retained.

Grid	Condition	AUC [95% CI]	Final window [95% CI]	Target	RMD
8 $\times$ 8	Disco	0.563 [0.509, 0.610]	0.965 [0.904, 0.997]	18/18	798.9
8 $\times$ 8	Frozen	0.553 [0.499, 0.612]	0.928 [0.860, 0.985]	17/18	855.6
8 $\times$ 8	No aux.	0.618 [0.594, 0.643]	0.958 [0.950, 0.966]	18/18	631.1
8 $\times$ 8	DQN 100k	0.049 [0.041, 0.057]	0.351 [0.335, 0.370]	1/18	1500.0
8 $\times$ 8	DQN 28,672	0.656 [0.648, 0.662]	0.940 [0.935, 0.943]	18/18	537.8
8 $\times$ 8	DQN cleared	0.881 [0.874, 0.888]	0.934 [0.928, 0.938]	18/18	165.6

Appendix

Table 34: Action flip: fixed-origin Catch analysis, 18 seeds per condition. Post-switch AUC averages iterations 1500–3000; final-window capability averages 2700–3000. Intervals are pointwise 95% bootstrap intervals. The common target is normalized score

\geq 0.8

after 102 newly completed episodes. Restricted mean delay (RMD) includes all runs, assigning unattained runs the 1500-iteration horizon.

Grid	Condition	AUC [95% CI]	Final window [95% CI]	Target	RMD
8 $\times$ 8	Disco	0.523 [0.503, 0.553]	1.000 [0.999, 1.000]	18/18	761.1
8 $\times$ 8	Frozen	0.498 [0.484, 0.509]	1.000 [0.999, 1.000]	18/18	798.9
8 $\times$ 8	No aux.	0.655 [0.636, 0.681]	0.980 [0.972, 0.984]	18/18	526.7
8 $\times$ 8	DQN 100k	0.050 [0.028, 0.071]	0.526 [0.472, 0.576]	0/18	1500.0
8 $\times$ 8	DQN 28,672	0.453 [0.435, 0.476]	0.736 [0.692, 0.781]	11/18	1257.8
8 $\times$ 8	DQN cleared	0.622 [0.599, 0.651]	0.788 [0.738, 0.850]	14/18	1082.2

Appendix

Table 35: Reward move: fixed-origin Catch analysis, 18 seeds per condition. Post-switch AUC averages iterations 1500–3000; final-window capability averages 2700–3000. Intervals are pointwise 95% bootstrap intervals. The common target is normalized score

\geq 0.8

after 102 newly completed episodes. Restricted mean delay (RMD) includes all runs, assigning unattained runs the 1500-iteration horizon.

Grid	Condition	AUC [95% CI]	Final window [95% CI]	Target	RMD
8 $\times$ 8	Disco	0.994 [0.992, 0.996]	0.998 [0.994, 0.999]	18/18	23.3
8 $\times$ 8	Frozen	0.995 [0.994, 0.996]	0.999 [0.996, 1.000]	18/18	20.0
8 $\times$ 8	No aux.	0.974 [0.953, 0.983]	0.997 [0.990, 0.999]	18/18	80.0
8 $\times$ 8	DQN 100k	0.934 [0.932, 0.936]	0.929 [0.923, 0.934]	18/18	20.0
8 $\times$ 8	DQN 28,672	0.932 [0.930, 0.933]	0.934 [0.929, 0.938]	18/18	20.0
8 $\times$ 8	DQN cleared	0.932 [0.929, 0.935]	0.933 [0.928, 0.939]	18/18	20.0

Appendix

Table 36: No change: fixed-origin Catch analysis, 18 seeds per condition. Post-switch AUC averages iterations 1500–3000; final-window capability averages 2700–3000. Intervals are pointwise 95% bootstrap intervals. The common target is normalized score

\geq 0.8

after 102 newly completed episodes. Restricted mean delay (RMD) includes all runs, assigning unattained runs the 1500-iteration horizon.

Grid	Change	Metric	Paired difference [95% CI]
8 $\times$ 8	Action flip	AUC	0.062 [0.009, 0.107]
8 $\times$ 8	Action flip	Final window	-0.010 [-0.037, 0.052]
8 $\times$ 8	Reward move	AUC	0.125 [0.091, 0.172]
8 $\times$ 8	Reward move	Final window	-0.020 [-0.027, -0.015]
8 $\times$ 8	No change	AUC	-0.018 [-0.039, -0.009]
8 $\times$ 8	No change	Final window	-0.000 [-0.005, 0.002]

Appendix

Table 37: No-auxiliary minus Disco on all 18 paired seeds per condition. Each difference is formed within seed before trimming; pointwise 95% intervals use 10,000 paired bootstrap replicates. No successful-run conditioning is applied.

Figure 13: Action-flip adaptation at two Catch grid sizes. (A,C) Normalized post-switch AUC and final-window capability at

8\times 8

and

16\times 16

. (B,D) Restricted mean time to normalized score 0.8, measured from the switch with all runs and censored non-attainers included. Each configuration has 18 seeds, a switch at 87,000 interactions, and 87,000 post-switch interactions. Labels show common-threshold attained/all. The

8\times 8

matrix includes three Disco variants and three DQN replay configurations;

16\times 16

includes three Disco variants. Bars are pointwise 95% seed-bootstrap intervals.

Change	Condition	Pre	Post-switch AUC [95% CI]	Endpoint [95% CI]
Action flip	A2C	5.029	3.511 [3.126, 3.794]	5.218 [4.678, 5.476]
Action flip	Disco	9.804	5.373 [5.070, 5.741]	8.763 [8.134, 9.341]
Action flip	Frozen	9.439	5.349 [4.914, 5.934]	8.462 [7.616, 9.228]
Action flip	PPO	5.120	3.978 [3.775, 4.189]	5.321 [5.040, 5.671]
No change	A2C	4.879	5.466 [5.349, 5.813]	5.691 [5.555, 6.267]
No change	Disco	8.851	10.772 [9.823, 11.724]	12.299 [10.988, 13.625]

Appendix

Table 38: MinAtar Breakout: absolute return, 18 seeds per condition, switch at 4000 and endpoint at 8000. AUC is the time-average from the declared switch to the endpoint, recomputed from complete trajectories. Endpoint values are independently reproduced from all run endpoints. Intervals are pointwise 95% bootstrap intervals; no own-performance threshold is used.

Change	Condition	Pre	Post-switch AUC [95% CI]	Endpoint [95% CI]
No change	A2C	0.990	0.993 [0.992, 0.994]	0.992 [0.989, 0.998]
No change	Disco	0.997	0.995 [0.991, 0.997]	1.000 [1.000, 1.000]
No change	Frozen	1.000	0.997 [0.996, 0.998]	1.000 [1.000, 1.000]
Observation flip	A2C	0.990	0.243 [0.181, 0.347]	0.326 [0.253, 0.463]
Observation flip	Disco	0.997	0.626 [0.577, 0.690]	0.992 [0.941, 0.998]
Observation flip	Frozen	1.000	0.604 [0.566, 0.673]	1.000 [0.996, 1.000]

Appendix

Table 39: Catch mapping: random-zero/oracle-one normalized return, 12 seeds per condition, switch at 1500 and endpoint at 3000. AUC is the time-average from the declared switch to the endpoint, recomputed from complete trajectories. Endpoint values are independently reproduced from all run endpoints. Intervals are pointwise 95% bootstrap intervals; no own-performance threshold is used.

Figure 14: Responses to observation and reward changes. (A) Pre-switch, trough, and endpoint normalized scores for observation reversal, reward-sign reversal, and stationary exposure. (B) Post-switch AUC integrates from the configured switch at iteration 1,500 to 3,000. Each condition has 12 seeds and reference indexing; bars are pointwise 95% seed-bootstrap intervals. The trough is a descriptive score, not the clock’s origin. Frozen-state agents continue updating parameters.

Algorithm	Depth	$n$	Solved fraction	Ever found	Tail return [95% CI]	Deepest majority solved
a2c	8	15	1	1	0.99057 [0.99, 0.99105]	12
a2c	10	15	1	1	0.99025 [0.98994, 0.99068]	12
a2c	12	15	0.73333	0.86667	0.84451 [0.47917, 0.99021]	12
a2c	14	15	0	0.46667	-0.0046868 [-0.0048272, -0.0044202]	12
a2c	16	15	0	0.13333	-0.0048606 [-0.004884, -0.0048315]	12
disco	8	15	1	1	0.94435 [0.89122, 0.98534]	12

Appendix

Table 40: E-A5: complete reference-convention staircase. A run is solved when its final-20%-of-time mean return reaches 0.495. Majority solved uses fraction >= 0.5; ever found checks any positive logged return. The column

n

gives the reported seed/run count.

Convention	Depth	$n$	Disco solved	A2C solved	Disco minus A2C [95% CI]
legacy	12	15	12	11	0.00034352 [-0.22028, 0.31887]
reference	12	15	10	11	-0.0030685 [-0.33376, 0.2212]
legacy	14	15	6	0	0.32822 [0.004777, 0.7605]
reference	14	15	4	0	0.10269 [0.0047881, 0.44483]

Appendix

Table 41: F1: paired convention checks at DeepSea depths 12 and 14. F1 is a separately indexed paired slice. Its counts remain attached to that slice; the complete E-A5 staircase is reported in a separate table. The column

n

gives the reported seed/run count.

Figure 15: Reward discovery and sustained solution. A–C show all reference-indexed E-A5 cells: solved fraction, ever-found fraction and final-return IQM, with 15 seeds per cell. C carries pointwise 95% seed-bootstrap intervals. D displays the independent F1 reference-indexed check at depths 12 and 14, which uses different runs. Lines connect tested depths.

Figure 16: State, scale, inheritance, and the value interface. (A) Catch’s four-arm state/EMA scale response. (B) All four arms’ usable windows at score 0.8 in three environments, using the E1 final-20% score and pointwise 95% intervals. (C) E2 matched/cross donor scores under natural transfer and repeated clamping, at both recipient scales. The plotted coordinate is the marginal score; the separately labeled interaction is the paired contrast in Equation 8 . (D) Unit-scale value-interface performance and effective categorical use. All cells contain 12 seeds. Marginal score intervals are pointwise; interaction intervals follow the family-11 procedure.

Figure 17: Behavior across retained data, streams, visual change, and exploration. (A) Catch action-flip Disco-minus-DQN post-switch AUC under capacity 100,000, matched capacity 28,672, and switch-time clearing; 18 paired seeds and family-11 intervals. (B) CatchStream and Tracking normalized reward rates, 12 and 18 seeds per arm, respectively. (C) Breakout action-reversal pre-switch and endpoint absolute returns, 18 seeds per arm. (D) The five-method DeepSea staircase, five depths and 15 seeds per cell. These panels use the same quantities as the corresponding main-text and detailed appendix analyses.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses
Aug 25, 2026Zhaochen Yu, Yingcheng Wu, Zhenfei Yin +5Long-Horizon LLM Agents Memory-Augmented Agents
NUS · Princeton University · Stanford University +1
Dream-RSI: Recursive Self-Improvement through Evolving Worlds
Sep 14, 2026Tong Zheng, Xidong Wu, Zheng Zhang +14RL Exploration Self-Improving Agents
Google · University of Maryland, College Park · Google Deepmind +1
MMLA-RSI: Recursive Self-Improvement through RTT and RTMA
Oct 14, 2026Junyi Zou, Avrova DonzSelf-Improving Agents Recursive Self-Improvement
MMLA-org · Communication University of China (CUC), No. 1 Dingfuzhuang East Street, Chaoyang District, Beijing 100024, China

Aug 25, 2026J/K move · Enter open · S save

Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses

Zhaochen Yu, Yingcheng Wu, Zhenfei Yin, Kaiyuan Chen+4

NUS · Princeton University · Stanford University+1

Long-Horizon LLM Agents Memory-Augmented Agents Self-Improving Agents Recursive Self-Improvement LLM Agent Skill Learning Long-Horizon Reasoning

Open paper AskPDF arXiv

Recursive self-improvement (RSI) remains hard in long-horizon tasks, where growing histories obscure the task state and misalign skill invocation. We introduce Recuris, a recursive Experiential-Working Memory architecture for long-horizon agent harnesses, in which Working Memory tracks task progress and guides skill selection from Experiential Memory, grounding skill use in current needs rather than the full history. This coupling also turns execution into structured evidence that localizes failures to specific memory components. Across tasks, a fixed Meta-Agent turns that evidence into localized, validation-gated updates to Skill Memory that reshape execution and yield new evidence, forming a bounded recursive memory-evolution loop. Across four long-horizon benchmarks and ten models, Recuris improves task success in 35 of the 37 completed model-benchmark pairs, carrying frontier models to SOTA-level task success: on tau-bench it adds +17.8 points to GPT-5.6 Sol and +15.6 to Claude Opus 5, taking Opus 5 to 87.9%, and +16.6/+13.5 points on Qwen3.6-27B/35B on SkillFlow. The advantage widens as the interaction horizon grows, to +32.2 points on the longest tasks, and common long-horizon failures fall by up to 80%. These results position recursively evolving memory as a scalable foundation for RSI, enabling agents to continuously transform accumulated experience into increasingly effective long-horizon behavior. Code: https://github.com/Gen-Verse/Recuris

Abstract

Reinforcement Learning

Explore similar work

Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses

Dream-RSI: Recursive Self-Improvement through Evolving Worlds

MMLA-RSI: Recursive Self-Improvement through RTT and RTMA