The pursuit of recursive self-improvement (RSI) toward general intelligence is divided between macro-level language model scaling and the interaction-driven principles of "Era of Experience". Yet, any self-improving architecture ultimately rests upon its underlying optimization engine: if general intelligence requires learning from grounded interaction, the reinforcement learning (RL) update rule itself must be capable of cumulative adaptation. While algorithm self-discovery has produced Disco103 that surpassed PPO to achieve SOTA benchmark performance -- its internal update machinery remains an uninspected black box. We present the first causal mechanistic audit of a self-discovered RL rule, structured directly around the five pillars of the Era of Experience: extended horizon, grounded reward scales, continuing streams, within-lifetime change, and exploration depth. By surgically pinning, freezing, and transplanting recurrent states while holding meta-parameters fixed, we test when learning history acts as an asset or a burden. Three findings organize the audit: (1) Recurrent history actively expands usable reward scales, sustaining a six-decade window versus three under zero-pinning. (2) Decoupling historical content from its maintenance reveals that the penalty of mismatched history stems from perpetual clamping; allowing imported state to evolve naturally attenuates this burden. (3) Under environmental change, controlling replay retention reverses the apparent adaptation advantage over DQN, demonstrating that external data turnover can confound internal plasticity. Validated through capability thresholds and ported to a second rule (OPEN), this work grounds macro-RSI ambitions in micro-level learning dynamics, establishing a foundational audit standard for next-generation, self-evolving RL algorithms.
Figures & tables
Figure 1: From the Era of Experience to a five-axis causal audit. Five demands become operational transformations of the learning loop. Rule weights η remain fixed while agent parameters and recurrent history (h,c) evolve. State interventions, value-interface analysis, and replay controls distinguish internal history, numerical representation, and external memory; boundary-signal and discovery-event controls complete the profile.
Figure 2: Interventions reach from learning history to numerical meaning. (A) Zero-pinning and timed freezing control the availability and acquisition time of history while agent parameters update. (B) A 400-iteration donor supplies hidden and cell state to a fresh 2,000-iteration recipient. Natural transfer initializes once and then evolves; clamping restores the donor pair after every iteration. Matched and cross-scale donors complete the comparison. (C) The detached target is transformed, clipped, and projected onto neighboring atoms; scalar readout applies the inverse transform after expectation. Appendices A and Q provide execution details and derivations.
Axis
Question
Readout
Control
A1 Horizon
How does progress accumulate?
Endpoint; own-target mastery
Rate sweep; freeze time
A2 Reward units
Which magnitudes remain usable?
Contiguous scale window
State/EMA; donor; interface
A3 Streams
What changes without resets?
Continuing reward rate
Masking; synthetic boundaries
A4 Change
How is competence rebuilt?
Levels; AUC; attainment
Change type; replay; state
A5 Exploration
When does experience inform?
Discovery; sustained solution
Depth; entropy; random policy
Table 1: Five axes form a capability profile. Each axis has a distinct question, readout, and explanatory control.
Figure 3: History, reward units, and numerical geometry. (A) Reference-indexed Catch, 12 seeds per cell, integer-trimmed final-20% scores with pointwise 95% intervals. (B) Absolute-capability window widths and pointwise marginal intervals across three environments. (C) Low-scale matched/cross donors under natural transfer or repeated clamping, 12 seeds and pointwise score intervals. The interaction aggregates within-seed difference-of-differences before trimming and uses a family-11 interval (Equation 8 ). (D) Unit-scale interface assay: score (zoomed axis) and effective bins, each with pointwise 95% seed-bootstrap intervals. Full grids, both recipient scales, all eight inheritance conditions, and the analytical interface appear in the appendix.
Figure 4: A capability profile across time, streams, change, and exploration. (A) Reference CatchBig: endpoint and time to each run’s own attained-level target, seven configurations, 12 seeds. (B) CatchStream ( n=12 ) and Tracking ( n=18 ): normalized reward rate with pointwise 95% seed-bootstrap intervals. (C) Breakout action reversal: pre/final absolute returns, four arms and 18 seeds per arm. (D) Reference DeepSea staircase: four algorithms and a Disco entropy ablation, five depths, 15 seeds per cell (375 runs); solved fractions use final-window threshold 0.495, with the dotted line marking majority solution. Complete tables and controls appear in the atlas.
Appendix figures & tables53 assets
Supplementary material from the paper’s appendix.
Appendix
Condition
Lifetime state
Other recipient components
Live
Update
Update
Zero-pinned
Restore zero after each iteration
Update
Frozen at tf
Restore own state captured at tf
Update
Natural transfer
Initialize with donor; then update
Fresh recipient; then update
Perpetual clamp
Initialize with donor; restore each iteration
Fresh recipient; then update
OPEN reset-once
Zero both GRU carries once; then update
Preserve inherited agent and other optimizer state
Appendix
Table 2: State-operation semantics. “Update” means the component follows its usual learner update. Transplantation transfers the lifetime pair only.
Test family
Outputs
Max. absolute error
Forward outputs
26
0.002304
Inner outputs/gradients
16
4.470×10−5
Updated parameters
12
2.733×10−4
Value estimator
9
5.471×10−14
Meta-gradient
42
4.170×10−7
Float32/float64 control
42
2.945×10−7
Appendix
Table 3: Cross-framework numerical checks on shared inputs. Errors are maximum absolute discrepancies over the recorded outputs in each family. Updated parameters are measured after one Adam step; the final row compares precisions within PyTorch.
Task
Criterion
Live
Pin
EMA
Both
ΔW
Adjusted CI
Catch
Half-best
6
3
4
3
3
[2, 3]
Catch
S≥0.8
6
3
4
3
3
[2, 3]
Catchdense
Half-best
6
4
5
4
2
[2, 2]
Catchdense
S≥0.8
6
3
4
3
3
[2, 3]
Catchlong
Half-best
5
2
3
2
3
[0, 4]
Catchlong
S≥0.8
3
2
3
2
1
[1, 3]
Appendix
Table 4: All scale-window estimates, 12 seeds per cell. Integer-trimmed and exact-IQM results coincide for these 12-seed summaries. Half-best rows belong to the primary family; absolute rows to its threshold-sensitivity companion. Intervals use the corresponding family-11 tails.
Figure 5: Reference-indexed Catch scale response and window uncertainty. Twelve seeds per condition; curves show final-20% normalized scores and pointwise 95% bootstrap bands. Width intervals are pointwise; the live-minus-pinned difference is three decades with adjusted interval [2,3] . The source arrays and complete environment table retain both absolute and half-best criteria.
Condition
10−3
95% CI
103
95% CI
live
0.6785
[0.345, 0.905]
0.9961
[0.992, 0.998]
ownfreeze400
0.6735
[0.417, 0.891]
0.9926
[0.990, 0.994]
match clamp
0.9326
[0.700, 0.982]
0.9952
[0.992, 0.997]
match init
0.8971
[0.690, 0.982]
0.9937
[0.990, 0.997]
cross clamp
0.0165
[0.004, 0.068]
0.9954
[0.993, 0.998]
cross init
0.5307
[0.201, 0.868]
0.9957
[0.991, 0.999]
Appendix
Table 5: All eight state conditions at each recipient scale, 12 seeds per cell. Tail scores use the common final-20% grid; intervals are pointwise 95%. Execution settings are specified in Appendix A .
Figure 6: Full inheritance trajectories at both recipient scales. Matched donors use the recipient scale; cross donors use the opposite extreme. Donors train for 400 iterations and recipients for 2,000. Twelve seeds per cell; curves and marginal scores have pointwise 95% intervals. Low-scale interaction 0.39265[0.07896,0.86094] and high-scale interaction 0.00236[−0.00783,0.00889] use the family-11 procedure, estimated separately within each recipient stratum.
Change / replay
Trim
Adjusted CI
IQM
Adjusted CI
action flip / 100k
0.5161
[0.427, 0.584]
0.5159
[0.428, 0.585]
action flip / 28,672
-0.0898
[-0.175, -0.030]
-0.0897
[-0.176, -0.030]
action flip / clear
-0.3154
[-0.403, -0.253]
-0.3163
[-0.402, -0.252]
reward move / 100k
0.4841
[0.440, 0.516]
0.4856
[0.439, 0.516]
reward move / 28,672
0.0701
[0.030, 0.125]
0.0689
[0.030, 0.128]
reward move / clear
-0.0901
[-0.151, -0.042]
-0.0893
[-0.153, -0.042]
Appendix
Table 6: All six planned Disco-minus-DQN AUC comparisons, 18 seeds, family-11 adjusted intervals. Exact middle-50% IQM is the estimator-sensitivity companion.
Change
System
Threshold
Attained
RM time ( 103 )
action flip
disco
0.8
18/18
46.3
action flip
disco
0.9
17/18
60.9
action flip
DQN clear
0.8
18/18
9.6
action flip
DQN clear
0.9
18/18
12.1
action flip
DQN 28,672
0.8
18/18
31.2
action flip
DQN 28,672
0.9
18/18
32.7
Appendix
Table 7: Common-capability attainment, fixed origin at the switch. Restricted mean time includes censored runs at the 87,000-interaction horizon; an observed threshold crossing is an attainment event, not a sustained-return guarantee.
Figure 7: Replay controls and post-switch learning trajectories. Catch 8×8 , 18 seeds, switch at 87,000 interactions. Left: normalized training-return streams with pointwise intervals. Right: paired Disco-minus-DQN post-switch AUC, with family-11 adjusted intervals. The capacities are 100,000 and 28,672 transitions; the cleared configuration uses capacity 100,000 and removes its contents at the switch. Update schedules remain those of each system. Fixed-origin threshold analyses use a completed-episode washout.
Contrast
Game
Scale
Estimate
Adjusted CI
Live - zero
breakout
0.1
0.001087
[-0.000839, 0.004019]
Live - zero
breakout
1.0
0.002232
[-0.001190, 0.004171]
Live - zero
breakout
10.0
0.001520
[0.000607, 0.002340]
Live - zero
freeway
0.1
0.000322
[-0.001213, 0.000946]
Live - zero
freeway
1.0
0.000099
[-0.000380, 0.000843]
Live - zero
freeway
10.0
0.000267
[-0.000591, 0.001192]
Appendix
Table 8: OPEN terminal-state contrasts, eight seed blocks per cell. The original family-34 adjustment and 200,000 bootstrap draws are retained. Units are unscaled evaluation reward per interaction.
Contrast
Game
Scale
Estimate
Adjusted CI
Live - initial
breakout
0.1
0.038862
[0.037537, 0.041191]
Live - random
breakout
0.1
0.038857
[0.037670, 0.040970]
Live - initial
breakout
1.0
0.039761
[0.038012, 0.040659]
Live - random
breakout
1.0
0.039755
[0.038452, 0.040359]
Live - initial
breakout
10.0
0.039524
[0.038656, 0.040489]
Live - random
breakout
10.0
0.039518
[0.038597, 0.040272]
Appendix
Table 9: OPEN terminal capability checks, eight seed blocks per cell. Family-34 adjusted intervals use 200,000 bootstrap draws. Units are unscaled evaluation reward per interaction.
Scale
OPEN − Adam
Disco − OPEN
Disco − Adam
Breakout
0.01
-0.00190 [-0.00533, 0.00194]
0.00976 [0.00094, 0.02234]
0.00786 [-0.00139, 0.02124]
0.1
-0.00128 [-0.00931, 0.00161]
0.00518 [0.00056, 0.01644]
0.00389 [-0.00588, 0.01504]
1
-0.00017 [-0.00676, 0.00156]
0.00369 [-0.00057, 0.01234]
0.00352 [-0.00558, 0.01271]
10
-0.00204 [-0.00286, -0.00031]
0.00397 [-0.00038, 0.01603]
0.00194 [-0.00236, 0.01393]
100
-0.00169 [-0.00258, -0.00097]
0.00288 [0.00165, 0.00448]
0.00119 [-0.00014, 0.00234]
Appendix
Table 10: Task-conditional system profiles at equal interaction budgets. All 30 contrasts use eight seeds per cell and the original family-30 adjusted intervals. Each cell gives the difference of marginal final-window trimmed means, followed by its interval in unscaled reward-per-interaction units. Adam denotes PPO with Adam. This cohort is distinct from the OPEN carry interventions.
Depth
Learner
Found
Wilson CI
Tail success
95% CI
12
disco
11/15
[0.480, 0.891]
0.517
[0.152, 0.866]
12
a2c
15/15
[0.796, 1.000]
0.889
[0.445, 1.000]
12
dqn
13/15
[0.621, 0.963]
0.737
[0.570, 0.757]
12
random
100/100
[0.963, 1.000]
0.000
[0.000, 0.000]
14
disco
6/15
[0.198, 0.643]
0.000
[0.000, 0.282]
14
a2c
12/15
[0.548, 0.930]
0.000
[0.000, 0.000]
Appendix
Table 11: DeepSea discovery and final-window success: 15 runs per learner cell and 100 random-policy campaigns per depth. Discovery intervals are Wilson 95%; tail intervals are pointwise seed bootstrap. The horizon is 12,000 episodes at each depth.
Contrast
Scale
Estimate
Pointwise 95%
Family-15 interval
ΔDefault
10−3
0.8225
[0.5456, 0.9301]
[0.3536, 0.9768]
ΔDefault
1
0.0024
[-0.0069, 0.0109]
[-0.0165, 0.0186]
ΔDefault
103
0.0024
[-0.0024, 0.0073]
[-0.0054, 0.0093]
ΔS30
10−3
0.9615
[0.8220, 0.9828]
[0.6680, 0.9843]
ΔS30
1
0.0035
[-0.0003, 0.0098]
[-0.0041, 0.0140]
ΔS30
103
-0.0004
[-0.0031, 0.0013]
[-0.0045, 0.0025]
Appendix
Table 12: Complete P1 interface-by-state analysis: all 15 final-20% (tail20) contrasts from 216 runs, with 12 paired seeds per cell and 10,000 seed-bootstrap draws. Δi is the live-minus-zero state effect for interface i ; interaction rows compare those per-seed effects before aggregation. Default and S30 use signed-hyperbolic supports 300 and 30; Linear uses linear support 300. The last two columns separately retain pointwise 95% intervals and Bonferroni-adjusted intervals for this secondary exploratory family of 15 contrasts. Integer-trimmed means and exact middle-50% IQM coincide at the displayed precision.
Cell
n
Legacy gap [95% CI]
Reference gap [95% CI]
Shift [95% CI]
A1 catchbig, disco-A2C
12
-0.39 [-0.59333, -0.17658]
-0.016667 [-0.063333, 0.013333]
0.34333 [0.10333, 0.59008]
A2 catch x 1e-3, disco-pinned@0
12
0.00166 [0.0015133, 0.00174]
0.0017033 [0.00164, 0.0017533]
3.3333×10−5 [ −2.6667×10−5 , 0.00017667]
A2 catch x 1, disco-pinned@0
12
0 [0, 0.0066667]
0 [0, 0.0066667]
0 [-0.01, 0.0066667]
A2 catch x 1e3, disco-pinned@0
12
826.67 [3.3333, 1666.7]
1413.3 [546.67, 1730]
56.667 [-20, 886.67]
A3 masked, disco-A2C
18
0.249 [0.24635, 0.25125]
0.2487 [0.24575, 0.2511]
-0.00015 [-0.00205, 0.0005]
A4 action-flip 8x8, disco-A2C
18
0.95 [0.79595, 1.108]
0.95 [0.802, 1.082]
-0.01 [-0.074, 0.064]
Appendix
Table 13: Paired sensitivity to the Retrace convention. Each row retains its source metric and paired contrast. Scale cells use raw reward units; other cells use their reported normalized score or return metric. The column n gives the reported seed/run count.
Assay
Conditions
Seeds
Iterations
E-A4m
Breakout: 2 protocols × 4 arms
18
8,000
E1
3 tasks × 7 scales × 4 state/EMA arms
12
2,000
E2
2 recipient scales × 8 inheritance conditions
12
2,000
E3
27 grid/change/agent/replay cells
18
3,000
E-A5
5 depths × 5 arms
15
1,655–3,310
E-LR
3 baselines × 4 learning rates
12
20,000
Appendix
Table 14: Reference-indexed matrices. Counts are seeds per algorithm–condition cell. Each iteration uses 58 interactions. Transplants additionally train a donor for 400 iterations; DeepSea budgets scale with depth.
Table 15: Reference-indexed CatchBig: endpoint and mastery. Groups identify the reference arm: Disco103 at learning rate 10−2 , or the 3×10−4 arm. Each row compares that reference with a comparator, using 12 paired seeds. Differences are reference minus comparator, with 95% intervals. Mastery uses each run’s own final-score target, in 103 environment steps. Component IQMs and paired-difference IQMs are distinct estimands.
Figure 8: Learning-rate and normalization sensitivity. A–C show all baseline rates, 12 seeds per cell; black rings identify measured default settings. D compares A2C and A2C-PopArt at all three scales. Points are tail IQMs with pointwise 95% seed-bootstrap intervals; lines connect tested settings.
Figure 9: Reward-scale response, usable windows, and terminal-state geometry. (A–C) The E1 state/EMA factorial contains four arms at seven scales in each environment, with 12 seeds per cell. Curves use the same final-20% scores and pointwise 95% seed-block intervals as Figure 3 . (D–F) Widest contiguous passing spans at absolute scores 0.7, 0.8, and 0.9, with pointwise window intervals. At 0.8, live/pinned widths are 6/3, 6/3, and 3/2 decades. (G–I) Live hidden/cell-state centroid displacement from unit scale, normalized by the corresponding unit-scale radius; geometry uses terminal states from the same runs.
Environment
Rrandom
Roptimal
Reference span
Catch
-0.75000
1.00000
1.75000
CatchDense
-3.04774
0.75000
3.79774
CatchLong
-0.87500
1.00000
1.87500
Appendix
Table 16: Reward-scale normalization references. Scores are (R/s−Rrandom)/(Roptimal−Rrandom) with no clipping; s is removed exactly once. CatchDense uses its fixed 40,000-episode reference estimate.
Environment
State
Score ≥0.7
Score ≥0.8
Score ≥0.9
Catch
Live
6 [5, 6]
6 [5, 6]
5 [5, 6]
Catch
Pin 0
3 [3, 3]
3 [3, 3]
3 [3, 3]
Catch
EMA held
4 [4, 4]
4 [4, 4]
4 [4, 4]
Catch
Both
3 [3, 4]
3 [3, 4]
3 [3, 3]
CatchDense
Live
6 [6, 6]
6 [6, 6]
5 [5, 6]
CatchDense
Pin 0
4 [3, 4]
3 [3, 4]
3 [3, 3]
Appendix
Table 17: Absolute usable scale windows in decades. Each value is the widest contiguous passing segment of the seven tested multipliers; bracketed intervals are pointwise 95% seed-block bootstrap intervals. The three thresholds are sensitivity views of the same final-20% scores and the same 1,008 runs, not additional experiments.
Environment
State
Width [95% interval]
Point threshold
Catch
Live
6 [6, 6]
0.498
Catch
Pin 0
3 [3, 4]
0.497
Catch
EMA held
4 [4, 4]
0.500
Catch
Both
3 [3, 4]
0.499
CatchDense
Live
6 [6, 6]
0.498
CatchDense
Pin 0
4 [4, 4]
0.500
Appendix
Table 18: Half-best scale-window sensitivity using the same 1,008 runs. The threshold is half the best IQM score of the arm and is recomputed within each bootstrap draw; this relative criterion is shown alongside the absolute-capability windows, not substituted for them.
Scale
Live
Pin 0
EMA held
Both
10−3
0.828 [0.553, 0.931]
0.004 [-0.003, 0.009]
0.005 [-0.003, 0.011]
0.005 [-0.002, 0.010]
10−2
0.977 [0.956, 0.992]
0.012 [0.001, 0.021]
0.012 [0.003, 0.296]
0.006 [-0.002, 0.010]
10−1
0.976 [0.944, 0.991]
0.176 [0.007, 0.555]
0.996 [0.993, 0.998]
0.419 [0.072, 0.829]
100
0.994 [0.985, 0.999]
0.991 [0.985, 0.995]
0.999 [0.998, 1.000]
0.995 [0.985, 0.998]
101
0.997 [0.995, 0.999]
0.993 [0.987, 0.997]
1.000 [0.999, 1.000]
0.994 [0.983, 0.997]
102
0.997 [0.995, 0.999]
0.988 [0.983, 0.992]
1.000 [0.998, 1.000]
0.991 [0.983, 0.997]
Appendix
Table 19: Catch scale response. Each cell gives the IQM of final-20% normalized scores over 12 seeds and its pointwise 95% seed-bootstrap interval. All 28 conditions use 116,000 interactions and reference indexing.
Scale
Live
Pin 0
EMA held
Both
10−3
0.881 [0.838, 0.920]
-0.131 [-0.183, -0.023]
0.007 [-0.039, 0.025]
-0.019 [-0.145, 0.024]
10−2
0.957 [0.938, 0.971]
0.126 [-0.027, 0.246]
0.586 [0.244, 0.778]
-0.015 [-0.139, 0.034]
10−1
0.972 [0.965, 0.980]
0.781 [0.644, 0.870]
0.801 [0.764, 0.835]
0.584 [0.377, 0.800]
100
0.988 [0.973, 0.994]
0.997 [0.992, 0.999]
0.921 [0.911, 0.930]
0.913 [0.910, 0.915]
101
0.996 [0.992, 0.999]
0.999 [0.995, 1.000]
0.999 [0.998, 1.000]
0.945 [0.944, 0.948]
102
0.997 [0.993, 0.998]
0.999 [0.998, 1.000]
0.999 [0.998, 1.000]
0.952 [0.936, 0.964]
Appendix
Table 20: CatchDense scale response. Each cell gives the IQM of final-20% normalized scores over 12 seeds and its pointwise 95% seed-bootstrap interval. All 28 conditions use 116,000 interactions and reference indexing.
Scale
Live
Pin 0
EMA held
Both
10−3
0.144 [0.034, 0.429]
-0.002 [-0.008, 0.004]
0.005 [-0.001, 0.007]
0.003 [-0.002, 0.007]
10−2
0.741 [0.457, 0.854]
0.001 [-0.007, 0.011]
0.002 [-0.005, 0.009]
0.003 [-0.002, 0.007]
10−1
0.772 [0.470, 0.855]
0.014 [-0.001, 0.034]
0.159 [0.009, 0.575]
0.002 [-0.002, 0.006]
100
0.961 [0.926, 0.983]
0.338 [0.136, 0.538]
0.944 [0.892, 0.986]
0.275 [0.007, 0.682]
101
0.988 [0.986, 0.992]
0.996 [0.994, 0.997]
0.997 [0.994, 0.998]
0.971 [0.940, 0.984]
102
0.987 [0.983, 0.990]
0.998 [0.997, 0.999]
0.995 [0.993, 0.997]
0.992 [0.990, 0.994]
Appendix
Table 21: CatchLong scale response. Each cell gives the IQM of final-20% normalized scores over 12 seeds and its pointwise 95% seed-bootstrap interval. All 28 conditions use 116,000 interactions and reference indexing.
Environment
State
Scale
h : shift / radius
c : shift / radius
Catch
Live
0.001
2.92 [2.22, 4.82]
3.35 [2.55, 5.58]
Catch
Live
0.01
2.55 [1.95, 4.21]
3.26 [2.46, 5.43]
Catch
Live
0.1
2.32 [1.77, 3.67]
2.91 [2.21, 4.77]
Catch
Live
1
0.00 [0.00, 0.00]
0.00 [0.00, 0.00]
Catch
Live
10
4.12 [2.68, 7.62]
8.39 [5.29, 16.29]
Catch
Live
100
10.88 [7.48, 19.13]
22.49 [14.88, 42.55]
Appendix
Table 22: Terminal-state displacement in the scale assay. For each environment and arm, the centroid displacement from unit scale is divided by the mean Euclidean radius of the 12 unit-scale states; bootstrap draws recompute that denominator. All 7 scales and 12 seeds are retained. Brackets are pointwise 95% intervals. The two pinned arms have identically zero states, so their normalized ratios are undefined rather than zero.
Environment
State
Vector
Between-scale fraction [95% interval]
PC1+2 fraction
Unit radius
Catch
Live
h
0.974 [0.966, 0.986]
0.970
0.113
Catch
Live
c
0.989 [0.983, 0.995]
0.994
1.372
Catch
Live
hc
0.989 [0.983, 0.995]
0.994
1.377
Catch
EMA held
h
0.941 [0.934, 0.964]
0.943
0.049
Catch
EMA held
c
0.991 [0.990, 0.994]
0.990
0.585
Catch
EMA held
hc
0.991 [0.990, 0.994]
0.990
0.587
Appendix
Table 23: Descriptive terminal-state geometry from the same scale assay. Each row contains 84 terminal states (7 scales by 12 seeds). Between-scale fraction is centroid sum of squares divided by total centered sum of squares. PCA uses unstandardized centered states. These are descriptive associations; pinned states have zero total variation and undefined PCA fractions.
Algorithm
Reward scale
n
Score [95% CI]
a2c
0.001
12
0.007746 [0.0025397, 0.011556]
a2c-popart
0.001
12
0.0054603 [0.00037776, 0.0093968]
a2c
1
12
0.99263 [0.9906, 0.99492]
a2c-popart
1
12
0.9746 [0.97244, 0.97879]
a2c
1000
12
0.0022857 [-0.0044508, 0.0062222]
a2c-popart
1000
12
0.99581 [0.94539, 0.99898]
Appendix
Table 24: E-POP: complete PopArt reward-scale comparison. The column n gives the reported seed/run count.
Scale
n
Descaled rate
Retention [95% CI]
Passing
Breakout / A2C ; unit-scale rate 0.08442
0.01
8
0.0652
0.7695 [0.7621, 0.7791]
no
0.1
8
0.08611
1.018 [1.003, 1.034]
yes
1
8
0.08442
1 [1, 1]
yes
10
8
0.08328
0.9848 [0.9687, 1.004]
yes
100
8
0.07725
0.916 [0.9023, 0.9316]
yes
Appendix
Table 25: MinAtar reward-scale response. Rate is reward per interaction after removing the multiplier. Retention forms each seed’s ratio to its unit-scale rate before IQM aggregation and the 95% interval. It is reported only for arms with unit-scale rate IQM at least 10−3 ; otherwise retention and passing status are N/A and absolute rates remain visible. Passing requires retention at least 0.8.
Environment
Algorithm
Usable scales
Low scale
High scale
minatar_breakout
a2c
4
0.1
100
minatar_breakout
disco
5
0.01
100
minatar_breakout
disco-noema
5
0.01
100
minatar_breakout
disco-zerornn
5
0.01
100
minatar_freeway
a2c
2
1
10
minatar_freeway
disco
3
1
100
Appendix
Table 26: MinAtar usable reward scales. N/A indicates a unit-scale reference below the 10−3 reliability threshold.
Recipient scale
Condition
n
Tail score [95% interval]
0.001
Live
12
0.67855 [0.34494, 0.90503]
0.001
Own freeze at 400
12
0.67347 [0.41660, 0.89117]
0.001
Matched / init
12
0.89714 [0.68961, 0.98168]
0.001
Cross / init
12
0.53070 [0.20054, 0.86776]
0.001
Random / init
12
0.68372 [0.31915, 0.91384]
0.001
Matched / clamp
12
0.93261 [0.69950, 0.98213]
Appendix
Table 27: Inherited-state assay: all 192 recipient runs. Each arm uses 12 seeds at each recipient scale; scores are IQMs over final-20% normalized scores and brackets are pointwise 95% intervals. Init transfers donor h,c once and then permits evolution; clamp overwrites h,c each learner step. These marginal scores are distinct from the within-seed mismatch interaction, which is formed before aggregation.
Figure 10: Analytical and measured views of the value interface. (A) Analytical forward-transform values h(x), shown separately from measured occupancy. (B) An illustrative two-hot projection of transformed target 0.35 onto adjacent atoms 0 and 1; weights 0.65 and 0.35 preserve the first moment. (C–D) Catch interface experiments at scale 1, 12 runs per condition, with pointwise 95% seed-bootstrap intervals for IQM score and effective bins. (E) Executed scalar prediction and target paths: inverse transformation follows the categorical expectation; the target is forward transformed before clipping and two-hot projection. Panel B illustrates the mathematical projection.
Interface
n
Score [95% CI]
Effective bins [95% CI]
Score difference
default
12
0.99695 [0.98806, 0.99975]
2.2221 [2.1918, 2.2528]
0
linear
12
0.99949 [0.99517, 1]
1.4692 [1.4151, 1.5199]
0.0025397
support30
12
0.99962 [0.9939, 1]
2.098 [2.0282, 2.1705]
0.0026667
Appendix
Table 28: E-IF: categorical-interface interventions. Effective bins are the exponentiated entropy of the mean categorical distribution. Score differences subtract marginal IQMs. The column n gives the reported seed/run count.
Figure 11: Continuing experience and boundary information. A–D show all arms in episodic Catch, masked Catch, Tracking and CatchStream on their task-specific normalized reward-rate scales. A–C use 18 runs per arm; D uses 12. E retains the raw reward-rate unit for Breakout’s paired masking effect. F distinguishes terminal signals and physical resets. Intervals are pointwise 95% seed-bootstrap intervals.
Exposure
Algorithm
n
Reward-rate score [95% CI]
Threshold steps [95% CI]
Fraction reaching threshold
episodic
a2c
18
0.99615 [0.9958, 0.99657]
33640 [32712, 34568]
1
episodic
disco
18
0.99757 [0.99641, 0.9981]
22272 [21344, 23200]
1
episodic
dqn
18
0.92842 [0.92598, 0.93111]
64960 [64960, 65656]
1
episodic
ppo
18
0.98472 [0.94737, 0.98701]
13688 [11832, 17864]
1
masked
a2c
18
0.00070186 [-0.0015157, 0.0027543]
—
0
masked
disco
18
0.99651 [0.99433, 0.99761]
28304 [26442, 31320]
1
Appendix
Table 29: Termination-channel and reset-exposure comparisons. A blank threshold time denotes an unavailable finite estimate; fractions retain all evaluated runs. Threshold times use environment steps. The column n gives the reported seed/run count.
Exposure
Algorithm
n
Reward-rate score [95% CI]
Threshold steps [95% CI]
Fraction reaching threshold
synth100
a2c
18
-0.00027205 [-0.0037605, 0.0027645]
—
0
synth100
disco
18
0.99776 [0.99696, 0.99855]
30160 [28536, 32016]
1
synth29
a2c
18
0.99231 [0.98917, 0.99435]
1.0347×105 [94656, 1.29×105 ]
1
synth29
disco
18
0.99651 [0.99433, 0.99761]
28304 [26442, 31320]
1
synth7
a2c
18
0.99615 [0.9958, 0.99657]
33640 [32712, 34568]
1
synth7
disco
18
0.99757 [0.99641, 0.9981]
22272 [21344, 23200]
1
Appendix
Table 30: Synthetic-terminal intervals in masked Catch. Synthetic terminal intervals are 7, 29, and 100 environment steps; the environment’s underlying reset dynamics follow masked Catch. The column n gives the reported seed/run count.
Algorithm
n
Reward-rate score [95% CI]
Ever reaches 0.8
a2c
12
0.00622 [-0.00067487, 0.012812]
0
disco
12
0.93896 [0.92218, 0.95135]
1
dqn
12
0.76918 [0.76285, 0.77705]
1
Appendix
Table 31: E-A3c: learning with continuous CatchStream dynamics. The final score averages the last 15% of log rows. The final column records whether a run reaches 0.8 at any logged time. The column n gives the reported seed/run count.
Algorithm
n
Episodic IQM
Masked IQM
Masked minus episodic [95% CI]
disco
18
0.08749
0.0841
-0.0041612 [-0.0055838, -0.0011324]
a2c
18
0.084701
0.07773
-0.0071963 [-0.0082076, -0.0066663]
ppo
18
0.084262
0.081661
-0.0029975 [-0.0048288, -0.001675]
dqn
18
0.085854
0.083464
-0.002365 [-0.0029163, -0.0018238]
Appendix
Table 32: MinAtar termination masking by algorithm. The column n gives the reported seed/run count.
Contrast
n
Paired difference [95% CI]
masking penalty:disco -a2c
18
0.0036713 [0.0022662, 0.0067003]
masking penalty:disco -ppo
18
-0.000915 [-0.0033301, 0.0026551]
masking penalty:disco -dqn
18
-0.0014237 [-0.0033904, 0.001704]
Appendix
Table 33: Paired contrasts of MinAtar masking effects. The column n gives the reported seed/run count.
Figure 12: Fixed-origin adaptation profiles and auxiliary-loss effects. (A) Post-switch normalized AUC for 27 Catch cells, 18 seeds each. (B) Nine observation/reward-mapping cells, 12 seeds each, in normalized score units. (C) Eight Breakout cells, 18 seeds each, in absolute-return units; its color scale is separate from A–B. Gray cells are unexecuted configurations. All AUCs are divided by the post-switch duration. (D) Paired no-auxiliary-minus-live differences in AUC and final-window score; all 18 seeds and pointwise 95% bootstrap intervals are retained.
Grid
Condition
AUC [95% CI]
Final window [95% CI]
Target
RMD
8 × 8
Disco
0.563 [0.509, 0.610]
0.965 [0.904, 0.997]
18/18
798.9
8 × 8
Frozen
0.553 [0.499, 0.612]
0.928 [0.860, 0.985]
17/18
855.6
8 × 8
No aux.
0.618 [0.594, 0.643]
0.958 [0.950, 0.966]
18/18
631.1
8 × 8
DQN 100k
0.049 [0.041, 0.057]
0.351 [0.335, 0.370]
1/18
1500.0
8 × 8
DQN 28,672
0.656 [0.648, 0.662]
0.940 [0.935, 0.943]
18/18
537.8
8 × 8
DQN cleared
0.881 [0.874, 0.888]
0.934 [0.928, 0.938]
18/18
165.6
Appendix
Table 34: Action flip: fixed-origin Catch analysis, 18 seeds per condition. Post-switch AUC averages iterations 1500–3000; final-window capability averages 2700–3000. Intervals are pointwise 95% bootstrap intervals. The common target is normalized score ≥0.8 after 102 newly completed episodes. Restricted mean delay (RMD) includes all runs, assigning unattained runs the 1500-iteration horizon.
Grid
Condition
AUC [95% CI]
Final window [95% CI]
Target
RMD
8 × 8
Disco
0.523 [0.503, 0.553]
1.000 [0.999, 1.000]
18/18
761.1
8 × 8
Frozen
0.498 [0.484, 0.509]
1.000 [0.999, 1.000]
18/18
798.9
8 × 8
No aux.
0.655 [0.636, 0.681]
0.980 [0.972, 0.984]
18/18
526.7
8 × 8
DQN 100k
0.050 [0.028, 0.071]
0.526 [0.472, 0.576]
0/18
1500.0
8 × 8
DQN 28,672
0.453 [0.435, 0.476]
0.736 [0.692, 0.781]
11/18
1257.8
8 × 8
DQN cleared
0.622 [0.599, 0.651]
0.788 [0.738, 0.850]
14/18
1082.2
Appendix
Table 35: Reward move: fixed-origin Catch analysis, 18 seeds per condition. Post-switch AUC averages iterations 1500–3000; final-window capability averages 2700–3000. Intervals are pointwise 95% bootstrap intervals. The common target is normalized score ≥0.8 after 102 newly completed episodes. Restricted mean delay (RMD) includes all runs, assigning unattained runs the 1500-iteration horizon.
Grid
Condition
AUC [95% CI]
Final window [95% CI]
Target
RMD
8 × 8
Disco
0.994 [0.992, 0.996]
0.998 [0.994, 0.999]
18/18
23.3
8 × 8
Frozen
0.995 [0.994, 0.996]
0.999 [0.996, 1.000]
18/18
20.0
8 × 8
No aux.
0.974 [0.953, 0.983]
0.997 [0.990, 0.999]
18/18
80.0
8 × 8
DQN 100k
0.934 [0.932, 0.936]
0.929 [0.923, 0.934]
18/18
20.0
8 × 8
DQN 28,672
0.932 [0.930, 0.933]
0.934 [0.929, 0.938]
18/18
20.0
8 × 8
DQN cleared
0.932 [0.929, 0.935]
0.933 [0.928, 0.939]
18/18
20.0
Appendix
Table 36: No change: fixed-origin Catch analysis, 18 seeds per condition. Post-switch AUC averages iterations 1500–3000; final-window capability averages 2700–3000. Intervals are pointwise 95% bootstrap intervals. The common target is normalized score ≥0.8 after 102 newly completed episodes. Restricted mean delay (RMD) includes all runs, assigning unattained runs the 1500-iteration horizon.
Grid
Change
Metric
Paired difference [95% CI]
8 × 8
Action flip
AUC
0.062 [0.009, 0.107]
8 × 8
Action flip
Final window
-0.010 [-0.037, 0.052]
8 × 8
Reward move
AUC
0.125 [0.091, 0.172]
8 × 8
Reward move
Final window
-0.020 [-0.027, -0.015]
8 × 8
No change
AUC
-0.018 [-0.039, -0.009]
8 × 8
No change
Final window
-0.000 [-0.005, 0.002]
Appendix
Table 37: No-auxiliary minus Disco on all 18 paired seeds per condition. Each difference is formed within seed before trimming; pointwise 95% intervals use 10,000 paired bootstrap replicates. No successful-run conditioning is applied.
Figure 13: Action-flip adaptation at two Catch grid sizes. (A,C) Normalized post-switch AUC and final-window capability at 8×8 and 16×16 . (B,D) Restricted mean time to normalized score 0.8, measured from the switch with all runs and censored non-attainers included. Each configuration has 18 seeds, a switch at 87,000 interactions, and 87,000 post-switch interactions. Labels show common-threshold attained/all. The 8×8 matrix includes three Disco variants and three DQN replay configurations; 16×16 includes three Disco variants. Bars are pointwise 95% seed-bootstrap intervals.
Change
Condition
Pre
Post-switch AUC [95% CI]
Endpoint [95% CI]
Action flip
A2C
5.029
3.511 [3.126, 3.794]
5.218 [4.678, 5.476]
Action flip
Disco
9.804
5.373 [5.070, 5.741]
8.763 [8.134, 9.341]
Action flip
Frozen
9.439
5.349 [4.914, 5.934]
8.462 [7.616, 9.228]
Action flip
PPO
5.120
3.978 [3.775, 4.189]
5.321 [5.040, 5.671]
No change
A2C
4.879
5.466 [5.349, 5.813]
5.691 [5.555, 6.267]
No change
Disco
8.851
10.772 [9.823, 11.724]
12.299 [10.988, 13.625]
Appendix
Table 38: MinAtar Breakout: absolute return, 18 seeds per condition, switch at 4000 and endpoint at 8000. AUC is the time-average from the declared switch to the endpoint, recomputed from complete trajectories. Endpoint values are independently reproduced from all run endpoints. Intervals are pointwise 95% bootstrap intervals; no own-performance threshold is used.
Change
Condition
Pre
Post-switch AUC [95% CI]
Endpoint [95% CI]
No change
A2C
0.990
0.993 [0.992, 0.994]
0.992 [0.989, 0.998]
No change
Disco
0.997
0.995 [0.991, 0.997]
1.000 [1.000, 1.000]
No change
Frozen
1.000
0.997 [0.996, 0.998]
1.000 [1.000, 1.000]
Observation flip
A2C
0.990
0.243 [0.181, 0.347]
0.326 [0.253, 0.463]
Observation flip
Disco
0.997
0.626 [0.577, 0.690]
0.992 [0.941, 0.998]
Observation flip
Frozen
1.000
0.604 [0.566, 0.673]
1.000 [0.996, 1.000]
Appendix
Table 39: Catch mapping: random-zero/oracle-one normalized return, 12 seeds per condition, switch at 1500 and endpoint at 3000. AUC is the time-average from the declared switch to the endpoint, recomputed from complete trajectories. Endpoint values are independently reproduced from all run endpoints. Intervals are pointwise 95% bootstrap intervals; no own-performance threshold is used.
Figure 14: Responses to observation and reward changes. (A) Pre-switch, trough, and endpoint normalized scores for observation reversal, reward-sign reversal, and stationary exposure. (B) Post-switch AUC integrates from the configured switch at iteration 1,500 to 3,000. Each condition has 12 seeds and reference indexing; bars are pointwise 95% seed-bootstrap intervals. The trough is a descriptive score, not the clock’s origin. Frozen-state agents continue updating parameters.
Algorithm
Depth
n
Solved fraction
Ever found
Tail return [95% CI]
Deepest majority solved
a2c
8
15
1
1
0.99057 [0.99, 0.99105]
12
a2c
10
15
1
1
0.99025 [0.98994, 0.99068]
12
a2c
12
15
0.73333
0.86667
0.84451 [0.47917, 0.99021]
12
a2c
14
15
0
0.46667
-0.0046868 [-0.0048272, -0.0044202]
12
a2c
16
15
0
0.13333
-0.0048606 [-0.004884, -0.0048315]
12
disco
8
15
1
1
0.94435 [0.89122, 0.98534]
12
Appendix
Table 40: E-A5: complete reference-convention staircase. A run is solved when its final-20%-of-time mean return reaches 0.495. Majority solved uses fraction >= 0.5; ever found checks any positive logged return. The column n gives the reported seed/run count.
Convention
Depth
n
Disco solved
A2C solved
Disco minus A2C [95% CI]
legacy
12
15
12
11
0.00034352 [-0.22028, 0.31887]
reference
12
15
10
11
-0.0030685 [-0.33376, 0.2212]
legacy
14
15
6
0
0.32822 [0.004777, 0.7605]
reference
14
15
4
0
0.10269 [0.0047881, 0.44483]
Appendix
Table 41: F1: paired convention checks at DeepSea depths 12 and 14. F1 is a separately indexed paired slice. Its counts remain attached to that slice; the complete E-A5 staircase is reported in a separate table. The column n gives the reported seed/run count.
Figure 15: Reward discovery and sustained solution. A–C show all reference-indexed E-A5 cells: solved fraction, ever-found fraction and final-return IQM, with 15 seeds per cell. C carries pointwise 95% seed-bootstrap intervals. D displays the independent F1 reference-indexed check at depths 12 and 14, which uses different runs. Lines connect tested depths.
Figure 16: State, scale, inheritance, and the value interface. (A) Catch’s four-arm state/EMA scale response. (B) All four arms’ usable windows at score 0.8 in three environments, using the E1 final-20% score and pointwise 95% intervals. (C) E2 matched/cross donor scores under natural transfer and repeated clamping, at both recipient scales. The plotted coordinate is the marginal score; the separately labeled interaction is the paired contrast in Equation 8 . (D) Unit-scale value-interface performance and effective categorical use. All cells contain 12 seeds. Marginal score intervals are pointwise; interaction intervals follow the family-11 procedure.
Figure 17: Behavior across retained data, streams, visual change, and exploration. (A) Catch action-flip Disco-minus-DQN post-switch AUC under capacity 100,000, matched capacity 28,672, and switch-time clearing; 18 paired seeds and family-11 intervals. (B) CatchStream and Tracking normalized reward rates, 12 and 18 seeds per arm, respectively. (C) Breakout action-reversal pre-switch and endpoint absolute returns, 18 seeds per arm. (D) The five-method DeepSea staircase, five depths and 15 seeds per cell. These panels use the same quantities as the corresponding main-text and detailed appendix analyses.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Recursive self-improvement (RSI) remains hard in long-horizon tasks, where growing histories obscure the task state and misalign skill invocation. We introduce Recuris, a recursive Experiential-Working Memory architecture for long-horizon agent harnesses, in which Working Memory tracks task progress and guides skill selection from Experiential Memory, grounding skill use in current needs rather than the full history. This coupling also turns execution into structured evidence that localizes failures to specific memory components. Across tasks, a fixed Meta-Agent turns that evidence into localized, validation-gated updates to Skill Memory that reshape execution and yield new evidence, forming a bounded recursive memory-evolution loop. Across four long-horizon benchmarks and ten models, Recuris improves task success in 35 of the 37 completed model-benchmark pairs, carrying frontier models to SOTA-level task success: on tau-bench it adds +17.8 points to GPT-5.6 Sol and +15.6 to Claude Opus 5, taking Opus 5 to 87.9%, and +16.6/+13.5 points on Qwen3.6-27B/35B on SkillFlow. The advantage widens as the interaction horizon grows, to +32.2 points on the longest tasks, and common long-horizon failures fall by up to 80%. These results position recursively evolving memory as a scalable foundation for RSI, enabling agents to continuously transform accumulated experience into increasingly effective long-horizon behavior. Code: https://github.com/Gen-Verse/Recuris
Zhaochen Yu, Yingcheng Wu, Zhenfei Yin +5
NUS · Princeton University · Stanford University +1
Recursive self-improvement is becoming essential for autonomous AI agents, whose progress depends on discovering high-value solutions across complex domains. Effective exploration drives this process, yet managing and improving exploration strategies remains a major bottleneck. Current systems face a fundamental dilemma: fixed strategies fail to adapt as search spaces scale, while online policy optimization must navigate vast meta-search spaces under delayed, expensive feedback from long-horizon rollouts. We introduce Dream-RSI, a framework for scalable, recursively self-improving exploration. A lightweight orchestration layer makes exploration explicit and programmable while leaving the underlying base agent unchanged. Our key insight is that accumulated discovery history can act as a replay simulator over the realized search space. By dreaming within this simulator built from historical discovery trees, Dream-RSI obtains immediate, low-cost off-policy feedback to evaluate and refine exploration policies without repeated, expensive online evaluation. The improved policy is then redeployed online to drive further discovery, continuously expanding the simulator pool in a self-improving loop. Across 9 tasks in 4 domains, Dream-RSI achieves competitive quality and improves discovery efficiency in several settings.
Tong Zheng, Xidong Wu, Zheng Zhang +14
Google · University of Maryland, College Park · Google Deepmind +1
An agent can improve its behavior within an unresolved problem by changing how it acts and which experience it retains. We propose MMLA-RSI, a recursive self-improvement scheme with two implementations in Memory-Mediated Learning Architecture: Reasoning-Time Training (RTT) changes the policy state Φ, while Reasoning-Time Memory Adaptation (RTMA) changes authoritative memory M. Each route also makes its adaptation method an object of improvement. A current numerical updater trains the policy proposing its successor; a current memory manager chooses the records available to the process proposing its successor. An independently admitted successor takes over the same role in the next generation. We give bounded executable rule languages, shared task/proposal interfaces, and one algorithm for either route or their composition. The RTT instance changes scalar schedules in a clipped-momentum update. The RTMA instance chooses NULL or a complete-row write with possible eviction. A fixed interpreter, trusted committer, and independent evaluator bound what can change. State versions, read closures, and conditional properties describe causal execution, accepted transitions, and finite work. Existing numerical-component studies motivate the construction. In four-candidate program selection, retaining LARC feedback updates reduces expected error by 24.65 and 36.65 percentage points for two trained initializations. In chronological continuous-integration replay, feedback binding is positive while retention increases loss. Completed SQL studies improve substantially over an unadapted substrate but do not establish the required additional online benefit. These results characterize their declared components. The two recursive routes and their joint realization are specified here; their beneficial recursive chains and discovery efficiency remain to be evaluated.
Junyi Zou, Avrova Donz
MMLA-org · Communication University of China (CUC), No. 1 Dingfuzhuang East Street, Chaoyang District, Beijing 100024, China