MMLA-RSI: Recursive Self-Improvement through RTT and RTMA

Junyi Zou; Avrova Donz

cs.CLOct 14, 2026

MMLA-RSI: Recursive Self-Improvement through RTT and RTMA

Authors: Junyi Zou, Avrova Donz

Organizations: MMLA-org · Communication University of China (CUC), No. 1 Dingfuzhuang East Street, Chaoyang District, Beijing 100024, China

Abstract

An agent can improve its behavior within an unresolved problem by changing how it acts and which experience it retains. We propose MMLA-RSI, a recursive self-improvement scheme with two implementations in Memory-Mediated Learning Architecture: Reasoning-Time Training (RTT) changes the policy state $Φ$ , while Reasoning-Time Memory Adaptation (RTMA) changes authoritative memory $M$ . Each route also makes its adaptation method an object of improvement. A current numerical updater trains the policy proposing its successor; a current memory manager chooses the records available to the process proposing its successor. An independently admitted successor takes over the same role in the next generation. We give bounded executable rule languages, shared task/proposal interfaces, and one algorithm for either route or their composition. The RTT instance changes scalar schedules in a clipped-momentum update. The RTMA instance chooses NULL or a complete-row write with possible eviction. A fixed interpreter, trusted committer, and independent evaluator bound what can change. State versions, read closures, and conditional properties describe causal execution, accepted transitions, and finite work. Existing numerical-component studies motivate the construction. In four-candidate program selection, retaining LARC feedback updates reduces expected error by 24.65 and 36.65 percentage points for two trained initializations. In chronological continuous-integration replay, feedback binding is positive while retention increases loss. Completed SQL studies improve substantially over an unadapted substrate but do not establish the required additional online benefit. These results characterize their declared components. The two recursive routes and their joint realization are specified here; their beneficial recursive chains and discovery efficiency remain to be evaluated.

Figures & tables

Object	Operation and lifetime	Restoration or transition
$\theta_{0},\omega$	Bound pretrained substrate and shared offline adaptation	Verify original identity; never recreate by retraining.
$\rho$	Slow initialization of the numerical carrier	Copied at problem start; bound for each comparison.
$\Phi$	Private state changed by feedback in the current problem	Reset to the problem’s bound $\rho$ .
$\eta^{\Phi},\eta^{M}$	Numerical updater and memory-management rule	The selected domain changes only by admitted version replacement.
$M$	Bounded authoritative records with verified complete-row commits	Starts from bound $M_{0}$ ; independently retained, reset, or held fixed.
$\Xi^{\rm read}$	Every additional dependency of a future action	Restored with the intervention’s declared closure.

Table 1: Ownership and lifetime. Policy learning, memory learning, and changes to their adaptation methods are distinct operations.

Figure 2: The shared timing condition. The unresolved task persists from attempt to feedback, commit, and a later pinned read. RTT commits a policy-state update; RTMA commits a non-NULL memory successor. Independent future audit outcomes remain outside both paths.

Witness	Required evidence
W1	The input and task identity remain the same and the problem is unresolved.
W2	Feedback is bound to a completed attempt and has arrived before the update.
W3	A genuine learning-state change is committed, with a version and read closure.
W4	The update is causal, belongs to the current epoch, and does not use future information.
W5	The commit precedes a fresh proposal and both occur before resolution.
W6	The next proposal actually loads the committed state version.

Table 2: The RTT witness. W1–W6 and W8 establish the process; W7 adds functional use. Task benefit is a separate empirical quantity.

Figure 3: Two controlled state paths and their read closure. Purple marks the policy path; teal marks authoritative memory. Context, caches, selectors, random streams, and callbacks are additional dependencies. Each intervention specifies its target and the other paths held fixed.

Figure 4: Two realizations of parent participation. The RTT parent updates the numerical proposal state. The RTMA parent manages the authoritative evidence read by the next proposal. Selection and independent audit are shared; an admitted rule returns to its own role in the next generation. Joint runs carry both rules.

Component study	Main observation	Strong comparison or boundary
Four-program selection	Retention reduces expected error by 24.65 and 36.65 points for static and adapted initializations.	Real-keep error 29.302% / 24.611%; support-loss selection 0.78125%.
Chronological CI replay	$G=-0.05337$ while $D=+0.14969$ : real binding helps relative to sham, retention harms.	Half-Brier loss rises from 0.12744 to 0.18081; HEDGE4 reaches 0.10243.
SParC interaction	Whole-interaction success 18.42% versus frozen six-proposal 20.18%; paired gain $-1.75$ points.	97.5% interval $[-4.39,0.88]$ ; strict prefix-read task gains are zero.
CoSQL correction transfer	Final-query success 46.965% versus frozen 47.944%; paired gain $-0.979$ points.	97.5% interval $[-4.247,1.759]$ ; neither arm passes its online $\Phi$ requirements.

Table 3: Existing component evidence, not an evaluation of the proposed recursive routes. Percentage-point differences and half-Brier differences have different units; the studies are not pooled.

Route	Main changed object	Dependence carried into later improvement
aTTT	Episode-specific numerical adapter	Updated policy changes later trajectory text under a prescribed update rule.
RRSI	Agent harness around a frozen model	Successor harness produces the next round’s evidence and evolution history.
MemEvolve	Programmable memory architecture	Architecture-dependent execution traces inform selection, diagnosis, and redesign of memory modules.
OPSD	Student policy	Student sampling changes; the experimental teacher remains fixed.
DCE+SRCL	Student and privileged teacher checkpoints	Updated ability changes the next round’s supervision.
STOP / DGM	Improver program / self-modifying agent	Successor can act in producing a later descendant.

Table 4: Different realizations of self-improvement. The table identifies demonstrated dependencies in the cited methods and the specified dependency of the proposed MMLA-RSI system. It is not a performance ranking.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Initialization	Real keep	Real reset	Sham keep	Sham reset
Static objective	29.302	53.948	57.666	53.948
Adapted objective	24.611	61.259	60.379	61.259

Appendix

Table 5: Expected program-selection error (%, lower is better). Reset restores each arm’s own trained initialization. The support-loss selector reaches 0.78125% error.

Contrast	Mean (percentage points)	99% group interval
Static retention $G$	24.65	[19.39, 29.26]
Static binding $D$	28.36	[21.85, 34.93]
Adapted retention $G$	36.65	[31.65, 41.20]
Adapted binding $D$	35.77	[30.33, 41.01]
Static error $-$ adapted error, real keep	4.69	[1.19, 8.88]

Appendix

Table 6: Within-arm feedback effects and between-objective increment answer separate questions. Intervals are conditional on the three paired training seeds.

Method, with recorded memory condition	Mean half-Brier loss
Shared real/sham reset read, real $M$	0.12744
Real feedback, keep $\Phi$ , real $M$	0.18081
Permuted feedback, keep $\Phi$ , real $M$	0.33050
HEDGE4	0.10243
Real retention $G$	$-0.05337$
Feedback binding $D$	$+0.14969$

Appendix

Table 7: CI replay. The three-seed means are descriptive summaries on two recorded repositories, not population confidence intervals. HEDGE4 has the lowest mean loss among the methods shown.

Figure 5: Binding is not the same as benefit. Left: program-selection retention and binding effects with original 99% development-group intervals. Right: CI effects in half-Brier units, shown as three-seed means without population intervals. Positive values favor the named effect in both panels; the units and populations differ.

Confirmation method	Final-query success (%)
Original frozen substrate, six proposals	7.295
Development-selected copy-edit rule	1.824
Retrieval rule	5.348
Search rule	0.961
Static task model, six frozen proposals	47.944
Static task model, real-feedback procedure	48.230

Appendix

Table 8: Completed CoSQL correction-transfer evaluation. The copy-edit rule remains the selected rule even though retrieval scores higher on confirmation. The current, separate contrast-training evaluation is not included.

Arm and controlled effect	Mean (percentage points)	95% database interval
Static retention	0.775	$[-0.638,2.930]$
Static binding	0.387	$[-0.196,0.971]$
Adapted retention	0.580	$[0.000,1.565]$
Adapted binding	0.423	$[-0.125,1.260]$

Appendix

Table 9: CoSQL controlled fast-state effects. Each arm requires both binding and retention lower bounds strictly above zero; neither meets that requirement. The strict checks use the stored unrounded bounds; the adapted-retention lower bound is exactly zero in the saved aggregate. These paired prefix reads differ from the complete procedures in Table 8 .

Study	Accounting scope	Recorded quantity	Value
Program selection	Full entry, including production preflight	Wall seconds	1,634.50
	Contract charge, including data work	Seconds	1,640.27
	Physical / effective token positions	Millions	49.026 / 19.521
CI	Qualification, task fit, and repeated evaluation	Native wall hours	3.675
SQL interaction R5	This successful R5 invocation	GPU residency upper hours	4.819
	Budgeted process-tree CPU	CPU hours	5.641

Appendix

Table 10: Recorded component costs. Wall time, process-tree CPU, GPU residency, and effective computation are separate quantities. They must not be summed as though they were one resource measure.

Self-Improving Agents

Self-improving agents use experience and feedback to update their models, memories, skills, or workflows for better future task performance.

See all 167 papers

Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks
Oct 8, 2026Haoyu Zhao, Zhengxu Yu, Zhiyuan He +5Continual Learning for LLM Agents Recursive Self-Improvement
University College London · Huawei Noah’s Ark Lab, UK · University of Liverpool
Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses
Aug 25, 2026Zhaochen Yu, Yingcheng Wu, Zhenfei Yin +5Long-Horizon LLM Agents Memory-Augmented Agents
NUS · Princeton University · Stanford University +1
RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
Sep 21, 2026Peng Xia, Rujun Han, Zifeng Wang +11OOD Generalization Agent Harness Optimization
Google Cloud AI Research · UNC-Chapel Hill · Stanford University +1

Oct 8, 2026J/K move · Enter open · S save

Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks

Haoyu Zhao, Zhengxu Yu, Zhiyuan He, Meng Fang+4

University College London · Huawei Noah’s Ark Lab, UK · University of Liverpool

Continual Learning for LLM Agents Recursive Self-Improvement Memory-Augmented Language Model Agents Model-Based RL ARC-AGI

Open paper AskPDF arXiv

Learning to act in unfamiliar environments requires agents to infer how the world works and revise that understanding as new evidence arrives. Yet limited observations can support multiple world models that explain past interactions but predict different outcomes in unseen states. We introduce Memento 3, building on the Memento series to enable frozen LLM agents to continually learn explicit world models through external memory. The agent maintains a natural-language rulebook as persistent semantic memory, recording revisable hypotheses about environment dynamics while leaving unknown aspects underspecified. It compiles this rulebook into executable code for prediction and planning. Through a continual loop of observation, reflection, rule revision, compilation, and verification, the agent uses prediction errors to refine both the rulebook and its code. Updated code is accepted only when the LLM judges it faithful to the rulebook and cell-exact replay reproduces the observed transitions. We investigate this process as a model-based route to recursive self-improvement (RSI): the agent autonomously explores the environment, revises its world model, and uses verified updates to guide subsequent interaction and learning, while the underlying LLM remains fixed. A population extension maintains multiple world models in parallel, sharing interaction evidence and using their predictions to guide exploration. On ARC-AGI-3, the single-model agent clears every level of all 25 public games, achieves a mean Relative Human Action Efficiency (RHAE) of 100.0, and uses 44% of the human action count. In an Atari Pong case study, a learned feedback controller wins 21:0 in each of three evaluated episodes with different openings, without further LLM calls.

Abstract

Self-Improving Agents

Explore similar work

Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks

Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses

RRSI: Regularized Recursive Self-Improvement of Agent Harnesses