MMLA-RSI: Recursive Self-Improvement through RTT and RTMA

Junyi Zou; Avrova Donz

cs.CLOct 14, 2026

MMLA-RSI: Recursive Self-Improvement through RTT and RTMA

Authors: Junyi Zou, Avrova Donz

Organizations: MMLA-org · Communication University of China (CUC), No. 1 Dingfuzhuang East Street, Chaoyang District, Beijing 100024, China

Abstract

An agent can improve its behavior within an unresolved problem by changing how it acts and which experience it retains. We propose MMLA-RSI, a recursive self-improvement scheme with two implementations in Memory-Mediated Learning Architecture: Reasoning-Time Training (RTT) changes the policy state $Φ$ , while Reasoning-Time Memory Adaptation (RTMA) changes authoritative memory $M$ . Each route also makes its adaptation method an object of improvement. A current numerical updater trains the policy proposing its successor; a current memory manager chooses the records available to the process proposing its successor. An independently admitted successor takes over the same role in the next generation. We give bounded executable rule languages, shared task/proposal interfaces, and one algorithm for either route or their composition. The RTT instance changes scalar schedules in a clipped-momentum update. The RTMA instance chooses NULL or a complete-row write with possible eviction. A fixed interpreter, trusted committer, and independent evaluator bound what can change. State versions, read closures, and conditional properties describe causal execution, accepted transitions, and finite work. Existing numerical-component studies motivate the construction. In four-candidate program selection, retaining LARC feedback updates reduces expected error by 24.65 and 36.65 percentage points for two trained initializations. In chronological continuous-integration replay, feedback binding is positive while retention increases loss. Completed SQL studies improve substantially over an unadapted substrate but do not establish the required additional online benefit. These results characterize their declared components. The two recursive routes and their joint realization are specified here; their beneficial recursive chains and discovery efficiency remain to be evaluated.

Figures & tables

Object	Operation and lifetime	Restoration or transition
$\theta_{0},\omega$	Bound pretrained substrate and shared offline adaptation	Verify original identity; never recreate by retraining.
$\rho$	Slow initialization of the numerical carrier	Copied at problem start; bound for each comparison.
$\Phi$	Private state changed by feedback in the current problem	Reset to the problem’s bound $\rho$ .
$\eta^{\Phi},\eta^{M}$	Numerical updater and memory-management rule	The selected domain changes only by admitted version replacement.
$M$	Bounded authoritative records with verified complete-row commits	Starts from bound $M_{0}$ ; independently retained, reset, or held fixed.
$\Xi^{\rm read}$	Every additional dependency of a future action	Restored with the intervention’s declared closure.

Table 1: Ownership and lifetime. Policy learning, memory learning, and changes to their adaptation methods are distinct operations.

Figure 2: The shared timing condition. The unresolved task persists from attempt to feedback, commit, and a later pinned read. RTT commits a policy-state update; RTMA commits a non-NULL memory successor. Independent future audit outcomes remain outside both paths.

Witness	Required evidence
W1	The input and task identity remain the same and the problem is unresolved.
W2	Feedback is bound to a completed attempt and has arrived before the update.
W3	A genuine learning-state change is committed, with a version and read closure.
W4	The update is causal, belongs to the current epoch, and does not use future information.
W5	The commit precedes a fresh proposal and both occur before resolution.
W6	The next proposal actually loads the committed state version.

Table 2: The RTT witness. W1–W6 and W8 establish the process; W7 adds functional use. Task benefit is a separate empirical quantity.

Figure 3: Two controlled state paths and their read closure. Purple marks the policy path; teal marks authoritative memory. Context, caches, selectors, random streams, and callbacks are additional dependencies. Each intervention specifies its target and the other paths held fixed.

Figure 4: Two realizations of parent participation. The RTT parent updates the numerical proposal state. The RTMA parent manages the authoritative evidence read by the next proposal. Selection and independent audit are shared; an admitted rule returns to its own role in the next generation. Joint runs carry both rules.

Component study	Main observation	Strong comparison or boundary
Four-program selection	Retention reduces expected error by 24.65 and 36.65 points for static and adapted initializations.	Real-keep error 29.302% / 24.611%; support-loss selection 0.78125%.
Chronological CI replay	$G=-0.05337$ while $D=+0.14969$ : real binding helps relative to sham, retention harms.	Half-Brier loss rises from 0.12744 to 0.18081; HEDGE4 reaches 0.10243.
SParC interaction	Whole-interaction success 18.42% versus frozen six-proposal 20.18%; paired gain $-1.75$ points.	97.5% interval $[-4.39,0.88]$ ; strict prefix-read task gains are zero.
CoSQL correction transfer	Final-query success 46.965% versus frozen 47.944%; paired gain $-0.979$ points.	97.5% interval $[-4.247,1.759]$ ; neither arm passes its online $\Phi$ requirements.

Table 3: Existing component evidence, not an evaluation of the proposed recursive routes. Percentage-point differences and half-Brier differences have different units; the studies are not pooled.

Route	Main changed object	Dependence carried into later improvement
aTTT	Episode-specific numerical adapter	Updated policy changes later trajectory text under a prescribed update rule.
RRSI	Agent harness around a frozen model	Successor harness produces the next round’s evidence and evolution history.
MemEvolve	Programmable memory architecture	Architecture-dependent execution traces inform selection, diagnosis, and redesign of memory modules.
OPSD	Student policy	Student sampling changes; the experimental teacher remains fixed.
DCE+SRCL	Student and privileged teacher checkpoints	Updated ability changes the next round’s supervision.
STOP / DGM	Improver program / self-modifying agent	Successor can act in producing a later descendant.

Table 4: Different realizations of self-improvement. The table identifies demonstrated dependencies in the cited methods and the specified dependency of the proposed MMLA-RSI system. It is not a performance ranking.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Initialization	Real keep	Real reset	Sham keep	Sham reset
Static objective	29.302	53.948	57.666	53.948
Adapted objective	24.611	61.259	60.379	61.259

Appendix

Table 5: Expected program-selection error (%, lower is better). Reset restores each arm’s own trained initialization. The support-loss selector reaches 0.78125% error.

Contrast	Mean (percentage points)	99% group interval
Static retention $G$	24.65	[19.39, 29.26]
Static binding $D$	28.36	[21.85, 34.93]
Adapted retention $G$	36.65	[31.65, 41.20]
Adapted binding $D$	35.77	[30.33, 41.01]
Static error $-$ adapted error, real keep	4.69	[1.19, 8.88]

Appendix

Table 6: Within-arm feedback effects and between-objective increment answer separate questions. Intervals are conditional on the three paired training seeds.

Method, with recorded memory condition	Mean half-Brier loss
Shared real/sham reset read, real $M$	0.12744
Real feedback, keep $\Phi$ , real $M$	0.18081
Permuted feedback, keep $\Phi$ , real $M$	0.33050
HEDGE4	0.10243
Real retention $G$	$-0.05337$
Feedback binding $D$	$+0.14969$

Appendix

Table 7: CI replay. The three-seed means are descriptive summaries on two recorded repositories, not population confidence intervals. HEDGE4 has the lowest mean loss among the methods shown.

Figure 5: Binding is not the same as benefit. Left: program-selection retention and binding effects with original 99% development-group intervals. Right: CI effects in half-Brier units, shown as three-seed means without population intervals. Positive values favor the named effect in both panels; the units and populations differ.

Confirmation method	Final-query success (%)
Original frozen substrate, six proposals	7.295
Development-selected copy-edit rule	1.824
Retrieval rule	5.348
Search rule	0.961
Static task model, six frozen proposals	47.944
Static task model, real-feedback procedure	48.230

Appendix

Table 8: Completed CoSQL correction-transfer evaluation. The copy-edit rule remains the selected rule even though retrieval scores higher on confirmation. The current, separate contrast-training evaluation is not included.

Arm and controlled effect	Mean (percentage points)	95% database interval
Static retention	0.775	$[-0.638,2.930]$
Static binding	0.387	$[-0.196,0.971]$
Adapted retention	0.580	$[0.000,1.565]$
Adapted binding	0.423	$[-0.125,1.260]$

Appendix

Table 9: CoSQL controlled fast-state effects. Each arm requires both binding and retention lower bounds strictly above zero; neither meets that requirement. The strict checks use the stored unrounded bounds; the adapted-retention lower bound is exactly zero in the saved aggregate. These paired prefix reads differ from the complete procedures in Table 8 .

Study	Accounting scope	Recorded quantity	Value
Program selection	Full entry, including production preflight	Wall seconds	1,634.50
	Contract charge, including data work	Seconds	1,640.27
	Physical / effective token positions	Millions	49.026 / 19.521
CI	Qualification, task fit, and repeated evaluation	Native wall hours	3.675
SQL interaction R5	This successful R5 invocation	GPU residency upper hours	4.819
	Budgeted process-tree CPU	CPU hours	5.641

Appendix

Table 10: Recorded component costs. Wall time, process-tree CPU, GPU residency, and effective computation are separate quantities. They must not be summed as though they were one resource measure.

Self-Improving Agents

Self-improving agents use experience and feedback to update their models, memories, skills, or workflows for better future task performance.

See all 167 papers

Oct 8, 2026

Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks

Learning to act in unfamiliar environments requires agents to infer how the world works and revise that understanding as new evidence arrives. Yet limited observations can support multiple world models that explain past interactions but predict different outcomes in unseen states. We introduce Memento 3, building on the Memento series to enable frozen LLM agents to continually learn explicit world models through external memory. The agent maintains a natural-language rulebook as persistent semantic memory, recording revisable hypotheses about environment dynamics while leaving unknown aspects underspecified. It compiles this rulebook into executable code for prediction and planning. Through a continual loop of observation, reflection, rule revision, compilation, and verification, the agent uses prediction errors to refine both the rulebook and its code. Updated code is accepted only when the LLM judges it faithful to the rulebook and cell-exact replay reproduces the observed transitions. We investigate this process as a model-based route to recursive self-improvement (RSI): the agent autonomously explores the environment, revises its world model, and uses verified updates to guide subsequent interaction and learning, while the underlying LLM remains fixed. A population extension maintains multiple world models in parallel, sharing interaction evidence and using their predictions to guide exploration. On ARC-AGI-3, the single-model agent clears every level of all 25 public games, achieves a mean Relative Human Action Efficiency (RHAE) of 100.0, and uses 44% of the human action count. In an Atari Pong case study, a learned feedback controller wins 21:0 in each of three evaluated episodes with different openings, without further LLM calls.

Haoyu Zhao, Zhengxu Yu, Zhiyuan He +5

University College London · Huawei Noah’s Ark Lab, UK · University of Liverpool

Continual Learning for LLM Agents Recursive Self-Improvement Memory-Augmented Language Model Agents Model-Based RL ARC-AGI

Ask PDF

Aug 25, 2026

Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses

Recursive self-improvement (RSI) remains hard in long-horizon tasks, where growing histories obscure the task state and misalign skill invocation. We introduce Recuris, a recursive Experiential-Working Memory architecture for long-horizon agent harnesses, in which Working Memory tracks task progress and guides skill selection from Experiential Memory, grounding skill use in current needs rather than the full history. This coupling also turns execution into structured evidence that localizes failures to specific memory components. Across tasks, a fixed Meta-Agent turns that evidence into localized, validation-gated updates to Skill Memory that reshape execution and yield new evidence, forming a bounded recursive memory-evolution loop. Across four long-horizon benchmarks and ten models, Recuris improves task success in 35 of the 37 completed model-benchmark pairs, carrying frontier models to SOTA-level task success: on tau-bench it adds +17.8 points to GPT-5.6 Sol and +15.6 to Claude Opus 5, taking Opus 5 to 87.9%, and +16.6/+13.5 points on Qwen3.6-27B/35B on SkillFlow. The advantage widens as the interaction horizon grows, to +32.2 points on the longest tasks, and common long-horizon failures fall by up to 80%. These results position recursively evolving memory as a scalable foundation for RSI, enabling agents to continuously transform accumulated experience into increasingly effective long-horizon behavior. Code: https://github.com/Gen-Verse/Recuris

Zhaochen Yu, Yingcheng Wu, Zhenfei Yin +5

NUS · Princeton University · Stanford University +1

Long-Horizon LLM Agents Memory-Augmented Agents Self-Improving Agents Recursive Self-Improvement LLM Agent Skill Learning Long-Horizon Reasoning

Ask PDF

Sep 21, 2026

RRSI: Regularized Recursive Self-Improvement of Agent Harnesses

An LLM agent's capability is largely magnified by its harness, namely the prompts, control flow, tooling, memory, and context management surrounding the frozen backbone model. Recent methods increasingly automate this process by iteratively proposing and selecting component-wise edits of an agent harness, practically establishing a form of recursive self-improvement (RSI) at the agent-system level. However, such recursive evolution may overfit by memorizing the training tasks, showing large in-distribution gains that shrink or even vanish on out-of-distribution benchmarks. We introduce Regularized Recursive Self-Improvement of Agent Harnesses (RRSI), which incorporates the principles of regularizations into harness self-improvement by constraining the evolution candidate proposal and selection. The proposer operates with a temporally annealed budget, limiting how many edits a candidate can bundle, and it encourages unexplored trajectories based on evolution history. The selector is equipped with a critic and a pruner: the critic screens benchmark-specific proposals, while the pruner, removes changes that are too small, too expensive, or no longer useful. Together these constraints favor reusable agent mechanisms over benchmark-specific ones or even noises. Across eight benchmarks spanning coding, agentic workspace and engineering design tasks, RRSI gains up to 6.0 points on the split it evolves against and up to 4.3 points on the five out-of-distribution benchmarks, while producing a harness that runs on 30% fewer policy tokens than the unregularized evolution. Code is available at https://github.com/google-research/rrsi and project page is https://regularized-rsi.com/.

Peng Xia, Rujun Han, Zifeng Wang +11

Google Cloud AI Research · UNC-Chapel Hill · Stanford University +1

OOD Generalization Agent Harness Optimization LLM Agent Self-Improvement Recursive Self-Improvement

Ask PDF

Abstract

Self-Improving Agents

Explore similar work

Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks

Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses

RRSI: Regularized Recursive Self-Improvement of Agent Harnesses