Recursive textual memory can carry earlier errors into later updates, yet task performance alone does not establish memory reliability. We define self-consistency bias as the persistence of incorrect claims already present in the incoming memory. We present KEEPER, a comparative analysis of four auxiliary objectives along information-content and update-sensitivity axes within a shared recursive interface, examining how supervision targets relate to agent performance. Among evaluated conditions, reward-related prediction achieve the highest task performance before and after offline reinforcement learning, outperforming joint supervision. A claim-level analysis of one baseline trajectory documents error persistence despite corrective observations. Together, these analyses motivate evaluating error persistence and valid-information retention alongside task outcomes.
Figures & tables
Figure 1: Overview of auxiliary memory supervision and evaluation. (a) Supervised initialization: the memory generator fθ encodes the update context with the reference memory pt∗ under teacher forcing. Reward Preservation (RP), Predictive Preservation (PP), and Semantic Anchoring (SA) supervise the information content of the representation ht or its projection qt ( ϕ : fixed text encoder); Update Stability (US) penalizes the sensitivity of qt to a perturbed incoming memory p~t−1 . We compare behavioral cloning (BC) alone, BC with each objective, and BC with all four. (b) Recursive inference, shared across conditions with auxiliary objectives inactive. The generated memory keeps a valid fact but also an incorrect claim inherited from pt−1 (self-consistency bias), which misleads the policy; error persistence is therefore evaluated together with valid-fact retention. Numbered badges mark the three evaluation levels: representation, generated memory, and decision.
ScienceWorld
ALFWorld
Variant
Initialization
Seen ↑
Unseen ↑
Seen ↑
Unseen ↑
Supervised initialization (BC)
A
BC
48.39
45.23
93.57
92.54
B
BC + RP (Sec. 3.2.1 )
49.39
49.01
96.43
95.52
C
BC + PP (Sec. 3.2.2 )
45.96
44.46
92.86
91.79
D
BC + SA (Sec. 3.2.3 )
41.85
40.90
92.14
90.30
Table 1: Comparison of auxiliary supervision before and after offline RL on ScienceWorld and ALFWorld using Llama-3.2-3B-Instruct and the STEP-HRL Zhen et al. (2026) protocol. ScienceWorld reports average task scores, and ALFWorld reports success rates (%); higher values are better. Variants A through F differ in their supervised objectives. Primed variants apply the unchanged STEP-HRL offline RL procedure to the corresponding initialized checkpoints, with auxiliary objectives inactive. BC: behavioral cloning; RP: Reward Preservation; PP: Predictive Preservation; SA: Semantic Anchoring; US: Update Stability. The best and second-best scores in each column within each training stage are marked in bold and underlined , respectively.
Variant
Initialization
Seen ↑
Unseen ↑
A
BC
94.29
94.03
B
BC + RP
96.43
96.27
C
BC + PP
95.00
94.78
D
BC + SA
92.14
89.55
E
BC + US
95.71
94.03
F
BC + RP + PP + SA + US
93.57
93.28
Table 2: Comparison of auxiliary supervision using Qwen3.5-2B on ALFWorld with the STEP-HRL Zhen et al. (2026) protocol. The checkpoint is Qwen/Qwen3.5-2B . All six conditions use supervised initialization without offline RL. We report success rates (%) on 140 seen and 134 unseen episodes; higher values are better. Variant definitions match Tab. 1 .
Figure 2: Different responses to corrective evidence within one memory update. The observation contradicts both the incoming location claim and the claim about pot contents. The updated memory corrects the location while retaining the incorrect claim about pot contents. The figure presents excerpts from the selected trajectory, with line breaks added for readability.
Figure 3: Claim trajectories in the same episode as Fig. 2 . The recorded environment state remains unchanged: the metal pot is empty and the agent is in the kitchen. Symbols distinguish contradicted claims ( × ), claims consistent with the environment state ( + ), omitted claims ( − ), and ambiguous claims ( ? ). The incorrect claim about pot contents persists before becoming ambiguous and then omitted, while the location claim becomes incorrect again after correction.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Optimizer
AdamW
Adam coefficients
(0.9,0.999)
Optimizer epsilon
10−8
Weight decay
0.01
Gradient clipping
1.0
Model precision
bfloat16
Appendix
Table 3: Settings inherited from the official STEP-HRL implementation. The batch size is the effective batch size in the released DeepSpeed configuration. The model identifier follows the main comparison.
Function
Architecture
Reward predictor r
H→⌊H/4⌋→1
Successor predictor F
H→⌊H/2⌋→d
Projection for qt
H→d
Appendix
Table 4: Trainable auxiliary functions. Each arrow denotes a linear layer with a bias; hidden layers use ReLU. H is the backbone hidden dimension and d=384 is the default text-embedding dimension.
Condition
λRP
λPP
λSA
λUS
BC
0
0
0
0
BC + RP
0.5
0
0
0
BC + PP
0
1
0
0
BC + SA
0
0
1
0
BC + US
0
0
0
1
BC + all four
0.5
1
1
1
Appendix
Table 5: Reference coefficient settings using the implementation defaults for active terms. Columns follow the objective names in the main text. All coefficients are zero during subsequent offline RL.
LLM agents can improve across tasks without updating model parameters by storing experience from their own executions in external memory and reusing it to guide later decisions. A central design question is whether to retain only observed trajectories and outcomes or also the rules, summaries, and skills inferred from them. Retaining these abstractions allows agents to reuse strategies without repeatedly deriving them from raw trajectories. However, errors in stored abstractions can propagate to subsequent decisions through retrieval and reuse. They can also shape execution trajectories, which then become the basis for new abstractions that reproduce the original errors. We term this feedback mechanism error reproduction: errors are regenerated through a cycle of abstraction, execution, and re-abstraction, extending their persistence beyond direct reuse of the original faulty knowledge. We propose Anti-Persistence, a memory design that accumulates trajectories and outcomes while reconstructing task-specific guidance on demand. An adaptive selector uses prior outcomes to decide whether and how to synthesize guidance, which is discarded after execution. This prevents its direct reuse as a stored object, although indirect feedback through retained trajectories remains possible. Across different benchmarks and backbones, Anti-Persistence achieves higher overall held-out success and the lowest average LLM-call cost among the evaluated memory designs. These results support separating experience accumulation from abstraction persistence in memorybased self-improvement.
Memory is an indispensable capability for long-horizon LLM agents, enabling them to preserve and utilize information accumulated across extended interactions. Existing memory-agent approaches are typically trained end-to-end with reinforcement learning on downstream tasks. However, collecting high-quality annotated problems for memory-intensive scenarios is costly, and the resulting training data often lack sufficient diversity to cover general memory behaviors. In this work, we propose MemTrain, a self-supervised training framework for generally enhancing the context-memory capability of LLM agents for more effective downstream post-training. MemTrain introduces two coupled proxy tasks over unlabeled Wikipedia corpora: (1) an end-to-end masked reconstruction objective, which requires the model to recover masked entities after multiple rounds of memory updates, thereby encouraging memory maintenance from the final outcome perspective; and (2) an intermediate memory recall objective, which requires the model to reconstruct masked historical information using intermediate memory states, encouraging faithful compression and memory completeness throughout the interaction process. The two objectives are jointly optimized using GRPO. Extensive experiments on long-text QA and search-based QA benchmarks demonstrate that MemTrain consistently improves downstream memory-intensive reasoning performance across different models, achieving gains of up to 17.67 points over direct task-specific post-training.
Ziheng Li, Xingrun Xing, Haoqing Wang +2
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Samsung Research, Beijing, China
LLM agents have made impressive progress in coding and mathematical reasoning, motivating growing interest in learning from long-horizon interaction. Memory supports this process, but plausible lessons can be redundant or reinforce inappropriate actions. Can a memory's source decision provide a practical test for whether to retain it? We localize consequential trajectory steps, extract candidate memories, and replay each source decision with and without its candidate under the same context. This reveals two properties: most candidates barely shift the probed distribution, and both successful and failed source actions can be reinforced or suppressed. These observations motivate Anchor, which tests shift magnitude and source-role alignment before retention. Admitted memories guide later trajectories, enabling experience-driven improvement under a frozen policy. On held-out ALFWorld and WebShop tasks, Anchor improves success rate and mean score by up to 19.9% and 17.2%, respectively, relative to the strongest baseline for each backbone. With Qwen3.5-27B, it retains 41% fewer ALFWorld cards than unfiltered retention and outperforms all four WebShop cost-comparison baselines while using fewer tokens.
Xinxiang Yin, Xinglang Zhang, Haodong Chen +7
The University of Hong Kong · Huazhong University of Science and Technology · The Hong Kong University of Science and Technology +4