Recursive textual memory can carry earlier errors into later updates, yet task performance alone does not establish memory reliability. We define self-consistency bias as the persistence of incorrect claims already present in the incoming memory. We present KEEPER, a comparative analysis of four auxiliary objectives along information-content and update-sensitivity axes within a shared recursive interface, examining how supervision targets relate to agent performance. Among evaluated conditions, reward-related prediction achieve the highest task performance before and after offline reinforcement learning, outperforming joint supervision. A claim-level analysis of one baseline trajectory documents error persistence despite corrective observations. Together, these analyses motivate evaluating error persistence and valid-information retention alongside task outcomes.
Figures & tables
Figure 1: Overview of auxiliary memory supervision and evaluation. (a) Supervised initialization: the memory generator fθ encodes the update context with the reference memory pt∗ under teacher forcing. Reward Preservation (RP), Predictive Preservation (PP), and Semantic Anchoring (SA) supervise the information content of the representation ht or its projection qt ( ϕ : fixed text encoder); Update Stability (US) penalizes the sensitivity of qt to a perturbed incoming memory p~t−1 . We compare behavioral cloning (BC) alone, BC with each objective, and BC with all four. (b) Recursive inference, shared across conditions with auxiliary objectives inactive. The generated memory keeps a valid fact but also an incorrect claim inherited from pt−1 (self-consistency bias), which misleads the policy; error persistence is therefore evaluated together with valid-fact retention. Numbered badges mark the three evaluation levels: representation, generated memory, and decision.
ScienceWorld
ALFWorld
Variant
Initialization
Seen ↑
Unseen ↑
Seen ↑
Unseen ↑
Supervised initialization (BC)
A
BC
48.39
45.23
93.57
92.54
B
BC + RP (Sec. 3.2.1 )
49.39
49.01
96.43
95.52
C
BC + PP (Sec. 3.2.2 )
45.96
44.46
92.86
91.79
D
BC + SA (Sec. 3.2.3 )
41.85
40.90
92.14
90.30
Table 1: Comparison of auxiliary supervision before and after offline RL on ScienceWorld and ALFWorld using Llama-3.2-3B-Instruct and the STEP-HRL Zhen et al. (2026) protocol. ScienceWorld reports average task scores, and ALFWorld reports success rates (%); higher values are better. Variants A through F differ in their supervised objectives. Primed variants apply the unchanged STEP-HRL offline RL procedure to the corresponding initialized checkpoints, with auxiliary objectives inactive. BC: behavioral cloning; RP: Reward Preservation; PP: Predictive Preservation; SA: Semantic Anchoring; US: Update Stability. The best and second-best scores in each column within each training stage are marked in bold and underlined , respectively.
Variant
Initialization
Seen ↑
Unseen ↑
A
BC
94.29
94.03
B
BC + RP
96.43
96.27
C
BC + PP
95.00
94.78
D
BC + SA
92.14
89.55
E
BC + US
95.71
94.03
F
BC + RP + PP + SA + US
93.57
93.28
Table 2: Comparison of auxiliary supervision using Qwen3.5-2B on ALFWorld with the STEP-HRL Zhen et al. (2026) protocol. The checkpoint is Qwen/Qwen3.5-2B . All six conditions use supervised initialization without offline RL. We report success rates (%) on 140 seen and 134 unseen episodes; higher values are better. Variant definitions match Tab. 1 .
Figure 2: Different responses to corrective evidence within one memory update. The observation contradicts both the incoming location claim and the claim about pot contents. The updated memory corrects the location while retaining the incorrect claim about pot contents. The figure presents excerpts from the selected trajectory, with line breaks added for readability.
Figure 3: Claim trajectories in the same episode as Fig. 2 . The recorded environment state remains unchanged: the metal pot is empty and the agent is in the kitchen. Symbols distinguish contradicted claims ( × ), claims consistent with the environment state ( + ), omitted claims ( − ), and ambiguous claims ( ? ). The incorrect claim about pot contents persists before becoming ambiguous and then omitted, while the location claim becomes incorrect again after correction.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Optimizer
AdamW
Adam coefficients
(0.9,0.999)
Optimizer epsilon
10−8
Weight decay
0.01
Gradient clipping
1.0
Model precision
bfloat16
Appendix
Table 3: Settings inherited from the official STEP-HRL implementation. The batch size is the effective batch size in the released DeepSpeed configuration. The model identifier follows the main comparison.
Function
Architecture
Reward predictor r
H→⌊H/4⌋→1
Successor predictor F
H→⌊H/2⌋→d
Projection for qt
H→d
Appendix
Table 4: Trainable auxiliary functions. Each arrow denotes a linear layer with a bias; hidden layers use ReLU. H is the backbone hidden dimension and d=384 is the default text-embedding dimension.
Condition
λRP
λPP
λSA
λUS
BC
0
0
0
0
BC + RP
0.5
0
0
0
BC + PP
0
1
0
0
BC + SA
0
0
1
0
BC + US
0
0
0
1
BC + all four
0.5
1
1
1
Appendix
Table 5: Reference coefficient settings using the implementation defaults for active terms. Columns follow the objective names in the main text. All coefficients are zero during subsequent offline RL.
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Samsung Research, Beijing, China