Hindsight Memory-PRM: Supervising Memory Management with Auditable Hindsight Credit
Authors: Haoxuan Jia, Yang Liu, Yingguang Yang, Yancheng Chen, Chongyang Zhang, Hao Zheng, Qian Li, Yulin Huang, Jianshen Zhang, Yongzhi Qi, Shang Luo, Kefu Xu, Hao Peng, Junyu Lu, Du Cheng, Philip S. Yu, Bin Chong
Organizations: Fullive-AI · Nanyang Technological University · Supply Chain Tech Team Y, JD.com · Peking University · University of Chinese Academy of Sciences · Beijing University of Posts and Telecommunications · Beihang University · Beijing Institute of Technology, Zhuhai · Northeastern University · University of Illinois Chicago
Memory operations of long-horizon LLM agents are hard to supervise: an operation's value is unobservable when it is taken. But they are special -- they leave machine-readable evidence in the trajectory: retrieval hits and answer-time citations. Hindsight Memory-PRM exploits this audit trail twice: offline to train an operation-conditioned memory-utility critic, and online, where retrievals, citations, and one controlled deletion-and-reanswer per probe settle an intervention-calibrated entry-level presence credit, propagated along version chains as an action-level proxy reward -- no per-operation human labels, no Monte-Carlo replay of continuations. On held-out LoCoMo a local 8B policy reaches 77.5% under a fixed shared reader, surpassing its API teacher (65.1%) and all reproduced external systems, at one eighth the context of Mem0's official operating point; on LongMemEval, 79.0%. Ablations attribute the gain to causal calibration rather than signal density, and the policy converges to a multi-version memory organization whose gains no tested open-loop baseline reproduces.
Figures & tables
Figure 1: Overview. Probe exams retrieve, answer, judge, and run one controlled deletion-and-reanswer per anchored probe; the audit trail (retrieved/cited/flips) trains the critic offline and settles the signed credit of Eq. ( 6 ) online, forming Eq. ( 7 ).
Group
Method
k
Context (tok)
Acc (%)
System-level reference
Heuristic teacher (our protocol)
10
1,320
65.1
Mem0 (top-10 / official k=200 , 19.6k tok)
10
1,150
69.7 / 74.7
Same-scaffold controlled
Untrained (same scaffold)
10
1,180
58.6 ± 0.5
SFT
10
1,640
62.5 ± 0.6
Outcome-only GRPO
10
1,720
63.4 ± 0.7
MC-rollout critic
10
1,900
65.0 ± 0.9
Table 1: LoCoMo held-out test (5 dialogues, 975 questions). Answerer and judge are fixed; external comparisons are system-level , “local 8B” denotes the manager. Retrieval-only results (46.8–53.4), further prompt-based systems, and the cited Memory-R1 result (62.7): Appendix E , Table 4 . ± : 3-seed standard deviation; dialogue-level uncertainty: Appendix K .
Figure 2: (a) The credit-signal ladder. (b) Ablations, one change per variant. Orange rows: the two 2×2 pillars ( − 6.5 / − 8.1 alone; − 14.1 both removed, gray row), near-additive, interaction + 0.5; full table in Appendix E .
Policy \ schema
LoCoMo schema
LME-adapted schema
Ours (LoCoMo-trained, no retraining)
58.4 (74%)
71.2 (90%)
Outcome-only (LoCoMo-trained, no retraining)
44.1 (56%)
52.6 (67%)
Ours (LongMemEval native)
63.5 (80%)
79.0
Table 2: Policy × schema factorization on the LongMemEval test split; parentheses: accuracy relative to native training (79.0). “No retraining” rebuilds the bank under the indicated schema without updating the policy (Appendix J ).
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Role
Model
Decoding
Where
Policy (trained)
Qwen3-8B + LoRA
rollout T=1.0 ; eval greedy
§ 3.5 , § 4.1
Critic (trained)
Qwen3-1.7B + LoRA
scoring head, deterministic
§ 3.4
Cold-start teacher
DeepSeek-V4-Flash
T=0
§ 3.3 , Tab. 1
Fixed answering model
DeepSeek-V4-Flash
T=0
§ 3.5 exams, all evals
Judge
DeepSeek-V4-Flash
T=0
§ 3.5 , § 4.1
Probe generator
DeepSeek-V4-Flash
T=0
§ 3.2 , App. F
Appendix
Table 3: Model roles and decoding configurations. Policy rollout sampling uses temperature 1.0 (for within-group diversity); everything else uses temperature 0.
Manager
Method
k
Context (tok)
Acc (%)
System-level ref.: retrieval-only
RAG-Session
5 sess
6,300
46.8
RAG-Utterance
10
4,100
50.5
Full-store (verbatim turns)
10
760
53.4
System-level ref.: prompt-based (API LLM)
A-Mem
10
980
47.9
MemoryOS
10
1,240
55.4
MemBuilder-prompt
10
1,410
59.2
Appendix
Table 4: LoCoMo held-out test set (5 dialogues, 975 questions). Every system runs its official construction and reading pipeline end to end; construction model, answering model, and judge are fixed across all prompt-based systems. Comparisons with external systems are system-level : our protocol’s arms share the three-layer store, per-version embeddings, and the automatically written verbatim layer, while external systems use their official storage and retrieval interfaces; controlled conclusions rest on the same-scaffold arms (teacher, SFT, outcome-only, observational, ours), with interface-equalized controls in Appendix L . Local-8B rows use the policy under evaluation as constructor; “local 8B” denotes the memory manager—answering and judging are performed by the same fixed API reader as all other arms. k is the number of retrieved entries; since entry lengths differ across systems we also give the actual answering-context tokens, and we report our method at a token budget matched to Mem0. † Cited from the original paper (own answerer and judge; code unreleased), indirect reference only.
Method
Acc (%)
Context (tok)
Entries/bank
Untrained (same scaffold)
61.0
1,540
364
SFT
64.0
980
76
Mem0 ( Chhikara et al., 2025 )
64.0
1,210
185
MemBuilder ( Shen et al., 2026 ) bank + unified reader
66.0
1,880
458
Outcome-only GRPO (same scaffold)
66.0
1,060
121
Observational attribution (online, same scaffold)
72.5
1,030
98
Appendix
Table 5: LongMemEval (400-question test split under MemBuilder’s protocol; identical questions per row; 3-seed means, std ± 0.4–0.9). Retrieval-only rows (38.0–52.0) in Appendix E ; questions are independent units, so LoCoMo’s nesting issue does not arise (Appendix K ).
Component
Observational (online)
Full method
Offline labels (incl. counterfactual completion) and critic checkpoint
identical
identical
Probe pool, exams, and miss construction
identical
identical
Online positive branch
cited capped at 0.6
four tiers (flip to 1.0)
Online negative branch (toxicity) and deletion-reanswer
absent
present
Appendix
Table 6: Observational-attribution (online) arm vs. the full method.
Variant
Hypothesis tested
Acc (%)
Δ
Remove process reward ( λ≡0 )
net contribution of critic shaping
71.0
−6.5
Remove hindsight credit
net contribution of verified ground truth
69.4
−8.1
Remove coverage penalty
complementarity of negative signal
68.2
−9.3
Constant λ=1.2
annealing (Prop. 1 : direction only)
67.0
−10.5
Sum credit
pessimistic min credit (not erosion evidence, App. B )
66.1
−11.4
No SFT cold start
can intervention-calibrated credit bootstrap from zero
63.0
−14.5
Appendix
Table 7: Component and mechanism ablations (LoCoMo, 975 questions). Each row removes or replaces exactly one component under the same checkpoint-selection protocol; the sum-credit arm’s shaping is rescaled to the min arm’s mean magnitude (Appendix B , Remark 4).
Gate
Context provided
Pass condition
Failure excluded
① closed book
no dialogue content
must fail or abstain
answerable from priors
② oracle
anchor turn t∗ only
must answer correctly
dud: anchor does not entail answer
③ uniqueness
all content minus t∗
must fail or abstain
redundant second source
Appendix
Table 8: The three gates.
Question source
Wrong after removal
Replays
Correct after removal
Real questions
5.2%
≈ 340
94.8%
Synthetic (one gate)
71.5%
1
28.5%
Synthetic (three gates)
98.7%
1
1.3%
Appendix
Table 9: Effect size, replays required, and label purity of counterfactual measurement. Replays estimated by Eq. ( 4 ) (one-sided α=0.05 , power 0.95, ε=0.05 ); δ is the difference in wrong-answer rate before vs. after removing the anchor turn (baseline wrong-answer rate 1.3%); the last column is the share of questions still answered correctly after removal.
Check
Measures
Threshold
Result
Dev ranking
Spearman of scores vs. hindsight targets
≥0.55
0.639
Miss detection
true Noop vs. counterfactual miss (AUC)
≥0.85
0.951
Mis-attach detection
true merges vs. forced attaches (AUC)
≥0.80
0.985
Cross-expert scale
correct action outscores wrong action
≥75%
89%
Style adversary
false-positive rate on keyword-stuffed probes
≤5%
3.2%
Deletion causality
loss deleting top / random / bottom 10%
≥8 / mid / ≤1
9.6 / 2.1 / 0.8
Appendix
Table 10: Pre-specified critic acceptance (development dialogue D5). Thresholds fixed before the experiment; all checks run on D5, which takes part in neither critic nor policy training; final test D6–D10 takes part in no check.
Phase
Shaping (forgeable)
Hindsight credit (audited)
Coverage (miss)
Task (measured acc)
First third
0.563 (9.6%)
1.043 (17.8%)
0.420 (7.2%)
3.826 (65.4%)
Middle third
0.218 (4.0%)
1.436 (26.6%)
0.246 (4.6%)
3.497 (64.8%)
Last third
0.106 (1.8%)
2.007 (34.1%)
0.118 (2.0%)
3.658 (62.1%)
Appendix
Table 11: Reward composition over training (LoCoMo, full method). Thirds of training steps; parentheses give the share of the four-term absolute sum; the miss share falls as the policy stops missing anchors (cf. Figure 5 ).
Deleted entry set
Share of bank
Δ Acc
Never retrieved ( uRR=0 )
74.6%
−0.1
Retrieved but uncited
18.2%
−1.9
Cited ( uOC=1 )
7.2%
−12.4
Appendix
Table 12: Deletion effects by credit stratum (development calibration segment D5-cal; of its 110 questions, 104 admit unique anchor mapping and controlled deletion—no sampling). Rows are 3-seed means; confidence intervals use session-level block bootstrap over D5-cal’s 10 sessions (blocks absorb within-dialogue correlation), 95% half-widths 0.6–1.4 points.
Type
Share
Asks about
multi-session
27/100
cross-session aggregation and counting
temporal-reasoning
26/100
temporal location and arithmetic
knowledge-update
16/100
attributes overwritten by later statements
single-session-user
14/100
user-stated attributes
single-session-assistant
11/100
content the assistant previously produced
single-session-preference
6/100
answering preferences
Appendix
Table 13: Question-type distribution (LongMemEval training split).
Figure 3: Per-dialogue paired slopes on the LoCoMo test split (D6–D10, 3-seed means; data of Table 14 ). Each line is one test dialogue; every pairing rises toward ours—the visual form of the 5/5 sign-test wins and of the recomputable block intervals.
Threshold (absolute acc)
Outcome-only (within 28,000)
Ours
Outcome-only plateau (64.0)
9,600
120
50% of gain (70.6)
not reached (plateau 64.0)
430
75% of gain (74.3)
not reached
1,140
90% of gain (76.5)
not reached
1,540
Final plateau (dev)
64.0
78.0
Appendix
Table 15: Rollouts to reach gain thresholds (LoCoMo development set; both arms start from SFT at 63.1; thresholds are percentages of the gain to our plateau 78.0).
Figure 4: Learning curves (LoCoMo dev, log-scale rollouts); dotted lines mark 50/75/90% of the gain. Exact numbers in Appendix K .
Figure 5: Action-share evolution over LoCoMo training (3-seed mean, ± band). From the SFT start (teacher-like: write-heavy, merge-light) the merge share rises fivefold ( 3.0%→14.5% ) and overtakes write by step ≈50 , while write halves and Noop settles at the ≈ 30/37 level of § 4.5 : the policy reallocates actions toward consolidation rather than writing more, the behavioral trace of the organization in § 4.7 .
Configuration
Acc (%)
Versions/entry
Note
Main (shared along chain, un-normalized)
77.5 ± 0.8
6.1
Normalized by chain length ( ui/L per op)
77.0 ± 0.9
5.4
hubs persist, accuracy unchanged
Cited-version-only scoring (rest score 0)
76.6 ± 1.0
5.2
hubs persist
Chain sharing disabled (last contributor only)
75.4 ± 1.1
3.9
merge rate and accuracy both drop
Appendix
Table 16: Controls on the credit-sharing rule (LoCoMo, 975 questions).
Long-horizon agents compress interaction histories into memory, but final outcomes entangle memory-writing credit with tool failures and reasoning errors. We propose HiMPO, a Hindsight-Informed Memory Policy Optimization framework that supplements trajectory-level reinforcement learning with memory-specific credit. For each update, HiMPO compares the target answer's log-likelihood under the previous and updated memories while holding the remaining pre-write context fixed. A soft gate based on group-relative hindsight relevance then weights the local utility advantage. Trajectory-level credit applies to all generated tokens; memory tokens additionally receive the auxiliary advantage. Across judge-based open-domain tasks and objective compressive-memory QA, HiMPO improves task performance while preserving compressed-context efficiency. On Local Wiki Search, it improves average F1 and exact match over MemPO by 2.6 and 2.2 percentage points, respectively. Controlled interventions and live memory-reversion studies provide evidence of reduced blame leakage and alignment between memory credit and behavioral effects. With the policy fixed, the credit estimator remains robust to the tested forms of target noise.
Jiangze Yan, Yi Shen, Wenjing Zhang +6
Unicom Data Intelligence, China Unicom · Data Science & Artificial Intelligence Research Institute, China Unicom
Memory is an indispensable capability for long-horizon LLM agents, enabling them to preserve and utilize information accumulated across extended interactions. Existing memory-agent approaches are typically trained end-to-end with reinforcement learning on downstream tasks. However, collecting high-quality annotated problems for memory-intensive scenarios is costly, and the resulting training data often lack sufficient diversity to cover general memory behaviors. In this work, we propose MemTrain, a self-supervised training framework for generally enhancing the context-memory capability of LLM agents for more effective downstream post-training. MemTrain introduces two coupled proxy tasks over unlabeled Wikipedia corpora: (1) an end-to-end masked reconstruction objective, which requires the model to recover masked entities after multiple rounds of memory updates, thereby encouraging memory maintenance from the final outcome perspective; and (2) an intermediate memory recall objective, which requires the model to reconstruct masked historical information using intermediate memory states, encouraging faithful compression and memory completeness throughout the interaction process. The two objectives are jointly optimized using GRPO. Extensive experiments on long-text QA and search-based QA benchmarks demonstrate that MemTrain consistently improves downstream memory-intensive reasoning performance across different models, achieving gains of up to 17.67 points over direct task-specific post-training.
Ziheng Li, Xingrun Xing, Haoqing Wang +2
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Samsung Research, Beijing, China
Long-term memory systems allow LLM agents to preserve information beyond a single context window, but most systems focus on storing and retrieving facts after extraction, leaving the write decision under-specified. What deserves memory can depend on the user's current task, topic, activity, or interaction partner, while uniform extraction applies one notion of importance across these different situations. We formulate this challenge as preference-conditioned write control and introduce AdaMem, which uses adaptive natural-language Memory Policies to personalize what an agent writes to memory. Each policy represents the user's memory preference for a particular interaction context, is updated from periodic feedback, and controls subsequent memory writing. We evaluate this loop in AdaMem-Bench, which assigns different memory preferences to six concurrent interaction personas across five ten-week stories. Across two extraction models and two feedback modes, AdaMem improves average QA accuracy over Mem0 from 80.0% to 84.35% while reducing persistent memory by 9.27%. Our analyses show that explicit feedback helps models learn better memory policies, but current models still struggle to translate those policies into reliably selective writing behavior. AdaMem thus demonstrates the promise of adaptive write control while exposing policy execution as a central limitation of current memory agents. Our code is publicly available: https://github.com/galaxyChen/AdaMem