Hindsight Memory-PRM: Supervising Memory Management with Auditable Hindsight Credit

Haoxuan Jia; Yang Liu; Yingguang Yang; Yancheng Chen; Chongyang Zhang; Hao Zheng; Qian Li; Yulin Huang

cs.CLAug 30, 2026

Hindsight Memory-PRM: Supervising Memory Management with Auditable Hindsight Credit

Authors: Haoxuan Jia, Yang Liu, Yingguang Yang, Yancheng Chen, Chongyang Zhang, Hao Zheng, Qian Li, Yulin Huang,

Organizations: Fullive-AI · Nanyang Technological University · Supply Chain Tech Team Y, JD.com · Peking University ·

Abstract

Memory operations of long-horizon LLM agents are hard to supervise: an operation's value is unobservable when it is taken. But they are special -- they leave machine-readable evidence in the trajectory: retrieval hits and answer-time citations. Hindsight Memory-PRM exploits this audit trail twice: offline to train an operation-conditioned memory-utility critic, and online, where retrievals, citations, and one controlled deletion-and-reanswer per probe settle an intervention-calibrated entry-level presence credit, propagated along version chains as an action-level proxy reward -- no per-operation human labels, no Monte-Carlo replay of continuations. On held-out LoCoMo a local 8B policy reaches 77.5% under a fixed shared reader, surpassing its API teacher (65.1%) and all reproduced external systems, at one eighth the context of Mem0's official operating point; on LongMemEval, 79.0%. Ablations attribute the gain to causal calibration rather than signal density, and the policy converges to a multi-version memory organization whose gains no tested open-loop baseline reproduces.

Figures & tables

Figure 1: Overview. Probe exams retrieve, answer, judge, and run one controlled deletion-and-reanswer per anchored probe; the audit trail (retrieved/cited/flips) trains the critic offline and settles the signed credit of Eq. ( 6 ) online, forming Eq. ( 7 ).

Group	Method	$k$	Context (tok)	Acc (%)
System-level reference	Heuristic teacher (our protocol)	10	1,320	65.1
System-level reference	Mem0 (top-10 / official $k{=}200$ , 19.6k tok)	10	1,150	69.7 / 74.7
Same-scaffold controlled	Untrained (same scaffold)	10	1,180	58.6 $\pm$ 0.5
	SFT	10	1,640	62.5 $\pm$ 0.6
	Outcome-only GRPO	10	1,720	63.4 $\pm$ 0.7
	MC-rollout critic	10	1,900	65.0 $\pm$ 0.9

Table 1: LoCoMo held-out test (5 dialogues, 975 questions). Answerer and judge are fixed; external comparisons are system-level , “local 8B” denotes the manager. Retrieval-only results (46.8–53.4), further prompt-based systems, and the cited Memory-R1 result (62.7): Appendix E , Table 4 .

\pm

: 3-seed standard deviation; dialogue-level uncertainty: Appendix K .

Figure 2: (a) The credit-signal ladder. (b) Ablations, one change per variant. Orange rows: the two

2\times 2

pillars (

-

6.5 /

-

8.1 alone;

-

14.1 both removed, gray row), near-additive, interaction

+

0.5; full table in Appendix E .

Policy $\backslash$ schema	LoCoMo schema	LME-adapted schema
Ours (LoCoMo-trained, no retraining)	58.4 (74%)	71.2 (90%)
Outcome-only (LoCoMo-trained, no retraining)	44.1 (56%)	52.6 (67%)
Ours (LongMemEval native)	63.5 (80%)	79.0

Table 2: Policy

\times

schema factorization on the LongMemEval test split; parentheses: accuracy relative to native training (79.0). “No retraining” rebuilds the bank under the indicated schema without updating the policy (Appendix J ).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Role	Model	Decoding	Where
Policy (trained)	Qwen3-8B + LoRA	rollout $T{=}1.0$ ; eval greedy	§ 3.5 , § 4.1
Critic (trained)	Qwen3-1.7B + LoRA	scoring head, deterministic	§ 3.4
Cold-start teacher	DeepSeek-V4-Flash	$T{=}0$	§ 3.3 , Tab. 1
Fixed answering model	DeepSeek-V4-Flash	$T{=}0$	§ 3.5 exams, all evals
Judge	DeepSeek-V4-Flash	$T{=}0$	§ 3.5 , § 4.1
Probe generator	DeepSeek-V4-Flash	$T{=}0$	§ 3.2 , App. F

Appendix

Table 3: Model roles and decoding configurations. Policy rollout sampling uses temperature 1.0 (for within-group diversity); everything else uses temperature 0.

Manager	Method	$k$	Context (tok)	Acc (%)
System-level ref.: retrieval-only	RAG-Session	5 sess	6,300	46.8
	RAG-Utterance	10	4,100	50.5
	Full-store (verbatim turns)	10	760	53.4
System-level ref.: prompt-based (API LLM)	A-Mem	10	980	47.9
	MemoryOS	10	1,240	55.4
	MemBuilder-prompt	10	1,410	59.2

Appendix

Table 4: LoCoMo held-out test set (5 dialogues, 975 questions). Every system runs its official construction and reading pipeline end to end; construction model, answering model, and judge are fixed across all prompt-based systems. Comparisons with external systems are system-level : our protocol’s arms share the three-layer store, per-version embeddings, and the automatically written verbatim layer, while external systems use their official storage and retrieval interfaces; controlled conclusions rest on the same-scaffold arms (teacher, SFT, outcome-only, observational, ours), with interface-equalized controls in Appendix L . Local-8B rows use the policy under evaluation as constructor; “local 8B” denotes the memory manager—answering and judging are performed by the same fixed API reader as all other arms.

k

is the number of retrieved entries; since entry lengths differ across systems we also give the actual answering-context tokens, and we report our method at a token budget matched to Mem0. † Cited from the original paper (own answerer and judge; code unreleased), indirect reference only.

Method	Acc (%)	Context (tok)	Entries/bank
Untrained (same scaffold)	61.0	1,540	364
SFT	64.0	980	76
Mem0 ( Chhikara et al., 2025 )	64.0	1,210	185
MemBuilder ( Shen et al., 2026 ) bank + unified reader	66.0	1,880	458
Outcome-only GRPO (same scaffold)	66.0	1,060	121
Observational attribution (online, same scaffold)	72.5	1,030	98

Appendix

Table 5: LongMemEval (400-question test split under MemBuilder’s protocol; identical questions per row; 3-seed means, std

\pm

0.4–0.9). Retrieval-only rows (38.0–52.0) in Appendix E ; questions are independent units, so LoCoMo’s nesting issue does not arise (Appendix K ).

Component	Observational (online)	Full method
Offline labels (incl. counterfactual completion) and critic checkpoint	identical	identical
Probe pool, exams, and miss construction	identical	identical
Online positive branch	cited capped at 0.6	four tiers (flip to 1.0)
Online negative branch (toxicity) and deletion-reanswer	absent	present

Appendix

Table 6: Observational-attribution (online) arm vs. the full method.

Variant	Hypothesis tested	Acc (%)	$\Delta$
Remove process reward ( $\lambda{\equiv}0$ )	net contribution of critic shaping	71.0	$-6.5$
Remove hindsight credit	net contribution of verified ground truth	69.4	$-8.1$
Remove coverage penalty	complementarity of negative signal	68.2	$-9.3$
Constant $\lambda{=}1.2$	annealing (Prop. 1 : direction only)	67.0	$-10.5$
Sum credit	pessimistic min credit (not erosion evidence, App. B )	66.1	$-11.4$
No SFT cold start	can intervention-calibrated credit bootstrap from zero	63.0	$-14.5$

Appendix

Table 7: Component and mechanism ablations (LoCoMo, 975 questions). Each row removes or replaces exactly one component under the same checkpoint-selection protocol; the sum-credit arm’s shaping is rescaled to the min arm’s mean magnitude (Appendix B , Remark 4).

Gate	Context provided	Pass condition	Failure excluded
① closed book	no dialogue content	must fail or abstain	answerable from priors
② oracle	anchor turn $t^{\ast}$ only	must answer correctly	dud: anchor does not entail answer
③ uniqueness	all content minus $t^{\ast}$	must fail or abstain	redundant second source

Appendix

Table 8: The three gates.

Question source	Wrong after removal	Replays	Correct after removal
Real questions	5.2%	$\approx$ 340	94.8%
Synthetic (one gate)	71.5%	1	28.5%
Synthetic (three gates)	98.7%	1	1.3%

Appendix

Table 9: Effect size, replays required, and label purity of counterfactual measurement. Replays estimated by Eq. ( 4 ) (one-sided

\alpha{=}0.05

, power 0.95,

\varepsilon{=}0.05

);

\delta

is the difference in wrong-answer rate before vs. after removing the anchor turn (baseline wrong-answer rate 1.3%); the last column is the share of questions still answered correctly after removal.

Check	Measures	Threshold	Result
Dev ranking	Spearman of scores vs. hindsight targets	$\geq 0.55$	0.639
Miss detection	true Noop vs. counterfactual miss (AUC)	$\geq 0.85$	0.951
Mis-attach detection	true merges vs. forced attaches (AUC)	$\geq 0.80$	0.985
Cross-expert scale	correct action outscores wrong action	$\geq 75\%$	89%
Style adversary	false-positive rate on keyword-stuffed probes	$\leq 5\%$	3.2%
Deletion causality	loss deleting top / random / bottom 10%	$\geq 8$ / mid / $\leq 1$	$9.6$ / $2.1$ / $0.8$

Appendix

Table 10: Pre-specified critic acceptance (development dialogue D5). Thresholds fixed before the experiment; all checks run on D5, which takes part in neither critic nor policy training; final test D6–D10 takes part in no check.

Phase	Shaping (forgeable)	Hindsight credit (audited)	Coverage (miss)	Task (measured acc)
First third	0.563 (9.6%)	1.043 (17.8%)	0.420 (7.2%)	3.826 (65.4%)
Middle third	0.218 (4.0%)	1.436 (26.6%)	0.246 (4.6%)	3.497 (64.8%)
Last third	0.106 (1.8%)	2.007 (34.1%)	0.118 (2.0%)	3.658 (62.1%)

Appendix

Table 11: Reward composition over training (LoCoMo, full method). Thirds of training steps; parentheses give the share of the four-term absolute sum; the miss share falls as the policy stops missing anchors (cf. Figure 5 ).

Deleted entry set	Share of bank	$\Delta$ Acc
Never retrieved ( $u^{\mathrm{RR}}{=}0$ )	74.6%	$-0.1$
Retrieved but uncited	18.2%	$-1.9$
Cited ( $u^{\mathrm{OC}}{=}1$ )	7.2%	$-12.4$

Appendix

Table 12: Deletion effects by credit stratum (development calibration segment D5-cal; of its 110 questions, 104 admit unique anchor mapping and controlled deletion—no sampling). Rows are 3-seed means; confidence intervals use session-level block bootstrap over D5-cal’s 10 sessions (blocks absorb within-dialogue correlation), 95% half-widths 0.6–1.4 points.

Type	Share	Asks about
multi-session	27/100	cross-session aggregation and counting
temporal-reasoning	26/100	temporal location and arithmetic
knowledge-update	16/100	attributes overwritten by later statements
single-session-user	14/100	user-stated attributes
single-session-assistant	11/100	content the assistant previously produced
single-session-preference	6/100	answering preferences

Appendix

Table 13: Question-type distribution (LongMemEval training split).

System	D6	D7	D8	D9	D10	Mean
Mem0 (shared top-10)	68.0	71.2	69.5	70.4	69.4	69.7
Mem0 (official $k{=}200$ )	74.0	75.6	74.3	75.2	74.4	74.7
SFT (distillation start)	61.8	63.9	62.1	62.6	62.1	62.5
Outcome-only	62.8	64.6	63.1	63.3	63.2	63.4
Observational attribution	68.4	71.6	69.9	70.7	70.4	70.2
Ours	75.2	79.1	76.8	78.3	78.1	77.5

Appendix

Table 14: Per-dialogue accuracy (LoCoMo test split, 5 dialogues; 3-seed means).

Figure 3: Per-dialogue paired slopes on the LoCoMo test split (D6–D10, 3-seed means; data of Table 14 ). Each line is one test dialogue; every pairing rises toward ours—the visual form of the 5/5 sign-test wins and of the recomputable block intervals.

Threshold (absolute acc)	Outcome-only (within 28,000)	Ours
Outcome-only plateau (64.0)	9,600	120
50% of gain (70.6)	not reached (plateau 64.0)	430
75% of gain (74.3)	not reached	1,140
90% of gain (76.5)	not reached	1,540
Final plateau (dev)	64.0	78.0

Appendix

Table 15: Rollouts to reach gain thresholds (LoCoMo development set; both arms start from SFT at 63.1; thresholds are percentages of the gain to our plateau 78.0).

Figure 4: Learning curves (LoCoMo dev, log-scale rollouts); dotted lines mark 50/75/90% of the gain. Exact numbers in Appendix K .

Figure 5: Action-share evolution over LoCoMo training (3-seed mean,

\pm

band). From the SFT start (teacher-like: write-heavy, merge-light) the merge share rises fivefold (

3.0\%\to 14.5\%

) and overtakes write by step

{\approx}50

, while write halves and Noop settles at the

\approx

30/37 level of § 4.5 : the policy reallocates actions toward consolidation rather than writing more, the behavioral trace of the organization in § 4.7 .

Configuration	Acc (%)	Versions/entry	Note
Main (shared along chain, un-normalized)	77.5 $\pm$ 0.8	6.1
Normalized by chain length ( $u_{i}/L$ per op)	77.0 $\pm$ 0.9	5.4	hubs persist, accuracy unchanged
Cited-version-only scoring (rest score 0)	76.6 $\pm$ 1.0	5.2	hubs persist
Chain sharing disabled (last contributor only)	75.4 $\pm$ 1.1	3.9	merge rate and accuracy both drop

Appendix

Table 16: Controls on the credit-sharing rule (LoCoMo, 975 questions).

Configuration	Acc (%)
Mem0 (official pipeline, shared top-10 reading)	69.7
Mem0 + our automatic verbatim-layer fallback	70.9
Heuristic teacher + explicit multi-version merge prompt	65.9
Rule-based merging (similarity clustering, same interface)	66.5
Heuristic teacher + usage-log feedback (two prompt iterations)	67.4
Ours (hit-version-only return, fixed token budget)	75.6

Appendix

Table 17: Scaffold equalization controls (LoCoMo, 975 questions).

Figure 6: Action-level replay audit: settled presence credit vs. do(

a\to

Noop ) replay marginals over the 300 stratified operations. Pooled Spearman 0.72 ( Write 0.78, Merge 0.61), the correlations reported above.

Agent Memory Management

Agent memory management studies how AI agents store, update, retrieve, compress, and forget information across long-running tasks.

See all 446 papers

HiMPO: Hindsight-Informed Memory Policy Optimization for Less-Entangled Credit in Long-Horizon Agents
Jun 15, 2026Jiangze Yan, Yi Shen, Wenjing Zhang +6Credit Assignment in RL Long-Horizon Agent Tasks
Unicom Data Intelligence, China Unicom · Data Science & Artificial Intelligence Research Institute, China Unicom
MemTrain: Self-Supervised Context Memory Training
Jun 2, 2026Ziheng Li, Xingrun Xing, Haoqing Wang +2LLM Agent Memory Persistent Memory for Language Models
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Samsung Research, Beijing, China
AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents
Jun 19, 2026Xingyu Chen, Rui Wang, Zhaopeng Tu +1LLM Agent Memory Persistent Memory for Language Models
Shanghai Jiao Tong University, Tencent · Shanghai Jiao Tong University · Tencent

Jun 15, 2026J/K move · Enter open · S save

HiMPO: Hindsight-Informed Memory Policy Optimization for Less-Entangled Credit in Long-Horizon Agents

Jiangze Yan, Yi Shen, Wenjing Zhang, Jieyun Huang+5

Unicom Data Intelligence, China Unicom · Data Science & Artificial Intelligence Research Institute, China Unicom

Credit Assignment in RL Long-Horizon Agent Tasks Memory-Augmented Agents Agent Memory Management

Open paper AskPDF arXiv

Long-horizon agents compress interaction histories into memory, but final outcomes entangle memory-writing credit with tool failures and reasoning errors. We propose HiMPO, a Hindsight-Informed Memory Policy Optimization framework that supplements trajectory-level reinforcement learning with memory-specific credit. For each update, HiMPO compares the target answer's log-likelihood under the previous and updated memories while holding the remaining pre-write context fixed. A soft gate based on group-relative hindsight relevance then weights the local utility advantage. Trajectory-level credit applies to all generated tokens; memory tokens additionally receive the auxiliary advantage. Across judge-based open-domain tasks and objective compressive-memory QA, HiMPO improves task performance while preserving compressed-context efficiency. On Local Wiki Search, it improves average F1 and exact match over MemPO by 2.6 and 2.2 percentage points, respectively. Controlled interventions and live memory-reversion studies provide evidence of reduced blame leakage and alignment between memory credit and behavioral effects. With the policy fixed, the credit estimator remains robust to the tested forms of target noise.

Abstract

Agent Memory Management

Explore similar work

HiMPO: Hindsight-Informed Memory Policy Optimization for Less-Entangled Credit in Long-Horizon Agents

MemTrain: Self-Supervised Context Memory Training

AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents