An effective agent memory system should adapt to a user's history and task requirements. Human-annotated supervision can guide this adaptation by providing feedback for evaluating and improving memory designs. Yet suitable labels are rarely available for new user histories, and manually constructing QA pairs and supporting evidence is costly. We introduce MemSteer, which turns unlabeled user histories into question-answer (QA) supervision with supporting evidence for memory-design optimization. MemSteer constructs an evidence-grounded answer plan that specifies how facts across related records jointly support a reference answer. The plan guides question generation, while source-grounded auditing and blind answer recovery check the resulting annotations. To study the utility of this supervision, we develop MemSteer-Opt, which combines QA-based design evaluation with evidence-guided failure diagnosis to revise memory store, retrieval, and context construction without updating model parameters. Experiments on ATM-Bench and RHELM show that this supervision is competitive with official-QA-guided optimization. Relative to Base-BM25, MemSteer-Opt improves ATM-Bench Question Score by 13.4 points and RHELM accuracy by 4.1 and 6.6 points without and with external data, respectively. The supervision also benefits two existing optimizers on ATM-Bench, while new-user experiments demonstrate continued design optimization using supervision generated from the target histories.
Figures & tables
Figure 1: Motivation for MemSteer. User histories vary in data characteristics, so one memory design may not suit every user. Automatic design adaptation through iterative optimization requires feedback on memory performance. MemSteer supports this optimization by automatically generating QA–evidence supervision from unlabeled histories, avoiding labor-intensive manual labeling.
Figure 2: Overview of MemSteer. (a) MemSteer turns unlabeled user histories into QA–evidence supervision through evidence-grounded answer planning, question generation, and source-grounded verification. (b) The resulting supervision supports the evaluation and refinement of memory designs with either MemSteer-Opt, our evidence-guided optimizer, or existing memory-design optimizers. Dashed E denotes optional evidence input.
Table 3
Evaluation users
Base-BM25
6-user design
Personal continuation
Source users (6)
31.5
34.3
–
Martin Keller
41.2
37.7
44.3
David R. Ellis
37.3
38.6
39.2
Table 3: RHELM cross-user accuracy (%, with external data). Martin and David are outside the six-user pool; personal continuation adapts its shared design using target-history-generated supervision. Source-user scores pool evaluation questions.
QS
QA quality (%)
Kept
Method
Full
Hard
GPT-5- mini
DS-V4F
Avg.
(%)
Naive-QA
54.2
5.3
63.3
63.8
63.5
–
+ checks
64.8
24.4
77.8
77.0
77.4
21.2
Ragas-QA
63.0
23.3
73.5
69.3
71.4
–
+ checks
65.1
20.7
84.3
80.8
82.5
42.1
MemSteer w/o checks
65.2
22.0
83.5
67.3
75.4
–
Table 4: QA-generation methods and verification on ATM-Bench. QA quality scores are reported for two judges (DS-V4F: DeepSeek-V4-Flash); Avg. is their mean. QS measures downstream utility; Kept is the percentage of candidates retained after verification (“–”: no filtering).
Generation
Quality (%)
Tokens/q (k)
Direct QA
67.8
64.0
Joint Plan–Q
73.1
68.2
Two-stage Plan–Q
85.9
105.1
Table 5: Generated QA quality evaluation. q denotes a question.
Figure 3: Attribution and sampling ablation on ATM-Bench and ATM-Bench-Hard. Bars report QS and R@10 (%). Optimized conditions use MemSteer-generated supervision and start from Base-BM25; Diagnosis + stratified sampling is the full MemSteer-Opt configuration.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
ATM-Bench
ATM-Bench-Hard
Method
Backbone
Supervision
QS ↑
R@10 ↑
QS ↑
R@10 ↑
Reference memory systems
Memexa
DeepSeek-V4
None
65.3
78.9
–
–
A-Mem
DeepSeek-V4
None
–
–
16.7
38.5
MemPalace
DeepSeek-V4
None
–
–
13.6
28.9
HippoRAG2
DeepSeek-V4
None
–
–
12.3
32.2
Appendix
Table 6: Performance on ATM-Bench and ATM-Bench-Hard (%). DeepSeek-V4-Flash is used for answer generation and as the judge for open-ended answers. Arrows indicate percentage-point changes relative to the corresponding initial design: Base-BM25 for MemSteer-Opt and the respective starting systems for the external optimizers. Bold and underlined values indicate the best and second-best results per metric, excluding the oracle, which uses gold evidence. † Official-QA training items are included in evaluation.
ATM-Bench
ATM-Bench-Hard
Method
Backbone
Supervision
QS ↑
R@10 ↑
QS ↑
R@10 ↑
Reference memory systems
MemPalace
Qwen3-VL-2B/8B
None
56.8
76.4
9.7
28.3
ATM-RAG
Qwen3-VL-2B/8B
None
51.0
68.7
13.8
30.4
Self-RAG
Qwen3-VL-2B/8B
None
50.3
68.7
–
–
ScrapMem
Qwen3-VL-2B/8B
None
48.4
66.1
–
–
Appendix
Table 7: Performance on ATM-Bench and ATM-Bench-Hard (%). MemSteer-Opt uses DeepSeek-V4-Flash for answer generation. Open-ended answers are evaluated by the official GPT-5-mini judge. All MemSteer-Opt runs start from Base-BM25. Green arrows indicate percentage-point improvements over Base-BM25. Bold and underlined values indicate the best and second-best results per metric among the MemSteer-Opt variants, respectively. The oracle uses gold evidence. † Official-QA training items are included in evaluation.
Figure 4: Effect of the supervision budget on ATM-Bench and ATM-Bench-Hard (%). All runs start from the same Base-BM25 design, shown as Base at zero training questions. The horizontal axis indicates the number of training questions; the validation set scales proportionally to maintain an approximately 60:40 training-to-validation split.
Model
Without External Data Sources
With External Data Sources
FC
TP
AG
HL
MI
EX
MX
Avg
FC
TP
AG
HL
MI
EX
MX
Avg
RAG Baselines
GPT-4.1-mini ( k=5 )
35.8
17.3
17.7
15.2
3.1
8.0
10.0
16.3
34.8
14.1
19.3
11.7
3.1
16.9
12.4
17.5
GPT-4.1-mini ( k=20 )
44.0
32.4
31.8
18.3
3.1
12.1
12.9
23.5
42.5
28.7
30.7
13.2
3.1
28.5
13.8
25.1
GPT-4.1-mini ( k=50 )
59.9
41.6
40.1
15.7
1.5
12.9
16.7
28.9
54.6
39.5
38.0
11.2
1.5
38.6
22.4
32.6
Hybrid ( k=5 )
34.3
20.5
14.1
19.8
1.5
8.0
10.5
16.7
31.9
19.5
14.1
19.3
1.5
16.1
10.5
17.6
Appendix
Table 8: RHELM accuracy (%) by question type, with and without external data sources. FC, TP, AG, HL, and MI denote fact, temporal, aggregation, hallucination, and misleading questions, respectively; EX covers attachments and emails, and MX denotes mixed-context questions. Avg reports overall accuracy across evaluation questions. All memory-framework methods and our systems use GPT-4.1-mini for answer generation. ATM-Bench-evolved (frozen) transfers the ATM-Bench-optimized memory design without further optimization on RHELM; its + Official-QA and + MemSteer variants continue optimization on RHELM using the corresponding supervision. Base-BM25+Official-QA and Base-BM25+MemSteer instead initialize optimization from Base-BM25. Bold and underlined values indicate the best and second-best results per column, respectively. † Official-QA training and validation items are included in evaluation.
Persona
Role
N
Base-BM25
Shared design
Jonas Richter
Training
106
25.5
33.0 (+7.5)
David Reyes
Training
132
34.8
37.1 (+2.3)
Dr. Elias Romero
Training
122
31.1
31.1 (+0.0)
Justice Arvind P. Mehra
Training
123
33.3
34.1 (+0.8)
Elena Morales
Validation
87
35.6
39.1 (+3.5)
Dr. Elena Markovic
Validation
109
28.4
32.1 (+3.7)
Appendix
Table 9: Per-persona accuracy on RHELM with external data sources (%). (a) Six-user optimization: roles indicate participation in training, validation, or neither (held-out). (b) Ten-user optimization: all personas participate as training and validation, with Official-QA or MemSteer supervision. N is the number of evaluation questions; parentheses show percentage-point changes from Base-BM25. Group summaries pool evaluation questions from the six source users (training and validation) or the four held-out users; Overall (pooled) pools all 1,305 questions. † Official-QA training and validation items are included in evaluation.
Figure 5: Accepted memory-design optimization trajectory on ATM-Bench, starting from Base-BM25 and comprising nine accepted revisions. Numbered badges and colored arrows indicate the revision order.