Beyond Downstream QA: Rethinking How We Evaluate Agent Memory

Sebastian-Antonio Toma; Jean de Kernier; Mike Ply; Oliviu Matei

cs.CLOct 15, 2026

Beyond Downstream QA: Rethinking How We Evaluate Agent Memory

Authors: Sebastian-Antonio Toma, Jean de Kernier, Mike Ply, Oliviu Matei

Organizations: Amazon Web Services · Technical University of Cluj-Napoca, Romania

Abstract

Agent memory benchmarks evaluate whether an agent's responses reflect correct knowledge, measuring downstream question-answering accuracy as a proxy for memory quality. We show this evaluation paradigm is incomplete: it conflates memory correctness with retrieval and generation quality, producing both false positives where correct responses mask broken memory and false negatives where incorrect responses blame correct memory for retrieval failures. We introduce FEB-30, a benchmark of 30 fact-evolution trajectories spanning 6 diagnostic categories and 3 difficulty levels, constructed via paradigm inversion: documents are generated from known fact trajectories, yielding deterministic ground truth without human annotation. Each trajectory defines how a fact should evolve across multi-author artifacts (documents, emails, conversations), enabling direct inspection of the memory store after each ingestion step. Applying FEB-30 to a representative memory system, we demonstrate that intrinsic and extrinsic evaluation diverge by an average of 16 percentage points, with 44% of trajectories exceeding meaningful divergence thresholds. Diagnostic decomposition reveals that the system correctly classifies memory operations (create, update, reinforce, supersede) only 20% of the time, a failure entirely invisible to extrinsic evaluation, and that failure modes differ systematically by category and difficulty. We release the benchmark, document generation pipeline, and scoring framework to support reproducible evaluation of agent memory systems.

Figures & tables

Cat	Name	Design constraint
C1	LLM Compensation	Memory wrong in a way QA still works
C2	Retrieval Bottleneck	Memory correct but hard to retrieve
C3	Classification	Focus on create/update/reinforce/conflict decision correctness
C4	Synthesis Quality	Multi-observation $\rightarrow$ correct fact formation
C5	Temporal Correctness	Fact must evolve correctly over time
C6	Multi-Author	Artifacts from different author roles

Table 1: FEB-30 diagnostic categories with design constraints. C1 and C2 are designed to produce intrinsic-extrinsic divergence; C3–C6 test dimensions where intrinsic evaluation provides richer signal than extrinsic.

Level	Docs	States	Signal clarity
L1	1–2	2–3	Claims nearly explicit
L2	3–4	3–4	Claims embedded in context
L3	4–5	4–5	Requires inference

Table 2: Difficulty levels in FEB-30.

Panel of Figure 1 — Figure 1: System architecture showing the write path (ingestion and fact synthesis), intrinsic and extrinsic scoring pipelines. For each trajectory state $s_{i}$ , intrinsic evaluation compares ground-truth claims $C_{i}$ against the fact store via embedding retrieval and LLM entailment. Extrinsic evaluation queries the store and judges the response against expected answers. $\star$ = proposed here.

Category	Intr. (%)	Extr. (%)	MAD	DivRate (%)
C1: LLM Comp.	82.2 $\pm$ 2.9	54.5 $\pm$ 10.8	.295 $\pm$ .084	67.3
C2: Retrieval	60.3 $\pm$ 4.7	50.9 $\pm$ 2.9	.135 $\pm$ .034	45.5
C3: Classif.	70.8 $\pm$ 8.2	48.9 $\pm$ 16.4	.258 $\pm$ .117	54.5
C4: Synthesis	52.8 $\pm$ 3.1	49.2 $\pm$ 2.4	.104 $\pm$ .036	21.2
C5: Temporal	59.3 $\pm$ 5.2	47.3 $\pm$ 4.5	.144 $\pm$ .062	45.5
C6: Multi-Auth.	41.5 $\pm$ 4.8	41.8 $\pm$ 5.7	.106 $\pm$ .049	34.5

Table 3: Intrinsic (fact accuracy %) and extrinsic (QA accuracy %) scores by category (left) and difficulty (right), averaged across

N{=}11

runs (

\pm

std). MAD = mean absolute divergence between intrinsic and extrinsic per trajectory. DivRate = percentage with divergence

>\tau{=}0.15

. C1 shows highest MAD (.295): memory quality is systematically better than what QA evaluation reveals. Per-trajectory scores are in Appendix I .

Category	Miss (%)	Fact Count	ClassAcc (%)
C1: LLM Comp.	54.0 $\pm$ 4.5	13.6 $\pm$ 1.0	41.7 $\pm$ 2.8
C2: Retrieval	74.5 $\pm$ 4.2	20.8 $\pm$ 1.3	4.5 $\pm$ 5.7
C3: Classif.	60.5 $\pm$ 7.9	14.8 $\pm$ 1.4	30.7 $\pm$ 12.3
C4: Synthesis	72.2 $\pm$ 2.6	14.9 $\pm$ 1.0	17.4 $\pm$ 2.4
C5: Temporal	68.0 $\pm$ 5.3	21.0 $\pm$ 1.0	0.4 $\pm$ 1.1
C6: Multi-Auth.	87.5 $\pm$ 3.5	12.0 $\pm$ 0.8	28.8 $\pm$ 7.3

Table 4: Diagnostic decomposition by category (left) and difficulty (right), averaged across

N{=}11

runs (

\pm

std). Miss Rate = percentage of expected claims not found in the fact store. Fact Count = average active facts per trajectory (lower indicates better synthesis). ClassAcc = action classification accuracy. C6 shows highest miss rate (88%); fact count inflates from 9.6 at L1 to 22.4 at L3.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

$s_{i}$	Action	Expected claims $C_{i}$ (cumulative)
0	create	“PostgreSQL 14 on-premise in Frankfurt”; “3 TB database”
1	update	+“Evaluating Aurora and AlloyDB”
2	update	+“Chose Aurora”; “Cutover Q3 2025”
3	update	+“DMS replication running”; “95% stored procedures validated”

Appendix

Table 5: State evolution for c5-linear.

$s_{i}$	Timestamp	Action	Expected claims $C_{i}$ (cumulative)
0	2025-03-10	create	“Chose ECS Fargate”; “Sub-100ms response”
1	2025-01-20	update	+“Lambda rejected (cold starts)”; “EKS rejected (complexity)”
2	2025-05-15	update	+“12 Fargate services at 50K rps”

Appendix

Table 6: State evolution for c4-out-of-order. Note

s_{1}

timestamp is older than

s_{0}

Component	Model	Temp.
System under test
Observation extraction	Frontier LLM	0
Fact synthesis	Frontier LLM	0
Embedding	Titan Embeddings V2 (1024-dim)	–
Retrieval	BM25 (0.3) + k-NN (0.7)	–
Evaluation scorer

Appendix

Table 7: All models used in the system and evaluation pipeline.

State	All (%)	L1 (%)	L2 (%)	L3 (%)
$s_{0}$ (n=330)	64.7 $\pm$ 28.8	67.5 $\pm$ 24.4	76.4 $\pm$ 22.1	50.2 $\pm$ 32.4
$s_{1}$ (n=330)	65.3 $\pm$ 22.1	72.7 $\pm$ 23.6	70.3 $\pm$ 20.3	52.9 $\pm$ 16.0
$s_{2}$ (n=319)	58.2 $\pm$ 20.8	68.2 $\pm$ 23.0	60.6 $\pm$ 17.9	46.1 $\pm$ 13.4
$s_{3}$ (n=242)	57.8 $\pm$ 18.8	62.3 $\pm$ 27.1	68.4 $\pm$ 14.8	48.2 $\pm$ 8.6
$s_{4}$ (n=110)	49.7 $\pm$ 11.4	–	57.3 $\pm$ 10.9	47.8 $\pm$ 10.7
$s_{5}$ (n=44)	50.8 $\pm$ 10.8	–	–	50.8 $\pm$ 10.8

Appendix

Table 8: Claim coverage (%) by state index, averaged across all trajectories and

N{=}11

runs.

n

= number of trajectory

\times

run pairs that reach this state (not all trajectories have the same number of states). Coverage decreases at later states as the system must retain earlier claims while integrating new information. L3 trajectories start lower and degrade further.

Category	UDR (%)	FUR (%)	Conf. MAE
C1: LLM Comp.	51.4 $\pm$ 6.0	39.4 $\pm$ 1.9	0.105 $\pm$ 0.005
C2: Retrieval	94.8 $\pm$ 4.3	0.0 $\pm$ 0.0	0.092 $\pm$ 0.009
C3: Classif.	70.5 $\pm$ 9.6	45.5 $\pm$ 6.0	0.117 $\pm$ 0.007
C4: Synthesis	81.4 $\pm$ 3.3	15.7 $\pm$ 2.1	0.097 $\pm$ 0.006
C5: Temporal	90.6 $\pm$ 2.2	0.0 $\pm$ 0.0	0.108 $\pm$ 0.007
C6: Multi-Auth.	50.8 $\pm$ 7.4	38.6 $\pm$ 3.1	0.124 $\pm$ 0.005

Appendix

Table 9: Update detection recall (UDR), false update rate (FUR), and confidence MAE per category and difficulty, averaged across

N{=}11

runs (

\pm

std across runs). C2 and C5 never produce false updates but C1 and C6 frequently misclassify reinforcements as updates. Confidence calibration is worst for C6 (multi-author) and best for C4 (synthesis). FUR decreases with difficulty because at L3 the system rarely modifies facts.

Comparison	Pearson $r$	$p$ -value
Miss rate vs. Intrinsic accuracy	$-0.841$	$5.9\times 10^{-9}$
Miss rate vs. Extrinsic accuracy	$-0.391$	$3.2\times 10^{-2}$
Intrinsic vs. Extrinsic accuracy	$+0.552$	$1.6\times 10^{-3}$

Appendix

Table 10: Pearson correlations across 30 trajectories. Miss rate strongly predicts intrinsic accuracy (

r=-0.84

) but weakly predicts extrinsic accuracy (

r=-0.39

), confirming that intrinsic metrics capture signal that extrinsic evaluation misses. The moderate intrinsic-extrinsic correlation (

r=0.55

) indicates the two methods share some signal but measure substantially different properties.

Category	Signed Div.	Interpretation
C1: LLM Comp.	$+0.276\pm 0.260$	Memory much better than QA reveals
C2: Retrieval	$+0.094\pm 0.137$	Memory slightly better
C3: Classif.	$+0.220\pm 0.276$	Memory much better than QA reveals
C4: Synthesis	$+0.036\pm 0.173$	Nearly neutral
C5: Temporal	$+0.120\pm 0.160$	Memory moderately better
C6: Multi-Auth.	$-0.003\pm 0.158$	No directional bias

Appendix

Table 11: Signed divergence (intrinsic

-

extrinsic) per category. C1 shows strong positive bias by design: memory is correct but QA underestimates. C6 shows no directional bias: both metrics are equally poor at capturing multi-author attribution.

Metric	Simpler sys.	Main sys.	Main (released)	Upgraded backbone
Intrinsic (%)	$57.6\pm 1.8$	$61.1\pm 1.6$	$60.6\pm 1.6$	$61.4\pm 1.3$
Extrinsic (%)	$48.8\pm 2.7$	$48.5\pm 2.1$	$48.8\pm 2.9$	$49.2\pm 3.4$
MAD	$0.164\pm 0.027$	$0.181\pm 0.022$	$0.168\pm 0.025$	$0.158\pm 0.035$
DivRate (%)	$43.3\pm 6.0$	$49.4\pm 7.2$	$43.6\pm 6.1$	$39.7\pm 6.1$
Signed divergence	$+0.088\pm 0.034$	$+0.126\pm 0.025$	$+0.118\pm 0.029$	$+0.121\pm 0.036$
Cohen’s $\kappa$	$0.104$	$0.082$	$0.118$	$0.035$

Appendix

Table 12: Headline statistics across the four configurations of Finding 5, each averaged over

N{=}11

runs. Main (released) is the configuration used for all other results in this paper. Positive signed divergence and near-chance

\kappa

persist in every configuration.

Trajectory	Cat	Diff	Intr. (%)	Extr. (%)	MAD	Miss (%)	FC	CA (%)	Dir
c1-confidence-001	C1	L1	100.0 $\pm$ 0.0	95.5 $\pm$ 14.4	.045	11 $\pm$ 21	6.4 $\pm$ 4.8	100 $\pm$ 0	$=$
c1-duplicate-facts-001	C1	L1	100.0 $\pm$ 0.0	50.0 $\pm$ 0.0	.500	17 $\pm$ 5	8.0 $\pm$ 3.7	100 $\pm$ 0	$\uparrow$
c1-stale-state-001	C1	L2	83.3 $\pm$ 12.3	45.5 $\pm$ 33.4	.470	69 $\pm$ 9	14.5 $\pm$ 8.2	0 $\pm$ 0	$\uparrow$
c1-wrong-provenance-001	C1	L2	79.1 $\pm$ 5.1	45.5 $\pm$ 14.4	.336	34 $\pm$ 14	21.3 $\pm$ 5.0	6 $\pm$ 13	$\uparrow$
c1-unresolved-conflict-001	C1	L3	48.5 $\pm$ 4.8	36.4 $\pm$ 22.3	.121	99 $\pm$ 4	15.8 $\pm$ 5.0	2 $\pm$ 7	$\uparrow$
c2-phrasing-mismatch-001	C2	L1	76.1 $\pm$ 3.6	50.0 $\pm$ 0.0	.261	31 $\pm$ 3	10.6 $\pm$ 5.0	0 $\pm$ 0	$\uparrow$

Appendix

Table 13: Per-trajectory results averaged across

N{=}11

runs (

\pm

std). MAD = mean absolute divergence between intrinsic and extrinsic for that trajectory. Miss = claim miss rate (%, complement of claim coverage Eq. 1 in the main paper, computed across all claims in all states). FC = average fact count. CA = classification accuracy (%). Dir = direction of divergence:

\uparrow

= intrinsic

>

extrinsic (memory underestimated),

\downarrow

= extrinsic

>

intrinsic (LLM compensates),

=

= gap

<

5 points.

Agent Memory

Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.

See all 190 papers

DolphinBench: Mapping the Pareto Frontier of Agent Memory
Sep 21, 2026Soumil Rathi, Deshraj Yadav, Taranjeet SinghAgent Memory Persistent Memory for Language Models
Mem0
MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
Jul 1, 2026Zhishang Xiang, Zerui Chen, Yunbo Tang +5Agent Memory LLM Sycophancy
Xiamen University · Jilin University
Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings
Jul 24, 2026Quentin SpencerLong-Horizon Agent Evaluation Memory Agent Benchmarks
Independent Researcher

Sep 21, 2026J/K move · Enter open · S save

DolphinBench: Mapping the Pareto Frontier of Agent Memory

Soumil Rathi, Deshraj Yadav, Taranjeet Singh

Mem0

Agent Memory Persistent Memory for Language Models Long-Horizon Agent Evaluation Memory Agent Benchmarks

Open paper AskPDF arXiv

Agents today often take real-world actions that depend on long-term memory and context recall over time. However, most current memory benchmarks are built for a conversational question-answer format, where the question itself signals that some fact must be retrieved, and often which one. Moreover, benchmarks rarely require anything beyond accuracy from submissions, allowing memory systems to make unreasonable cost/time tradeoffs to achieve higher scores. We present DolphinBench, a benchmark that evaluates memory directly through an agent's task completion. DolphinBench includes three knowledge-work personas with roughly 500k tokens of user messages per persona and evaluates agents on tasks that depend on information from that history. We verify all 200 tasks per persona by running an agent with and without the relevant history, requiring success with it and failure without it. Finally, we require all evaluations to report total cost and latency alongside accuracy, which enables us to evaluate agent memory systems holistically. No existing memory benchmark combines all three. The dataset and evaluation code are available at https://dolphinbench.ai.

Abstract

Agent Memory

Explore similar work

DolphinBench: Mapping the Pareto Frontier of Agent Memory

MemSyco-Bench: Benchmarking Sycophancy in Agent Memory

Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings