cs.AISep 28, 2026
SaveWhen Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model
Organizations: Independent Researcher
Abstract
Does conversational memory need LLM-extracted facts, or is selecting the right raw turns enough? Published results disagree. Extraction-based systems report gains from distilled facts. Recent studies find raw history with good ranking does as well, but disagree about whether ranking matters. We ran a pre-registered study on held-out LoCoMo conversations and LongMemEval. At a tight budget on LoCoMo, raw turns selected by a single call to Jev, a typed decision model, are non-inferior to an LLM-extraction memory (one-sided 95% bound -3.0 points against a -5-point margin). Blind human grading narrows the margin but does not change the result. Raw turns cost 3,061 times less to write, and the result holds with a second answer model. Within this study, reranking's gain shrinks as the budget grows. It adds 17.4 points on LoCoMo and 9.1 on LongMemEval when three of 30 candidates are kept. At generous budgets it adds 1.5 and 1.1, and extraction systems are more accurate. This suggests why published results disagree. At matched context, Jev selects as accurately as an LLM reranker (non-inferiority bound -2.0) at a third of the latency, and more accurately than a multi-call graph traversal. Reranking lowers correct abstention. Plans, code and graded answers are released.
Figures & tables
Figure 1: Write path (per turn, top) and read path (per question, bottom) of Turns + Jev, Turns + cosine, engram v2 and Jev-Mem. Border colour says what does the work: code (blue), an LLM call (amber), a Jev typed decision (purple), a store (green), the answer model (red) and the judge (teal). The grid gives LLM calls and Jev requests per turn and Jev requests per question, from each system’s code (Jev-Mem: its default profile); Turns + Jev and Turns + cosine share a write path and differ only per question, where Turns + Jev makes one Jev request and Turns + cosine none. A design diagram; no measured data.
Figure 2: One held-out question traced through both read paths, replayed offline from the frozen stores and the call cache (no API call; both contexts match the recorded token counts: Turns + Jev 292, engram v2 309). Each column lists the top four of the 30-item cosine shortlist and every item the answer model read, in cosine order, with Jev’s P(relevant): purple rows were kept by Jev, blue rows by the cosine floor, and the dashed row was kept by Jev but ranked below the cut at 3. An illustration chosen by the rule in section 3 , not evidence. Appendix J shows a question where extraction wins, chosen by the same kind of rule.
| Grading | Turns + Jev | engram v2 | Difference | One-sided 95% bound | Two-sided 95% CI | Non-inferior (margin 5) |
|---|---|---|---|---|---|---|
| Judge (registered) | 77.0% | 77.5% | 0.5 | 3.0 | [ 3.5, +2.5] | yes |
| Human, strict | 76.3% | 78.0% | 1.7 | 3.9 | [ 4.3, +0.9] | yes |
| Human, lenient | 77.4% | 79.9% | 2.6 | 4.7 | [ 5.1, 0.03] | yes |
Table 1: H1: Turns + Jev at 6 (265 tokens per question) against engram v2 at 3 (251 tokens), 778 scored questions of the five held-out conversations. The judge row is the registered test; the human rows replace the judge’s labels on the 141 graded discordant questions. Differences and bounds in points.
Figure 3: H1 (registered) as a forest plot: Turns + Jev at 6 minus engram v2 at 3, in points, on the 778 questions of the five held-out conversations. Bars are two-sided 95% intervals; the red tick is the one-sided 95% lower bound, tested against the 5-point margin (dashed). The judge row is the registered test; the human rows replace the judge’s labels on the 141 graded discordant questions ( section 5.4 ); the Llama 3.3 70B row re-answers from the same contexts (the answer-model check of section 4 ).
| Test | Turns + Jev vs | Turns + Jev k | Turns + Jev | Other | Only Turns + Jev / only other | p | Holm p |
|---|---|---|---|---|---|---|---|
| S1 | Turns + cosine (LoCoMo) | 3 | 77.2% | 59.9% | 152 / 17 | 2.7e 28 | 1.9e 27 |
| S2 | Jev-Mem (LoCoMo) | 4 | 77.0% | 70.6% | 98 / 48 | 4.3e 5 | 1.3e 4 |
| S3 | mem0 (LoCoMo) | 3 | 77.2% | 68.5% | 134 / 66 | 1.7e 6 | 7.6e 6 |
| S4 | Turns + LLM (LoCoMo, non-inferiority) | 3 | 77.2% | 77.6% | 28 / 31 | 1.5e 6 | 7.6e 6 |
| S5 | mem0 (LongMemEval, 30 knowledge-update) | 2 | 70.0% | 70.0% | 4 / 4 | 1.00 | 1.00 |
| S6 | Turns + cosine (LongMemEval sample, 70) | 3 | 68.6% | 65.7% | 7 / 5 | 0.77 | 1.00 |
Table 2: Secondary tests. In each, the comparator runs at 3 and Turns + Jev at its matched k. “Only Turns + Jev” and “only other” count questions answered correctly by one system. S4 is a non-inferiority test (one-sided p); the rest are exact two-sided McNemar tests. Holm adjustment over S1–S7. LoCoMo tests use 778 questions; LongMemEval ingestion: user turns for S5–S6, user and assistant turns for S7. Bold: rejected after Holm adjustment (family-wise 0.05).
Figure 4: Secondary tests S1–S7 (registered): Turns + Jev minus the comparator, in points, with paired 95% intervals; the Holm-adjusted p is printed at the right, and purple rows are rejected after Holm correction (grey rows are not). S4 is a non-inferiority test against the 5-point margin (dashed). LoCoMo tests use 778 questions; S5 and S6 use the LongMemEval sample (30 and 70 questions), S7 the full set (470).
Figure 5: The rerank’s gain over similarity search (Turns + Jev minus Turns + cosine, paired, in points, with 95% intervals) against k. LoCoMo: 778 questions of the five held-out conversations at 3, 6 and 20; LongMemEval: 470 non-abstention questions, user and assistant turns, at 3 and 20. The 3 points are registered tests (S1, S7); the others are descriptive.
| System, setting | Accuracy | Tokens | Write $/1k | Read $/query | Multi-hop | Temporal | Open-domain | Single-hop |
| Tight budget ( 3 or 6) | ||||||||
| Turns + cosine, 3 | 59.9% | 130 | $0.0006 | 0 † | 54.3 | 55.2 | 42.0 | 65.7 |
| Turns + cosine, 6 | 68.6% | 254 | $0.0006 | 0 † | 61.4 | 60.6 | 54.0 | 75.9 |
| Turns + Jev, 3 | 77.2% | 139 | $0.0006 | $0.00020 | 75.7 | 69.1 | 58.0 | 83.2 |
| Turns + Jev, 6 | 77.0% | 265 | $0.0006 | $0.00020 | 74.3 | 72.1 | 56.0 | 82.3 |
| Turns + LLM, 3 | 77.6% | 143 | $0.0006 | $0.00023 | 73.6 | 72.7 | 58.0 | 83.2 |
Table 3: LoCoMo, five held-out conversations, 778 scored questions: accuracy (%), tokens per question, write cost per 1,000 turns and read cost per query at list prices (as in Table 5 : the read cost is the Jev or LLM rerank call, excluding the answer call; 0 † : embedding only, the read path makes no model call, only a query embedding, which is not priced; “–”: none), and accuracy by category. Rows are grouped by budget. Bold: best in column within the budget group (highest accuracy, lowest write cost); read costs are not bolded, because the lowest are the unpriced embedding-only reads.
Figure 6: Accuracy against retrieved tokens per question (log scale), with Wilson 95% intervals. Left: LoCoMo, 778 questions of the five held-out conversations, each system at each k it was run. Right: LongMemEval, 470 non-abstention questions, user and assistant turns; only Turns + Jev, Turns + cosine and full context ran on all 500 LongMemEval questions (mem0 ran only on the 30 knowledge-update questions of the registered sample, and engram v2 and Jev-Mem not at all), which is why the right panel has three systems. Shaded: the tight budget (at most 300 tokens). the hollow point, Turns + Jev (wide), is post-hoc; the other points are registered runs, compared descriptively except in the tests of section 5.1 – section 5.3 .
| System | Accuracy | Tokens | KU (72) | MS (121) | SS-A (56) | SS-P (30) | SS-U (64) | TR (127) | Abs |
| Tight budget ( 3) | |||||||||
| Turns + cosine, 3 | 57.7% | 522 | 58.3 | 31.4 | 85.7 | 53.3 | 95.3 | 52.0 | 43.3% |
| Turns + Jev, 3 | 66.8% | 534 | 77.8 | 47.9 | 98.2 | 46.7 | 98.4 | 53.5 | 43.3% |
| Generous budget ( 20, and full context) | |||||||||
| Turns + cosine, 20 | 72.8% | 4,352 | 84.7 | 58.7 | 98.2 | 40.0 | 96.9 | 63.8 | 70.0% |
| Turns + Jev, 20 | 73.8% | 2,278 | 84.7 | 67.8 | 96.4 | 46.7 | 96.9 | 58.3 | 63.3% |
Table 4: LongMemEval, all 500 questions, user and assistant turns ingested: accuracy (%) on the 470 non-abstention questions and by question type (n in parentheses), and the share of the 30 abstention questions answered by abstaining. KU: knowledge update; MS: multi-session; SS-A, SS-P, SS-U: single-session assistant, preference and user; TR: temporal reasoning; Abs: abstention. Bold: best in column within the budget group.
Figure 7: Where Turns + Jev’s accuracy stops, by category, on the 778 questions of the five held-out conversations: the rerank kept at least one evidence turn (purple), evidence was in the shortlist but the rerank kept none of it (amber), or no evidence turn was in the shortlist (grey). Upper bar of each pair: Turns + Jev’s 30-turn shortlist (shortlist recall, exploratory as registered). Lower, lighter bar: the wide variant’s 150-turn shortlist (post-hoc). Percentages are printed where the segment is wide enough.
| System | Write $/1k turns | LLM | Jev | Embeddings | Write p50 (s) | Read $/query | Jev calls/query | Read p50 (ms) | Read p90 (ms) |
|---|---|---|---|---|---|---|---|---|---|
| Turns + Jev | $0.0006 | – | – | $0.0006 | 0.2 | $0.00020 | one | 273 | 359 |
| Turns + cosine | $0.0006 | – | – | $0.0006 | 0.2 | 0 † | none | 219 | 263 |
| Turns + LLM | $0.0006 | – | – | $0.0006 | 0.2 | $0.00023 | one (no-op) | 816 | 1,107 |
| engram v2 | $1.865 | $1.322 | $0.542 | $0.0015 | 2.0–2.1 | $0.00018 | one | 276 | 319 |
| mem0 | $1.321 | $1.319 | – | $0.0017 | 2.1–2.3 | 0 † | none | 486 | 718 |
| Jev-Mem | $0.212 | – | $0.211 | $0.0011 | 0.5 | $0.00120 | 5.3 | 1,329 | 1,622 |
Table 5: Cost and latency (exploratory as registered), all at 3 (the tight budget). Write cost per 1,000 turns at list prices, split by LLM, Jev and embeddings; read cost per query; read latency measured live on a fixed sample of 40 questions at 3, one query at a time, including the query-embedding call. Jev-Mem’s latency comes from its reads of the same questions, measured live when they ran (42 reads: two question texts repeat). A write-latency range spans the conversations and is compared by its lower end. 0 † : embedding only, no model call on the read path, only an unpriced query embedding. Bold: best in column within the budget group (lowest cost and latency); read costs are not bolded, as in Table 3 .
Figure 8: Accuracy against total cost per question (log scale) at 3, on the 778 questions of the five held-out conversations (exploratory as registered): write cost amortised at the benchmark’s 4.0 turns written per question, plus read cost and answer cost (judge excluded), at list prices; full context has no write or read cost. In a read-heavy use with one turn written per question, engram v2’s total falls to 0.00142 and Jev-Mem’s to $0.00151; the other systems’ totals do not change at this precision.
| System | 3 | 20 |
|---|---|---|
| Turns + cosine | 63.6% | 47.8% |
| engram v2 | 59.8% | 52.2% |
| mem0 | 59.8% | 53.6% |
| Turns + Jev | 54.1% | 49.3% |
| Turns + LLM | 48.8% | 53.1% |
Table 6: Share of LoCoMo adversarial questions (209, five held-out conversations) answered by abstaining (exploratory as registered). Each column is a budget group. Bold: best in column within the budget group (highest share).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| System, setting | conv-44 (123) | conv-47 (150) | conv-48 (191) | conv-49 (156) | conv-50 (158) |
|---|---|---|---|---|---|
| Turns + Jev, 3 | 78.0 | 76.7 | 79.1 | 73.7 | 78.5 |
| Turns + Jev, 6 | 79.7 | 76.7 | 78.5 | 73.7 | 76.6 |
| Turns + Jev, 20 | 79.7 | 74.7 | 81.7 | 75.0 | 76.6 |
| Turns + cosine, 3 | 65.9 | 57.3 | 62.8 | 58.3 | 55.7 |
| Turns + cosine, 20 | 78.0 | 74.0 | 79.1 | 74.4 | 74.7 |
| Turns + LLM, 3 | 76.4 | 72.7 | 80.6 | 77.6 | 79.7 |
Table 7: Accuracy (%) per conversation, LoCoMo scored categories.
| Date | Change | Reason | Effect on results |
|---|---|---|---|
| 2026-09-26 | Amendment, deposited after Batch A (S1 and S2 known) and before any H1 result: shortlist recall; LongMemEval on all 500 questions with user and assistant turns, with the new test S7; the second answer model; the outcome paragraphs and the rule for “LongMemEval holds”; budget caps | Extend the study before the primary test was run | S7 joins the Holm family; new robustness checks; H1, its margin and the other tests unchanged |
| 2026-09-26 | Outcome paragraphs revised before upload | The first author’s own wording | None; made before any H1 result |
| 2026-09-26 | mem0’s extraction calls went through OpenRouter, about half served by Azure, instead of the OpenAI API; the ledger was corrected and guards added before any later run | A mem0 library default routes calls to OpenRouter when its key is set ( Appendix G ) | S3 is reported with a caveat; no other test involves mem0 |
| 2026-09-26 | Runs repeated after OpenAI rate limits, with more retries and a Jev throttle; completed calls replayed from the call cache | Rate limits | None: replayed calls are identical |
| 2026-09-26 | Turns + LLM also asks Jev’s query-relation question once per query | Shared read-path code | None: the question is a no-op on turns |
| 2026-09-26 | Token counting treats text that spells a special token ( <|endoftext|> , in one LongMemEval haystack) as ordinary text | The tokenizer refused that text | One question re-run; every other count unchanged |
Table 8: Deviations from the registered plan.
| Mapping | Agreement with judge | On Turns + Jev’s answers | On engram v2’s answers | Only Turns + Jev right | Only engram v2 right | Both right | Both wrong |
| Strict | 81% | 81% | 82% | 47 | 59 | 17 | 18 |
| Lenient | 79% | 82% | 77% | 41 | 60 | 31 | 9 |
| Mapping, treatment | Questions | Turns + Jev | engram v2 | Difference | One-sided 95% bound | Two-sided 95% CI |
|---|---|---|---|---|---|---|
| Strict, (a) judge’s labels | 778 | 76.3% | 78.0% | 1.7 | 3.9 | [ 4.3, +0.9] |
| Strict, (b) dropped | 777 | 76.4% | 78.0% | 1.5 | 3.7 | [ 4.1, +1.1] |
| Lenient, (a) judge’s labels | 778 | 77.4% | 79.9% | 2.6 | 4.7 | [ 5.1, 0.03] |
| Lenient, (b) dropped | 777 | 77.5% | 79.9% | 2.4 | 4.6 | [ 5.0, +0.09] |
Table 9: H1 with human grades under both treatments of the ungraded question. Differences and bounds in points.
| Category | Questions | All evidence in shortlist | At least one | Rerank keeps none |
|---|---|---|---|---|
| Multi-hop | 250 | 42.8% | 88.4% | 5.9% |
| Temporal | 284 | 83.5% | 88.4% | 22.7% |
| Open-domain | 83 | 42.0% | 63.0% | 31.4% |
| Single-hop | 771 | 89.2% | 91.2% | 5.8% |
| All | 1,388 | 76.9% | 88.5% | 10.4% |
Figure 9: The counter-example of Appendix J , drawn as Figure 2 (both contexts match the recorded token counts: Turns + Jev 279, engram v2 199). The evidence turn for “a breeder” is not in Turns + Jev’s 30-turn shortlist; engram v2’s fact from it is, and Jev keeps it. An illustration chosen by the rule above, not evidence.