A growing class of conversational-memory systems compresses dialogue history into structured artifacts (extracted facts, decisions, or events). The premise is that distilled structure retrieves better than raw text. We test this premise with a controlled ablation. Within one fixed retrieval--rerank--reasoning pipeline, we swap only the stored representation (LLM-extracted typed artifacts versus verbatim conversation chunks) and hold the model, retriever, reranker, and judge constant. Verbatim chunks win by 15.9 points on LoCoMo (43.9% vs. 28.0%) and by 22.0 points on LongMemEval-S (67.4% vs. 45.4%). A semantic graph does not recover the gap, and six confound controls reproduce the effect. The mechanism is lossy distillation rather than structure per se. Accuracy tracks how much source text survives in the store, and the extracted-artifact pipeline does not beat naive RAG in overall accuracy (chunks do abstain worse; see Limitations). For the extraction designs we test, structured memory should augment verbatim text rather than replace it. Adding artifacts alongside chunks costs no detected accuracy, while substituting them forfeits the gap. Code and data are available at https://github.com/tao-hpu/cog-canvas.
Figures & tables
Figure 1: The controlled ablation design. At write time the conversation passes through exactly one manipulated stage (Axis 1), either verbatim 512-character chunking or LLM extraction into typed artifacts. The output enters a shared embedded store, optionally linked by a similarity graph (Axis 2). The query-time path (hybrid retrieval, reranking, token-budgeted context selection, answerer, judge) is identical across all four resulting pipelines. The Artifacts + Graph cell is the full CogCanvas system as originally engineered.
Stored representation
Overall
Multi-hop
Temporal
Open-domain
Reference ceiling (no compression; conversations fit the context window)
Full context (entire conversation in prompt)
59.9
57.4
62.3
59.4
Verbatim conversation chunks
Chunks, no graph
43.9
35.5
50.2
47.9
Chunks + graph
43.1
36.5
47.7
46.9
Chunks, zero overlap (control ii)
42.8
34.8
49.2
44.8
Table 1: LoCoMo (Cat 1–3; 10 conversations, 699 questions; LLM-Judge via GPT-4o-mini). Within one fixed retrieval–rerank–reasoning pipeline we swap only the stored representation; all rows share one retriever (bge-m3), reranker (bge-reranker-v2-m3), answerer (gpt-4o), and judge. The full-context row is a reference ceiling and not a memory system. LoCoMo conversations ( ∼ 26K tokens) fit gpt-4o’s window, so nothing is compressed. Bold marks the best memory-system entry per column (baseline rows eligible). Verbatim chunks beat LLM-extracted artifacts by 15.9 points overall and on every question type. The gap is significant (McNemar exact test: 154 vs. 43 discordant pairs, p<10−15 ; cluster bootstrap resampling whole conversations: 95% CI [13.0,18.4] pp), while artifacts vs. naive RAG is not (106 vs. 103, p=0.89 ). The graph, overlap, budget, dense-only, and union controls are analyzed in the text.
Condition
Chunks
Artif.
Gap
Headline (Table 1 )
43.9
28.0
+ 15.9
Zero-overlap chunks
42.8
28.0
+ 14.7
Budget-matched artifacts ( k=60 )
43.9
29.2
+ 14.7
Dense-only retrieval
41.8
27.5
+ 14.3
Session-level extraction
43.9
29.2
+ 14.7
Decontaminated extraction prompt
43.9
28.5
+ 15.5
Table 2: Robustness of the chunks-minus-artifacts gap on LoCoMo categories 1–3 (LLM-Judge %, 699 questions; last row: LongMemEval-S, 500 questions). Each row changes one element of the headline comparison; the Claude row changes both LLM roles. Gaps use unrounded counts. The budget-matched and session-level rows are separate runs that coincide at 204 of 699. Details: § 4 and Appendices L.5 , H , L.7 .
Store
IE
MSR
KU
TR
Abs
All
T-Avg
n
150
121
72
127
30
500
—
Chunks
84.7
62.8
77.8
50.4
46.7
67.4
64.5
Union
80.7
58.7
76.4
51.2
36.7
64.6
60.7
RAG+RR
85.3
61.2
72.2
38.6
70.0
64.8
65.5
RAG
86.7
58.7
72.2
37.0
70.0
64.2
64.9
Artifacts
50.7
34.7
69.4
36.2
43.3
45.4
46.9
Table 3: LongMemEval-S (500 questions; LLM-Judge via GPT-4o-mini). The same representation swap as Table 1 , on a second benchmark. Rows: Chunks = chunks, no graph; Union = chunks ∪ artifacts + graph; RAG + RR = RAG + Rerank; Artifacts = artifacts + graph (CogCanvas); k=60 = its budget-matched variant; Trunc. = truncation, recent-5 turns; Δ C − A = chunks minus artifacts. IE = information extraction, MSR = multi-session reasoning, KU = knowledge updates, TR = temporal reasoning, Abs = abstention; Overall is question-weighted, T-Avg the unweighted category mean. Truncation’s 93.3 abstention is degenerate, since it answers almost nothing. McNemar exact tests: chunks vs. artifacts p<10−15 (152 vs. 42 discordant pairs; bootstrap 95% CI [17.0,27.0] pp); artifacts trail even naive RAG, p<10−11 (48 vs. 142). Δ s are computed from unrounded counts; ns marks per-type deltas that are not significant (KU: 15 vs. 9 discordant, p=0.31 ; Abs: 4 vs. 3, p=1.0 , a one-question margin).
Method
Extract
Gen
Total
Total †
Ratio
Truncation
$0.00
$1.25
$1.25
$0.09
13.2 ×
RAG
$0.00
$3.50
$3.50
$0.23
15.3 ×
RAG+Rerank
$0.00
$3.50
$3.50
$0.23
15.3 ×
Chunks (no graph) §
$0.00
$3.84
$3.84
$0.23
16.7 ×
Summarization
$0.67
$1.74
$2.41
$0.17
14.2 ×
Artifacts (CogCanvas) §
$0.13
$2.20
$2.33
$0.27
8.7 ×
Table 4: Measured cost on LoCoMo10 (699 questions, 2,938 indexed turns; USD). Extract and Gen aggregate response.usage tokens per phase. Total † is the gpt-4o-mini answerer tier and Ratio is gpt-4o / mini. Bold marks the cheapest retrieval-augmented method. § Priced from the token logs of the two Table 1 runs; Total † re-prices the same tokens. A separately instrumented artifact run with deeper retrieval (top-20, four hops) costs $2.92. ‡ Not instrumented. Details in Appendix N .
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Recall
Exact
Cmplx.
Lossy stores
Truncation
12.5%
7.0%
Low
Summarization
19.0%
14.0%
Low
GraphRAG
83.5%
70.0%
High
Verbatim stores
RAG (k=10)
93.5%
89.5%
Low
Appendix
Table 5: Information retention on the standard synthetic benchmark (a fixed 50-conversation subset, 50 turns, 4 planted facts each; rolling compression every 40 turns; gpt-4o-mini answerer). The two chunks rows were measured in the 2026-06 batch. All other rows are the 2025-12 batch under the identical protocol (Table 16 ). Same-configuration cross-batch drift on this harness is 1–3.7pp with direction preserved (Table 16 ). The summarization-vs-chunks exact-match gap (14.0% vs. 91.0%), the 77.0-point figure quoted in §1, is therefore a cross-batch difference, far larger than any observed drift. Cmplx. = qualitative implementation complexity (write-time machinery and infrastructure). Truncation keeps the most recent 5 turns (§ 4 ).
Method
Pass
KW
Causal
Impact
Summarization
0.0%
42.5%
48.3%
35.8%
Chunks + graph
24.5%
60.8%
50.6%
71.0%
Chunks, no graph
25.0%
60.8%
50.6%
71.0%
GraphRAG
40.0%
64.8%
62.5%
67.0%
RAG+Rerank
45.0%
68.2%
64.2%
73.3%
RAG
55.5%
74.8%
68.8%
79.5%
Appendix
Table 6: Multi-hop causal reasoning on the synthetic probe (50 conversations, rolling compression every 40 turns, gpt-4o-mini answerer). The two chunks rows and the RAG+Rerank row are the 2026-06 batch. All other rows, including both terms of the Δ row, are the 2025-12 batch under the identical protocol (Table 16 ). The RAG configuration re-measures at 52.5% in 2026-06 (Appendix O ). KW = Keyword Coverage. This is the one setting in the paper where the artifact pipeline wins, and it is extraction’s best case (see text). The advantage inverts on the public benchmarks (Tables 1 and 3 ).
Figure 2: A controlled fidelity dial (§ 4.7 ). Keeping a random fraction of each verbatim chunk’s tokens (solid line; pipeline held fixed) drives LoCoMo Cat 1–3 accuracy down monotonically. At retain = 1.0 the dial reproduces the verbatim-chunks anchor (43.9%). The dashed line repeats the dial under a gpt-4o-mini answerer (accuracy 43.5→30.8→21.2→12.2→8.4 as retain goes 1.0→0.2 ). The near-coincident curve is consistent with a collapse that belongs to the stored representation and does not depend on the strong answerer model. Real memory systems (right column) land on the same accuracy axis in fidelity order. Extraction-based Mem0/A-Mem fall in the dial’s low-fidelity regime, verbatim-preserving SeCom/chunks fall in the high one, and full context (no compression) sits above all. SeCom’s edge over fixed chunks is a segmentation effect within the verbatim tier and does not reverse the fidelity order.
Method
TF
State
Infra
P&P
Incr.
Truncation
✓
✗
✗
✓
✓
Summarization
✓
✗
✗
✓
✓
RAG
✓
✗
✗
✓
✓
RAG+Rerank
✓
✗
✗
✓
✓
GraphRAG
✓
✗
✗
✓
✗
Letta ( Letta AI, 2025 )
✓
✓
✓
✗
✓
Appendix
Table 7: Design-space positioning across five operational axes. TF = training-free (no fine-tuning required). State = maintains memory state across queries. Infra = requires persistent infrastructure (e.g. PostgreSQL, Docker). P&P = plug-and-play (drop-in for any answerer LLM, no API contract). Incr. = supports incremental updates without re-indexing the full conversation. All four pipelines under test (chunks/artifacts × graph on/off) occupy the same stateless, training-free, infrastructure-free cell. Our comparison therefore swaps representations within one corner of this space and does not promote any one system. GraphRAG’s lack of incremental support stems from its community-summarization stage, which must re-cluster the entity graph when new turns arrive. Letta’s stateful design places it outside our stateless evaluation pipeline (§ 4 , Appendix C ). We cite its 74.0% LoCoMo result only as an external anchor.
Figure 3: Walked example of the synthetic benchmark protocol on conv 922f651b (50 turns, “hard” difficulty). Turns row. A 50-turn conversation has four facts planted at turns 3, 5, 7, 9 (colored cells). Filler dialogue (turns 11–39) and the post-compression query window (turns 41–50) are collapsed into ellipsis blocks for space. The dashed line at turn 40 marks the rolling-compression boundary that discards short-term context. Cards row. Each planted fact carries a cognitive type drawn from the broader extraction ontology of § 3 (here Reminder in orange, KeyFact in green, or Decision in red). Each card also shows the fact’s structured key: value content, the post-compression test query, and the ground-truth answer used to score Recall and Exact Match.
Figure 4: Planted-fact retention on the synthetic benchmark (protocol of Table 5 ). Methods cluster by storage fidelity. Verbatim stores (RAG, chunks, quote-grounded artifacts) all exceed 90% recall (89.5–93.0% exact match), and lossy stores collapse. The annotated 77.0-point exact-match gap is the figure quoted in §1.
Figure 5: The synthetic multi-hop probe is the one setting where the artifact pipeline wins (81.0% vs. 55.5% for the best verbatim method). It is extraction’s best case, where every fact is pre-shaped into exactly one artifact. The advantage inverts on both public benchmarks.
Figure 6: LoCoMo breakdown by question type (per-question micro-averages, LLM Judge). Swapping LLM-extracted artifacts (red) for verbatim chunks (green) inside the same pipeline costs accuracy on every question type. † GraphRAG uses a gpt-4o-mini answerer (Table 1 , footnote).
Archival representation
Overall
M-hop
Temp.
O-dom.
Verbatim turns
24.0
27.7
13.4
49.0
Extracted artifacts
13.2
14.5
7.2
29.2
Δ (verbatim − artifacts)
+10.9
+13.1
+6.2
+19.8
Appendix
Table 8: Statefulness does not overturn the ordering. We swap only the archival representation inside a stateful MemGPT-style agent (core memory + rolling compression + archival retrieval) and hold everything else fixed. Evaluation is on LoCoMo Cat 1–3 (699 questions, gpt-4o answerer, gpt-4o-mini judge). Verbatim archival dominates extracted-artifact archival on every question type (overall + 10.9pp, McNemar exact p<10−11 , 100 vs. 24 discordant). The Δ row is computed from unrounded counts (168 vs. 92 of 699 correct), and cells are rounded to one decimal.
Configuration
All
M-hop
Temp.
O-dom.
Full System
28.6%
20.9%
30.5%
44.8%
− Graph Expansion
25.8%
19.5%
29.0%
33.3%
− Reranking
20.9%
16.7%
23.1%
26.0%
Minimal (Graph only)
14.3%
11.7%
13.7%
24.0%
Appendix
Table 9: Ablation study on LoCoMo (Cat 1/2/3, all 10 conversations, LLM Judge, question-micro; January batch). Full system achieves 28.6%. Reranking is the largest contributor ( − 7.7pp when removed), followed by graph expansion ( − 2.8pp overall, − 11.5pp on open-domain).
compress_rate
Overall
M-hop
Temp.
O-dom.
1.00 (verbatim segments)
47.4
37
56
47
0.75 (SeCom default)
46.9
39
53
49
0.50 (aggressive)
36.9
30
41
44
ref: verbatim chunks
45.2
ref: full context
59.9
Appendix
Table 10: SeCom accuracy as a function of its own extractive-compression rate (LoCoMo Cat 1–3, micro-accuracy). Accuracy falls monotonically as less verbatim text is retained. At aggressive compression a real extractive compressor drops below the fixed-chunk baseline. Ref. rows re-measure the chunks and full-context anchors in this batch. Chunks re-measures at 45.2 against the 43.9 headline of Table 1 , a same-configuration run-to-run difference of 1.3pp (Appendix L.1 ).
Store (all same backbone)
Tok/q
Acc.
Verbatim chunks (coarse)
2,127
42.8
Sentence-verbatim, k=90 (2.2 × budget)
4,674
39.1
Sentence-verbatim, k=38 (budget match)
2,140
36.2
Sentence-verbatim, k=15
1,010
32.2
Typed artifacts (lossy, fine)
1,159
26.5
Appendix
Table 11: Sentence-verbatim control (question-micro accuracy over 699 questions; gpt-4o-mini answerer). The chunks, k=90 , k=15 and artifacts rows come from one batch; the k=38 row was added in a later batch under the same configuration. “Tok/q” is mean answerer-prompt tokens per question. At matched fine granularity and budget, verbatim beats lossy by 5.7pp (32.2 vs. 26.5). At the chunk token budget ( k=38 , 2,140 vs. 2,127 tokens), the sentence store trails chunks by 6.6pp, which is granularity’s share of the gap. The k=90 row (2.2 × the chunk budget) is kept for reference. This batch’s chunks and artifacts anchors (42.8, 26.5) re-measure the mini-answerer anchors reported elsewhere (41.9, Appendix T ; 28.5–29.9, Appendix P ). The two sets agree within 0.9–3.4pp, the cross-batch range documented in Appendix L.1 .
Store (same pipeline)
GPT pair
Claude pair
Verbatim chunks
43.9
53.9 (54.7)
Artifacts + graph
28.0
33.2 (34.0)
Δ Chunks − Artifacts
+ 15.9
+ 20.7
Prompt tok/q, chunks / artifacts
2,127 / 1,191
2,400 / 1,518
Appendix
Table 12: The Table 1 pair under two vendors for answerer and extractor (LoCoMo Cat 1–3, 699 questions; question-micro, mean over the 10 conversations in parentheses; judge gpt-4o-mini in both). The GPT pair is gpt-4o / gpt-4o-mini, and the Claude pair is claude-sonnet-4-6 / claude-haiku-4-5. On the Claude pair, the McNemar exact test counts 199 chunks-only correct and 54 artifacts-only correct ( p=1.1×10−20 ). Token counts are the vendors’ own accounting and are not directly comparable across columns. The within-column asymmetry is the one analysed in Table 1 .
Store
k
Tok/q
All
M-hop
Temp.
O-dom.
Chunks
15
2,127
47.4
37.9
53.9
53.1
EDU
90
∼ 2,700 e
36.5
23.4
46.7
40.6
EDU
60
∼ 1,900 e
36.3
22.0
47.4
41.7
EDU
15
∼ 700 e
29.8
17.0
40.5
31.2
Artifacts
15
1,159
30.2
26.2
30.8
39.6
Appendix
Table 13: EMem-style EDU control (LoCoMo Cat 1–3, question-micro accuracy; Chunks = verbatim chunks, EDU = EMem-style EDU store, Artifacts = typed artifacts; one 2026-07 batch, gpt-4o answerer). The chunks and artifacts rows are the same-batch re-runs of Appendix T , so no cross-snapshot comparison is involved. e EDU token budgets are estimated from mean unit length (103 chars ≈ 26 tokens) times k plus prompt overhead. The chunks and artifacts figures are instrumented means (Appendix G ).
Store (same backbone, Chinese)
Acc.
Naive RAG (verbatim)
86.1
Verbatim chunks
82.2
Typed artifacts (lossy)
35.6
Appendix
Table 14: PerLTQA dialogue-category QA (gpt-4o-mini answerer, 31 conversations, 902 questions). Both verbatim stores beat the lossy artifact store by ∼ 47–50 points; artifacts again never beat naive RAG.
Direct experimental finding
Interpretation we draw
Chunks beat artifacts by 15.9pp (LoCoMo) and 22.0pp (LongMemEval-S) in one fixed pipeline; both gaps significant (Tables 1 , 3 ).
The stored representation, not the retrieval stack, is what loses.
The gap survives six confound controls, three retriever families and a second answerer/extractor vendor (§ 4 , Appendices P , H ).
The effect is not an artifact of one configuration.
Accuracy falls monotonically as verbatim tokens are removed, at random or by a real compressor (Figure 2 , Appendix F ).
Fidelity is causal for accuracy in this pipeline.
At equal budget, a sentence-verbatim store trails chunks by 6.6pp and beats artifacts by 5.7pp at matched granularity (Appendix G ).
Granularity explains part of the gap; fidelity explains more of it.
Union store: 42.5 vs. chunks 43.9, p=0.39 (no significant difference detected; not an equivalence test).
Adding structure to verbatim text costs no detected accuracy; replacing text does.
Of 58 diagnosable discordant questions, 29.3% have no trace in any artifact and 39.7% survive only in a raw-context field (Appendix D ).
Extraction commits to relevance before the question exists; only the first class is irreversible loss.
Appendix
Table 15: Measured results (left) and the interpretations placed on them (right). Left-column entries are reproducible from the released run artifacts; right-column entries are the account of § 4.7 and can be rejected without touching the left column.
Result
Batch
Aggregation
Answerer
Tab. 5 (synthetic std.)
2025-12/06 d
per-fact
gpt-4o-mini
Tab. 6 (multi-hop)
2025-12/06 d
conv. macro
gpt-4o-mini
Tab. 1 (LoCoMo)
2026-06
question micro
gpt-4o a
cat. 4/5 paragraphs
2026-06
question micro
gpt-4o
Tab. 3 (LME-S)
2026-06
question micro
gpt-4o
Tab. 4 (cost)
2026-05 b
usage logs
both tiers
Appendix
Table 16: Provenance of the main quantitative results. a GraphRAG row uses gpt-4o-mini (Table 1 footnote). b Instrumented cost runs of 2026-05-27; the chunks and artifacts rows are priced from the 2026-06 runs (Table 4 footnote). c Dashed dial curve uses gpt-4o-mini (Appendix T ). d Two-batch tables. In these, the chunks rows (and, in Table 6 , the RAG+Rerank row) are 2026-06, and all other rows are 2025-12 under the identical protocol (table captions mark the affected comparisons). “Conv. mean/macro” rows are comparable with each other and are not commensurate with question-micro rows. Unless a caption states otherwise, Δ /gap values are computed from unrounded counts and may differ from the difference of the rounded cells by 0.1pp. Same-configuration repeats of the headline pipelines agree to ≤ 0.4pp within a batch (three-run study, § 6 ). The widest same-configuration repeat we observed is 2.0pp (SeCom at rate 0.75, Appendix F ). Cross-batch anchors drift 1–3.7pp with direction preserved, and the drift is answerer-side (Appendix T ). The drifting anchors are chunks 43.9/45.2/47.4/47.6 (gpt-4o), 41.9/42.8/43.1 (gpt-4o-mini), and RAG grid 55.5/52.5 (Table 18 ).
Config.
Overall
M-hop
Temp.
O-dom.
Full System
71.8%
87.3%
63.2%
55.0%
– Graph Exp.
25.8%
32.3%
20.5%
24.7%
Improvement
+45.9pp
+55.0pp
+42.6pp
+30.3pp
Chunk store ∗
59.0%
56.1%
67.6%
38.7%
+ rerank ∗
61.6%
61.2%
68.9%
38.4%
Appendix
Table 17: Retrieval recall: percentage of ground-truth keywords found in retrieved context (no LLM answering; artifact rows January batch, same protocol as Table 9 ). Graph expansion improves the artifact store’s keyword recall by +46pp. ∗ Chunk-store rows (no graph; second row after reranking): June batch, identical script and k .
Chunk size
Overlap
k=5
k=10
256
50
43.5%
50.5%
512
100
49.0%
52.5%
1024
200
63.0%
63.0%
Appendix
Table 18: RAG pass rate on the synthetic multi-hop probe across chunk size and retrieval depth (same protocol and answerer as Table 6 ). The configuration used throughout the paper (512 chars, k=10 ) is not the grid optimum.
Retriever family
Chunks
Artifacts
Gap
bge-m3 (dense, default)
43.1
28.5
+ 14.6
text-embedding-3-small (dense)
43.5
29.9
+ 13.6
BM25 (sparse lexical)
43.8
29.0
+ 14.7
Appendix
Table 19: LoCoMo categories 1–3 (699 questions, LLM-Judge %), chunks vs. artifacts under three retriever families, with the answerer fixed at gpt-4o-mini and extraction recomputed without cache. The chunk advantage is stable at + 13.6 to + 14.7pp; chunks barely move (43.1–43.8) as the retriever changes.
Cat-5
Cat 1–3
Refusal mechanism
abstention
answerable
Score-threshold gate (no gate: 10.8 / 43.1)
τ=0.1
55.6
25.3
τ=0.2
59.2
21.9
τ=0.3
63.5
17.9
τ=0.4
65.5
—
Appendix
Table 20: Refusal mechanisms on the chunk pipeline (LoCoMo, gpt-4o-mini, LLM-Judge %). Top: an evidence-sufficiency gate on the top reranker score; as τ rises, category-5 refusal climbs but answerable accuracy (categories 1–3) falls monotonically below even the artifact pipeline (the τ=0.4 answerable cell is omitted because abstention already saturates). Bottom: two stronger semantic refusers, an LLM evidence-sufficiency gate (M1, swept over a coarse score pre-filter τ ) and answer-then-verify (M2). M1 modestly Pareto-dominates the score gate at matched abstention, but no mechanism escapes the trade-off. Both blocks use gpt-4o-mini no-gate anchors: category-5 refusal 10.8 in both, answerable accuracy 43.1 and 44.6 from two runs (equal within run-to-run noise). The 6.5% of § 4 is the gpt-4o answerer.
Store
Cat 1–3
Cat 5
Chunks (union’s chunk subset)
47.6
4.7
Artifacts (union’s artifact subset)
30.3
10.3
Union (chunks + artifacts)
46.4
7.0
Artifacts, independent extraction
29.3
11.7
Appendix
Table 21: Same-store three-way comparison on LoCoMo (question-micro LLM-Judge %, gpt-4o answerer; categories 1–3: 699 answerable questions; category 5: 446 unanswerable questions, where the score is the correct-refusal rate). All arms are 2026-11 runs on the same code. This batch sits above the 2026-06 headline batch on categories 1–3 (chunks 43.9, artifacts 28.0, union 42.5), the answerer-side drift documented in Appendix T and Table 16 . The chunks-minus-artifacts gap here is 17.3pp (172 vs. 51 discordant, p<10−15 ). Union vs. chunks is not significant (62 vs. 71, p=0.49 ). A 2026-09 run of the union arm gave 45.9 and 6.3.
Configuration
Overall
M-hop
Temp.
O-dom.
cs=800, g=3, cl=2, local
13.5
16.2
2.2
52.4
cs=800, g=1, cl=2, local †
13.0
18.9
2.2
38.1
cs=800, g=2, cl=2, local
12.4
16.2
2.2
42.9
cs=800, g=1, cl=3, local
12.4
16.2
2.2
42.9
cs=800, g=1, cl=2, drift
11.9
13.5
4.4
38.1
cs=400, g=1, cl=2, local
11.9
16.2
1.1
42.9
Appendix
Table 22: Tier 1 grid search on a 3-conversation LoCoMo subset, 185 questions (LLM Judge, %). The default configuration ( † ) ranks second; the top row is the max_gleanings = 3 leader. The open-domain column shows substantial variance across cells (23.8–52.4) reflecting the small denominator (21 open-domain questions on the subset).
Configuration
Overall
M-hop
Temp.
O-dom.
cs=800, g=1, cl=2, drift (tuned)
13.0
12.8
7.5
32.3
cs=800, g=3, cl=2, local
11.4
13.8
3.4
31.2
cs=800, g=1, cl=2, local †
11.2
13.5
3.1
31.2
Verbatim chunks, no graph ∗
43.9
35.5
50.2
47.9
Artifacts + graph ∗
28.0
22.3
31.2
34.4
Gap: tuned GraphRAG − chunks
–30.9
–22.7
–42.7
–15.6
Appendix
Table 23: Tier 2 grid search on the full LoCoMo benchmark (10 conversations, 699 questions, LLM Judge %). The drift cell is the tuned ceiling reported in the main results table. † Default configuration. ∗ Reference rows reproduce the main-table numbers (Table 1 ); all rows are per-question micro-averages (see aggregation note above).
Config
Ref.
Causal
Pass
Low
0.3
0.25
62.5%
Default
0.5
0.45
67.5%
High
0.7
0.6
60.0%
Very High
0.8
0.7
57.5%
Appendix
Table 24: Threshold sensitivity on the synthetic multi-hop probe (10-conversation pilot, pre-rolling protocol, gpt-4o-mini answerer; rows internally comparable only). Performance is stable across values.
Large language models still struggle with reliable long-term conversational memory: simply enlarging context windows or applying naive retrieval often introduces noise and destabilizes responses. We present APEX-MEM, a conversational memory system that combines three key innovations: (1) a property graph which uses domain-agnostic ontology to structure conversations as temporally grounded events in an entity-centric framework, (2) append-only storage that preserves the full temporal evolution of information, and (3) a multi-tool retrieval agent that understands and resolves conflicting or evolving information at query time, producing a compact and contextually relevant memory summary. This retrieval-time resolution preserves the full interaction history while suppressing irrelevant details. APEX-MEM achieves 88.88% accuracy on LOCOMO's Question Answering task and 86.2% on LongMemEval, outperforming state-of-the-art session-aware approaches and demonstrating that structured property graphs enable more temporally coherent long-term conversational reasoning.
Does conversational memory need LLM-extracted facts, or is selecting the right raw turns enough? Published results disagree. Extraction-based systems report gains from distilled facts. Recent studies find raw history with good ranking does as well, but disagree about whether ranking matters. We ran a pre-registered study on held-out LoCoMo conversations and LongMemEval. At a tight budget on LoCoMo, raw turns selected by a single call to Jev, a typed decision model, are non-inferior to an LLM-extraction memory (one-sided 95% bound -3.0 points against a -5-point margin). Blind human grading narrows the margin but does not change the result. Raw turns cost 3,061 times less to write, and the result holds with a second answer model. Within this study, reranking's gain shrinks as the budget grows. It adds 17.4 points on LoCoMo and 9.1 on LongMemEval when three of 30 candidates are kept. At generous budgets it adds 1.5 and 1.1, and extraction systems are more accurate. This suggests why published results disagree. At matched context, Jev selects as accurately as an LLM reranker (non-inferiority bound -2.0) at a third of the latency, and more accurately than a multi-call graph traversal. Reranking lowers correct abstention. Plans, code and graded answers are released.
Large language model assistants are increasingly expected to retain and reason over information accumulated across many sessions. We introduce EngramaBench, a benchmark for long-term conversational memory built around five personas, one hundred multi-session conversations, and one hundred fifty queries spanning factual recall, cross-space integration, temporal reasoning, adversarial abstention, and emergent synthesis. We evaluate Engrama, a graph-structured memory system, against GPT-4o full-context prompting and Mem0, an open-source vector-retrieval memory system. All three use the same answering model (GPT-4o), isolating the effect of memory architecture. GPT-4o full-context achieves the highest composite score (0.6186), while Engrama scores 0.5367 globally but is the only system to score higher than full-context prompting on cross-space reasoning (0.6532 vs. 0.6291, n=30). Mem0 is cheapest but substantially weaker (0.4809). Ablations reveal that the components driving Engrama's cross-space advantage trade off against global composite score, exposing a systems-level tension between structured memory specialization and aggregate optimization.