Reconstructing the Right Episode: Evaluating Interleaved Conversational Memory Beyond Long Context

Zhexi Feng; Ruiyi Zhang; Yongbo Yang; Pengtao Xie

cs.CLAug 26, 2026

Reconstructing the Right Episode: Evaluating Interleaved Conversational Memory Beyond Long Context

Authors: Zhexi Feng, Ruiyi Zhang, Yongbo Yang, Pengtao Xie

Organizations: Department of Electrical and Computer Engineering University of California San Diego

Abstract

Conversations with chat assistants increasingly span many topics in a single long-running thread, challenging memory systems. Existing long-context and memory benchmarks often expose session or topic boundaries, or probe direct personal-memory questions. These settings understate a harder assistant-memory regime: a flat mixed-topic thread where the system must infer which earlier episode makes a later task decision valid. We introduce SCALE-QA, a constraint-grounded task QA benchmark for flat unsegmented threads targeting episode integrity failure. The dataset contains 3,000 audited questions across 10 domains, uses deterministic four-way multiple-choice grading, and includes a deterministic runtime builder; experiments use all 3,000 questions through 128k and a stratified 400-question diagnostic at 1M. SCALE-QA questions are ordinary task-oriented requests whose correct answer depends on causally related evidence introduced earlier in the conversation. We also propose Temporal-Semantic Interleaved Memory Reconstruction (TSIM), which segments the turn stream into coherent episodes and indexes them through a hierarchical multi-view memory stack with deterministic episode-level summary and cluster-routing views. Experiments show that SCALE-QA challenges strong RAG baselines and long-context LLMs alike; across three open-source and proprietary LLM backends, TSIM achieves the highest accuracy in every backend setting, gaining 5.6-17.6 accuracy points over the strongest corresponding baseline.

Figures & tables

Failure mode	Failure pattern	Bottleneck	Recovery unit
Retrieval miss	evidence not retrieved	visibility	passage
Lost-in-the-middle	evidence underused in prompt	position	token span
State tracking failure	updates not maintained	update order	state value
Episode integrity failure	evidence visible but fragmented	episode integrity	operative episode

Table 1: Episode integrity failure compared with related long-context and memory failures. The categories are non-exclusive, but each foregrounds a different bottleneck and recovery unit. Representative prior anchors include dense retrieval for retrieval miss ( Karpukhin et al., 2020 ) , lost-in-the-middle behavior ( Liu et al., 2024 ) , and state tracking in long conversations ( Laban et al., 2026 ) .

Figure 1: Representative SCALE-QA example. A later advisor question is answerable only by reconstructing an earlier resource-policy episode. Standard RAG and partial-recovery systems retrieve incomplete fragments and select invalid models (BERT-Large, DistilBERT), whereas TSIM reconstructs the operative episode and selects the constraint-consistent answer (FastText). The lower panel quotes LongMemEval question/evidence text ( Wu et al., 2025 , Figure 1) as a direct-evidence contrast.

Figure 2: Overview of the SCALE-QA construction and runtime-packaging pipeline. LLM-generated scenarios are realized as chat-plus-MCQ records, then normalized, machine-filtered, human-reviewed, and stored in a validated QA pool. The runtime builder mixes accepted records with reproducible chat-history noise and packs them into length-controlled packages with aligned question and answer lists.

Figure 3: Episode-centered multi-view memory in TSIM . M1 segments the turn stream into reconstructed episodes. M2 indexes each episode through raw, summary, and cluster views. M3 converts hits from all views into episode-level scores and returns top-ranked episodes, not isolated turns, to the answer model.

Backend	Retriever	Acc	CL Hit	Rat. Sim.	CtxTok	Lat.
Gemma2:9b	TSIM	69.6	70.7	0.472	1060.8	4.20
	Standard RAG	24.4	7.7	0.296	929.5	3.35
	Hybrid-RRF Chunk RAG	31.1	11.2	0.254	1193.6	4.00
	RAPTOR	40.6	35.3	0.403	790.5	20.67
	MemGPT	60.1	62.6	0.413	2301.9	3.88
	HippoRAG	25.2	12.5	0.380	748.9	6.51

Table 2: Cross-backend main comparison on the 3,000-question SCALE-QA dataset at 128k. Accuracy is the primary metric; CL Hit, rationale similarity, context tokens, and latency are supporting diagnostics. CL Hit is backend-conditioned closed-loop evidence hit and should be compared within backend blocks. Accuracy, CL Hit, rationale similarity, and context size use full-dataset runs; API latency uses serial Quick100 controls rather than the parallel scheduler. The daggered GPT-4o-mini row is a tuned non-episodic retrieval control (BM25 + BGE dense/rerank + HyDE + RRF + parent-window + MMR; Appendix C ) without TSIM episode segmentation or multi-granularity memory; its GPU-assisted latency is a cost diagnostic, not a hardware-normalized leaderboard value. Appendix Table 19 breaks out Accuracy by domain.

Figure 4: Context scaling on all 3,000 SCALE-QA questions under GPT-4o-mini. The shared

0

k point is an evidence-only prompt; 16k–128k points use length-controlled mixed-session packages, with Full Context receiving the constructed context directly. The

x

-axis is log2-spaced; 128k callouts report measured packed prompt/context estimates from Appendix F .

Figure 5: Context-scaling stress test on a stratified 400-question SCALE-QA subset. The log2-spaced

x

-axis shows added mixed-session context;

0

k is evidence-only. Plot A uses DeepSeek R1; Plot B uses Gemini 2.5 Flash at 1M. Standard RAG is a lower-anchor control; latency uses serial averages.

Stage	Variant	Acc	CL	Rat.	Lat.	Tok.
L0	Std. RAG top- $5$	26.2	5.6	0.246	1.61	968.4
L1	Fixed-token direct	43.4	35.0	0.416	3.52	1269.2
L2	Semantic-drift direct	55.5	52.2	0.430	3.62	1179.8
L3	Full TSIM stack	74.2	74.3	0.485	3.56	1044.4

Table 3: Main-module ablation under GPT-4o-mini. Accuracy is the primary metric; CL/Rat./Tok. denote supporting CL Hit, rationale similarity, and context-token diagnostics; latency is within-table only.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Axis	LongMemEval	SCALE-QA
History format	Timestamped chat histories with session structure	Flat mixed-topic threads with no boundary metadata
Question type	Flexible personal-memory QA	Constraint-grounded task QA
Domain ontology	Personal-life ontology (health, hobbies, work-life, etc.)	Ten task-oriented operational domains
Target failure mode	Long-term memory over sessions and updates	Episode integrity failure in unsegmented threads
Evaluation protocol	Judged open-ended responses	Deterministic four-way MCQ with exact evidence traces

Appendix

Table 4: Axis-by-axis distinction between LongMemEval and SCALE-QA . SCALE-QA builds on scalable long-memory evaluation but isolates boundary-free episode reconstruction in flat task-oriented threads.

Domain	QA	Typical evidence forms	Common retrieval stress cues
CS-Software	300	code/version/deployment rules	stale defaults; tool exceptions
Network/Hardware	300	port, routing, hardware policies	constraint traps; local compliance
Finance	300	credit, reimbursement, risk rules	overwritten eligibility; denials
Legal	300	contracts, filings, exceptions	exception resolution; operative clauses
Biomed	300	protocols, safety notes, exclusions	dormant safety constraints
Engineering	300	materials, tests, device specs	incompatibilities; hidden failures

Appendix

Table 5: Supplementary SCALE-QA domain overview. Every domain contributes exactly 300 questions; correct labels are globally balanced A/B/C/D = 750/750/750/750. Stress cues are representative rather than mutually exclusive, since many examples combine stale state, long-range bridges, and local exception traps.

Release property	Value
Total QA records	3,000
Public split labels	2 $\times$ 1,500
Topics	10 $\times$ 300
Split-topic cells	20 $\times$ 150
Correct labels	A/B/C/D = 750/750/750/750
Audited evidence snippets	4,346

Appendix

Table 6: SCALE-QA dataset audit. Counts are taken from the validation report and expected-document audit.

Split	QA	Dom.	Evid.	Zero	Oracle
Full dataset	3,000	$10{\times}300$	3,000/3,000	7.63/18.10	100.00/100.00
Codex	1,500	$10{\times}150$	1,500/1,500	7.40/19.27	100.00/100.00
Claude	1,500	$10{\times}150$	1,500/1,500	7.87/16.93	100.00/100.00

Appendix

Table 7: Supplementary source-level calibration. Zero columns report gemma2:9b /Gemini 2.5 Flash zero-shot solvability without supporting chat evidence; oracle columns give the model the relevant context.

Metric	Mean	Median	Std.	Direction
Naturalness	3.80	3.67	0.25	higher better
Answerability	4.91	5.00	0.17	higher better
Ambiguity risk	1.45	1.33	0.32	lower better
Constraint plausibility	3.99	4.00	0.29	higher better

Appendix

Table 8: Overall blind human realism audit on 300 stratified examples with three anonymous annotators.

Ann.	Gold Acc.	Max ans.	Natural	Answerable	Ambig.	Plausible
Ann. 1	99.0	25.7	4.17	4.90	1.19	3.98
Ann. 2	91.0	30.3	4.04	4.87	1.92	4.08
Ann. 3	96.3	25.3	3.20	4.95	1.25	3.90

Appendix

Table 9: Annotator-level descriptive checks for the three-annotator human audit. Gold Acc. is agreement with the benchmark answer; Max ans. is the largest selected-answer share, used as a response-balance check.

Pair	Ans. agr.	Ans. $\kappa$	Primary agr.	State	Long	Trap	Cons. gold
Ann. 1–2	91.3	0.884	56.7	43.3	83.0	84.7	272/274
Ann. 1–3	96.0	0.947	16.3	45.3	83.0	100.0	287/288
Ann. 2–3	89.0	0.853	32.7	57.3	72.0	84.7	265/267

Appendix

Table 10: Pairwise agreement in the final human audit. Answer-choice agreement is high, including chance-corrected pairwise Cohen’s

\kappa

; primary-stress agreement is lower because many examples contain overlapping stress cues. Cons. gold reports gold agreement among pairwise majority-resolved cases.

View	Label	Count	Rate
Multi-label cue	State overwrite	196	65.3
Multi-label cue	Long-range bridge	291	97.0
Multi-label cue	Constraint trap	300	100.0
Multi-label cue	Any multi-label	298	99.3
Multi-label cue	All three cues	189	63.0
Primary cue	Constraint trap	158	52.7

Appendix

Table 11: Human stress-label distribution on the same 300 examples. Stress cues are intentionally multi-label; primary labels summarize the dominant cue only. Needs adjudication indicates cases where annotators did not form a stable dominant-stress label, not invalid examples.

Topic	Natural	Answerable	Ambig.	Plausible
Biomed	3.66	4.86	1.49	4.00
Biz-Ops	3.79	4.92	1.31	4.13
CS-Software	3.74	4.94	1.34	4.09
Daily-Life	4.09	4.91	1.38	4.04
Engineering	3.72	4.90	1.42	4.13
Finance	3.79	4.88	1.41	4.06

Appendix

Table 12: Per-topic realism means in the 300-example human audit. Ambig. denotes ambiguity risk, where lower is better.

Protocol item	Definition
Token accounting	Benchmark-side estimate $\mathrm{tokens}\approx 1.3\times$ whitespace word count, used for deterministic packing and constructed-length reporting, not for asserting exact provider-native token parity.
Length scaling	No built-in length cap; practical limits are distractor-pool size and computational budget. This paper reports settings through 1M to match evaluated native-context backends.
Full-corpus regime	Used when selected truth tokens fit inside the target context length; the full selected truth background is retained and noise fills remaining space.
Length-batched regime	Used when selected truth tokens exceed the target length; records are deterministically partitioned into budget-controlled batches.
Packing rule	Deterministic capacity-constrained best fit: records are ordered by descending truth-token count and then by question name.
Truth cap	In length-batched mode, the default truth-cap ratio is $0.82$ , leaving room for noise and prompt/interface overhead.

Appendix

Table 13: Length-controlled evaluation protocol used by the deterministic runtime-package builder.

System	Retrieval unit	Context assembly	Purpose in comparison
Standard RAG top-5	Individual retrieved chunks/turns	Top five retrieved units are passed to the answer backend.	Tests whether short semantic retrieval alone can recover the decisive evidence.
Hybrid-RRF Chunk RAG	Dense chunks + BM25 sparse hits + neighboring turns	Dense and sparse hits are fused with reciprocal-rank fusion, expanded with local neighbors, and packed under a matched context cap.	Tests whether lightweight hybrid chunk retrieval closes the episode-reconstruction gap.
Tuned Hybrid-Rerank Chunk RAG	Dense/sparse candidates + HyDE + cross-encoder reranking	Adds query expansion, RRF fusion, BGE reranking, parent-window expansion, and MMR packing under the same GPT-4o-mini 128k task.	Tests whether a strongly tuned but non-episodic chunk pipeline can close the gap, and exposes its reranking cost.
RAPTOR strict no-block	Hierarchical summaries built without gold blocks	Retrieved hierarchy outputs are mapped into the same no-block evaluation regime.	Tests whether hierarchical abstraction alone solves interleaved conversational memory.
MemGPT paper-default	Explicit memory-management substrate	Retrieved memory material is inserted under the same answer and writeback protocol.	Tests whether higher recall from a memory-style substrate converts into final accuracy.
HippoRAG paper-default	Graph-style retrieval substrate	Retrieved graph/memory evidence is evaluated with the same question set and scoring protocol.	Tests transfer of graph-centric memory retrieval to flat-thread writeback evaluation.

Appendix

Table 14: High-level implementation distinctions for the main retrieval and memory systems.

Streaming semantic-drift segmentation in Official TSIM
1. Flatten the mixed conversation into a turn stream and encode each turn as a normalized dense vector.
2. Maintain the current segment start $s$ and token count $L$ .
3. For incoming turn $i$ , average only the recent turns still inside the current segment to form local center $c_{i}$ .
4. Score the newest turn by cosine similarity to $c_{i}$ , plus a small bonus for a model $\rightarrow$ user transition.
5. If $L\geq\tau_{\min}$ and the score falls below similarity threshold $\theta_{s}$ , close the segment before turn $i$ .
6. If $L\geq\tau_{\max}$ , force a boundary even if semantic drift is weak.

Appendix

Table 15: Compact pseudocode view of the semantic-drift segmenter used by Official TSIM . The final system uses a simple local-window cosine rule rather than a heavier offline clustering procedure for boundary decisions.

Stage	Data	Purpose
Search	300 QA	Fast screening of candidate retrieval configurations.
Confirm	600 QA	Check stability across 64k, 128k, 256k, and 512k targets.
Answer validation	200 QA	Confirm that retrieval gains transfer to answer accuracy.
Full retrieval check	3,000 QA	Verify evidence coverage and context stability at 128k.

Appendix

Table 16: Frozen-configuration selection protocol for the SCALE-QA experiments.

Candidate	Representative variation	Search Hit	Confirm Hit	Answer Hit	Answer Acc	Answer Ctx
Official calibrated	$\theta=.70,k_{f}=5,k_{r}/k_{s}=28/20,w_{l}=.75$	86.33	79.33	87.50	78.50	1011.6
High-threshold compact	$\theta=.74,k_{r}=44,k_{s}=16,w_{l}=.85$	85.67	77.62	86.00	79.00	757.7
Lean memory stack	$k_{f}=4,k_{r}/k_{s}=36/12,w_{l}=.65$	84.00	76.58	87.50	77.50	818.2
Lean L2-exp $=1$	Same as lean, one L2-to-episode expansion	83.67	76.85	87.50	77.50	818.1

Appendix

Table 17: Representative neighborhood stability from the development sweep. Search300 and confirm600 report CL Hit; answer200 reports CL Hit, Accuracy, and average retrieved context tokens. The answer-validation prompt constrains outputs to valid answer choices, so the reported answer accuracy follows the paper’s main Accuracy convention.

Figure 6: Single-variable M1 threshold sensitivity on the frozen confirmation split. Points report all-evidence Recall@5 and bars show Wilson 95% confidence intervals; the question IDs, runtime construction, and all non-threshold settings are fixed.

Component	Frozen reported constants
Segmenter	min_tokens =120, max_tokens =320, recent_window =4, $\theta_{s}=0.70$ , speaker bonus $b=0.03$ , minimum two-message buffer.
Retrieval breadth	$k_{r}=28$ , $k_{s}=20$ , $k_{l}=2$ , $k_{\mathrm{final}}=5$ .
Ranking weights	$w_{r}=1.15$ , $w_{s}=1.20$ , $w_{l}=0.75$ , $\lambda_{\mathrm{sem}}=0.55$ , $\lambda_{l}=0.70$ .
Expansion	Cluster expansion $k=2$ , L2-to-episode expansion $k=2$ , cluster threshold $0.42$ , soft margin $0.08$ , temporal expansion hops $0$ .
Prompt assembly	Reconstructed episodes capped at eight messages with two-message overlap; L2 cluster summaries route and boost candidates but are not inserted directly into the prompt.

Appendix

Table 18: Exact TSIM constants used in the reported SCALE-QA runs. These values are moved out of the main text to avoid visually overloading the method narrative with implementation constants.

Gemma2:9b
Domain	TSIM	Std. RAG	Hybrid	RAPTOR	MemGPT	HippoRAG
CS	79.0	29.3	33.7	48.0	66.7	25.3
Net	77.3	26.7	33.7	49.3	67.0	27.0
Fin	54.3	22.7	29.3	32.7	46.7	23.3
Legal	60.0	22.3	28.0	36.3	56.3	26.7
Bio	74.7	25.3	36.7	39.7	66.7	21.3

Appendix

Table 19: Supplementary per-domain Accuracy for the 128k cross-backend comparison. Hybrid denotes Hybrid-RRF Chunk RAG. Domain abbreviations: CS = software, Net = network/hardware, Fin = finance, Bio = biomedicine, Eng = engineering, Biz = business operations.

Stress view	$N$	Std. RAG	Hybrid-RRF	RAPTOR	MemGPT	HippoRAG	TSIM
All audited	300	26.7/5.7	29.0/9.3	41.7/29.7	52.7/58.3	29.3/8.7	73.7/78.7
State overwrite	196	24.0/3.6	27.0/8.2	38.3/26.5	49.5/56.6	28.6/7.1	70.9/76.0
Long-range bridge	291	26.8/5.8	28.9/8.9	42.3/29.9	52.9/58.1	29.6/8.6	73.9/78.4
Constraint trap	300	26.7/5.7	29.0/9.3	41.7/29.7	52.7/58.3	29.3/8.7	73.7/78.7

Appendix

Table 20: Full stress-type method audit on the 300-example three-annotator human-audited subset under GPT-4o-mini at 128k. Cells report Accuracy / CL Hit. Stress views are multi-label, so counts are not expected to sum to 300. Full Context is omitted because only 11 audited items have matched rows in this diagnostic slice.

Retrieved unit	All recall@5	Co-contain
TSIM episode	0.810	0.890
Fixed 128-token window	0.719	0.675
Fixed 256-token window	0.647	0.826
Fixed 320-token window	0.577	0.831
Standard RAG chunk	0.456	0.004

Appendix

Table 21: Exact-evidence reconstruction diagnostics on the full 3,000-question runtime. All recall@5 uses all questions; co-containment uses the multi-snippet subset. Larger fixed windows improve co-containment but reduce recall, whereas TSIM improves both.

Method	Embedder	All recall@5
Standard RAG	BGE-large	0.456
Standard RAG	MiniLM	0.408
TSIM	BGE-large	0.810
TSIM	MiniLM	0.649

Appendix

Table 22: Retrieval-only embedding sensitivity on the same 3,000 questions. MiniLM denotes all-MiniLM-L6-v2 ; BGE-large denotes BAAI/bge-large-en-v1.5 .

Method	Boundary information	Judged Acc. (%)	All-evidence recall (%)	Avg. answer- context tokens
TSIM	None; episodes reconstructed	71.0	84.04	4,723.5
BM25 fixed-chunk top-5	None	61.2	65.74	4,616.3
BGE turn top-5	None	56.6	70.64	340.8
BM25 session top-5	Official session boundaries	74.4	76.60	13,086.5

Appendix

Table 23: Targeted transductive diagnostic on LongMemEval-S cleaned V1 (500 questions). Judged accuracy follows the official free-form response protocol; all-evidence recall is computed on the 470 non-abstention questions with annotated evidence, and average context denotes answer-context tokens. The session row is boundary-assisted and not information-condition matched.

System	Views	Vectors	Vector MB	Text amp.	Ingest ms/turn	Ret. p50 ms	Ret. p95 ms	Summary API
Standard RAG	1	5,554	21.7	1.056 $\times$	4.178	21.478	36.791	0
TSIM	3	6,281	24.5	2.087 $\times$	24.986	141.093	159.978	0

Appendix

Table 24: Measured retrieval-side system accounting. Vector MB is a lower-bound dense-vector estimate; text amplification counts indexed textual views relative to the source history. Latencies characterize this implementation and hardware, not a hardware-normalized leaderboard.

Failure type	Example	Diagnostic observation
Episode-incomplete retrieval	codex:Network-Hardware-075	Standard RAG retrieves related text but omits the neighboring license clause that makes the local decision operative.
Fixed-window low SNR	claude-code:Network-Hardware-032	A fixed window contains the evidence within 1,755 tokens but at lower evidence density than the 1,127-token reconstructed episode.
Answer-side override misuse	claude-code:Biomed-003	TSIM retrieves the decisive episode, yet the answer model follows a plausible public default instead of the local override.
Routing/packing miss	codex:Biomed-124	Two gold units are required; final selected context covers only one, exposing a residual routing and packing failure.

Appendix

Table 25: Compact qualitative cases illustrating retrieval fragmentation, low signal-to-noise windows, answer-side evidence misuse, and residual TSIM routing errors.

Budget	Hit	Acc	HitRank	Lat.	CtxTok
16k	100.00	62.5	335.4	4.32	29,938
32k	100.00	57.8	669.2	4.57	60,145
64k	100.00	41.7	836.3	10.03	86,315
128k	100.00	29.8	711.7	14.61	88,425

Appendix

Table 26: Full Context sanity audit for Figure 4 . Hit is evidence containment in the constructed prompt. The collapse is therefore not explained by missing gold evidence alone.

System	Prompt	Rows	Acc.	Prompt tok.	Lat.
Full Context	Vanilla	100	49.0	98,339	5.52
Full Context	Dev-selected	100	49.0	98,298	2.80
TSIM	–	100	77.0	1,389	2.40

Appendix

Table 27: Prompt-robust Full Context diagnostic under GPT-4o-mini in an independent no-writeback setting. The dev-selected prompt is evidence-first, chosen on a disjoint 20-example development split and frozen before evaluation. Rows are the same stratified 100-example evaluation subset with balanced domains, answer labels, and logical-context quartiles. Prompt tokens and latency are provider-reported averages; TSIM uses its standard memory pipeline rather than a Full Context prompt template. Numbers are not directly comparable to the main stateful results in Table 2 because this diagnostic uses a 100-example subset under a no-writeback evaluation regime; it isolates prompt-template effects on Full Context rather than re-evaluating system rankings.

Backend	Method	Acc	Hit-C	Hit-W	Miss-C	Miss-W
Gemma2:9b	Standard RAG	24.4	5.2	2.1	10.6	47.7
	Hybrid-RRF Chunk RAG	31.1	8.4	2.1	9.5	27.2
	RAPTOR	40.6	25.2	10.0	14.6	47.0
	MemGPT	60.1	48.6	14.0	11.4	25.9
	HippoRAG	25.2	8.7	3.7	15.6	68.4
	TSIM	69.6	58.2	12.4	11.3	17.7

Appendix

Table 28: Backend-output audit for the 128k main comparison. Hit-C/W and Miss-C/W split closed-loop evidence hit by correct and wrong answers, all in percentages.

Backend	Baseline	Base	TSIM	$\Delta$	95% CI
Gemma2:9b	Standard RAG	24.4	69.6	+45.21	[43.48, 47.03]
	Hybrid-RRF Chunk RAG	31.1	69.6	+38.52	[36.72, 40.24]
	RAPTOR	40.6	69.6	+28.98	[27.13, 30.76]
	MemGPT	60.1	69.6	+9.53	[7.90, 11.22]
	HippoRAG	25.2	69.6	+44.42	[42.41, 46.36]
Gemini 2.5 Flash	Standard RAG	29.8	80.2	+50.33	[48.98, 51.63]

Appendix

Table 29: Paired bootstrap audit for the final 3,000-question main-table runs. The closest final API-backed comparison is Gemini 2.5 Flash with MemGPT ; the improvement remains significant under paired resampling.

Method	Domain Acc Std.	Batch Acc Range	Ctx Mean	Ctx Median	Ctx P95	Ctx Max
Standard RAG	3.73	20.4–30.0	929.5	937	1076	1259
Hybrid-RRF Chunk RAG	3.65	25.0–43.2	1193.6	1194	1200	1200
RAPTOR	7.62	37.4–45.5	790.5	780	996	1219
MemGPT	7.70	52.6–72.7	2301.9	2317	2608	3359
HippoRAG	3.97	20.8–29.7	748.9	757	891	1043
TSIM	8.08	64.5–76.9	1060.8	1037	1408	1950

Appendix

Table 30: Variance and retrieved-context token audit for the final Gemma2:9b main-table run. Context-token columns summarize selected answer context, not benchmark-side constructed length.

Explore similar work

Sep 7, 2026cs.CL

Where to Look and What to Use: Retrieve-Localize-Generate for Long-Term Conversational Memory Question Answering

Retrieval-augmented generation (RAG) enables large language models (LLMs) to answer questions by accessing external knowledge and has been widely adopted for long-term conversational memory question answering. However, existing methods suffer from two key challenges: (1) fragmented evidence scattered across temporally distant sessions, and (2) noisy content within retrieved sessions that triggers the lost-in-the-middle effect. To address these challenges, we propose MemLoc, a unified Retrieve-Localize-Generate framework for long-term conversational memory QA. For retrieval, MemLoc decomposes each session into multi-granularity memory units and performs query routing via an inner-memory graph with entropy-based granularity selection. It further models cross-session semantic and temporal dependencies through a cross-memory graph, enabling coarse-to-fine retrieval of top-K relevant memory candidates. For localization, we introduce a reasoning-based evidence locator trained with Self-reflective Hint Policy Optimization (SHPO), which performs progressive refinement by extracting query-relevant fragments within memory units to suppress noise and reranking across candidates to remove redundancy, producing a compact evidence set with lightweight location IDs. For generation, these IDs act as precise grounding signals that guide the LLM to the correct memory positions, mitigating the lost-in-the-middle effect while preserving original contextual integrity. Extensive experiments on four benchmarks demonstrate that MemLoc achieves state-of-the-art retrieval accuracy and response quality while maintaining efficiency. Our code is available at: https://github.com/Nikol-coder/MemLoc.

Yifan Wang, Xinkui Lin, Yongxiu Xu +9

University of Electronic Science and Technology of China, Chengdu, China 4State Key Laboratory of Internet Architecture, Tsinghua University, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 3School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · 5Independent Researcher

Conversational Memory Peak Memory Question

Ask PDF

Apr 23, 2026cs.CL

EngramaBench: Evaluating Long-Term Conversational Memory with Structured Graph Retrieval

Large language model assistants are increasingly expected to retain and reason over information accumulated across many sessions. We introduce EngramaBench, a benchmark for long-term conversational memory built around five personas, one hundred multi-session conversations, and one hundred fifty queries spanning factual recall, cross-space integration, temporal reasoning, adversarial abstention, and emergent synthesis. We evaluate Engrama, a graph-structured memory system, against GPT-4o full-context prompting and Mem0, an open-source vector-retrieval memory system. All three use the same answering model (GPT-4o), isolating the effect of memory architecture. GPT-4o full-context achieves the highest composite score (0.6186), while Engrama scores 0.5367 globally but is the only system to score higher than full-context prompting on cross-space reasoning (0.6532 vs. 0.6291, n=30). Mem0 is cheapest but substantially weaker (0.4809). Ablations reveal that the components driving Engrama's cross-space advantage trade off against global composite score, exposing a systems-level tension between structured memory specialization and aggregate optimization.

Julian Acuna

Conversational Memory Question-Answering Benchmarks Graph-Based Retrieval Vector Database

Ask PDF

Jul 23, 2026cs.CL

RUMBA: Russian User Memory Benchmark

The ability to handle long-term memory in LLMs is becoming increasingly critical, yet existing benchmarks remain English-centric and rely on aggregate retrieval metrics, failing to capture interactions between long-range context, temporal information, and reasoning. To address this, we introduce RUMBA (Russian User Memory BenchmArk) - a new benchmark for long-term conversational memory that provides a fine-grained taxonomy of memory-centric question types and a unified methodology accounting for semantic type, session scope, temporal reasoning, and the explicitness of temporal expressions. RUMBA consists of timestamped user-assistant dialogues with QA pairs requiring retrieval, combination, and reasoning across sessions. While designed for Russian, we also provide an aligned English subset under the same methodology. We evaluate contemporary memory systems and long-context models, and show how RUMBA serves as a diagnostic tool to analyze model behavior across benchmark slices and identify strengths and failure modes of different memory mechanisms.

Elizaveta Shevtsova, Inna Glebkina, Mark Baushenko +2

1DAIMLD

Conversational Memory Question-Answering Benchmarks Large Language Model Benchmarks Russian

Ask PDF