SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval

Jesper Derehag; Carlos Calva; Timmy Ghiurau

cs.LGMar 16, 2026

SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval

Authors: Jesper Derehag, Carlos Calva, Timmy Ghiurau

Organizations: Midbrain.

Abstract

Recent conversational memory systems invest heavily in LLM-based structuring at ingestion time and learned retrieval policies at query time. We show that neither is necessary. SmartSearch retrieves from raw, unstructured conversation history using a fully deterministic pipeline: NER-weighted substring matching for recall, rule-based entity discovery for multi-hop expansion, and a CrossEncoder+ColBERT rank fusion stage -- the only learned component -- running on CPU in ~650ms. Oracle analysis on two benchmarks identifies a compilation bottleneck: retrieval recall reaches 98.6%, but without intelligent ranking only 22.5% of gold evidence survives truncation to the token budget. With score-adaptive truncation and no per-dataset tuning, SmartSearch achieves 93.5% on LoCoMo and 88.4% on LongMemEval-S, exceeding all known memory systems under the same evaluation protocol on both benchmarks while using 8.5x fewer tokens than full-context baselines.

Figures & tables

Panel of Figure 1 — Figure 1: SmartSearch pipeline. Stages 1–2 are fully deterministic; Stage 3 (Rank) is the only learned component, running two models in parallel on CPU. Dashed arrow: optional multi-hop entity expansion (3% of queries).

Hop distribution (oracle, LoCoMo only)
Property	LoCoMo	LME-S
Single-hop	97.0%	—
Two-hop	2.8%	—
Three-hop	0.2%	—
Tool selection (oracle, LoCoMo only)
Resolved by grep	98.9%	—

Table 1: Oracle trace analysis (LoCoMo, 1,317 traces) and ranking effect on both benchmarks. Mean gold rank: average position of the first gold passage among grep candidates.

Rank
Stage	Indexed	Index-free
Retrieve	I n	IF n
No reranker	R0
…
CrossEncoder + RRF	R n
Truncate

Table 2: Ablation structure and naming scheme. Configurations are composites of retrieval, rank, and budget choices (e.g., I0-R5-B0 ). Section 4 tables vary one dimension and hold the others constant.

		LoCoMo		LME-S
ID	Config	Acc.	$\Delta$	Acc.	$\Delta$
I0-R5-B1	Indexed	91.9	—	81.2	—
IF0-R4-B1	Index-free	91.0	—	78.4	—
IF1-R4-B1	+ PRF	90.9	$-$ 0.1	84.4	+6.0
IF2-R4-B1	+ entity discovery	91.0	+0.1	79.8	+1.4
IF3-R4-B1	+ both	90.7	$-$ 0.3	87.6	+9.2

Table 3: Query expansion ablation.

\Delta

relative to the index-free baseline. Expansion is neutral on short conversations but strongly positive on long ones.

ID	Configuration	Acc.	Tok.	$\Delta$
I0-R0-B1	No reranker	76.8	1,547	—
I0-R1-B1	+ MiniLM-L-12 (33M)	84.7	1,547	+7.9
I0-R1-B1 †	+ Inference prompt †	86.4	1,691	+1.7
I0-R2-B1	bge-base (278M)	88.6	3,203	+2.2
I0-R3-B1	bge-large (560M)	90.7	3,192	+2.1
I0-R4a-B1	+ ColBERT RRF	91.2	3,196	+0.5

Table 4: Ranking pipeline ablation on LoCoMo. Each row changes one factor. † Answer-prompt change, not a reranker change. Full 27-configuration table in Appendix A .

Fixed budget
		LoCoMo		LME-S
Budget	$\alpha$	Tokens	Recall	Tokens	Recall
1,000	—	1,458	.927	1,133	.850
2,000	—	2,967 (+104%)	.958 (+3.1%)	2,443 (+116%)	.928 (+7.8%)
3,000	—	4,495 (+208%)	.972 (+4.5%)	3,740 (+230%)	.959 (+10.9%)
4,000	—	6,038 (+314%)	.979 (+5.2%)	5,043 (+345%)	.973 (+12.3%)

Table 5: Truncation ablation (I0-R5 pipeline, simulated from offline score matrix). Top: fixed word budget; recall plateaus on LoCoMo but keeps improving on LME-S. Bottom: score-adaptive truncation with fixed budget ceiling 4,000 and top-

K{=}60

, sweeping

\alpha

; the threshold adaptively allocates fewer tokens on short corpora and more on long ones. All deltas relative to budget 1K /

\alpha{=}0

baseline.

EverMemOS protocol (gpt-4.1-mini answer+judge, 7-step CoT)
System	Overall	1-hop	M-hop	Temp.	Open	Tokens
SmartSearch (indexed)	93.5	91.1	91.3	80.2	96.7	3,141
EverMemOS a	92.3	96.1	91.1	89.7	70.8	2,298
Full-context a†	91.2	94.9	90.4	88.0	71.9	20,281
Zep a	85.2	90.8	81.9	77.3	75.0	1,411
MemOS a	80.8	85.4	79.4	75.1	64.6	2,498

Table 6: LoCoMo accuracy (%). Results under two incompatible evaluation protocols should not be compared across groups. a Hu et al. (2026) . b Xia et al. (2026) . † Within 1.1 pp of EverMemOS despite no memory system.

System	Ovr.	KU	MS	SA	SP	SU	TR	Tok.
SmartSearch (index-free)	88.4	93.6	84.2	85.7	96.7	100.0	82.7	3,392
SmartSearch (indexed)	87.6	92.3	79.7	87.5	96.7	95.7	86.5	3,393
Memora b	87.4	97.4	78.2	78.6	83.3	98.6	89.5	${\sim}$ 2,900
EverMemOS a	83.0	89.7	73.7	85.7	93.3	97.1	77.4	2,800
MemOS a	77.8	74.3	70.7	67.9	96.7	95.7	77.4	1,400
Nemori b	74.6	79.5	55.6	92.9	86.7	90.0	72.2	${\sim}$ 4,300

Table 7: LongMemEval-S accuracy (%). All systems use gpt-4.1-mini as answer LLM and gpt-4o-mini as judge. a Hu et al. (2026) . b Xia et al. (2026) .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Configuration	Overall	1-hop	M-hop	Temp.	Open	Tokens	$\Delta$
Phase 1: Search & prompt ablations
Base	MiniLM-L-12, thresh= $-$ 4, budget=2K	84.7	84.4	76.9	69.8	89.5	1,547	—
A1v1	Cumulative-score pruning (ratio=0.20)	82.4	81.2	75.7	66.7	87.2	677	$-$ 2.3
A1v2	Score-floor pruning (score $>$ 0)	81.2	79.4	74.8	66.7	85.9	688	$-$ 3.6
A2	Date-aware search	85.3	86.2	77.6	70.8	89.5	1,547	+0.5
A3	Inference reasoning prompt	86.4	87.2	78.2	71.9	90.8	1,691	+1.6

Appendix

Table 8: Complete ablation results on LoCoMo (1,540 questions, gpt-4o-mini judge).

\Delta

is change vs. baseline (84.7%). Experiments grouped by phase. Baseline: MiniLM-L-12 reranker, threshold=

-

4, 2,000-word budget.

Model	Params	Gold top-62	Recovery (%)	Median rank	Latency
ms-marco-MiniLM-L-12-v2	33M	1,949	83.1	3	${\sim}$ 130 ms
mxbai-rerank-xsmall-v1	71M	1,998	85.2	3	${\sim}$ 168 ms
mxbai-rerank-base-v1	184M	2,087	89.0	2	${\sim}$ 182 ms
bge-reranker-base	278M	1,852	79.0	5	${\sim}$ 340 ms
ColBERT v2	—	1,979	84.4	4	${\sim}$ 500 ms
bge-reranker-large	560M	2,050	87.4	3	${\sim}$ 908 ms

Appendix

Table 9: Cross-encoder gold passage recovery at top-62 budget cutoff. 2,345 gold passages across 1,540 questions. Latency measured on CPU (single-threaded); in the full system, CrossEncoder and ColBERT run in parallel (

\text{wall-clock}=\max(\text{CE},\text{ColBERT})

Config	Proxy $\Delta$	Live $\Delta$	Gap
I0-R4b-B1 (3-way RRF)	+1.8 pp	+0.4 pp	1.4 pp
I0-R4c-B1 (score fusion)	+2.7 pp	+0.2 pp	2.5 pp
I0-R5-B1 (model swap)	+1.2 pp	+1.2 pp	0.0 pp

Appendix

Table 10: Offline proxy predictions vs. live accuracy for three configurations on the I0-R3-B1 (bge-large, 560M) baseline. Proxy

\Delta

: predicted gain from offline gold-passage simulation. Live

\Delta

: measured accuracy change. Fusion strategies (I0-R4b-B1, I0-R4c-B1) show large proxy-live gaps; model swaps (I0-R5-B1) do not.

Conversational Memory

Conversational memory equips AI assistants to retain, update, and retrieve useful information from past interactions for coherent, personalized dialogue.

See all 71 papers

JustMem: Just-Enough Memory Access for Long-Term Conversations
Sep 17, 2026Guanhua Chen, Yanting Wang, Wenjing Zhi +1Conversational Memory Conversational QA
Beihang University
ConvMemory v2: A Recall-Preserving Top-10 Evidence Reranker for Conversational Memory Retrieval
Jun 9, 2026Taiheng PanCross-Encoder Reranking Conversational Memory
School of Computing and Information Systems University of Melbourne
EngramaBench: Evaluating Long-Term Conversational Memory with Structured Graph Retrieval
Apr 23, 2026Julian AcunaLLM Evaluation Memorization in Language Models

Sep 17, 2026J/K move · Enter open · S save

JustMem: Just-Enough Memory Access for Long-Term Conversations

Guanhua Chen, Yanting Wang, Wenjing Zhi, Lei Sha

Beihang University

Conversational Memory Conversational QA Persistent Memory for Language Models Long-Term Conversational Memory Agent Memory Management

Open paper AskPDF arXiv

Efficient long-term conversational memory requires retrieving sufficient evidence without indiscriminately expanding the context presented to the language model. This is challenging because relevant evidence may be distributed across multiple sessions, while compression may discard details needed for answering. Different queries therefore require different forms of memory access. To capture these demands, we formulate memory access along two dimensions: discovery breadth, which controls how broadly evidence is searched, and reading fidelity, which controls whether evidence is read in compact form or recovered from the original conversation. Based on this formulation, we introduce JustMem, which stores conversation history as compact atomic memories and adapts memory access along these two dimensions to each query. Specifically, LOOKUP handles local evidence, COMPOSE broadens discovery for distributed evidence, and REPLAY increases reading fidelity for fidelity-sensitive evidence. On LoCoMo and LongMemEval-S, JustMem achieves the highest mean accuracy and retrieval recall among the compared memory systems while using substantially fewer generative-model tokens for memory construction and inference.

Abstract

Conversational Memory

Explore similar work

JustMem: Just-Enough Memory Access for Long-Term Conversations

ConvMemory v2: A Recall-Preserving Top-10 Evidence Reranker for Conversational Memory Retrieval

EngramaBench: Evaluating Long-Term Conversational Memory with Structured Graph Retrieval