Beyond the Final Embedding: Retrieving Memories from Token-Level Evidence

Han Luo; Zhan Zhuang; Xu He; Jujia Zhao; Yu Li; Suqin Yuan; Dong Li; Lei Feng

cs.IROct 9, 2026

Beyond the Final Embedding: Retrieving Memories from Token-Level Evidence

Authors: Han Luo, Zhan Zhuang, Xu He, Jujia Zhao, Yu Li, Suqin Yuan, Dong Li, Lei Feng

Organizations: Southeast University · MemoraX AI · Leiden University · University of Sydney

Abstract

Memory-augmented assistants retrieve evidence from stored conversations and documents to answer queries. Most embedding-based retrievers compare single-vector representations of queries and memories, even when relevance depends on a small part of the content. Such local evidence can be matched to queries using the contextual token representations already computed by the same embedding models. Our method uses sparse token matching to improve retrieval without additional training. Each query token retrieves a small set of matching tokens across candidate memories. The matching scores are aggregated for each memory and combined with whole-memory similarity to rank memories. An optional low-rank query adapter can be trained with supervision while keeping the encoder and stored memory representations fixed. Our method improves retrieval across eight benchmarks and multiple model families without additional training. Query adaptation further increases the gains to 13.3 and 14.3 percentage points in average Recall@10 and NDCG@10 on Qwen3-Embedding-4B. These retrieval gains translate into higher question-answering accuracy on long-term memory benchmarks.

Figures & tables

Figure 1: Local evidence from existing token representations. Training-free fusion raises the relevant memory from rank 28 to 1 and yields a net gain of 523 Top-10 hits on LoCoMo.

Figure 2: Sparse token matching for memory retrieval. Query tokens retrieve matching states from the frozen memory bank; matches are aggregated by source memory and fused with whole-memory similarity for final ranking. The orange query adapter is optional, and memory states and their index remain fixed when it is trained. Memory IDs and alignments are illustrative.

(a) Recall@10 $\uparrow$
Method	LoCoMo	LongMemEval	REALTALK	ESG-Reports	QASPER	PeerQA	CovidQA	MLDR	Overall
Standalone retrieval systems
BM25	48.3	81.3	40.0	50.5	61.4	30.7	94.1	100.0	63.3
ColBERTv2	61.8	86.3	52.6	56.0	73.4	44.5	94.3	97.0	70.8
XTR	43.6	89.4	42.7	62.9	71.2	38.8	93.9	98.0	67.5
ConstBERT	63.1	85.7	55.9	53.0	72.7	43.4	91.2	96.0	70.1

Table 1: Retrieval performance (%). Overall is the eight-task mean. Bold and underline denote best and second-best scores within each group; trained methods average three seeds.

	Recall@10 $\bm{\uparrow}$			NDCG@10 $\bm{\uparrow}$
Model	Original	Ours (training-free)	Ours	Original	Ours (training-free)	Ours
Qwen3-Embedding-0.6B	65.6	74.2	74.5	50.8	61.1	61.6
Qwen3-Embedding-4B	62.5	73.8	75.7	48.0	59.9	62.3
Llama-Nemotron-Embed-1B-v2	74.8	77.3	77.6	61.4	65.1	65.4
Llama-NV-Embed-Reasoning-3B	77.9	79.6	79.5	60.9	67.4	66.8
EmbeddingGemma-300M	72.8	79.8	79.8	59.4	67.1	67.1

Table 2: Cross-backbone retrieval (%). Eight-task means; Ours averages three training seeds. Bold and underline mark the best and second-best scores within each backbone and metric.

	LoCoMo						LongMemEval
	Question type				Overall		Question type						Overall
Model	S-hop	Multi	Temp.	Open	Acc.	F1	User	Asst.	Pref.	Multi	Temp.	Upd.	Acc.	F1
Qwen3-Embedding-0.6B	76.1	57.1	42.7	53.1	64.2	31.2	85.7	94.6	56.7	45.1	50.4	82.1	64.2	12.7
+ Ours	86.2	60.6	46.7	57.3	71.5	34.1	94.3	100.0	60.0	49.6	60.2	85.9	70.6	13.1
Qwen3-Embedding-4B	76.8	55.3	38.9	52.1	63.4	31.2	91.4	96.4	60.0	50.4	56.4	78.2	67.8	13.1
+ Ours	82.8	55.7	43.3	55.2	67.9	33.0	97.1	98.2	70.0	54.9	59.4	85.9	72.6	12.4

Table 3: Downstream question answering at top-10 (%). Question-type columns report accuracy, while overall columns report accuracy and F1. Bold marks the better score per backbone–Ours pair.

	Qwen3-Embedding-0.6B		Qwen3-Embedding-4B
Method	Recall@10	NDCG@10	Recall@10	NDCG@10
w/o query adaptation	74.2	61.1	73.8	59.9
w/o local–global fusion	73.5	60.1	74.5	60.6
w/o sparse matching	72.6	59.6	72.4	60.1
Ours	74.5	61.6	75.7	62.3

Table 4: Component analysis across eight tasks (%). Ours combines sparse matching, local–global fusion, and query adaptation. Each variant changes one component. Removing fusion retains only the sparse score. w/o sparse matching uses MaxSim. Adapted rows average three source-training seeds.

Figure 3: Native retrieval across memory budgets and eight backbones. Recall@

K

and NDCG@

K

on LoCoMo and LongMemEval for

K\in\{4,8,10,16\}

. Colors identify backbones; dashed lines show Original embedding and solid lines show Ours.

Figure 4: Sensitivity to the token retrieval budget

k

. Adapted curves average three training seeds. Dotted lines mark

k=64

Figure 5: Sensitivity to the pooled-score weight

\alpha

. Dotted lines mark

\alpha=0.5

Method	Recall@10 $\uparrow$	NDCG@10 $\uparrow$	Size (MiB) $\downarrow$	Query (ms) $\downarrow$
BM25	48.28	36.68	0.23	0.76
ColBERTv2	61.82	49.42	1.54	27.27
XTR	43.57	31.61	5.78	93.79
ConstBERT	63.09	50.33	4.45	10.18
Qwen3-Embedding-0.6B	46.87	34.24	2.23	16.66
Ours (Full)	68.53	54.45	153.93	29.21

Table 5: Retrieval quality, storage, and query latency on LoCoMo.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Dimension	Whole-vector pooling	Input limit
Qwen-0.6B / 4B	1024 / 2560	Last valid token	32,768
Qwen-8B	4096	Last valid token	32,768
Llama-1B / 3B	2048 / 3072	Masked mean	8,192
Llama-8B	4096	Masked mean	32,768
EmbeddingGemma	768	Masked mean + projection	2,048
Jina-v5-text-nano	768	Last valid token	8,192

Appendix

Table 6: Backbone encoding settings.

Setting	Value
Adapter	Rank-32 query residual; zero-initialized output matrix
Optimizer	AdamW; learning rate $10^{-4}$ ; weight decay $10^{-4}$
Training budget	25 epochs; batch size 16 queries
Loss	Multi-positive MaxSim cross-entropy; temperature $\tau=0.07$
Gradient clipping	Maximum norm 1

Appendix

Table 7: Query-adapter training settings.

Model	LC	LM	RT	ESG	QA	PQ	CQ	MLDR
Llama-1B	68.5	93.6	57.2	60.6	77.5	46.6	95.5	99.0
Ours (training-free)	72.3	95.7	60.5	67.1	79.9	47.7	97.4	98.0
Ours	72.6 $\pm$ 0.14	95.5 $\pm$ 0.11	61.2 $\pm$ 0.12	68.0 $\pm$ 0.80	80.2 $\pm$ 0.14	47.8 $\pm$ 0.47	97.5 $\pm$ 0.05	98.0 $\pm$ 0.00
Llama-3B	72.4	89.4	61.2	81.3	80.7	45.8	93.8	99.0
Ours (training-free)	76.2	95.7	64.9	75.6	81.9	47.4	97.1	98.0
Ours	76.6 $\pm$ 0.17	94.8 $\pm$ 0.42	63.9 $\pm$ 0.23	75.8 $\pm$ 1.06	82.2 $\pm$ 0.18	47.2 $\pm$ 0.47	97.2 $\pm$ 0.05	98.0 $\pm$ 0.00

Appendix

Table 8: Per-task capped Recall@10 for the six additional backbones in Table 2 (%). Backbone rows report Original embedding; Ours reports mean

\pm

SD.

Model	LC	LM	RT	ESG	QA	PQ	CQ	MLDR
Llama-1B	54.2	83.1	48.0	48.4	56.2	36.2	84.4	80.8
Ours (training-free)	59.0	89.1	51.2	54.1	56.6	37.3	89.6	83.4
Ours	59.1 $\pm$ 0.10	88.7 $\pm$ 0.30	52.1 $\pm$ 0.08	54.2 $\pm$ 0.33	57.1 $\pm$ 0.11	37.7 $\pm$ 0.10	89.8 $\pm$ 0.06	84.2 $\pm$ 0.16
Llama-3B	56.3	75.4	49.2	57.0	58.2	34.6	76.8	80.1
Ours (training-free)	62.4	87.5	54.1	64.2	59.1	38.8	87.7	85.1
Ours	62.7 $\pm$ 0.06	86.2 $\pm$ 0.02	53.9 $\pm$ 0.16	63.5 $\pm$ 0.57	59.0 $\pm$ 0.21	37.8 $\pm$ 0.19	87.3 $\pm$ 0.01	83.8 $\pm$ 0.64

Appendix

Table 9: Per-task NDCG@10 for the six additional backbones in Table 2 (%). Backbone rows report Original embedding; Ours reports mean

\pm

SD.

Model	Dataset	HNSW + floor	Exact + floor	HNSW + zero
Qwen-0.6B	LoCoMo	68.69/54.51	69.30/55.31	61.84/47.04
Qwen-0.6B	LongMemEval	94.06/87.48	94.01/87.50	92.46/82.35
Qwen-4B	LoCoMo	67.83/53.58	69.44/55.49	60.69/45.60
Qwen-4B	LongMemEval	95.94/88.15	95.96/88.68	93.99/85.58

Appendix

Table 10: Effects of exact search and missing-score estimation at

k=64

. Adapted Recall@10/NDCG@10 (%), averaged over three training seeds.

Qwen-0.6B
	MaxSim		MaxSim + Whole		Sparse		Sparse + Whole
Task	Training-free	Query-adapted	Training-free	Query-adapted	Training-free	Query-adapted	Training-free	Query-adapted
LoCoMo	64.2/51.3	66.3/53.7	61.6/48.9	63.6/50.3	66.7/52.1	67.4/53.3	68.0/53.4	68.7/54.5
LongMemEval	92.2/82.5	92.7/83.2	93.0/84.3	93.2/84.7	93.0/86.4	93.3/86.2	94.5/87.1	94.1/87.5
REALTALK	53.4/43.8	55.9/45.9	54.7/44.6	56.4/45.6	51.8/41.9	53.7/44.3	55.1/44.9	55.6/45.9
ESG	54.3/37.7	57.0/39.7	56.3/44.9	56.3/45.3	66.3/47.1	65.8/49.9	64.1/52.3	65.1/51.3

Appendix

Table 11: Scoring controls, Recall@10/NDCG@10 (%).

Qwen3-Embedding-0.6B
	Full		SQ4		RP128+SQ4		Pool 4		Pool 8		Pool 16
Task	R/N	MiB	R/N	MiB	R/N	MiB	R/N	MiB	R/N	MiB	R/N	MiB
LoCoMo	68.7/54.5	153.9	68.3/54.3	26.2	66.8/52.0	10.7	65.9/49.5	11.7	67.3/50.9	20.9	66.2/50.7	39.3
LongMemEval	94.1/87.5	597.0	94.6/87.6	95.1	94.9/86.5	32.9	90.1/76.6	9.8	92.0/80.4	17.6	92.9/82.8	33.0
REALTALK	55.6/45.9	173.1	55.1/45.4	29.5	54.6/44.8	12.0	53.3/42.0	13.6	51.6/42.2	24.3	51.4/41.1	45.8
ESG	65.1/51.3	86.3	68.3/52.5	14.1	59.4/47.4	5.5	66.7/41.8	2.3	64.6/42.9	4.0	57.0/40.2	7.3

Appendix

Table 12: Retrieval quality and storage under compression. Quality is Recall@10/NDCG@10, evaluated on all queries and averaged over three query adapters; storage is measured on a fixed representative candidate bank per task.

Qwen3-Embedding-0.6B
Method	Recall@10 $\uparrow$	NDCG@10 $\uparrow$	Size (MiB) $\downarrow$	Query (ms) $\downarrow$
Original embedding	46.87	34.24	2.23	16.66
Search-Adaptor	49.08	35.62	4.24	16.83
Ours (Full)	68.53	54.45	153.93	29.21
Ours (SQ4)	68.37	54.31	26.17	21.68
Ours (RP128+SQ4)	66.69	51.92	10.70	20.69

Appendix

Table 13: Retrieval quality, storage, and latency on LoCoMo.

Late Interaction Retrieval

Late interaction retrieval matches query and document token vectors after encoding to preserve fine-grained relevance while supporting efficient search.

See all 28 papers

Learning to Retrieve Missing Evidence for Long-Term Memory QA
Sep 29, 2026Yi-Xuan Deng, Yi Zhang, Wei Liu +2Long-Term Conversational Memory Evidence Retrieval
Tsinghua University
Where to Look and What to Use: Retrieve-Localize-Generate for Long-Term Conversational Memory Question Answering
Sep 7, 2026Yifan Wang, Xinkui Lin, Yongxiu Xu +9Graph Retrieval-Augmented Generation Persistent Memory for Language Models
University of Electronic Science and Technology of China, Chengdu, China 4State Key Laboratory of Internet Architecture, Tsinghua University, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 3School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · 5Independent Researcher
Recall What Is Missing: Composing Evidence from Long-Term Agent Memory
Oct 11, 2026Qi Feng, Jicong FanLong-Term Conversational Memory Information Retrieval
School of Data Science, The Chinese University of Hong Kong, Shenzhen

Sep 29, 2026J/K move · Enter open · S save

Learning to Retrieve Missing Evidence for Long-Term Memory QA

Yi-Xuan Deng, Yi Zhang, Wei Liu, Chao Xue+1

Tsinghua University

Long-Term Conversational Memory Evidence Retrieval Persistent Memory for Language Models Question Answering RL for IR Retrieval-Augmented Reasoning+3

Open paper AskPDF arXiv

Long-term memory enables language models to use past interactions in future conversations. However, evidence needed to answer a question may be scattered across distant turns, while the question itself omits clues needed to locate it. Retrieved facts can reveal these clues, motivating retrieval decisions conditioned on evidence already found. We introduce MERA (Missing-Evidence Retrieval Augmentation), which separates globally searchable memory from a question-specific evidence state. Verified evidence guides subsequent retrieval without restricting access to the global memory. We train a lightweight planner through reinforcement learning, rewarding queries that recover previously missing evidence. MERA achieves strong answer accuracy across Qwen3-30B and GPT-4o-mini backbones. With Qwen3-30B for evidence processing and answer generation, the trained 0.6B planner achieves 77.40% accuracy on LoCoMo and 71.29% on LongMemEval-S, exceeding a 30B planner without retrieval-grounded training by 4.10% and 3.96%, respectively. On LoCoMo, later retrieval rounds increase cumulative evidence recall from 55.5% to 80.5%.

Abstract

Late Interaction Retrieval

Explore similar work

Learning to Retrieve Missing Evidence for Long-Term Memory QA

Where to Look and What to Use: Retrieve-Localize-Generate for Long-Term Conversational Memory Question Answering

Recall What Is Missing: Composing Evidence from Long-Term Agent Memory