Small language model agents have limited space to read the memory needed for multi-hop questions. Relevant facts are often scattered across records whose surrounding text consumes much of that space. We propose LMMR, a learned multigranular memory retrieval method that selects both the next source and how much of it to read. LMMR links paragraphs and sentence fragments to their original sources and pairs views that supply the same missing support during training. Relevant distractors and repeated information provide contrasting examples. At inference, the reranker scores what each candidate adds to an anchor passage, and a selector chooses the evidence within a token budget. On 22,398 official validation questions from HotpotQA, 2WikiMultihopQA, and MuSiQue, a frozen 2B answer model reaches 61.47, 43.26, and 30.58 F1 under a 400-token budget. The gains over query expansion are 3.06, 2.14, and 4.55 points, while memory input decreases by 23.1-26.9%. LMMR also uses 23.5-27.7% fewer input tokens than a coarse retriever trained on the same questions. Against that retriever followed by LLMLingua-2 at comparable input lengths, F1 gains reach 3.99 points and are significant on HotpotQA and 2WikiMultihopQA after Holm correction. All eight small answer models evaluated on HotpotQA improve over query expansion and MMR in both F1 and exact match.
Figures & tables
Figure 1: The same question and source records yield different small-model answers. (a) Query expansion supplies two paragraphs (235 tokens); the 2B model returns the father’s name. (b) LMMR uses the Bob Buck paragraph followed by a Joe Buck sentence (122 tokens), and the same model returns the correct team. Blue text marks supporting content; grey lines abbreviate background. Bars show evidence length on the same 400-token scale. Answers and lengths are from the recorded HotpotQA case.
Figure 2: LMMR illustrated through concrete memory views. (a) A source paragraph and its first-sentence view retain the same father relation. Under the same question and anchor, both supply new support; relevant non-supporting and redundant examples supervise the shared conditional reranker. (b) The reranker first ranks sources, then compares expanded sentence views with the paragraphs still available. Selection merges the anchor and one additional view within the token budget. Blue marks corresponding support, ellipses abbreviate displayed text, and dashed arrows show parameter sharing.
Method
HotpotQA (7,405)
2Wiki (12,576)
MuSiQue (2,417)
F1 ↑
EM ↑
Tokens ↓
F1 ↑
EM ↑
Tokens ↓
F1 ↑
EM ↑
Tokens ↓
Query expansion
58.42
43.42
240.92
41.12
31.67
242.47
26.02
17.17
259.01
Normalized MMR
58.79
43.69
242.05
41.49
31.99
243.72
25.97
17.13
259.96
Matched coarse-only
61.25
45.75
242.30
42.71
32.82
245.97
28.50
19.07
260.77
QE + LLMLingua-2
55.90
40.20
185.35
37.54
26.06
186.38
26.39
16.80
200.18
Matched coarse + LLMLingua-2
58.71
42.36
186.42
39.27
27.59
189.05
29.22
18.87
201.32
Table 1: Training-matched retrieval and learned compression on the full validation sets. All methods use the 2B answer model and a 400-token evidence budget.
Figure 3: Answer-quality differences and evidence-input reduction relative to QE on HotpotQA, including the training-matched and learned-compression controls. F1 (a) and EM (b) use identical axis ranges. Panel (c) shows the reduction in mean evidence tokens. All settings use 7,405 questions; differences are calculated before rounding.
Configuration
Learning
Answer quality
Evidence
Adapted
Paired views
F1 ↑
EM ↑
Complete ↑
Density ↑
Tokens ↓
FH
–
–
56.65
42.08
63.46
42.25
179.22
FCH
–
–
57.90
42.92
65.91
42.13
183.22
Coarse-only adaptation
✓
–
60.04
44.71
80.85
33.90
240.61
Without score smoothing
✓
✓
61.49
45.86
73.09
43.76
185.16
Without redundant negatives
✓
✓
61.23
45.66
72.90
43.78
185.54
Table 2: Full-validation ablation of training and loss components on HotpotQA.
Figure 4: Answer F1 and EM as the HotpotQA candidate pool increases from 10 to 50 paragraphs. Each setting uses 7,405 validation questions.
Figure 5: Conditional-score weight and evidence-budget analyses. (a) Weight sensitivity on 7,405 HotpotQA validation questions. (b,c) Fixed-evidence clipping on 192 questions.
Figure 6: F1 and EM gains over QE for eight small answer models on 7,405 HotpotQA validation questions. Both panels use the same gain scale; the dotted line marks the QE reference at zero. Triangles denote MMR and blue circles denote LMMR. Labels give the LMMR gains, calculated before rounding. Absolute scores are reported in Table 3 .
Answer model
Query expansion
MMR
LMMR
F1 ↑
EM ↑
F1 ↑
EM ↑
F1 ↑
EM ↑
Qwen3.5 0.8B
46.19
33.48
46.41
33.56
49.18
35.79
Gemma 3 1B
42.70
29.76
42.85
29.90
45.43
31.65
Hunyuan 1.8B
39.36
27.75
39.33
27.75
42.21
30.11
Qwen3.5 2B
58.42
43.42
58.79
43.69
61.47
45.96
SmolLM3 3B
37.27
18.53
37.43
18.66
40.49
21.04
Table 3: Eight small answer models on 7,405 HotpotQA validation questions.
Figure 7: Question-type F1 gains over QE, with pointwise 95% paired-bootstrap intervals. The number of questions in each group is denoted by n .
Method
Answer quality
Support coverage (%)
Evidence quality (%)
Tokens ↓
F1 ↑
EM ↑
Source ↑
Complete ↑
Density ↑
AIC ↑
Query expansion
58.42
43.42
90.07
77.87
33.47
89.66
240.92
Normalized MMR
58.79
43.69
90.43
78.18
33.46
90.33
242.05
Coarse-only adaptation
60.04
44.71
91.53
80.85
33.90
92.23
240.61
FH
56.65
42.08
89.50
63.46
42.25
84.87
179.22
FCH
57.90
42.92
89.57
65.91
42.13
86.09
183.22
Table 4: Answer and evidence quality on 7,405 HotpotQA validation questions. Support-source recall, support completeness, support density, and AIC are percentages.
Figure 8: Mean evidence composition (a) and five mutually exclusive error categories (b) on HotpotQA. The three methods occupy separate tracks within each error category. Point labels are percentages of 7,405 validation questions. QE denotes query expansion.
Metric
QE
FCH
LMMR
Mean evidence tokens
232.34
178.01
182.18
Evidence selection, A100 (ms)
690.81
1116.36
1081.23
Prompt encoding, Orin (ms)
528.51
456.03
464.83
Answer generation, Orin (ms)
360.96
360.17
367.21
Total request latency (ms)
1927.87
2291.09
2271.93
Generation throughput, Orin (tokens/s)
19.68
19.73
19.78
Table 5: Measured execution on 192 questions: evidence selection on A100 and answering on Jetson Orin.
Figure 9: Recorded question-answering examples on Jetson Orin. The methods share a coarse anchor but select a full paragraph or a fine fragment as subsequent memory. Parallel evidence cards show original excerpts and recorded model answers. The lower annotations report evidence length and Orin prompt-encoding time.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Parameter
Value
Memory selection
B ; Ke ; Ka ; Ks
400 tokens; 2; 2; 3
Scoring
τ ; α ; β ; η
2; 1.2; 1; 0.7
Node lengths
Coarse / fine
192 / 96 reranker tokens
Training examples
Training / development; coarse–fine pairs
1,600 / 64; 1,206 paired-positive questions
Controls and negatives
Coarse-only / matched coarse-only; hard negatives
381 / 1,600 questions; top 2 non-supporting negatives per granularity
Loss weights
m ; λr ; λB ; λc
0.5; 0.5; 0.25; 0.1
Appendix
Table A1: Method and optimization parameters.
Provider
Model identifier
Quantization
Alibaba
Qwen/Qwen3.5-0.8B
Q4_K_M
Google
google/gemma-3-1b-it
Q4_K_M
Tencent
tencent/Hunyuan-1.8B-Instruct
Q4_K_M
Alibaba
Qwen/Qwen3.5-2B
Q4_K_M
Hugging Face
HuggingFaceTB/SmolLM3-3B
Q4_K_M
IBM
ibm-granite/granite-4.0-micro
Q4_K_M
Appendix
Table A2: Configurations of the eight answer models.
Component
Instruction or input
Relevance instruction
Given a question, retrieve relevant passages that answer the question.
Conditional instruction
Given the question and the known evidence, retrieve a different passage that supplies the missing information needed to answer the question. Prefer complementary evidence over repetition of the known evidence.
Conditional fields
<Query>: {question} Known evidence: {anchor} <Document>: {candidate} Decision tokens: yes / no .
Answer instruction
Answer using only the supplied evidence. Return a concise direct answer without explanation. If the evidence is insufficient, say "not enough information".
Table A3: Model instructions and input fields used in retrieval and answering.
Method
10 paragraphs
20 paragraphs
30 paragraphs
50 paragraphs
F1 ↑
EM ↑
F1 ↑
EM ↑
F1 ↑
EM ↑
F1 ↑
EM ↑
Query expansion
59.24
44.04
58.56
43.55
58.42
43.42
58.27
43.27
Normalized MMR
59.55
44.27
58.97
43.86
58.79
43.69
58.61
43.50
Coarse-only adaptation
60.90
45.36
60.14
44.79
60.04
44.71
59.91
44.63
FH
57.20
42.39
56.72
42.12
56.65
42.08
56.55
41.99
LMMR
62.05
46.40
61.61
46.10
61.47
45.96
61.42
45.87
Appendix
Table B1: Candidate-pool analysis on 7,405 HotpotQA validation questions.
Figure B1: Memory behavior as the HotpotQA pool grows from 10 to 50 paragraphs. Each condition uses 7,405 validation questions and the same 400-token budget. Panels report supporting-source recall, non-boolean answer-in-context rate (AIC), mean per-question support density, and mean evidence length. Markers show the evaluated settings.
α
F1 ↑
EM ↑
Tokens ↓
0.8
61.55
45.96
185.95
1.0
61.47
45.94
185.60
1.2
61.47
45.96
185.32
1.5
61.46
45.94
185.02
1.8
61.44
45.91
185.00
Appendix
Table B2: Conditional-score weight sensitivity on 7,405 HotpotQA validation questions.
Budget
Query expansion
MMR
LMMR
F1 ↑
EM ↑
Tokens ↓
F1 ↑
EM ↑
Tokens ↓
F1 ↑
EM ↑
Tokens ↓
140
36.21
24.48
134.76
36.83
23.96
135.11
39.97
27.08
129.08
180
41.81
27.08
168.58
42.50
26.56
169.69
46.65
30.21
153.44
220
49.12
34.90
196.84
48.39
32.81
198.45
55.63
36.98
169.94
280
53.98
38.54
221.94
53.27
36.46
224.48
58.90
39.06
179.64
400
54.83
40.10
232.34
54.11
38.02
234.97
59.37
39.06
183.03
Appendix
Table B3: Fixed-evidence budget analysis on 192 questions.
Figure B2: Supporting-source recall, answer presence (AIC), and mean evidence length by MuSiQue reasoning depth. The two-, three-, and four-hop groups contain 1,252, 760, and 405 questions, respectively. All groups use the same 400-token evidence budget and frozen 2B answer model. Lines connect the observed hop groups.
Dataset and type
n
F1 ↑
EM ↑
QE
LMMR
QE
LMMR
2Wiki bridge comparison
2751
42.44
41.49
40.31
38.71
2Wiki comparison
3040
48.43
49.32
42.34
42.57
2Wiki compositional
5236
39.32
43.72
27.12
29.76
2Wiki inference
1549
30.50
32.97
10.78
12.07
Hotpot bridge
5918
62.62
66.41
46.54
49.78
Appendix
Table B4: Answer F1 and EM by dataset and question type.
Method
Single reference
Alias-aware
F1 ↑
EM ↑
F1 ↑
EM ↑
Query expansion
26.02
17.17
28.02
20.07
Normalized MMR
25.97
17.13
27.93
20.02
Coarse-only adaptation
26.65
17.50
28.64
20.31
FH
26.05
16.84
27.94
19.74
LMMR
30.58
20.27
32.57
23.50
Appendix
Table B5: Single-reference and alias-aware scoring on MuSiQue.
Method
F1 ↑
EM ↑
Query expansion
65.13
50.83
Normalized MMR
65.47
51.10
Coarse-only adaptation
66.77
52.13
FH
63.08
49.32
LMMR
67.92
53.30
Appendix
Table B6: HotpotQA with a uniform short-answer instruction: 7,405 questions and 10 candidate paragraphs.
Baseline
F1 difference
95% interval
Holm p
2WikiMultihopQA
Query expansion
+2.14
[1.55, 2.72]
0.0052
Normalized MMR
+1.77
[1.20, 2.35]
0.0052
Coarse-only adaptation
+2.28
[1.69, 2.86]
0.0052
FH
+5.05
[4.52, 5.57]
0.0052
Matched coarse-only
+0.55
[0.05, 1.08]
0.1102
Appendix
Table B7: Primary F1 differences, 95% paired-bootstrap intervals, and Holm-adjusted p-values across 21 comparisons. Positive differences favor LMMR.
Method
2WikiMultihopQA (12,576 questions)
MuSiQue (2,417 questions)
F1 ↑
EM ↑
Tokens ↓
F1 ↑
EM ↑
Tokens ↓
Query expansion
41.12
31.67
242.47
26.02
17.17
259.01
Normalized MMR
41.49
31.99
243.72
25.97
17.13
259.96
Coarse-only adaptation
40.98
31.46
244.10
26.65
17.50
255.13
FH
38.22
28.97
173.23
26.05
16.84
188.03
LMMR
43.26
32.63
177.94
30.58
20.27
189.21
Appendix
Table B8: Zero-shot transfer to the full validation sets of 2WikiMultihopQA and MuSiQue. MuSiQue uses single-reference scoring.
Answer model
QE F1
QE EM
MMR F1
MMR EM
LMMR F1
LMMR EM
Nemotron Nano 4B
8.23
0.30
8.28
0.30
8.48
0.45
Appendix
Table B9: Additional Nemotron Nano 4B results: raw-output scoring on 7,405 HotpotQA questions with Q4_K_M quantization.
Method
Answer model
Evaluation setting
EM
F1
HotpotQA: overall evaluation sets
IRCoT Trivedi et al. [2023]
Flan-T5-XL 3B
Open domain / 500
—
56.70
IRCoT Trivedi et al. [2023]
Flan-T5-XXL 11B
Open domain / 500
—
59.10
IRCoT Trivedi et al. [2023]
GPT-3 175B
Open domain / 500
49.30
60.70
Smart-Searcher Song et al. [2025]
Qwen2.5-7B
Open domain / 500
—
59.00
PER-QA RAG Liu et al. [2025]
8B + GPT-4o planner
Processed set / overall
—
62.90
Appendix
Table C1: Published results and LMMR reference results under the indicated settings.
Long-term memory enables language models to use past interactions in future conversations. However, evidence needed to answer a question may be scattered across distant turns, while the question itself omits clues needed to locate it. Retrieved facts can reveal these clues, motivating retrieval decisions conditioned on evidence already found. We introduce MERA (Missing-Evidence Retrieval Augmentation), which separates globally searchable memory from a question-specific evidence state. Verified evidence guides subsequent retrieval without restricting access to the global memory. We train a lightweight planner through reinforcement learning, rewarding queries that recover previously missing evidence. MERA achieves strong answer accuracy across Qwen3-30B and GPT-4o-mini backbones. With Qwen3-30B for evidence processing and answer generation, the trained 0.6B planner achieves 77.40% accuracy on LoCoMo and 71.29% on LongMemEval-S, exceeding a 30B planner without retrieval-grounded training by 4.10% and 3.96%, respectively. On LoCoMo, later retrieval rounds increase cumulative evidence recall from 55.5% to 80.5%.
Modern language agents must operate over long-horizon, multi-turn histories, yet deploying such agents with Small Language Models (SLMs) remains fundamentally difficult. Full-context prompting causes context overflow, flat retrieval exposes the model to noisy evidence, and open-ended agentic loops are unreliable under limited reasoning capacity. We argue that a substantial portion of SLM memory failure arises from mismatched memory operations: different query types demand categorically different retrieval strategies, evidence transformations, and context budgets that SLMs cannot reliably self-orchestrate through open-ended reasoning. We introduce MemFlow, a training-free memory orchestration framework that externalizes memory planning from the SLM. A Router Agent classifies each query by intent and dispatches it to the Memory Agent, which executes one of three specialized tiers (Profile Lookup, Targeted Retrieval, or Deep Reasoning) and assembles the resulting evidence under a dynamic, tier-aware token budget. An Answer Agent then generates a response from this compact context, and a Validator Agent optionally retries with a heavier memory tier when the response is not supported by the provided evidence. This route-then-compile design avoids tool-selection hallucination and reasoning loops while keeping the answer context compact. Evaluated on a frozen Qwen3-1.7B backbone across long-horizon memory benchmarks - LongMemEval, LoCoMo, and LongBench - MemFlow improves accuracy by nearly 2x over full-context SLM baselines. These results suggest that structured intent routing and deterministic evidence preparation can make limited-capacity models substantially more effective in resource-constrained long-horizon agents.
Modern language models and agents increasingly require persistent memory for complete codebases, long interaction histories, and heterogeneous enterprise records. The key challenge is to keep hundreds of millions of tokens searchable while passing only bounded source evidence to the answer model. We introduce MegaMem, a source-resolved dual-view retrieval system that separates semantic access from generation evidence. Distilled records and detailed evidence are searched with original and transformed queries; every distilled hit resolves to an immutable source ID before reciprocal-rank fusion, deduplication, and cross-encoder reranking; and only the highest-ranked detailed evidence within a fixed budget supports generation. Post-answer attribution then identifies which loaded sources support the fixed answer. We evaluate MegaMem on EnterpriseRAG-Bench, which contains more than 500,000 heterogeneous enterprise documents and approximately 650M tokens. MegaMem improves Overall from 68.22 to 82.26 and reaches 86.50 Correctness. These results show that MegaMem supports ultra-large persistent memory while preserving strong answer accuracy under a bounded generation context. By separating searchable memory scale from answer-context size, MegaMem provides a practical path toward accurate retrieval over memories ranging from hundreds of millions to one billion tokens. Our code is available at https://github.com/ xfab-xinyuansong/MegaMem.git.