LMMR: Learned Multigranular Memory Retrieval for Small Language Model Agents
Organizations: School of Information and Intelligent Engineering, Zhejiang Wanli University, Ningbo 315100, China
Abstract
Small language model agents have limited space to read the memory needed for multi-hop questions. Relevant facts are often scattered across records whose surrounding text consumes much of that space. We propose LMMR, a learned multigranular memory retrieval method that selects both the next source and how much of it to read. LMMR links paragraphs and sentence fragments to their original sources and pairs views that supply the same missing support during training. Relevant distractors and repeated information provide contrasting examples. At inference, the reranker scores what each candidate adds to an anchor passage, and a selector chooses the evidence within a token budget. On 22,398 official validation questions from HotpotQA, 2WikiMultihopQA, and MuSiQue, a frozen 2B answer model reaches 61.47, 43.26, and 30.58 F1 under a 400-token budget. The gains over query expansion are 3.06, 2.14, and 4.55 points, while memory input decreases by 23.1-26.9%. LMMR also uses 23.5-27.7% fewer input tokens than a coarse retriever trained on the same questions. Against that retriever followed by LLMLingua-2 at comparable input lengths, F1 gains reach 3.99 points and are significant on HotpotQA and 2WikiMultihopQA after Holm correction. All eight small answer models evaluated on HotpotQA improve over query expansion and MMR in both F1 and exact match.
Figures & tables
| Method | HotpotQA (7,405) | 2Wiki (12,576) | MuSiQue (2,417) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| F1 | EM | Tokens | F1 | EM | Tokens | F1 | EM | Tokens | |
| Query expansion | 58.42 | 43.42 | 240.92 | 41.12 | 31.67 | 242.47 | 26.02 | 17.17 | 259.01 |
| Normalized MMR | 58.79 | 43.69 | 242.05 | 41.49 | 31.99 | 243.72 | 25.97 | 17.13 | 259.96 |
| Matched coarse-only | 61.25 | 45.75 | 242.30 | 42.71 | 32.82 | 245.97 | 28.50 | 19.07 | 260.77 |
| QE + LLMLingua-2 | 55.90 | 40.20 | 185.35 | 37.54 | 26.06 | 186.38 | 26.39 | 16.80 | 200.18 |
| Matched coarse + LLMLingua-2 | 58.71 | 42.36 | 186.42 | 39.27 | 27.59 | 189.05 | 29.22 | 18.87 | 201.32 |
| Configuration | Learning | Answer quality | Evidence | ||||
|---|---|---|---|---|---|---|---|
| Adapted | Paired views | F1 | EM | Complete | Density | Tokens | |
| FH | – | – | 56.65 | 42.08 | 63.46 | 42.25 | 179.22 |
| FCH | – | – | 57.90 | 42.92 | 65.91 | 42.13 | 183.22 |
| Coarse-only adaptation | – | 60.04 | 44.71 | 80.85 | 33.90 | 240.61 | |
| Without score smoothing | 61.49 | 45.86 | 73.09 | 43.76 | 185.16 | ||
| Without redundant negatives | 61.23 | 45.66 | 72.90 | 43.78 | 185.54 | ||
| Answer model | Query expansion | MMR | LMMR | |||
|---|---|---|---|---|---|---|
| F1 | EM | F1 | EM | F1 | EM | |
| Qwen3.5 0.8B | 46.19 | 33.48 | 46.41 | 33.56 | 49.18 | 35.79 |
| Gemma 3 1B | 42.70 | 29.76 | 42.85 | 29.90 | 45.43 | 31.65 |
| Hunyuan 1.8B | 39.36 | 27.75 | 39.33 | 27.75 | 42.21 | 30.11 |
| Qwen3.5 2B | 58.42 | 43.42 | 58.79 | 43.69 | 61.47 | 45.96 |
| SmolLM3 3B | 37.27 | 18.53 | 37.43 | 18.66 | 40.49 | 21.04 |
| Method | Answer quality | Support coverage (%) | Evidence quality (%) | Tokens | |||
|---|---|---|---|---|---|---|---|
| F1 | EM | Source | Complete | Density | AIC | ||
| Query expansion | 58.42 | 43.42 | 90.07 | 77.87 | 33.47 | 89.66 | 240.92 |
| Normalized MMR | 58.79 | 43.69 | 90.43 | 78.18 | 33.46 | 90.33 | 242.05 |
| Coarse-only adaptation | 60.04 | 44.71 | 91.53 | 80.85 | 33.90 | 92.23 | 240.61 |
| FH | 56.65 | 42.08 | 89.50 | 63.46 | 42.25 | 84.87 | 179.22 |
| FCH | 57.90 | 42.92 | 89.57 | 65.91 | 42.13 | 86.09 | 183.22 |
| Metric | QE | FCH | LMMR |
|---|---|---|---|
| Mean evidence tokens | 232.34 | 178.01 | 182.18 |
| Evidence selection, A100 (ms) | 690.81 | 1116.36 | 1081.23 |
| Prompt encoding, Orin (ms) | 528.51 | 456.03 | 464.83 |
| Answer generation, Orin (ms) | 360.96 | 360.17 | 367.21 |
| Total request latency (ms) | 1927.87 | 2291.09 | 2271.93 |
| Generation throughput, Orin (tokens/s) | 19.68 | 19.73 | 19.78 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Parameter | Value |
|---|---|---|
| Memory selection | ; ; ; | 400 tokens; 2; 2; 3 |
| Scoring | ; ; ; | 2; 1.2; 1; 0.7 |
| Node lengths | Coarse / fine | 192 / 96 reranker tokens |
| Training examples | Training / development; coarse–fine pairs | 1,600 / 64; 1,206 paired-positive questions |
| Controls and negatives | Coarse-only / matched coarse-only; hard negatives | 381 / 1,600 questions; top 2 non-supporting negatives per granularity |
| Loss weights | ; ; ; | 0.5; 0.5; 0.25; 0.1 |
| Provider | Model identifier | Quantization |
|---|---|---|
| Alibaba | Qwen/Qwen3.5-0.8B | Q4_K_M |
| google/gemma-3-1b-it | Q4_K_M | |
| Tencent | tencent/Hunyuan-1.8B-Instruct | Q4_K_M |
| Alibaba | Qwen/Qwen3.5-2B | Q4_K_M |
| Hugging Face | HuggingFaceTB/SmolLM3-3B | Q4_K_M |
| IBM | ibm-granite/granite-4.0-micro | Q4_K_M |
| Component | Instruction or input |
|---|---|
| Relevance instruction | Given a question, retrieve relevant passages that answer the question. |
| Conditional instruction | Given the question and the known evidence, retrieve a different passage that supplies the missing information needed to answer the question. Prefer complementary evidence over repetition of the known evidence. |
| Conditional fields | <Query>: {question} Known evidence: {anchor} <Document>: {candidate} Decision tokens: yes / no . |
| Answer instruction | Answer using only the supplied evidence. Return a concise direct answer without explanation. If the evidence is insufficient, say "not enough information". |
| Answer fields | Evidence: {selected evidence} Question: {question} Answer: |
| Method | 10 paragraphs | 20 paragraphs | 30 paragraphs | 50 paragraphs | ||||
|---|---|---|---|---|---|---|---|---|
| F1 | EM | F1 | EM | F1 | EM | F1 | EM | |
| Query expansion | 59.24 | 44.04 | 58.56 | 43.55 | 58.42 | 43.42 | 58.27 | 43.27 |
| Normalized MMR | 59.55 | 44.27 | 58.97 | 43.86 | 58.79 | 43.69 | 58.61 | 43.50 |
| Coarse-only adaptation | 60.90 | 45.36 | 60.14 | 44.79 | 60.04 | 44.71 | 59.91 | 44.63 |
| FH | 57.20 | 42.39 | 56.72 | 42.12 | 56.65 | 42.08 | 56.55 | 41.99 |
| LMMR | 62.05 | 46.40 | 61.61 | 46.10 | 61.47 | 45.96 | 61.42 | 45.87 |
| F1 | EM | Tokens | |
|---|---|---|---|
| 0.8 | 61.55 | 45.96 | 185.95 |
| 1.0 | 61.47 | 45.94 | 185.60 |
| 1.2 | 61.47 | 45.96 | 185.32 |
| 1.5 | 61.46 | 45.94 | 185.02 |
| 1.8 | 61.44 | 45.91 | 185.00 |
| Budget | Query expansion | MMR | LMMR | ||||||
|---|---|---|---|---|---|---|---|---|---|
| F1 | EM | Tokens | F1 | EM | Tokens | F1 | EM | Tokens | |
| 140 | 36.21 | 24.48 | 134.76 | 36.83 | 23.96 | 135.11 | 39.97 | 27.08 | 129.08 |
| 180 | 41.81 | 27.08 | 168.58 | 42.50 | 26.56 | 169.69 | 46.65 | 30.21 | 153.44 |
| 220 | 49.12 | 34.90 | 196.84 | 48.39 | 32.81 | 198.45 | 55.63 | 36.98 | 169.94 |
| 280 | 53.98 | 38.54 | 221.94 | 53.27 | 36.46 | 224.48 | 58.90 | 39.06 | 179.64 |
| 400 | 54.83 | 40.10 | 232.34 | 54.11 | 38.02 | 234.97 | 59.37 | 39.06 | 183.03 |
| Dataset and type | F1 | EM | |||
|---|---|---|---|---|---|
| QE | LMMR | QE | LMMR | ||
| 2Wiki bridge comparison | 2751 | 42.44 | 41.49 | 40.31 | 38.71 |
| 2Wiki comparison | 3040 | 48.43 | 49.32 | 42.34 | 42.57 |
| 2Wiki compositional | 5236 | 39.32 | 43.72 | 27.12 | 29.76 |
| 2Wiki inference | 1549 | 30.50 | 32.97 | 10.78 | 12.07 |
| Hotpot bridge | 5918 | 62.62 | 66.41 | 46.54 | 49.78 |
| Method | Single reference | Alias-aware | ||
|---|---|---|---|---|
| F1 | EM | F1 | EM | |
| Query expansion | 26.02 | 17.17 | 28.02 | 20.07 |
| Normalized MMR | 25.97 | 17.13 | 27.93 | 20.02 |
| Coarse-only adaptation | 26.65 | 17.50 | 28.64 | 20.31 |
| FH | 26.05 | 16.84 | 27.94 | 19.74 |
| LMMR | 30.58 | 20.27 | 32.57 | 23.50 |
| Method | F1 | EM |
|---|---|---|
| Query expansion | 65.13 | 50.83 |
| Normalized MMR | 65.47 | 51.10 |
| Coarse-only adaptation | 66.77 | 52.13 |
| FH | 63.08 | 49.32 |
| LMMR | 67.92 | 53.30 |
| Baseline | F1 difference | 95% interval | Holm p |
| 2WikiMultihopQA | |||
| Query expansion | +2.14 | [1.55, 2.72] | 0.0052 |
| Normalized MMR | +1.77 | [1.20, 2.35] | 0.0052 |
| Coarse-only adaptation | +2.28 | [1.69, 2.86] | 0.0052 |
| FH | +5.05 | [4.52, 5.57] | 0.0052 |
| Matched coarse-only | +0.55 | [0.05, 1.08] | 0.1102 |
| Method | 2WikiMultihopQA (12,576 questions) | MuSiQue (2,417 questions) | ||||
|---|---|---|---|---|---|---|
| F1 | EM | Tokens | F1 | EM | Tokens | |
| Query expansion | 41.12 | 31.67 | 242.47 | 26.02 | 17.17 | 259.01 |
| Normalized MMR | 41.49 | 31.99 | 243.72 | 25.97 | 17.13 | 259.96 |
| Coarse-only adaptation | 40.98 | 31.46 | 244.10 | 26.65 | 17.50 | 255.13 |
| FH | 38.22 | 28.97 | 173.23 | 26.05 | 16.84 | 188.03 |
| LMMR | 43.26 | 32.63 | 177.94 | 30.58 | 20.27 | 189.21 |
| Answer model | QE F1 | QE EM | MMR F1 | MMR EM | LMMR F1 | LMMR EM |
|---|---|---|---|---|---|---|
| Nemotron Nano 4B | 8.23 | 0.30 | 8.28 | 0.30 | 8.48 | 0.45 |
| Method | Answer model | Evaluation setting | EM | F1 |
| HotpotQA: overall evaluation sets | ||||
| IRCoT Trivedi et al. [2023] | Flan-T5-XL 3B | Open domain / 500 | — | 56.70 |
| IRCoT Trivedi et al. [2023] | Flan-T5-XXL 11B | Open domain / 500 | — | 59.10 |
| IRCoT Trivedi et al. [2023] | GPT-3 175B | Open domain / 500 | 49.30 | 60.70 |
| Smart-Searcher Song et al. [2025] | Qwen2.5-7B | Open domain / 500 | — | 59.00 |
| PER-QA RAG Liu et al. [2025] | 8B + GPT-4o planner | Processed set / overall | — | 62.90 |