cs.CLNov 17, 2026

LMMR: Learned Multigranular Memory Retrieval for Small Language Model Agents

Authors: Pengfeng Song, Yunkai Zhuang, Liye Shi

Organizations: School of Information and Intelligent Engineering, Zhejiang Wanli University, Ningbo 315100, China

Abstract

Small language model agents have limited space to read the memory needed for multi-hop questions. Relevant facts are often scattered across records whose surrounding text consumes much of that space. We propose LMMR, a learned multigranular memory retrieval method that selects both the next source and how much of it to read. LMMR links paragraphs and sentence fragments to their original sources and pairs views that supply the same missing support during training. Relevant distractors and repeated information provide contrasting examples. At inference, the reranker scores what each candidate adds to an anchor passage, and a selector chooses the evidence within a token budget. On 22,398 official validation questions from HotpotQA, 2WikiMultihopQA, and MuSiQue, a frozen 2B answer model reaches 61.47, 43.26, and 30.58 F1 under a 400-token budget. The gains over query expansion are 3.06, 2.14, and 4.55 points, while memory input decreases by 23.1-26.9%. LMMR also uses 23.5-27.7% fewer input tokens than a coarse retriever trained on the same questions. Against that retriever followed by LLMLingua-2 at comparable input lengths, F1 gains reach 3.99 points and are significant on HotpotQA and 2WikiMultihopQA after Holm correction. All eight small answer models evaluated on HotpotQA improve over query expansion and MMR in both F1 and exact match.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Learning to Retrieve Missing Evidence for Long-Term Memory QA

    Sep 29, 2026Yi-Xuan Deng, Yi Zhang, Wei Liu +2RL for IREvidence Retrieval

  2. MemFlow: Intent-Driven Memory Orchestration for Small Language Model Agents

    May 5, 2026Jiayi Chen, Yingcong Li, Guiling WangLLM Agent Workflow OptimizationPersistent Memory for Language Models

  3. MegaMem: A Retrieval Solution for Ultra-Large Context Windows

    Aug 22, 2026Xinyuan Song, Bowen Zhu, Hasibul Haque +1Retrieval-Augmented GenerationPersistent Memory for Language Models