Memorilla: Latent Semantic Memory for LLMs

Adibvafa Fallahpour; Parsa Idehpour; Vignesh Kothapalli; Nikita Mounier; Shirley Wu; Shayan Pardis; Jure Leskovec

cs.LGOct 15, 2026

Memorilla: Latent Semantic Memory for LLMs

Authors: Adibvafa Fallahpour, Parsa Idehpour, Vignesh Kothapalli, Nikita Mounier, Shirley Wu, Shayan Pardis, Jure Leskovec

Organizations: Stanford University · University of Toronto · Vector Institute · Arc Institute ·

Abstract

Memory enables large language models (LLMs) to use information beyond their parameters, such as past user conversations, long documents, or an agent's earlier actions. For a frozen LLM, access to this accumulated knowledge is limited to what a memory system places in its context window. However, gathering and combining relevant information needed for each query while keeping the model's context short remains challenging for existing systems. Here we introduce Memorilla, a memory system that gathers information from an entire document collection into a fixed number of query-dependent 'memory tokens' for a frozen LLM. Memorilla encodes documents (such as user chats, book chapters) once into reusable embeddings, then learns to retrieve and place query-specific information in the LLM's input embedding space. Using only 16 memory tokens, Memorilla achieves the best performance against baselines including Mem0 and MemoRAG on five of eight benchmarks spanning personalization, long-document and knowledge-intensive question answering. Notably, on FactKG, it reaches 84.33% accuracy versus 80.43% with full-context inference, while using 108x fewer tokens to represent the source documents. Furthermore, Memorilla can facilitate continual learning in agentic systems where the document count (the agent's past turns) increases throughout the trajectory. In a TextWorld environment, we show that Memorilla reduces the agent's tendency to wander between rooms it has already explored, and improves its average success rate from 18.6% with full-history in-context learning to 24.2%. Overall, Memorilla aids frozen LLMs in accumulating and using knowledge effectively across long-context interactions and takes a step towards building continually learning AI systems.

Figures & tables

Figure 1: Performance versus context length. Mean score over eight benchmarks against the geometric mean of document tokens given to a frozen Qwen3-8B decoder. Memorilla lies on the Pareto frontier (dashed) with the highest score at 16 memory tokens, 394 $\times$ fewer than full context.

Figure 2: Memorilla architecture. A frozen encoder embeds source documents (User Chats) once into a persistent memory store. For each Query, K query-conditioned slots attend over the store and are projected into the frozen LLM’s embedding space as a memory readout.

Panel of Figure 3 — Figure 3: Per-benchmark performance versus context length. Primary-metric score against mean document tokens per query, with a frozen Qwen3-8B decoder. Memorilla lies on the Pareto frontier (dashed) of every benchmark, leads on five and is second on NarrativeQA and PV4.

Panel of Figure 4 — Figure 4: Per-question breakdowns against RAG-50. (a) NarrativeQA question types are the question’s first word. Conceptual questions (‘Why’, ‘How’, ‘What’) favor Memorilla whereas entity-recall questions (‘Which’, ‘Who’) favor RAG. (b) Memorilla ’s advantage grows with entity count, indicating better capture of multi-entity relational patterns. (c) RAG-50 degrades as more facts are composed, while Memorilla remains stable across all depths.

Figure 5: Per-question complementarity between Memorilla and RAG-50. Share of questions answered by each method alone and by at least one of them (union), an oracle upper bound for a hybrid system. The union exceeds either method on every benchmark. † A question counts as answered when its ROUGE-L is at least $\max(0.20,m)$ , where $m$ is Memorilla ’s median.

Figure 6: Personalization Case Study. On PersonaMem-v2, Memorilla personalizes a generic query by leveraging the user’s remembered iron-deficiency anemia history, despite no lexical overlap between the question and the relevant conversation history.

Figure 7: Reinforcement-learned memory on TextWorld. (a) A frozen encoder embeds the trajectory, the trainable memory module compresses it to $K{=}8$ latents, and the frozen LLM agent acts; sparse rewards over $G{=}16$ rollouts drive a GRPO update to the memory module alone. (b) On a held-out game, ICL loops between explored rooms while (c) Memorilla reaches the goal ( $\star$ ).

Figure 8: Memorilla on TextWorld with a frozen Qwen3-4B-Instruct-2507 decoder. Terminal episodes end through the game or the turn limit before the history exceeds the 4,096-token evaluation budget.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Source	Category	Num. Queries
Pretraining	EnWiki Articles	Paraphrase	1,979,809
Finetuning	SQuAD v2 ( Rajpurkar et al., 2018 )	Reading Comprehension	86,821
	DROP ( Dua et al., 2019 )	Reading Comprehension	77,400
	CoQA ( Reddy et al., 2019 )	Reading Comprehension	108,647
	PubMedQA ( Jin et al., 2019 )	Reading Comprehension	900
	QuAIL ( Rogers et al., 2020 )	Reading Comprehension	10,246

Appendix

Table 1: Training datasets for Memorilla . The pretraining corpus is derived from English Wikipedia. The finetuning mixture spans reading comprehension, open-domain question answering, and summarization tasks across 10 sources. Sizes are numbers of training queries.

Benchmark	Chunk unit	Tokens per chunk (mean / median)
PMv2	One conversation session	280 / 262
PV4	One chat dialogue	252 / 224
PubMedQA	One abstract section	84 / 78
TriviaQA	Evidence passage, split sentence-greedily at 128 tokens	70 / 77
NarrativeQA	Story paragraph, split on sentences above ${\sim}$ 256 tokens	94 / 35
FactKG	One knowledge-graph triple	13 / 13

Appendix

Table 2: Document chunks per benchmark. Each chunk

d_{i}

is embedded once by the frozen encoder. Token counts use the Qwen3 tokenizer.

Component	Setting	Value
Memory module	Pooling mode	Adaptive cross-attention with softmax over documents
	Memory slots	$K{=}16$
	Cross-attention layers	2 layers, slots $\leftarrow$ documents
	Attention heads	8
	Encoder-side dimension	2560
	Decoder-side dimension	2560 for 4B decoder, 4096 for 8B decoder

Appendix

Table 3: Model configuration for Memorilla . The memory module maps document representations into a fixed set of learned memory tokens that are inserted into the frozen decoder context.

Setting	Stage 1	Stage 2	Stage 3
	EnWiki pretrain	Multi-task pretrain	Downstream fine-tune
Initialization	Random	Warm-start from Stage 1	Warm-start from Stage 2
Data	EnWiki paraphrase reconstruction	Mixed supervision corpus	Unified downstream mixture
Tasks	Paraphrase reconstruction	Reading comprehension, open-domain QA, summarization	PV4, PMv2, FactKG, TriviaQA, NQA, PubMedQA, LaMP-4, LaMP-7
Optimizer	AdamW	AdamW	AdamW
LR scheduler	Cosine annealing with warmup	Cosine annealing with warmup	Cosine annealing with warmup

Appendix

Table 4: Training stages for Memorilla . Stage 1 pretrains memory on EnWiki paraphrase reconstruction, Stage 2 warm-starts on the mixed supervision corpus in Table 1 , and Stage 3 performs downstream multi-task fine-tuning.

Item	Configuration
Stage 1 corpus	1.98M EnWiki paraphrase reconstruction examples, chunked at approximately 128 tokens with mean 1.73 documents per row
Stage 2 corpus	Mixed supervision over SQuAD v2, DROP, CoQA, PubMedQA, QuAIL, PwC, CNN/DailyMail, SAMSum, DialogSum, and MS MARCO
Stage 3 corpus	Unified concatenation of PV4, PMv2, FactKG, TriviaQA, NarrativeQA, PubMedQA, LaMP-4, and LaMP-7
PV4 and PMv2 specialization	After multi-task fine-tuning, PV4 and PMv2 are each further fine-tuned for 5 epochs on the corresponding benchmark training split using the same recipe
Other downstream tasks	FactKG, TriviaQA, NarrativeQA, PubMedQA, LaMP-4, and LaMP-7 use the multi-task checkpoint directly with no additional task-specific training
Evaluation split	Official test split for every task except LaMP-4 and LaMP-7, which use their development splits; TriviaQA, which uses a deterministic seed-23 validation subsample with $n{=}500$ ; and PubMedQA, which uses the 100-example test portion of a 900/100 split derived from PQA-L

Appendix

Table 5: Downstream training and evaluation details. Stage 3 trains on a unified downstream mixture. PV4 and PMv2 receive additional benchmark-specific fine-tuning, while all other benchmarks use the multi-task checkpoint directly.

Setting	Mem0	MemoRAG
Implementation	mem0ai 0.1.118 with its stock prompts and ADD/UPDATE/DELETE/NONE loop, unmodified	Official repository with its prompts unchanged
Memory-writing model	Frozen Qwen3-4B-Instruct-2507 extracts facts and consolidates memories	Frozen Qwen3-4B-Instruct-2507 reads the collection as its memory
Memory-model decoding	Greedy, up to 2,048 tokens	Greedy, up to 128 tokens per clue call
Write path	One add() per document, with each extracted fact reconciled against its 5 nearest stored memories	Collection read once per entity and cached, front-truncated only where it exceeds the context window (NarrativeQA)
Memory scope	One Qdrant collection per user (PMv2, PV4), story (NarrativeQA) or question (others)	One memory per user, story or question
Retrieval query	The question	Clue spans and surrogate questions longer than 3 words, plus the question

Appendix

Table 6: Mem0 and MemoRAG configuration. Rows below the rule are shared with the RAG-

k

baselines, so both methods differ from RAG only in the context given to the decoder.

Dataset	Task	$\bar{\mu}$ docs/q	$\bar{\mu}$ tok/q	Train	Val	Test
PersonalizationV4 (PV4) (Ours)	Personalization SA	200	50,476	15,058	1,755	4,237
PersonaMem-v2 (PMv2) ( Jiang et al., 2025 )	Personalization SA	116	31,618	18,521	2,059	5,000
NarrativeQA (NQA) ( Kočiskỳ et al., 2018 )	Long-document generative QA	1,420	76,225	32,747	3,461	10,557
FactKG ( Kim et al., 2023 )	KG fact verification	129	1,665	86,367	13,266	9,041
TriviaQA ( Joshi et al., 2017 )	Open-domain factoid QA	164	11,861	138,384	17,944	—
PubMedQA ( Jin et al., 2019 )	Biomedical QA	4	324	900	—	100

Appendix

Table 7: Benchmark datasets overview. Each row corresponds to an evaluation benchmark. Each data point consists of a textual query paired with a collection of source documents from which the model must produce an answer or generation.

\bar{\mu}

docs/q and

\bar{\mu}

tok/q report the mean number of source documents and total source tokens per query in the test split.

Persona	Name Maya Patel. Age 34. Location Austin, Texas, USA. Occupation Certified Welder & Part-Time CNC Operator at a regional aerospace supplier.
	… Maya grew up in a multicultural household where her Indian-American parents emphasized discipline, education, and the importance of physical well-being. After completing a two-year technical diploma in welding technology, she entered the workforce in a traditionally male-dominated industry, where she quickly earned a reputation for precision, reliability, and an unflappable calm under pressure. Though she never pursued a university degree, her practical knowledge of metallurgy, robotics-aided fabrication, and safety protocols rivals that of many engineers. …
Categories	Aerospace welding expertise & mental rehearsal habits. Gaming genre preferences & playstyle strategies. Morning mango ritual & mobility routine. … (3 of 10)
Chat topic	Maya asks for a list of cooperative building-and-craft games she can play online with classmates after welding class.
User turn	… looking for coop building-and-craft games I can jump into with the welding class after tonight’s session. Need stuff that’s accessible for mixed skill levels, supports voice chat, and has meaningful base-building or automation mechanics. Toss me a short list with platform notes and why each one fits.
Assistant turn	(seven games with platform notes, from Minecraft to Raft) … Each supports voice via Discord/Steam, scales with mixed experience, and offers clear building goals for group sessions.
Question	In a new open-world crafting RPG, players can specialize in one mastery for unique bonuses. Considering Maya’s welding expertise, love for base-building, and ergonomic focus from the Iron Heart project, which specialization best aligns with her likely in-game identity?

Appendix

Table 8: One held-out PersonalizationV4 user through the pipeline. Excerpts are verbatim apart from abridgement. The question draws on base-building, which appears only in the dialogue shown, and on the Iron Heart project, which appears in 18 other dialogues, so answering requires combining documents. The distractors are never shown to the model and serve only as scoring candidates.

Questions	Each question must require synthesizing information or understanding nuance from the persona.
	Questions must require combining two or more facts from the persona profile. Avoid questions that can be answered by recalling a single fact.
	Embed each question in a specific, realistic scenario that requires applying persona knowledge to a novel situation rather than simple recall.
	The question stem must not hint at characteristics of the answer.
Distractors	Every choice (A-E) must be a reasonable, professional, or logical preference for a human. Never use obviously wrong, absurd, or incompetent distractors.
	Ensure distractors include industry-standard best practices that most people in the persona’s role would choose, but which this specific persona rejects.
	Use information that is true about the persona but applies it to the wrong situation or frequency.

Appendix

Table 9: Question-generation instructions. Verbatim excerpts from the prompt given to GPT-5.1 for each category, which also contains the full persona profile, the target category and the number of questions to write.

Benchmark	Docs/q	Fits 32K	RAG 1	RAG 5	RAG 50	Full Context	LLM Lingua-2	xRAG	Mem0	Memo RAG	Memorilla
PMv2	116	Partial	125	652	9,009	31,587	250	1	665	9,254	16
PV4	200	No	201	1,021	10,832	50,455	180	1	978	11,046	16
PubMedQA	4	Yes	69	321	324 †	324	290	1	8	324	16
NarrativeQA	1,420	Partial	69	354	3,612	67,385	596	1	493	800	16
TriviaQA	164	Partial	51	226	2,357	11,888	243	1	269	303	16
FactKG	129	Yes	13	64	568	1,727	279	1	300	573	16

Appendix

Table 10: Document tokens per query. Mean number of document-derived tokens in the decoder input, excluding the question and all shared instructions. Mem0, MemoRAG and LLMLingua-2 are counted at the setting reported for each benchmark’s primary metric in Table 11 , and truncated inputs are counted as read. Fits 32K reports whether every Full Context prompt fits the enforced 32,768-token limit, and Partial means that YaRN is enabled for the remaining examples. † marks benchmarks where RAG-50 reads the same chunks as Full Context, for every PubMedQA question and 96.7% of LaMP-7 questions.

Task	Metric	Closed book	RAG 1	RAG 5	RAG 50	Full Context	LLM Lingua-2	xRAG	Mem0	Memo RAG	Memorilla	Compression
PMv2	Acc (SA)	25.64	26.10	25.92	26.72	27.92	26.92	30.90	27.48	28.76	44.60	1,974 $\times$
PV4	Acc (SA)	58.30	62.21	64.98	65.78	59.12	52.14	62.64	65.07	67.88	66.65	3,153 $\times$
	Acc (SA, Hard)	31.96	38.00	39.27	41.97	36.57	—	34.98	44.20	45.95	41.97
PubMedQA	Acc	21.00	34.00	67.00	67.00	67.00	69.00	36.00	24.00	67.00	59.00	20 $\times$
NarrativeQA	BLEU-1	7.09	7.36	7.85	8.29	5.89	6.87	16.86	7.77	9.14	16.36	4,212 $\times$
	ROUGE-1	11.54	13.32	15.60	18.45	19.12	7.26	17.33	14.24	21.80	19.68

Appendix

Table 11: Main results across eight benchmarks. All methods use a frozen Qwen3-Embedding-4B encoder and a frozen Qwen3-8B decoder; Mem0 and MemoRAG also use a frozen Qwen3-4B-Instruct-2507 to build their memories. RAG-

k

retrieves the top-

k

chunks by cosine similarity and Full Context reads the full collection. Memorilla uses

K{=}16

memory tokens. Best result per row is bold , second best is underlined . The Compression column reports how many times more document tokens Full Context gives the decoder than the 16 memory tokens of Memorilla (Table 10 ). LLMLingua-2 was not evaluated on the PV4 hard subset.

Method	Tok.	LaMP-4	LaMP-7
		ROUGE-L	ROUGE-L
ComMer	128	17.50	45.00
Memorilla (4B)	16	17.72	47.04
Memorilla (8B)	16	18.15	48.37

Appendix

Table 12: Comparison with ComMer on personalized generation, against ComMer’s reported results. ComMer uses Gemma-2b while Memorilla uses Qwen3.

Method	Query aware	Params	PMv2 SA
Global mean-pool	no	6,566,400	28.94
Segmented mean-pool	no	6,566,400	29.68
Segmented, parameter-matched	no	190,149,120	31.10
Cosine top-16, parameter-matched	yes (hard)	190,149,120	33.86
Documents only	no	268,856,320	34.72
Q-Former	yes (soft)	268,856,320	43.34

Appendix

Table 13: Memory architecture ablation. All methods use the frozen Qwen3-Embedding-4B encoder and Qwen3-4B-Instruct-2507 decoder and share the training recipe, prompts, decoding and scorer. They are trained from scratch, so values are not comparable to Table 11 . The Q-Former variants use more parameters than Memorilla .

Figure 9: Memory-token capacity. All models use a Qwen3-4B-Instruct-2507 decoder and are trained from scratch under identical settings, differing only in the number of memory tokens $K$ ( $K{=}16$ is the setting used throughout).

Panel of Figure 10 — Figure 10: Memory token budget at inference. Performance of Memorilla with a frozen Qwen3-8B decoder using the first $K$ memory tokens of the checkpoints trained with the full $16$ tokens budget.

Figure 11: Memory versus comparable-budget adaptation. Memorilla with an 8B decoder and 194.1M trainable parameters against a no-document rank-130 LoRA adapter with a 4B decoder and 268.4M trainable parameters.

Figure 12: Encoder size. The memory module is trained from scratch with a frozen Qwen3-8B decoder and either encoder; all other settings are fixed.

Path	Quantity	$N{=}1$	$N{=}512$	Growth
Write	Document encoding	22.92 ms	2,588.84 ms	linear in $N$
Write	Storage per document	5 KB	5 KB	flat
Read	Memory module forward	2.01 ms	1.96 ms	$-0.06$ ms
Read	CPU to GPU transfer	0.014 ms	0.060 ms	$+0.046$ ms
Decoder	Context tokens	16	16	flat
	Memorilla end to end	740 ms	739 ms	$-1.9$ ms

Appendix

Table 14: Serving cost as the collection grows. One NVIDIA H100-80GB, batch size 1, bf16, greedy, max_new_tokens =32, 250-token documents; median of 5 timings after 2 warmups. The write path is paid once per document and is shared with dense retrieval; read path and decoder cost are paid per query. Full Context at

N{=}512

is 128,064 context tokens, past the enforced 32,768-token limit, so it uses YaRN (factor 4). Timings use HF transformers generate without a paged KV cache, so Full Context latency depends on the serving stack.

Panel of Figure 13 — Figure 13: An example TextWorld prompt. A winning 12-turn rollout on a held-out medium game, at the choice of the turn-6 action; training prompts have the same format. The eight memory vectors sit before the system prompt and are recomputed every turn from one document per completed turn, with the current observation as the query, while the full trajectory stays in the prompt. Line breaks inside the documents are added for width.

Component	Setting	Value
Environment	Task suite	TextWorld held-out games up to 65 rooms
	Splits	800 train prompts; 40-prompt validation and test splits (30 games each)
	Agent state	Initial description plus prior observations/actions
	Reward	TextWorld game score; no shaping reward
Model interface	Decoder	Frozen Qwen3-4B-Instruct-2507
	Trajectory encoder	Frozen Qwen3-Embedding-4B

Appendix

Table 15: TextWorld RL configuration. The memory module is the only trainable component, and the full textual history is retained in every condition for a fair comparison.

Memory-Augmented Language Models

Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.

See all 291 papers

Memori: A Persistent Memory Layer for Efficient, Context-Aware LLM Agents
Mar 20, 2026Luiz C. Borro, Luiz A. B. Macarini, Gordon Tindall +2Persistent Memory for Language Models LLM Memory
Memori Labs Inc.
RaMem: Contextual Reinstatement for Long-term Agentic Memory
Jun 22, 2026Wei Yang, Bryce Kan, Shixuan Li +5Agent Memory LLM Agent Memory
University of Southern California
LeanMem: Simple and Efficient Long-Term Memory for LLM Agents
Aug 4, 2026Yuxin Liao, Le Wu, Min Hou +3LLM Agent Memory Persistent Memory for Language Models
Hefei University of Technology

Mar 20, 2026J/K move · Enter open · S save

Memori: A Persistent Memory Layer for Efficient, Context-Aware LLM Agents

Luiz C. Borro, Luiz A. B. Macarini, Gordon Tindall, Michael Montero+1

Memori Labs Inc.

Persistent Memory for Language Models LLM Memory Memory-Augmented Language Model Agents Long-Term Conversational Memory

Open paper AskPDF arXiv

As large language models (LLMs) evolve into autonomous agents, persistent memory at the API layer is essential for enabling context-aware behavior across LLMs and multi-session interactions. Existing approaches force vendor lock-in and rely on injecting large volumes of raw conversation into prompts, leading to high token costs and degraded performance. We introduce Memori, an LLM-agnostic persistent memory layer that treats memory as a data structuring problem. Its Advanced Augmentation pipeline converts unstructured dialogue into compact semantic triples and conversation summaries, enabling precise retrieval and coherent reasoning. Evaluated on the LoCoMo benchmark, Memori achieves 81.95% accuracy, outperforming existing memory systems while using only 1,294 tokens per query (~5% of full context). This results in substantial cost reductions, including 67% fewer tokens than competing approaches and over 20x savings compared to full-context methods. These results show that effective memory in LLM agents depends on structured representations instead of larger context windows, enabling scalable and cost-efficient deployment.

Abstract

Memory-Augmented Language Models

Explore similar work

Memori: A Persistent Memory Layer for Efficient, Context-Aware LLM Agents

RaMem: Contextual Reinstatement for Long-term Agentic Memory

LeanMem: Simple and Efficient Long-Term Memory for LLM Agents