Memory enables large language models (LLMs) to use information beyond their parameters, such as past user conversations, long documents, or an agent's earlier actions. For a frozen LLM, access to this accumulated knowledge is limited to what a memory system places in its context window. However, gathering and combining relevant information needed for each query while keeping the model's context short remains challenging for existing systems. Here we introduce Memorilla, a memory system that gathers information from an entire document collection into a fixed number of query-dependent 'memory tokens' for a frozen LLM. Memorilla encodes documents (such as user chats, book chapters) once into reusable embeddings, then learns to retrieve and place query-specific information in the LLM's input embedding space. Using only 16 memory tokens, Memorilla achieves the best performance against baselines including Mem0 and MemoRAG on five of eight benchmarks spanning personalization, long-document and knowledge-intensive question answering. Notably, on FactKG, it reaches 84.33% accuracy versus 80.43% with full-context inference, while using 108x fewer tokens to represent the source documents. Furthermore, Memorilla can facilitate continual learning in agentic systems where the document count (the agent's past turns) increases throughout the trajectory. In a TextWorld environment, we show that Memorilla reduces the agent's tendency to wander between rooms it has already explored, and improves its average success rate from 18.6% with full-history in-context learning to 24.2%. Overall, Memorilla aids frozen LLMs in accumulating and using knowledge effectively across long-context interactions and takes a step towards building continually learning AI systems.
Figures & tables
Figure 1: Performance versus context length. Mean score over eight benchmarks against the geometric mean of document tokens given to a frozen Qwen3-8B decoder. Memorilla lies on the Pareto frontier (dashed) with the highest score at 16 memory tokens, 394 × fewer than full context.
Figure 2: Memorilla architecture. A frozen encoder embeds source documents (User Chats) once into a persistent memory store. For each Query, K query-conditioned slots attend over the store and are projected into the frozen LLM’s embedding space as a memory readout.
Figure 3: Per-benchmark performance versus context length. Primary-metric score against mean document tokens per query, with a frozen Qwen3-8B decoder. Memorilla lies on the Pareto frontier (dashed) of every benchmark, leads on five and is second on NarrativeQA and PV4.
Figure 4: Per-question breakdowns against RAG-50. (a) NarrativeQA question types are the question’s first word. Conceptual questions (‘Why’, ‘How’, ‘What’) favor Memorilla whereas entity-recall questions (‘Which’, ‘Who’) favor RAG. (b) Memorilla ’s advantage grows with entity count, indicating better capture of multi-entity relational patterns. (c) RAG-50 degrades as more facts are composed, while Memorilla remains stable across all depths.
Figure 5: Per-question complementarity between Memorilla and RAG-50. Share of questions answered by each method alone and by at least one of them (union), an oracle upper bound for a hybrid system. The union exceeds either method on every benchmark. † A question counts as answered when its ROUGE-L is at least max(0.20,m) , where m is Memorilla ’s median.
Figure 6: Personalization Case Study. On PersonaMem-v2, Memorilla personalizes a generic query by leveraging the user’s remembered iron-deficiency anemia history, despite no lexical overlap between the question and the relevant conversation history.
Figure 7: Reinforcement-learned memory on TextWorld. (a) A frozen encoder embeds the trajectory, the trainable memory module compresses it to K=8 latents, and the frozen LLM agent acts; sparse rewards over G=16 rollouts drive a GRPO update to the memory module alone. (b) On a held-out game, ICL loops between explored rooms while (c) Memorilla reaches the goal ( ⋆ ).
Figure 8: Memorilla on TextWorld with a frozen Qwen3-4B-Instruct-2507 decoder. Terminal episodes end through the game or the turn limit before the history exceeds the 4,096-token evaluation budget.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Source
Category
Num. Queries
Pretraining
EnWiki Articles
Paraphrase
1,979,809
Finetuning
SQuAD v2 ( Rajpurkar et al., 2018 )
Reading Comprehension
86,821
DROP ( Dua et al., 2019 )
Reading Comprehension
77,400
CoQA ( Reddy et al., 2019 )
Reading Comprehension
108,647
PubMedQA ( Jin et al., 2019 )
Reading Comprehension
900
QuAIL ( Rogers et al., 2020 )
Reading Comprehension
10,246
Appendix
Table 1: Training datasets for Memorilla . The pretraining corpus is derived from English Wikipedia. The finetuning mixture spans reading comprehension, open-domain question answering, and summarization tasks across 10 sources. Sizes are numbers of training queries.
Benchmark
Chunk unit
Tokens per chunk (mean / median)
PMv2
One conversation session
280 / 262
PV4
One chat dialogue
252 / 224
PubMedQA
One abstract section
84 / 78
TriviaQA
Evidence passage, split sentence-greedily at 128 tokens
70 / 77
NarrativeQA
Story paragraph, split on sentences above ∼ 256 tokens
94 / 35
FactKG
One knowledge-graph triple
13 / 13
Appendix
Table 2: Document chunks per benchmark. Each chunk di is embedded once by the frozen encoder. Token counts use the Qwen3 tokenizer.
Component
Setting
Value
Memory module
Pooling mode
Adaptive cross-attention with softmax over documents
Memory slots
K=16
Cross-attention layers
2 layers, slots ← documents
Attention heads
8
Encoder-side dimension
2560
Decoder-side dimension
2560 for 4B decoder, 4096 for 8B decoder
Appendix
Table 3: Model configuration for Memorilla . The memory module maps document representations into a fixed set of learned memory tokens that are inserted into the frozen decoder context.
Table 4: Training stages for Memorilla . Stage 1 pretrains memory on EnWiki paraphrase reconstruction, Stage 2 warm-starts on the mixed supervision corpus in Table 1 , and Stage 3 performs downstream multi-task fine-tuning.
Item
Configuration
Stage 1 corpus
1.98M EnWiki paraphrase reconstruction examples, chunked at approximately 128 tokens with mean 1.73 documents per row
Stage 2 corpus
Mixed supervision over SQuAD v2, DROP, CoQA, PubMedQA, QuAIL, PwC, CNN/DailyMail, SAMSum, DialogSum, and MS MARCO
Stage 3 corpus
Unified concatenation of PV4, PMv2, FactKG, TriviaQA, NarrativeQA, PubMedQA, LaMP-4, and LaMP-7
PV4 and PMv2 specialization
After multi-task fine-tuning, PV4 and PMv2 are each further fine-tuned for 5 epochs on the corresponding benchmark training split using the same recipe
Other downstream tasks
FactKG, TriviaQA, NarrativeQA, PubMedQA, LaMP-4, and LaMP-7 use the multi-task checkpoint directly with no additional task-specific training
Evaluation split
Official test split for every task except LaMP-4 and LaMP-7, which use their development splits; TriviaQA, which uses a deterministic seed-23 validation subsample with n=500 ; and PubMedQA, which uses the 100-example test portion of a 900/100 split derived from PQA-L
Appendix
Table 5: Downstream training and evaluation details. Stage 3 trains on a unified downstream mixture. PV4 and PMv2 receive additional benchmark-specific fine-tuning, while all other benchmarks use the multi-task checkpoint directly.
Setting
Mem0
MemoRAG
Implementation
mem0ai 0.1.118 with its stock prompts and ADD/UPDATE/DELETE/NONE loop, unmodified
Official repository with its prompts unchanged
Memory-writing model
Frozen Qwen3-4B-Instruct-2507 extracts facts and consolidates memories
Frozen Qwen3-4B-Instruct-2507 reads the collection as its memory
Memory-model decoding
Greedy, up to 2,048 tokens
Greedy, up to 128 tokens per clue call
Write path
One add() per document, with each extracted fact reconciled against its 5 nearest stored memories
Collection read once per entity and cached, front-truncated only where it exceeds the context window (NarrativeQA)
Memory scope
One Qdrant collection per user (PMv2, PV4), story (NarrativeQA) or question (others)
One memory per user, story or question
Retrieval query
The question
Clue spans and surrogate questions longer than 3 words, plus the question
Appendix
Table 6: Mem0 and MemoRAG configuration. Rows below the rule are shared with the RAG- k baselines, so both methods differ from RAG only in the context given to the decoder.
Dataset
Task
μˉ docs/q
μˉ tok/q
Train
Val
Test
PersonalizationV4 (PV4) (Ours)
Personalization SA
200
50,476
15,058
1,755
4,237
PersonaMem-v2 (PMv2) ( Jiang et al., 2025 )
Personalization SA
116
31,618
18,521
2,059
5,000
NarrativeQA (NQA) ( Kočiskỳ et al., 2018 )
Long-document generative QA
1,420
76,225
32,747
3,461
10,557
FactKG ( Kim et al., 2023 )
KG fact verification
129
1,665
86,367
13,266
9,041
TriviaQA ( Joshi et al., 2017 )
Open-domain factoid QA
164
11,861
138,384
17,944
—
PubMedQA ( Jin et al., 2019 )
Biomedical QA
4
324
900
—
100
Appendix
Table 7: Benchmark datasets overview. Each row corresponds to an evaluation benchmark. Each data point consists of a textual query paired with a collection of source documents from which the model must produce an answer or generation. μˉ docs/q and μˉ tok/q report the mean number of source documents and total source tokens per query in the test split.
Persona
Name Maya Patel. Age 34. Location Austin, Texas, USA. Occupation Certified Welder & Part-Time CNC Operator at a regional aerospace supplier.
… Maya grew up in a multicultural household where her Indian-American parents emphasized discipline, education, and the importance of physical well-being. After completing a two-year technical diploma in welding technology, she entered the workforce in a traditionally male-dominated industry, where she quickly earned a reputation for precision, reliability, and an unflappable calm under pressure. Though she never pursued a university degree, her practical knowledge of metallurgy, robotics-aided fabrication, and safety protocols rivals that of many engineers. …
Maya asks for a list of cooperative building-and-craft games she can play online with classmates after welding class.
User turn
… looking for coop building-and-craft games I can jump into with the welding class after tonight’s session. Need stuff that’s accessible for mixed skill levels, supports voice chat, and has meaningful base-building or automation mechanics. Toss me a short list with platform notes and why each one fits.
Assistant turn
(seven games with platform notes, from Minecraft to Raft) … Each supports voice via Discord/Steam, scales with mixed experience, and offers clear building goals for group sessions.
Question
In a new open-world crafting RPG, players can specialize in one mastery for unique bonuses. Considering Maya’s welding expertise, love for base-building, and ergonomic focus from the Iron Heart project, which specialization best aligns with her likely in-game identity?
Appendix
Table 8: One held-out PersonalizationV4 user through the pipeline. Excerpts are verbatim apart from abridgement. The question draws on base-building, which appears only in the dialogue shown, and on the Iron Heart project, which appears in 18 other dialogues, so answering requires combining documents. The distractors are never shown to the model and serve only as scoring candidates.
Questions
Each question must require synthesizing information or understanding nuance from the persona.
Questions must require combining two or more facts from the persona profile. Avoid questions that can be answered by recalling a single fact.
Embed each question in a specific, realistic scenario that requires applying persona knowledge to a novel situation rather than simple recall.
The question stem must not hint at characteristics of the answer.
Distractors
Every choice (A-E) must be a reasonable, professional, or logical preference for a human. Never use obviously wrong, absurd, or incompetent distractors.
Ensure distractors include industry-standard best practices that most people in the persona’s role would choose, but which this specific persona rejects.
Use information that is true about the persona but applies it to the wrong situation or frequency.
Appendix
Table 9: Question-generation instructions. Verbatim excerpts from the prompt given to GPT-5.1 for each category, which also contains the full persona profile, the target category and the number of questions to write.
Benchmark
Docs/q
Fits 32K
RAG 1
RAG 5
RAG 50
Full Context
LLM Lingua-2
xRAG
Mem0
Memo RAG
Memorilla
PMv2
116
Partial
125
652
9,009
31,587
250
1
665
9,254
16
PV4
200
No
201
1,021
10,832
50,455
180
1
978
11,046
16
PubMedQA
4
Yes
69
321
324 †
324
290
1
8
324
16
NarrativeQA
1,420
Partial
69
354
3,612
67,385
596
1
493
800
16
TriviaQA
164
Partial
51
226
2,357
11,888
243
1
269
303
16
FactKG
129
Yes
13
64
568
1,727
279
1
300
573
16
Appendix
Table 10: Document tokens per query. Mean number of document-derived tokens in the decoder input, excluding the question and all shared instructions. Mem0, MemoRAG and LLMLingua-2 are counted at the setting reported for each benchmark’s primary metric in Table 11 , and truncated inputs are counted as read. Fits 32K reports whether every Full Context prompt fits the enforced 32,768-token limit, and Partial means that YaRN is enabled for the remaining examples. † marks benchmarks where RAG-50 reads the same chunks as Full Context, for every PubMedQA question and 96.7% of LaMP-7 questions.
Task
Metric
Closed book
RAG 1
RAG 5
RAG 50
Full Context
LLM Lingua-2
xRAG
Mem0
Memo RAG
Memorilla
Compression
PMv2
Acc (SA)
25.64
26.10
25.92
26.72
27.92
26.92
30.90
27.48
28.76
44.60
1,974 ×
PV4
Acc (SA)
58.30
62.21
64.98
65.78
59.12
52.14
62.64
65.07
67.88
66.65
3,153 ×
Acc (SA, Hard)
31.96
38.00
39.27
41.97
36.57
—
34.98
44.20
45.95
41.97
PubMedQA
Acc
21.00
34.00
67.00
67.00
67.00
69.00
36.00
24.00
67.00
59.00
20 ×
NarrativeQA
BLEU-1
7.09
7.36
7.85
8.29
5.89
6.87
16.86
7.77
9.14
16.36
4,212 ×
ROUGE-1
11.54
13.32
15.60
18.45
19.12
7.26
17.33
14.24
21.80
19.68
Appendix
Table 11: Main results across eight benchmarks. All methods use a frozen Qwen3-Embedding-4B encoder and a frozen Qwen3-8B decoder; Mem0 and MemoRAG also use a frozen Qwen3-4B-Instruct-2507 to build their memories. RAG- k retrieves the top- k chunks by cosine similarity and Full Context reads the full collection. Memorilla uses K=16 memory tokens. Best result per row is bold , second best is underlined . The Compression column reports how many times more document tokens Full Context gives the decoder than the 16 memory tokens of Memorilla (Table 10 ). LLMLingua-2 was not evaluated on the PV4 hard subset.
Method
Tok.
LaMP-4
LaMP-7
ROUGE-L
ROUGE-L
ComMer
128
17.50
45.00
Memorilla (4B)
16
17.72
47.04
Memorilla (8B)
16
18.15
48.37
Appendix
Table 12: Comparison with ComMer on personalized generation, against ComMer’s reported results. ComMer uses Gemma-2b while Memorilla uses Qwen3.
Method
Query aware
Params
PMv2 SA
Global mean-pool
no
6,566,400
28.94
Segmented mean-pool
no
6,566,400
29.68
Segmented, parameter-matched
no
190,149,120
31.10
Cosine top-16, parameter-matched
yes (hard)
190,149,120
33.86
Documents only
no
268,856,320
34.72
Q-Former
yes (soft)
268,856,320
43.34
Appendix
Table 13: Memory architecture ablation. All methods use the frozen Qwen3-Embedding-4B encoder and Qwen3-4B-Instruct-2507 decoder and share the training recipe, prompts, decoding and scorer. They are trained from scratch, so values are not comparable to Table 11 . The Q-Former variants use more parameters than Memorilla .
Figure 9: Memory-token capacity. All models use a Qwen3-4B-Instruct-2507 decoder and are trained from scratch under identical settings, differing only in the number of memory tokens K ( K=16 is the setting used throughout).
Figure 10: Memory token budget at inference. Performance of Memorilla with a frozen Qwen3-8B decoder using the first K memory tokens of the checkpoints trained with the full 16 tokens budget.
Figure 11: Memory versus comparable-budget adaptation. Memorilla with an 8B decoder and 194.1M trainable parameters against a no-document rank-130 LoRA adapter with a 4B decoder and 268.4M trainable parameters.
Figure 12: Encoder size. The memory module is trained from scratch with a frozen Qwen3-8B decoder and either encoder; all other settings are fixed.
Path
Quantity
N=1
N=512
Growth
Write
Document encoding
22.92 ms
2,588.84 ms
linear in N
Storage per document
5 KB
5 KB
flat
Read
Memory module forward
2.01 ms
1.96 ms
−0.06 ms
CPU to GPU transfer
0.014 ms
0.060 ms
+0.046 ms
Decoder
Context tokens
16
16
flat
Memorilla end to end
740 ms
739 ms
−1.9 ms
Appendix
Table 14: Serving cost as the collection grows. One NVIDIA H100-80GB, batch size 1, bf16, greedy, max_new_tokens =32, 250-token documents; median of 5 timings after 2 warmups. The write path is paid once per document and is shared with dense retrieval; read path and decoder cost are paid per query. Full Context at N=512 is 128,064 context tokens, past the enforced 32,768-token limit, so it uses YaRN (factor 4). Timings use HF transformers generate without a paged KV cache, so Full Context latency depends on the serving stack.
Figure 13: An example TextWorld prompt. A winning 12-turn rollout on a held-out medium game, at the choice of the turn-6 action; training prompts have the same format. The eight memory vectors sit before the system prompt and are recomputed every turn from one document per completed turn, with the current observation as the query, while the full trajectory stays in the prompt. Line breaks inside the documents are added for width.
Component
Setting
Value
Environment
Task suite
TextWorld held-out games up to 65 rooms
Splits
800 train prompts; 40-prompt validation and test splits (30 games each)
Agent state
Initial description plus prior observations/actions
Reward
TextWorld game score; no shaping reward
Model interface
Decoder
Frozen Qwen3-4B-Instruct-2507
Trajectory encoder
Frozen Qwen3-Embedding-4B
Appendix
Table 15: TextWorld RL configuration. The memory module is the only trainable component, and the full textual history is retained in every condition for a fair comparison.
Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.
As large language models (LLMs) evolve into autonomous agents, persistent memory at the API layer is essential for enabling context-aware behavior across LLMs and multi-session interactions. Existing approaches force vendor lock-in and rely on injecting large volumes of raw conversation into prompts, leading to high token costs and degraded performance. We introduce Memori, an LLM-agnostic persistent memory layer that treats memory as a data structuring problem. Its Advanced Augmentation pipeline converts unstructured dialogue into compact semantic triples and conversation summaries, enabling precise retrieval and coherent reasoning. Evaluated on the LoCoMo benchmark, Memori achieves 81.95% accuracy, outperforming existing memory systems while using only 1,294 tokens per query (~5% of full context). This results in substantial cost reductions, including 67% fewer tokens than competing approaches and over 20x savings compared to full-context methods. These results show that effective memory in LLM agents depends on structured representations instead of larger context windows, enabling scalable and cost-efficient deployment.
Luiz C. Borro, Luiz A. B. Macarini, Gordon Tindall +2
Long-term memory has become increasingly important for LLM agents that operate across extended interactions and evolving task contexts. Recent memory systems have made past experiences more persistent, compact, and retrievable, but retrieval alone does not ensure that a memory provides valid evidence for the current query. When experiences are compressed into reusable fragments, memories from different situations may appear equally relevant if they involve recurring entities or user states. We refer to this failure as context collapse: memories lose the surrounding context needed to judge whether they provide valid evidence for the current query. To address this problem, we propose Contextual Reinstatement for Agentic Memory (RaMem), a framework that turns retrieved memory fragments into contextually verifiable evidence. RaMem operates through four coordinated stages: (i) evidence anchoring grounds each memory in its original episodic conditions, especially event time, mention time, session span, and participants; (ii) recall condition induction derives the evidence conditions implied by the query; (iii) validity-aware retrieval uses these conditions to prioritize context-compatible memories while retaining content-relevant candidates as fallback evidence; and (iv) context-preserved synthesis keeps the selected memories' structured context available to the generator. Experiments on long-term memory benchmarks show that RaMem consistently improves performance over strong memory baselines, with average F1 gains of more than 10% across several backbones.
Long-term memory is essential for LLM-based agents to sustain interactions and reliably leverage distant history. However, existing memory systems typically process heterogeneous dialogue content through a uniform summarization and retrieval pipeline, leading to either excessive token consumption or irreversible loss of fine-grained evidence. We argue that historical dialogue content should be handled differently according to its compressibility, temporal dynamics, and fidelity requirements. Based on this insight, we propose LeanMem, a lightweight long-term memory framework. LeanMem first filters out low-value content, then stores informative segments as compact profile memory, temporally structured event memory, or source-grounded record memory, depending on the nature of the information. During maintenance, only dynamically evolving event memories are selectively updated, avoiding redundant consolidation of stable profiles and immutable records. During inference, LeanMem dynamically selects memory types and allocates retrieval budgets according to query-specific evidence demands, assembling relevant evidence on demand. On LoCoMo and LongMemEval-S with GPT-4.1-mini and Qwen3-8B, LeanMem improves accuracy over the strongest memory-based baseline in every setting, by up to 15.1 points, at the lowest or near-lowest construction cost, inference tokens, and latency. The code and datasets are included in the supplementary materials.