Although LLM agents can leverage tools for complex tasks, they still need memory to maintain cross-turn consistency and accumulate reusable information in long-horizon interactions. However, retrieval-based external memory systems incur low online overhead but suffer from unstable accuracy due to limited query construction and candidate filtering. In contrast, many systems use repeated large-model calls for online memory operations, improving accuracy but accumulating latency over long interactions. We propose LightMem, a lightweight memory system for better agent memory driven by Small Language Models (SLMs). LightMem modularizes memory retrieval, writing, and long-term consolidation, and separates online processing from offline consolidation to enable efficient memory invocation under bounded compute. We organize memory into short-term memory (STM) for immediate conversational context, mid-term memory (MTM) for reusable interaction summaries, and long-term memory (LTM) for consolidated knowledge, and uses user identifiers to support independent retrieval and incremental maintenance in multi-user settings. Online, LightMem operates under a fixed retrieval budget and selects memories via a two-stage procedure: vector-based coarse retrieval followed by semantic consistency re-ranking. Offline, it abstracts reusable interaction evidence and incrementally integrates it into LTM. Experiments show consistent gains across model scales, with an average F1 improvement of about 2.5 over A-MEM on LoCoMo, while achieving higher efficiency and low median latency (83 ms for retrieval and 581 ms end-to-end).
Figures & tables
Figure 1: LightMem combines enhanced retrieval with SLMs, achieving high retrieval accuracy while significantly reducing online latency compared to retrieval-based and LLM-based memory systems.
Figure 2: Multiple SLMs coordinate an online pathway for query-time routing and retrieval over STM/MTM, and an offline pathway that incrementally consolidates MTM into a graph-structured LTM.
HQ prompt step
Example
Detect missing info
Identify underspecified references (e.g., pronouns such as “it/that/he/the project”), implicit context dependencies, and vague time cues (e.g., “recently/last time/before”).
Generate HQs
Rewrite the user request into one or more standalone HQs, optionally splitting intent into user-specific vs. factual queries, and making implied context (time/location) explicit.
Route and budget
Assign each HQ to MTM (user-specific) or LTM (general/public knowledge) and allocate a retrieval budget under a fixed top- K constraint.
Example
Input: “Recommend a dinner spot.” The prompt yields two HQs: one asks about the user’s preferred cuisines or dietary constraints (routed to MTM), and the other asks about highly rated nearby restaurants (routed to LTM).
Table 1: Structured HQ prompt used by SLM-1 for query decomposition and routing.
Model
Method
Single-hop
Multi-hop
Temporal
Open-domain
Adversarial
Token Length
F1
BLEU
F1
BLEU
F1
BLEU
F1
BLEU
F1
BLEU
GPT-4o-mini
LoCoMo
40.36
29.05
25.02
19.75
18.41
14.77
12.04
11.16
69.23
68.75
16,910
ReadAgent
9.67
7.66
9.15
6.48
12.60
8.87
5.31
5.12
9.81
9.02
643
MemoryBank
6.61
5.16
5.00
4.77
9.68
6.99
5.56
5.94
7.36
6.48
432
MemGPT
41.04
34.34
26.65
17.72
25.52
19.44
9.15
7.44
43.29
42.73
16,977
A-MEM
44.65
37.06
27.02
20.09
45.85
36.67
12.14
12.00
50.03
49.47
2,520
Table 2: Main results on LoCoMo across question categories. We report F1 and BLEU-1 for each category, along with the token length of the effective context. Best results within each model block are in bold.
Method
F1
BLEU-1
ROUGE-L
ROUGE-2
METEOR
SBERT
LoCoMo
2.55
3.13
2.75
0.90
1.64
15.76
MemGPT
1.18
1.07
0.96
0.42
0.95
8.54
A-MEM
3.45
3.37
3.54
3.60
2.05
19.51
LightMem
4.12
3.95
4.20
4.15
2.48
23.40
Table 3: Comparison of different memory mechanisms on DialSim with GPT-4o-mini.
Figure 3: Ablation study on DialSim. We report F1, BLEU-1, ROUGE-L, ROUGE-2, METEOR, and SBERT similarity.
Method
Retrieval Latency P50 (ms) ↓
Retrieval Latency P95 (ms) ↓
End-to-End P50 (ms) ↓
End-to-End P95 (ms) ↓
LoCoMo
0
0
2054
3658
ReadAgent
34
97
614
1027
MemoryBank
16
51
439
1047
MemGPT
143
451
2087
3451
A-MEM
856
1583
914
3682
LightMem
83
167
581
1325
Table 4: Scalability and latency analysis across five baselines on GPT-4o-mini.
MTM size
Vector retrieval
LightMem
Δ F1
∼ 100
3.95
3.98
0.03
∼ 1,000
3.90
4.05
0.15
∼ 5,000
3.86
4.09
0.23
10,000
3.83
4.12
0.29
Table 5: Performance stability under natural MTM growth on DialSim with Llama-3.2-1B. Statistics are computed cumulatively along the same full dialogue trajectory at different MTM sizes.
Group
Setting
F1
SBERT
A
Full LightMem
4.12
23.40
B
SLM-1 50% HQ noise
3.95
23.08
C
w/o SLM-2 reranking
3.83
22.56
D
SLM-3 50% write noise
3.78
22.45
E
Cascading failure
1.85
11.20
Table 6: Error injection stress test on DialSim under a fixed final Top- K memory budget.
Group
Setting
Multi-hop F1
A
Full LightMem (normal gap)
28.85
B
LTM-only retrieval
19.45
C
MTM-only retrieval
20.12
D
MTM noise saturation
23.10
Table 7: Update gap stress test on the LoCoMo multi-hop subset with GPT-4o-mini.
Category
Method
F1 (mean ± std)
Δ F1
95% CI of Δ
p-value
Multi-hop
A-MEM
27.02 ± 0.31
—
—
—
LightMem
28.85 ± 0.28
1.83
[ +1.24, +2.41 ]
0.001
Temporal
A-MEM
45.85 ± 0.37
—
—
—
LightMem
46.20 ± 0.34
0.35
[ +0.12, +0.58 ]
0.008
Table 8: Statistical significance results against A-MEM under the same backbone and retrieval budget.
Category
Type
Description and Example
Nodes
Entity
Concrete objects, locations, or named items extracted from interactions (e.g., “Python script”, “Paris”, “Project X”).
Concept
Abstract categories or generalized classes that characterize shared properties of entities (e.g., “Programming Language”, “Capital City”, “Urgent Task”).
Edges
IsA
Hierarchical membership relation between an entity and a concept (e.g., Paris \textscIsA Capital City ).
HasProperty
Links an entity to a specific attribute or state (e.g., Project X \textscHasProperty Completed ).
RelatedTo
Generic semantic association based on co-occurrence or functional relatedness (e.g., SNN \textscRelatedTo Neural Operator ).
Implies
Logical or causal dependency extracted from reasoning traces (e.g., High Density \textscImplies Congestion ).
Table 9: Node and edge types used in the LTM knowledge graph.
Metric
Value
Description and Analysis
Batch update interval
Every 10–15 turns
Offline consolidation is periodically triggered when MTM accumulates sufficient new entries or reaches capacity pressure, enabling amortized long-term updates without interfering with online processing.
Node growth rate
∼ 1 node / 4 turns
On average, only one new semantic node is added to LTM every four dialogue turns. This sparsity reflects the high compression ratio of LightMem , where noise is filtered and only high-value evidence is retained for persistent storage.
Offline processing time
∼ 3.5 s / batch
Average time for the offline LLM to perform abstraction, graph linkage, and structural updates. This latency occurs entirely on the offline path and is fully decoupled from online retrieval (average retrieval latency ∼ 83 ms), making it invisible to users.
Reasoning accuracy (F1)
4.12 (full) vs. 3.96 (no update)
The full LightMem system (4.12) outperforms a variant without offline consolidation (3.96), corresponding to an approximately 4% performance drop when LTM evolution is disabled. This result highlights the importance of sustained LTM growth and consolidation for long-horizon reasoning.
Table 10: LTM update frequency, consolidation cost, and ablation results on reasoning accuracy.
Long-term memory systems for large language models (LLMs) have gained popularity for extending reasoning capabilities across applications. Current memory systems rely on LLM agents to organize and consolidate memory, resulting in costly, inefficient write operations. To address this limitation, we propose MemFit, a long-term memory system for conversational agents that reduces the cost and latency of memory operations. Unlike existing systems that rely on expensive LLM calls for memory construction or discard surface-level details through compression, MemFit stores each turn verbatim in an append-only store with near-instantaneous, LLM-free insertion, indexing turns with segment summaries rather than replacing them. Additionally, MemFit uses an LLM-free, multi-path retrieval strategy that combines lexical and semantic signals with cross-encoder reranking over caption- augmented episodes in both textual and multimodal settings. Empirical results on three widely used benchmarks, LoCoMo, MemGallery, and LongMemEval-S, show that MemFit achieves state-of-the-art performance while reducing memory construction time and cost several-fold, providing a scalable and efficient solution for persistent agentic memory.
Mitchell Piehl, Muchao Ye
Department of Computer Science The University of Iowa
Long-term memory is essential for LLM-based agents to sustain interactions and reliably leverage distant history. However, existing memory systems typically process heterogeneous dialogue content through a uniform summarization and retrieval pipeline, leading to either excessive token consumption or irreversible loss of fine-grained evidence. We argue that historical dialogue content should be handled differently according to its compressibility, temporal dynamics, and fidelity requirements. Based on this insight, we propose LeanMem, a lightweight long-term memory framework. LeanMem first filters out low-value content, then stores informative segments as compact profile memory, temporally structured event memory, or source-grounded record memory, depending on the nature of the information. During maintenance, only dynamically evolving event memories are selectively updated, avoiding redundant consolidation of stable profiles and immutable records. During inference, LeanMem dynamically selects memory types and allocates retrieval budgets according to query-specific evidence demands, assembling relevant evidence on demand. On LoCoMo and LongMemEval-S with GPT-4.1-mini and Qwen3-8B, LeanMem improves accuracy over the strongest memory-based baseline in every setting, by up to 15.1 points, at the lowest or near-lowest construction cost, inference tokens, and latency. The code and datasets are included in the supplementary materials.
Long-term memory enables LLM agents to leverage past interactions, but dialogue histories quickly exceed the context window, forcing agents to retrieve relevant subsets at query time. Because useful evidence is sparse and scattered across verbose conversations, retrieval faces a fundamental tension: broadening recall improves coverage but floods downstream reasoning with noise, while compressing memories at write time eases retrieval but irreversibly discards details that future queries may need. We introduce LazyMem, which resolves this tension by deferring all memory construction to query time. Given a retrieved candidate pool, a lightweight model processes it in overlapping parallel windows, selectively retaining and compressing only query-relevant content. The model is trained with supervised fine-tuning followed by reinforcement learning, using a reward that jointly encourages the identification of relevant messages and the generation of compressions that are faithful to the source and useful for answering the query. On LongMemEval, LazyMem-4B achieves an LLM-judge accuracy of 0.85, outperforming the strongest non-oracle baseline while using only 213 answer-context memory tokens, 21.0 times fewer than the baseline. It further generalizes to LoCoMo without target-domain training and reduces mean latency relative to the prior query-time baseline. Code is available at https://github.com/allacnobug/LazyMem.
Jing Yu, Yibo Zhao, Jiaming Zhang +1
School of Data Science and Engineering, East China Normal University