Although LLM agents can leverage tools for complex tasks, they still need memory to maintain cross-turn consistency and accumulate reusable information in long-horizon interactions. However, retrieval-based external memory systems incur low online overhead but suffer from unstable accuracy due to limited query construction and candidate filtering. In contrast, many systems use repeated large-model calls for online memory operations, improving accuracy but accumulating latency over long interactions. We propose LightMem, a lightweight memory system for better agent memory driven by Small Language Models (SLMs). LightMem modularizes memory retrieval, writing, and long-term consolidation, and separates online processing from offline consolidation to enable efficient memory invocation under bounded compute. We organize memory into short-term memory (STM) for immediate conversational context, mid-term memory (MTM) for reusable interaction summaries, and long-term memory (LTM) for consolidated knowledge, and uses user identifiers to support independent retrieval and incremental maintenance in multi-user settings. Online, LightMem operates under a fixed retrieval budget and selects memories via a two-stage procedure: vector-based coarse retrieval followed by semantic consistency re-ranking. Offline, it abstracts reusable interaction evidence and incrementally integrates it into LTM. Experiments show consistent gains across model scales, with an average F1 improvement of about 2.5 over A-MEM on LoCoMo, while achieving higher efficiency and low median latency (83 ms for retrieval and 581 ms end-to-end).
Figures & tables
Figure 1: LightMem combines enhanced retrieval with SLMs, achieving high retrieval accuracy while significantly reducing online latency compared to retrieval-based and LLM-based memory systems.
Figure 2: Multiple SLMs coordinate an online pathway for query-time routing and retrieval over STM/MTM, and an offline pathway that incrementally consolidates MTM into a graph-structured LTM.
HQ prompt step
Example
Detect missing info
Identify underspecified references (e.g., pronouns such as “it/that/he/the project”), implicit context dependencies, and vague time cues (e.g., “recently/last time/before”).
Generate HQs
Rewrite the user request into one or more standalone HQs, optionally splitting intent into user-specific vs. factual queries, and making implied context (time/location) explicit.
Route and budget
Assign each HQ to MTM (user-specific) or LTM (general/public knowledge) and allocate a retrieval budget under a fixed top- K constraint.
Example
Input: “Recommend a dinner spot.” The prompt yields two HQs: one asks about the user’s preferred cuisines or dietary constraints (routed to MTM), and the other asks about highly rated nearby restaurants (routed to LTM).
Table 1: Structured HQ prompt used by SLM-1 for query decomposition and routing.
Model
Method
Single-hop
Multi-hop
Temporal
Open-domain
Adversarial
Token Length
F1
BLEU
F1
BLEU
F1
BLEU
F1
BLEU
F1
BLEU
GPT-4o-mini
LoCoMo
40.36
29.05
25.02
19.75
18.41
14.77
12.04
11.16
69.23
68.75
16,910
ReadAgent
9.67
7.66
9.15
6.48
12.60
8.87
5.31
5.12
9.81
9.02
643
MemoryBank
6.61
5.16
5.00
4.77
9.68
6.99
5.56
5.94
7.36
6.48
432
MemGPT
41.04
34.34
26.65
17.72
25.52
19.44
9.15
7.44
43.29
42.73
16,977
A-MEM
44.65
37.06
27.02
20.09
45.85
36.67
12.14
12.00
50.03
49.47
2,520
Table 2: Main results on LoCoMo across question categories. We report F1 and BLEU-1 for each category, along with the token length of the effective context. Best results within each model block are in bold.
Method
F1
BLEU-1
ROUGE-L
ROUGE-2
METEOR
SBERT
LoCoMo
2.55
3.13
2.75
0.90
1.64
15.76
MemGPT
1.18
1.07
0.96
0.42
0.95
8.54
A-MEM
3.45
3.37
3.54
3.60
2.05
19.51
LightMem
4.12
3.95
4.20
4.15
2.48
23.40
Table 3: Comparison of different memory mechanisms on DialSim with GPT-4o-mini.
Figure 3: Ablation study on DialSim. We report F1, BLEU-1, ROUGE-L, ROUGE-2, METEOR, and SBERT similarity.
Method
Retrieval Latency P50 (ms) ↓
Retrieval Latency P95 (ms) ↓
End-to-End P50 (ms) ↓
End-to-End P95 (ms) ↓
LoCoMo
0
0
2054
3658
ReadAgent
34
97
614
1027
MemoryBank
16
51
439
1047
MemGPT
143
451
2087
3451
A-MEM
856
1583
914
3682
LightMem
83
167
581
1325
Table 4: Scalability and latency analysis across five baselines on GPT-4o-mini.
MTM size
Vector retrieval
LightMem
Δ F1
∼ 100
3.95
3.98
0.03
∼ 1,000
3.90
4.05
0.15
∼ 5,000
3.86
4.09
0.23
10,000
3.83
4.12
0.29
Table 5: Performance stability under natural MTM growth on DialSim with Llama-3.2-1B. Statistics are computed cumulatively along the same full dialogue trajectory at different MTM sizes.
Group
Setting
F1
SBERT
A
Full LightMem
4.12
23.40
B
SLM-1 50% HQ noise
3.95
23.08
C
w/o SLM-2 reranking
3.83
22.56
D
SLM-3 50% write noise
3.78
22.45
E
Cascading failure
1.85
11.20
Table 6: Error injection stress test on DialSim under a fixed final Top- K memory budget.
Group
Setting
Multi-hop F1
A
Full LightMem (normal gap)
28.85
B
LTM-only retrieval
19.45
C
MTM-only retrieval
20.12
D
MTM noise saturation
23.10
Table 7: Update gap stress test on the LoCoMo multi-hop subset with GPT-4o-mini.
Category
Method
F1 (mean ± std)
Δ F1
95% CI of Δ
p-value
Multi-hop
A-MEM
27.02 ± 0.31
—
—
—
LightMem
28.85 ± 0.28
1.83
[ +1.24, +2.41 ]
0.001
Temporal
A-MEM
45.85 ± 0.37
—
—
—
LightMem
46.20 ± 0.34
0.35
[ +0.12, +0.58 ]
0.008
Table 8: Statistical significance results against A-MEM under the same backbone and retrieval budget.
Category
Type
Description and Example
Nodes
Entity
Concrete objects, locations, or named items extracted from interactions (e.g., “Python script”, “Paris”, “Project X”).
Concept
Abstract categories or generalized classes that characterize shared properties of entities (e.g., “Programming Language”, “Capital City”, “Urgent Task”).
Edges
IsA
Hierarchical membership relation between an entity and a concept (e.g., Paris \textscIsA Capital City ).
HasProperty
Links an entity to a specific attribute or state (e.g., Project X \textscHasProperty Completed ).
RelatedTo
Generic semantic association based on co-occurrence or functional relatedness (e.g., SNN \textscRelatedTo Neural Operator ).
Implies
Logical or causal dependency extracted from reasoning traces (e.g., High Density \textscImplies Congestion ).
Table 9: Node and edge types used in the LTM knowledge graph.
Metric
Value
Description and Analysis
Batch update interval
Every 10–15 turns
Offline consolidation is periodically triggered when MTM accumulates sufficient new entries or reaches capacity pressure, enabling amortized long-term updates without interfering with online processing.
Node growth rate
∼ 1 node / 4 turns
On average, only one new semantic node is added to LTM every four dialogue turns. This sparsity reflects the high compression ratio of LightMem , where noise is filtered and only high-value evidence is retained for persistent storage.
Offline processing time
∼ 3.5 s / batch
Average time for the offline LLM to perform abstraction, graph linkage, and structural updates. This latency occurs entirely on the offline path and is fully decoupled from online retrieval (average retrieval latency ∼ 83 ms), making it invisible to users.
Reasoning accuracy (F1)
4.12 (full) vs. 3.96 (no update)
The full LightMem system (4.12) outperforms a variant without offline consolidation (3.96), corresponding to an approximately 4% performance drop when LTM evolution is disabled. This result highlights the importance of sustained LTM growth and consolidation for long-horizon reasoning.
Table 10: LTM update frequency, consolidation cost, and ablation results on reasoning accuracy.