MemX: A Local-First Long-Term Memory System for AI Assistants
Abstract
We present MemX, a local-first long-term memory system for AI assistants with stability-oriented retrieval design. MemX is implemented in Rust on top of libSQL and an OpenAI-compatible embedding API, providing persistent, searchable, and explainable memory for conversational agents. Its retrieval pipeline applies vector recall, keyword recall, Reciprocal Rank Fusion (RRF), four-factor re-ranking, and a low-confidence rejection rule that suppresses spurious recalls when no answer exists in the memory store. We evaluate MemX on two axes. First, two custom Chinese-language benchmark suites (43 queries, <=1,014 records) validate pipeline design: Hit@1=91.3% on a default scenario and 100% under high confusion, with conservative miss-query suppression. Second, the LongMemEval benchmark (500 queries, up to 220,349 records) quantifies system boundaries across four ability types and three storage granularities. At fact-level granularity the system reaches Hit@5=51.6% and MRR=0.380, doubling session-level performance, while temporal and multi-session reasoning remain challenging (<=43.6% Hit@5). FTS5 full-text indexing reduces keyword search latency by 1,100x at 100k-record scale, keeping end-to-end search under 90 ms. Unlike Mem0 and related work that targets end-to-end agent benchmarks, MemX focuses on a narrower, reproducible baseline: local-first deployment, structural simplicity, explainable retrieval, and stability-oriented design.
Figures & tables
| Factor | Signal source | Weight |
|---|---|---|
| Semantic similarity (from RRF score) | ||
| Recency (prefer last_retrieved_at , | ||
| fallback to created_at ) | ||
| Frequency (prefer retrieval_count , | ||
| fallback to access_count ) | ||
| Importance (explicit annotation) |
| Metric | Description |
|---|---|
| Hit@ ( ) | Fraction of relevant queries where at least one correct memory appears in the top- results. |
| Topic Coverage@ | For multi-topic queries, the fraction of expected topics covered in the top- results. |
| MRR | Mean Reciprocal Rank over relevant queries. |
| Miss-Empty-Rate | Fraction of miss queries (no correct answer exists) for which the system returns an empty result set. |
| Miss-Strict-Rate | Fraction of miss queries for which the top vector score stays below the miss threshold (more stringent than Miss-Empty-Rate). |
| Avg/P95 Search (ms) | Average and 95th-percentile end-to-end search latency. |
| Metric | Default | High-Confusion |
|---|---|---|
| Records | 1,014 | 600 |
| Queries (rel / miss) | 23 / 2 | 14 / 4 |
| Hit@1 | 91.3% [73, 98] | 100.0% [78, 100] |
| Hit@5 | 95.7% [79, 99] | 100.0% [78, 100] |
| Coverage@5 | 91.3% | 75.0% |
| MRR | 0.935 | 1.000 |
| Avg Hit@1 | Avg Miss-Empty | Avg Miss-Strict | Note | |
|---|---|---|---|---|
| 0.48 | 95.7% | 62.5% | 87.5% | Tied with 0.50 |
| 0.50 | 95.7% | 62.5% | 87.5% | Recommended |
| 0.52 | 93.5% | 62.5% | 87.5% | Begins to hurt recall |
| 0.64 | 87.0% | 100.0% | 87.5% | Significantly hurts recall |
| Granularity | Records | Hit@1 | Hit@5 | MRR | TC@5 |
|---|---|---|---|---|---|
| Session | 19,195 | 14.8% | 24.6% | 0.183 | 19.5% |
| Round | 100,486 | 17.2% | 27.0% | 0.207 | 21.0% |
| Fact | 220,349 | 29.8% | 51.6% | 0.380 | 41.5% |
| Session | Round | Fact | |||||
|---|---|---|---|---|---|---|---|
| Ability type | Hit@5 | MRR | Hit@5 | MRR | Hit@5 | MRR | |
| Information extraction | 156 | 36.5 | .298 | 39.7 | .324 | 55.8 | .379 |
| Knowledge update | 78 | 30.8 | .218 | 44.9 | .360 | 75.6 | .624 |
| Multi-session reasoning | 133 | 18.0 | .117 | 15.0 | .110 | 43.6 | .323 |
| Temporal reasoning | 133 | 13.5 | .093 | 13.5 | .078 | 40.6 | .296 |
| Config | Hit@1 | Hit@5 | MRR | TC@5 | Avg ms |
|---|---|---|---|---|---|
| V (vector-only) | 30.2% | 56.6% | 0.406 | 44.2% | 89.6 |
| Full pipeline | 29.8% | 51.6% | 0.380 | 41.5% | 88.2 |
| Scenario | Records | Vector | Keyword | Total | P95 |
| Custom scenarios (Section 5.2 ) | |||||
| Default | 1,014 | 545 | 2.2 | 538 | 679 |
| High-Confusion | 600 | 389 | 1.6 | 398 | 460 |
| LongMemEval (Section 5.4 ) | |||||
| Session (LIKE) | 19,195 | 98.6 | 286.7 | 385 | 1,418 |
| Session (FTS5) | 19,195 | 15.8 | 1.6 | 17.3 | 20 |
| Config | Hit@1 | Hit@3 | TC@ | Miss-Empty | Miss-Strict |
|---|---|---|---|---|---|
| V | 94.6 [82, 99] | 94.6 [82, 99] | 81.0 | 0.0 [0, 39] | 83.3 [44, 97] |
| V+K | 94.6 [82, 99] | 94.6 [82, 99] | 81.0 | 0.0 [0, 39] | 83.3 [44, 97] |
| V+K+Rej | 94.6 [82, 99] | 94.6 [82, 99] | 81.0 | 66.7 [30, 90] | 83.3 [44, 97] |
| Full | 94.6 [82, 99] | 97.3 [86, 100] | 83.2 | 66.7 [30, 90] | 83.3 [44, 97] |
| All values in %. Pooled over relevant and miss queries. Wilson 95% CI in brackets. | |||||
| Rule | Condition to reject | Intuition |
|---|---|---|
| R1 | Current system: reject only when both signals are weak. | |
| R2 | Vector-only gate; ignores keyword signal. | |
| R3 | Keyword-required gate; ignores vector signal. | |
| R4 | Strict dual gate: reject if either signal is weak. | |
| R5 | Same as R1 but with a raised threshold ( ). |
| R1 | R2 | R3 | R4 | R5 | |
| job_lookup (0.504, kw) | ✓ | ✓ | × | × | × |
| incident_kw (0.490, kw) | ✓ | × | ✓ | × | ✓ |
| hard_miss (0.453, kw) | × | × | × | × | × |
| cloud_region_miss (0.621, kw) | ✓ ∗ | ✓ ∗ | × | × | ✓ ∗ |
| FN (of 21 valid) | 0 | 1 | 18 | 19 | 1 |
| FP (of 3 miss) | 1 | 1 | 1 | 0 | 1 |
| Profile | Access | Retrieval | Last accessed | Last retrieved | |
|---|---|---|---|---|---|
| A | Admin-heavy | 50 | 2 | 1 d ago | 15 d ago |
| B | Search-heavy | 3 | 25 | 30 d ago | 1 d ago |
| C | Stale | 40 | 40 | 60 d ago | 60 d ago |
| Strategy 1 (retrieval) | Strategy 2 (access) | |||||
|---|---|---|---|---|---|---|
| Score | Score | |||||
| A | 0.707 | 0.110 | 0.577 | 0.977 | 0.393 | 0.659 |
| B | 0.977 | 0.326 | 0.656 | 0.500 | 0.139 | 0.527 |
| C | 0.250 | 0.371 | 0.476 | 0.250 | 0.371 | 0.476 |
| Ranking | B A C | A B C | ||||
Persistent Memory for Language Models
Persistent memory for language models studies how models retain, update, and use information across interactions and beyond their context windows.
See all 271 papers