KV-LTM: Persistent Per-Session KV Caches as Long-Term Memory for Personal Assistants
Abstract
Personal assistants accumulate months of conversations, but today's long-term memory stacks put derivation or selection between a question and its evidence. In a system-level comparison on LongMemEval, two commercial assistants scored well below the same model reading the history offline [1], consistent with, but not isolating, content loss between write and read. We propose KV-LTM: a memory tier storing each session's raw text plus its KV cache under a fixed reader model, answering by restoring cached sessions and appending the question. KV-LTM is complementary to, not a replacement for, retrieval and indexing. Text is the memory of record; the cache is a recompute-free prefill accelerator, degrading gracefully to text. We specify the design: storage record, budget-derived retention, append-only read path, and bounded aggregation. We size it analytically: across pinned BF16 full-attention (GQA) and latent-cached MLA configurations, per-token state spans approximately 31K-328K bytes (the pinned KDA+MLA hybrid differs: approximately 8.1K bytes/token plus a fixed per-session recurrent term), giving approximately 0.34-3.6 TB per user-year at a declared 30K-token/day scenario, against approximately 49 MB of raw text; per-query cost is modelled as regime-conditional transfer, attention, decode, aggregation, and compute. We define an evaluation protocol needing no persistent-cache implementation for logical QA evaluation: per-session prefill emulates valid cache reuse under the same model/input-state contract and ideal arithmetic, with named arms, controlled baselines, and a preregistered reader-model gate; a separate minimal real-cache replay tests numerical equivalence and measures the cost axis, with outcomes not assumed. KV-LTM trades storage for avoided re-prefill compute and write-time fidelity, within a declared retention budget.
Figures & tables
| Class | Mechanism | Keeps every token? | Status for KV-LTM |
|---|---|---|---|
| (i) Architectural footprint | GQA, MLA — smaller state per token by design [ 10 ] [ 11 ] | Yes (for that model) | On-thesis; exact for that model |
| (ii) Quantization | FP8/INT4 KV (KIVI, KVQuant) [ 12 ] | Yes | Numerically lossy ; must be validated per config |
| (iii) Token eviction | SnapKV, StreamingLLM — drops tokens; degrades in multi-turn reuse [ 13 ] [ 14 ] | No | Off-thesis as a memory mechanism |
| Hybrid native row | KDA recurrent layers (fixed-size state) + MLA layers (per-token state), e.g., Kimi Linear (Kimi-Linear-48B-A3B, config pinned in §5) [ 15 ] | Partially (recurrent state is fixed-size) | Separate accounting: sessions × S_recurrent + tokens × b_MLA (§5) |
| System | Paradigm | What it optimizes | Source |
|---|---|---|---|
| Mem0 | Extracted facts | Compact, fast personalization | [ 2 ] (vendor-authored paper, ECAI 2025) |
| Zep / Graphiti | Temporal knowledge graph | Changing relationships, current state | [ 3 ] (vendor preprint) |
| Letta / MemGPT | OS-style tiered memory | Self-managed paging between context and storage tiers | [ 16 ] |
| MemoryBank | Fact memory + forgetting curve | Human-like retention and decay | [ 17 ] |
| MemoryOS | Hierarchical OS-style store | Short/mid/long-term integration | [ 18 ] |
| MemPalace | Verbatim archive + spatial index | Full verbatim retention | [ 4 , project + independent analysis] |
| Paradigm | Exemplars (public primary) | What it optimizes | Originals retained? | Where content can be given up |
|---|---|---|---|---|
| RAG / vector indexing | RAG baselines evaluated in LoCoMo [ 5 ] | Read-time selectivity | Yes (configurations that keep raw chunks) | Query-time selection — evidence outside the retrieved subset never reaches the reader |
| Extracted facts | Mem0 [ 2 ] (vendor-authored paper, ECAI 2025) | Compact, fast personalization | No — facts replace the dialogue | Write-time derivation — wording, rationale, and uncaptured context do not survive into the store |
| Temporal graphs | Zep / Graphiti [ 3 ] (vendor preprint) | Changing relationships; current state | Yes — raw messages retained as episodes | Derivation + selection — the semantic entity/edge view is derived; superseded facts are marked invalid by timestamp, not deleted ; retrieval over the derived view mediates what reaches the reader |
| Hierarchical evidence stores | OpenViking [ 19 ] (vendor docs) | Progressive L0/L1/L2 loading; traceable retrieval | Yes (L2 originals) | Derived L0/L1 abstracts mediate access — selection among them decides what gets read in full |
| Verbatim archives | MemPalace [ 4 , project + independent analysis] | Full verbatim retention | Yes | Query-time selection via vector retrieval — headline recall claims are independently questioned in the cited analysis |
| OS-style tiered memory | Letta / MemGPT [ 16 ] ; MemoryOS [ 18 ] | Self-managed paging across memory tiers | May persist in a deep tier (documented for MemGPT: recall storage reached via agent-issued search) | Write-time derivation on eviction/consolidation — MemGPT: recursive summaries of evicted messages [ 16 ] ; MemoryOS: segmented-page organization with LLM-summarized segments [ 18 ] |
| Loss mode | Mechanism | Example | Number | Label |
|---|---|---|---|---|
| System-level QA gap (recursive summarization vs full conversation) | not isolated — system-level comparison; QA accuracy is not a retrieval-recall measurement | Zep paper’s DMR benchmark | recursive summarization 35.3% (MemGPT-reported) vs full conversation 94.4% (Zep-run); different GPT-4-Turbo snapshots | vendor preprint [ 3 ] |
| System-level QA gap (full context vs memory system vs RAG) | not isolated — system-level comparison; QA is not a candidate-recall metric; reported subset not independently verified (§8 pins its own) | Mem0 paper on LoCoMo | full-context J≈72.9 vs Mem0 66.9 vs best RAG ≈61 | vendor-authored paper (ECAI 2025) [ 2 ] |
| Write-time derivation (extraction) | controlled ablation — only the stored representation varies (model, retriever, reranker, judge fixed) | Fidelity-Before-Structure ablation | verbatim chunks vs extracted artifacts: LoCoMo cat 1–3, 43.9% vs 28.0% (699 Qs / 10 convs); LongMemEval-S 67.4% vs 45.4% (500 Qs) | preprint (single author, under review); self-comparison [ 32 , v4] |
| Summarization | constraint quantifier dropped (directly supported by the source) | [ 32 ] constraint probe (illustrative example: “use type hints everywhere”) | summarization 14.0% exact (19.0% recall); RAG 89.5% exact; quote-grounded artifacts 93.0% exact (97.5% recall) — 50 convs × 50 turns, 4 planted facts/conv, compression every 40 turns | preprint (single author, under review); self-comparison [ 32 , v4] |
| Compaction | fact loss per pass; constraint loss on cascade (directly supported by the source) | Knowledge Objects audit | 36.7× pass over 2,000 structured facts → 60% of stored facts irrecoverable; three cascading rounds → 54% of project constraints | preprint [ 33 ] |
| System-level QA gap (full context vs memory system, small scale) | not isolated — system-level comparison; setup-specific | ConvoMem | full context 70–82% vs Mem0 30–45% below 150 conversations | vendor-authored preprint (Salesforce) [ 34 ] |
| Field | Content |
|---|---|
| state | The session’s native cache state under the reader model: per-token K/V (or MLA latent) for attention layers, plus the fixed-size recurrent state for hybrid layers (§2.3, §5), plus any runtime auxiliary state required for replay — in a versioned state format |
| text | The raw session text — the memory of record |
| meta | Timestamps, session ID, model/config hash |
| serialization | Versioned schema: roles, timestamps, prompt template, tokenizer version, metadata schema version |
| text_digest / state_digest | Separate integrity hashes over the serialized text and the native state. Bad or missing state is rebuilt only from verified-intact source text; if source verification fails, the session is recovered from a known-good copy or marked unavailable/out-of-coverage — damaged text is never silently trusted (§4.4, §4.8) |
| Model (pinned config) | Class | Bytes/token (bf16) | TB per 1M tokens | Annual @30K tok/day* |
|---|---|---|---|---|
| MiniCPM3-4B (62 layers; d_c 256 + d_R 32) | MLA | 35,712 | 0.036 | ≈0.39 TB |
| DeepSeek-V2-Lite (27 layers; d_c 512 + d_R 64; MoE 16B/2.4B active) | MLA | 31,104 | 0.031 | ≈0.34 TB |
| Qwen2.5-7B (28 layers × 4 kv-heads × 128) | GQA | 57,344 | 0.057 | ≈0.63 TB |
| Llama-3.1-8B (32 layers × 8 × 128) | GQA | 131,072 | 0.131 | ≈1.44 TB |
| Llama-3.3-70B (80 layers × 8 × 128) | GQA | 327,680 | 0.328 | ≈3.59 TB |
| DeepSeek-V3-class (61 layers × 576) — published-config class assumption, not a pinned checkpoint | MLA | 70,272 | 0.070 | ≈0.77 TB bf16 / ≈0.39 TB ideal FP8** |
| Scenario | Coverage (declared assumption) | Tokens/day | Tokens/year |
|---|---|---|---|
| Low | 12% | ≈10,000 | 3.65M |
| Mid (anchor) | 35% | ≈30,000 | 10.95M |
| High | 100% | ≈85,500 | 31.2M |
| Model (class) | Low (3.65M tok) | Mid (10.95M tok) | High (31.2M tok) | Sensitivity |
|---|---|---|---|---|
| Llama-3.1-8B (GQA) | 0.48 TB | 1.44 TB | 4.09 TB | linear in coverage; achievable-N at a fixed budget scales inversely with coverage |
| Llama-3.3-70B (GQA) | 1.20 TB | 3.59 TB | 10.2 TB | same linearity |
| Qwen2.5-7B (GQA) | 0.21 TB | 0.63 TB | 1.79 TB | |
| DeepSeek-V3-class (MLA bf16) | 0.26 TB | 0.77 TB | 2.19 TB | ideal-FP8 halves the quantized payload only (before scales/metadata; lossy — validate) |
| DeepSeek-V2-Lite (MLA) | 0.11 TB | 0.34 TB | 0.97 TB | |
| MiniCPM3-4B (MLA) | 0.13 TB | 0.39 TB | 1.11 TB |
| Model | TB/yr (mid) | Achievable-N within 512 GB (upper estimate) |
|---|---|---|
| Llama-3.1-8B | 1.44 | ≈4.3 months |
| DeepSeek-V3-class (bf16) | 0.77 | ≈8.0 months |
| MiniCPM3-4B | 0.39 | ≈16 months |
| DeepSeek-V2-Lite | 0.34 | ≈18 months |
| Kimi-Linear (hybrid) | 0.088 + sessions×S_rec | formula below; 5.8 years is the optimistic MLA-only byte ceiling (S_rec → 0), not attainable full-hybrid retention |
| Config (BF16) | Weight budget | State per read | H200 (141 GB) | B200 (180 GB) | B300 (288 GB) |
|---|---|---|---|---|---|
| MiniCPM3-4B (MLA) | 8.15 GB | ≈78 MB | ≈1,500 | ≈2,000 | ≈3,200 |
| DeepSeek-V2-Lite (MLA, MoE; all experts resident) | 31.4 GB | ≈68 MB | ≈1,400 | ≈1,900 | ≈3,300 |
| Qwen2.5-7B (GQA) | 15.2 GB | ≈126 MB | ≈890 | ≈1,170 | ≈1,940 |
| Llama-3.1-8B (GQA) | 16.1 GB | ≈287 MB | ≈385 | ≈510 | ≈850 |
| Llama-3.3-70B (GQA) | 141.1 GB | ≈718 MB | 0 (no fit) | ≈30 | ≈165 |
| Arm | Definition | Role |
|---|---|---|
| A | Per-session prefill + appended question over ALL retained sessions → per-session outputs → aggregator (§4.6) + joint tier (§4.7) | The KV-LTM logical read path, emulated without an implementation |
| A_k ˆ pre / A_k ˆ probe | The frozen hybrid variants (§4.5): pre-load prefilter → top-k → read, resp. 1-token probe → top-k → read | Optional selected variants, reported as separate cost rows — the prefilter saves cold I/O, the probe saves decode; fusing them would conflate two mechanisms |
| C | Full-concat long-context baseline: the common retained history in one context (a whole-original-history variant is separate and labelled) | A different system (joint attention, lost-in-the-middle exposure) — not a bound. C uses the gate-selected reader where each full serialized input fits its effective context (prefix + history + question + output reserve; otherwise N/A with coverage stated); a named long-context model only as a separately labelled C variant |
| O | Oracle: per-session reading on gold-evidence sessions only (same aggregator and joint rules) | Diagnostic. Retained gold = original benchmark gold IDs ∩ the common retained archive; verified absent-evidence questions get the empty-read expected-abstention treatment; missing gold annotations mark O N/A (the A−O contrast is then reported unavailable); evidence removed by retention is out-of-retention coverage, not benchmark unanswerability; partial gold coverage is reported for multi-evidence items |
| R | Controlled retrieval baseline with the same reader and judge: headline R_session (top-k sessions , matching our unit) plus R_chunk reported separately; retrieval unit, k, index build, and any embedder frozen on dev data before runs. R_session’s final-answer path (joint concatenation of retrieved sessions vs per-session reading + aggregation, with the context-fit rule) is stated per run — or declared deliberately equal to A_kˆpre’s selection for parity | Required for the headline comparison |
| D | Published external scores (Mem0, Zep, …) | Context only — models, judges, and prompts differ; never the controlled headline |
| Tier | Benchmark | Measures | Feasibility notes |
|---|---|---|---|
| Tier-1 (run) | LongMemEval-S (+M for scale) [ 1 ] | 5 abilities over chat histories (S ≈ 115K tokens over ~30–50 sessions; M ≈ 1.5M tokens over ~500 sessions) | Arms A/A_k/O/R feasible where each full serialized input (prefix + session + question + output reserve) fits the reader’s effective context — routine at per-session scale for the candidate readers; C is N/A wherever the retained history exceeds effective context (a nominal ≥128K class does not establish that every LME-S item fits; per-item coverage stated); O has gold session IDs |
| Tier-1 | LoCoMo [ 5 ] | Single/multi-hop, temporal, open-domain, adversarial QA over ≤35 sessions | Joint-evidence class derived per question from gold evidence IDs (sessions spanned); gold IDs available (O feasible); scoring exclusion and abstention analysis share the same adversarial items |
| Tier-1 | PersonaMem [ 41 ] | Dynamic persona memory, ≤60 sessions, 32K/128K/1M versions | Only the 32K version is a C candidate on a 32K-class reader — and only where prefix + history + question + output reserve fit its effective context |
| Tier-1 | MemoryAgentBench [ 42 ] | Incremental injection; retrieval, test-time learning, conflict resolution | Broader agentic; injection-then-query matches the §4.3 write path |
| Tier-1 | BEAM [ 38 ] | 10 abilities at 100K–10M tokens (incl. contradiction, event ordering) | Scale frontier; arm C N/A at 1M/10M for 32K-class readers — coverage stated |
| Tier-1 | ConvoMem [ 34 ] (vendor-authored preprint, Salesforce) | Full-context-vs-memory crossover by conversation count | Setup-specific; used for the retention-window question |
| Failure mode | Mechanism | Where handled |
|---|---|---|
| Cross-session synthesis limit | Per-session reads cannot jointly attend; answers needing joint evidence depend on the aggregator and the joint tier | Exercised on per-question-annotated joint-evidence questions (§8.3, §8.6), with joint-fallback limits reported as their own rows (§8.7); the §8.2 contrasts cover aggregator, distractor, and evidence-format sensitivity (§4.6, §4.7) |
| Scan cost at year scale | The exhaustive reference path would read the whole retained state per query — an analytic cache-equivalent ≈1.44 TB at 8B-GQA (mid scenario, one full year retained, all-cold; §7; nothing measured, §8.7) — which is why the prefilter variant exists for this regime | §4.5, §7.1–§7.2 |
| Lost-in-the-middle inside long cached sessions | Evidence in the middle of a long session is used less reliably even when present [ 52 ] — KV-LTM inherits this per session | §5.2 session-length caps; §8.5 reader gate |
| Reader long-context limits | Claimed context ≠ effective context: many models’ effective context falls below their claimed context in controlled probes [ 39 ] [ 40 ] | §8.5 gate (diagnostics labelled as such) |
| Session mis-bounding | Sessions are the retention, cache, and retrieval unit; mis-bounded sessions can affect all three. A labelled mis-bounding-rate sensitivity sweep (accuracy vs mis-bounding rate) is planned, not implemented — no bare hypothetical rate is asserted | §9.3 next steps (planned sweep); §9.2 session-identification dependency |
| Quantization-induced degradation | KV quantization is numerically lossy and must be validated per config before counting its savings [ 12 ] | §2.3 class (ii), §5.2 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Substitution | Bytes/token | Pin status |
|---|---|---|---|
| MiniCPM3-4B | 62 × (256 + 32) × 2 | 35,712 | [config] [ 11 ] (HF config.json) |
| DeepSeek-V2-Lite | 27 × (512 + 64) × 2 | 31,104 | [config] [ 11 ] (HF config.json) |
| Qwen2.5-7B | 2 × 28 × 4 × 128 × 2 | 57,344 | [config] HF config.json, retrieved 2026-09-26 [ 11 ] |
| Llama-3.1-8B | 2 × 32 × 8 × 128 × 2 | 131,072 | [config] HF config.json, retrieved 2026-09-26 [ 11 ] |
| Llama-3.3-70B | 2 × 80 × 8 × 128 × 2 | 327,680 | [config] HF config.json, retrieved 2026-09-26 [ 11 ] |
| DeepSeek-V3-class MLA | 61 × (512 + 64) × 2 | 70,272 (bf16); 35,136 (ideal FP8, payload only, before scales/metadata) [assumption: ideal halving] | [assumption: published V3-class config — representative, not a pinned checkpoint] |
| Model | computation | TB/yr |
|---|---|---|
| Llama-3.1-8B | 10.95e6 × 131,072 = 1.435e12 | 1.44 |
| Llama-3.3-70B | 10.95e6 × 327,680 = 3.588e12 | 3.59 |
| Qwen2.5-7B | 10.95e6 × 57,344 = 6.279e11 | 0.63 |
| DeepSeek-V3-class | 10.95e6 × 70,272 = 7.69e11 | 0.77 (bf16) / 0.385 (ideal FP8, payload only) |
| MiniCPM3-4B | 10.95e6 × 35,712 = 3.91e11 | 0.39 |
| DeepSeek-V2-Lite | 10.95e6 × 31,104 = 3.41e11 | 0.34 |
Persistent Memory for Language Models
Persistent memory for language models studies how models retain, update, and use information across interactions and beyond their context windows.
See all 282 papers