Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligence

Souhail Bakkali; Ayoub Merimi

cs.CVSep 25, 2026

Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligence

Authors: Souhail Bakkali, Ayoub Merimi

Organizations: University of Rennes, CNRS, IRISA — UMR 6074 — 263 Av. G´en´eral Leclerc, Rennes, France · ENSA Berrechid, Hassan First University, Settat, Morocco

Abstract

Modern Document AI reads contracts, extracts fields, reasons over tables, and grounds answers to page regions, then forgets everything. Processing an amendment the next day begins from scratch: no schema retained, no contradiction detected, no experience carried forward. This is a structural choice, not a scale failure: current systems are stateless functions. We call this the statelessness bottleneck. This bottleneck lies beyond parameter scaling, context extension, and retrieval augmentation: storage provides persistence and retrieval provides access, but neither consolidates observations into knowledge that improves future processing. This survey formalizes persistent evidence-grounded document state as a unifying framework, specifying the operations and invariants required to convert multimodal evidence into durable, provenance-linked state. We introduce a statefulness audit showing that ten representative benchmarks, coded against eight statefulness criteria, leave cross-session state evolution untested, and derive a longitudinal benchmark harness with five counterfactual metrics: Experience Gain, Cost Efficiency, Memory Harm, Forgetting Fidelity, Coverage Retention, to characterize the benefit, cost, risk, and governability of persistent document state. Document AI lacks mechanisms coupling persistent state to document-native structure, provenance, and temporal validity. The next era of Document AI will be defined by what systems retain across documents, sessions, and time.

Figures & tables

Panel of Figure 1 — Fig. 1: The statelessness bottleneck . (a) Current systems process each document as an isolated function and reset after every invocation. (b) The proposed paradigm incorporates each document into persistent state at every session, producing an output at each step: $D_{1}$ yields $y_{1}$ and state $S_{1}$ ; $S_{1}$ is carried forward and merged with the validated evidence of $D_{2}$ to produce $S_{2}$ , from which the informed answer $y_{2}$ is generated. Processing $D_{1}$ therefore changes how $D_{2}$ is understood.

Panel of Figure 2 — Fig. 2: The capability map of Document AI under the placement rubric of Section II : horizontal axis = broadest document scope; vertical axis = highest persistence regime. The cross-modal representation lineage saturates at Page scope with no persistence; retrieval systems reach External-store persistence at Collection scope; text-only agent memory reaches Episodic persistence over conversation streams, off the document-structure axis. The Semantic and Governed band is empty at every scope; the red target marks the conjunctive capability $\mathcal{C}^{*}$ . That emptiness is the statelessness bottleneck .

Method	Year	Modality	Pretraining objective	Layout	Visual	OCR-free	Native state
BERT [ 66 ]	2019	Text	MLM	–	–	–	–
LayoutLMv1–v3 [ 4 , 5 ]	2020–22	Text+Layout+Vision	MLM/MIM	✓	✓	–	–
EAML [ 47 ]	2021	Image+Text	Mutual learning	–	✓	–	–
SelfDoc [ 48 ]	2021	Cross-modal	Self-supervised	✓	✓	–	–
UDoc [ 49 ]	2021	Cross-modal	Unified alignment	✓	✓	–	–
DiT [ 55 ]	2022	Vision	MIM	Implicit	✓	✓	–

TABLE I: Representation lineage in Document AI. The final column indicates whether the architecture natively maintains persistent state across sessions. No representation family produces state that survives the episode.

Paradigm	Multimodal	Doc structure	Persistent	Temporal	Provenance	Governance
Continual learning	–	–	$\checkmark$	–	–	–
Agent memory	Partial	–	$\checkmark$	Partial	Partial	–
Multimodal RAG	$\checkmark$	$\checkmark$	Ext. index	–	Page	–
Commercial LLM memory	Partial	–	Context only	–	–	–
Target $\mathcal{C}^{*}$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

TABLE II: Adjacent persistent-state paradigms against the conjunctive requirement

\mathcal{C}^{*}

. No existing paradigm satisfies all six properties jointly.

Panel of Figure 5 — Fig. 3: The persistent document memory architecture. Six lifecycle operations act on the six components of $S_{t}$ (Equation ( 4 )). Provenance ( $P_{t}$ ) is an invariant across all operations. The authorization state $A_{t}$ governs access control. Forget removes the target across all stores and repairs derived abstractions.

Substrate	Persist	Inspect	Relate	Generalize	Delete	Ground
Vector store	$\checkmark$	$\checkmark$	Partial	Partial	$\checkmark$	Partial
Knowledge graph	$\checkmark$	$\checkmark$	$\checkmark$	Partial	$\checkmark$	$\checkmark$
Document database	$\checkmark$	$\checkmark$	$\checkmark$	–	$\checkmark$	$\checkmark$
Parametric update	$\checkmark$	–	Implicit	$\checkmark$	–	–
Episodic memory	$\checkmark$	$\checkmark$	Partial	–	$\checkmark$	$\checkmark$
Hybrid	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

TABLE III: Candidate substrates for the six components of

S_{t}

. Cells record whether the substrate natively provides each property, under the coding rules stated in Section III-E . Columns: Persist (survives sessions), Inspect (human-readable), Relate (relational structure), Generalize (abstraction), Delete (verifiable removal), Ground (bounding-box traceability). No single substrate dominates; hybrid composition is required.

Benchmark	Sequential	Cross-doc	Family	Temporal	Provenance	Forgetting	Session	Answer.
DocVQA [ 71 ]	–	–	–	–	Partial	–	–	–
MP-DocVQA [ 93 ]	–	–	–	–	Page	–	–	–
DUDE [ 94 ]	–	–	–	–	Page	–	–	Yes
LongDocURL [ 1 ]	–	–	–	–	Box	–	–	–
M-LongDoc [ 97 ]	–	Partial	–	–	Partial	–	–	–
ViDoRe [ 7 ]	–	–	–	–	Page	–	Ext. index	–

TABLE IV: Statefulness audit of representative Document AI benchmarks. “Partial” indicates a neighboring capability without the full persistent-state criterion. The LoCoMo/SYNAPSE row represents adjacent-field (text-only conversational) evaluation. The final row is the specification from Section III , not an empirical result.

Panel of Figure 8 — Fig. 4: The longitudinal benchmark harness. The warm path processes $D_{1}\dots D_{k}$ and retains $S_{k}$ ; the cold baseline starts from $S_{0}=\emptyset$ . Both are evaluated on the identical future task $(D_{k+1},q_{k+1})$ . The difference $\Delta_{Q}$ (Experience Gain) isolates the contribution of persistent state. The remaining four metrics are computed from related conditions of this harness.

Panel of Figure 9 — Fig. 5: The three governance challenges of persistent evidence-grounded state. (a) Memory contamination through unvalidated writes. (b) Bitemporal supersession of evolving documents. (c) Five-level unlearning hierarchy; current Document AI implements Levels 1–2 only.

Panel of Figure 10 — Fig. 6: Five-level hierarchy for document-aware unlearning. Passing one level does not imply success at the next. Current Document AI implements Levels 1–2; Levels 3–5 remain unbuilt. Levels 1–3 are evaluated by Forgetting Fidelity ( $FF$ ) and Coverage Retention ( $CR$ ); Levels 4–5 require additional certification.

Persistent Memory for Language Models

Persistent memory for language models studies how models retain, update, and use information across interactions and beyond their context windows.

See all 272 papers

May 25, 2026

Is Agent Memory a Database? Rethinking Data Foundations for Long-Term AI Agent Memory

Long-running AI agents need persistent memory. Memory supports learning across sessions, reduces repeated context injection, and enables auditing of past decisions. Current agent memory systems and database paradigms treat memory as storage. They localize correctness at records, embeddings, or edges. Each supplies only some of the capabilities that long-term memory requires. The result is four recurring failure modes: unregulated growth, missing semantic revision, capacity-driven forgetting, and read-only retrieval. In our vision, long-term agent memory is a new data-management workload. Its correctness is a property of the state trajectory, not of individual records. We formalize this as Governed Evolving Memory (GEM). GEM replaces record-level database operations with four state-level operators: ingestion, revision, forgetting, and retrieval. Six correctness conditions govern how the state evolves. Three structural observations establish that no record-level system can satisfy these conditions, regardless of the storage model. We realize the abstraction in MemState, a prototype on a property-graph backend. MemState validates feasibility and exposes the gap to a native engine. We outline three research directions that define memory-centric data management as a workload.

Abdelghny Orogat, Essam Mansour

Concordia University

Agent Memory Memory-Augmented Agents Agent Memory Management Agent Memory Governance

Ask PDF

Jul 21, 2026

DocAtlas: Long-Document Understanding as Mutable-State Interaction

Long-document understanding requires models to find and combine evidence across many pages, layouts, tables, figures, and charts. Existing retrieval-augmented systems usually select evidence from a static index before generation, while recent agentic systems add multi-turn tool use but often rely on frozen proprietary backbones whose behavior is set by prompts. We present DocAtlas, a system that treats long-document understanding as a mutable-state information-seeking process. We instantiate DocAtlas as a mutable document harness: an external environment that determines what document information is searched, read, stored, reviewed, and shown to the model at each step. Given a document and question, the harness exposes search, reading, note-taking, and review tools, maintains a hierarchical tree and note store, and updates both as the agent records evidence. DocAtlas combines self-improving retrieval, selective evidence access, and active working memory under a fixed context budget. The same harness supports inference-time use with large VLMs and end-to-end reinforcement learning for compact VLM agents. With GPT-5.4, DocAtlas reaches 71.4% on MMLongBench-Doc, exceeding the human-expert reference of 65.8%. A Qwen3.5-4B VLM trained with end-to-end RL in the DocAtlas environment reaches 63.7%, compared with a 54.4% direct-input baseline, showing that mutable document-harness design can improve compact document agents by a large margin.

Hongchen Wei, Yuanzhe Wang, Bei Liu +8

Wuhan University · Microsoft

Document Understanding Long-Context QA Long-Context Modeling Agentic RAG Document QA

Ask PDF

Jun 9, 2026

Infini Memory: Maintainable Topic Documents for Long-Term LLM Agent Memory

Long-term LLM agents need persistent memory that can track changing facts and provide relevant evidence across sessions. Existing memory systems often store observations as isolated records, summaries, or indexed fragments, which makes evidence aggregation, fact revision, and memory maintenance difficult. We propose Infini Memory, a maintainable text-based persistent memory architecture that treats agent memory as topic-structured documents. Each topic document serves as a semantic unit for collecting related evidence, preserving metadata, and revising facts over time. New observations are first staged in a buffer and periodically consolidated into coherent textual contexts. At inference time, an agentic retrieval procedure lets the LLM read memory through iterative tool calls rather than a single retrieval step. On MemoryAgentBench, Infini Memory achieves 64.7% overall score. Ablations show that topic-structured maintenance and iterative evidence inspection improve complementary aspects of long-term memory use.

Suozhao Ji, Baodong Wu, Zehao Wang +8

1Infinigence AI · 2Tsinghua University · 3Shanghai Jiaotong University

Agentic RAG Persistent Memory for Language Models Memory Consolidation Agent Memory Management LLM Agent Memory Memory Management for LLM Agents+1

Ask PDF

Abstract

Persistent Memory for Language Models

Explore similar work

Is Agent Memory a Database? Rethinking Data Foundations for Long-Term AI Agent Memory

DocAtlas: Long-Document Understanding as Mutable-State Interaction

Infini Memory: Maintainable Topic Documents for Long-Term LLM Agent Memory