MEMONDEMAND: A Memory Management System for Large-Scale Enterprise Data

Xinyuan Song; Bowen Zhu; Hasibul Haque; Liang Zhao

cs.AIAug 22, 2026

MEMONDEMAND: A Memory Management System for Large-Scale Enterprise Data

Authors: Xinyuan Song, Bowen Zhu, Hasibul Haque, Liang Zhao

Organizations: Causal Dynamics Lab, USA · Emory University, USA

Abstract

Enterprise repositories are large, heteroge- neous, and continuously updated, making re- trieval difficult when efficient access, source- faithful evidence, and cross-query adaptation must be supported together. Enterprise mem- ory extends retrieval beyond the model con- text, but existing systems do not jointly address collection-specific hierarchy construction, low- cost routing, detailed evidence loading, and workload-aware memory updates at this scale. We introduce MEMONDEMAND, short for On- Demand Memory, a memory management sys- tem with three coordinated mechanisms: a dy- namic multi-level hierarchy that determines the abstraction structure and depth for each col- lection, dual memory at every hierarchy level that separates distilled routing from detailed evidence, and on-demand memory promotion that updates node priority under a bounded active-state budget. On EnterpriseRAG-Bench, MEMONDEMAND outperforms the strongest published LB#1 result at every evaluated scale from 10M tokens through the complete 618M- token collection, with gains of 12.23% at 10M and 4.66% at 618M. Results on FinanceBench, HotpotQA, and FRAMES further show strong performance across financial, multi-hop, and fact-retrieval settings. Together, these results establish MEMONDEMAND as an accurate, ef- ficient, and scalable memory solution for very large enterprise repositories across data scales, domains, and evidence requirements. Our code is available at https://github.com/ xfab-xinyuansong/MemOnDemand.git.

Figures & tables

Figure 1: MemOnDemand overview. (a) Bottom-up construction: A controller builds a data-dependent hierarchy from L0 sources using Create , Recluster , and Stop , with each node linking distilled and detailed memory. (b) Top-down retrieval: Queries traverse distilled hierarchy nodes and search L0 directly; both routes are fused and deduplicated by source ID. (c) Selective evidence loading: Promotion-aware ranking selects detailed L0 evidence under budget $B$ , and answers cite only loaded sources. (d) On-demand promotion: An active set of size at most $K$ prioritizes useful nodes through promotion, refresh, decay, and demotion, with transitions logged.

Method	Hierarchical organization	Dynamic hierarchy	Dual memory	Selective loading	Persistent priority	Separate $B/K$
RAPTOR ( Sarthi et al., 2024 )	✓	✗	✗	✓	✗	✗
HiRAG ( Huang et al., 2025 )	✓	✓	✗	✓	✗	✗
H-MEM ( Sun et al., 2026a )	✓	✗	✗	✓	✓	✗
HippoRAG 2 ( Gutiérrez et al., 2025 )	✓	✗	✗	✓	✗	✗
MemoRAG ( Qian et al., 2025 )	✗	✗	✓	✓	✗	✗
REFRAG ( Lin et al., 2025 )	✗	✗	✓	✓	✗	✗

Table 1: Comparison with closely related retrieval and memory systems. We compare dynamic hierarchy construction, separation of compact routing and detailed evidence (dual memory), selective evidence loading, persistent cross-query priorities, and separate budgets for answer context (

B

) and active state (

K

Tier	Combined $\uparrow$	$\Delta$ Combined	Correct $\uparrow$	Complete $\uparrow$	DocRcl $\uparrow$	Evid. F1 $\uparrow$	InvDoc $\downarrow$	Promo	Demote
	(%)	(%)	(%)	(%)	(%)
LB#1	68.22	–	81.60	72.86	79.02	–	0.4700	–	–
10M	82.75	+14.53	91.40	85.83	92.71	0.928	0.1200	790	762
20M	81.66	+13.44	90.80	84.08	90.04	0.900	0.1700	811	784
60M	73.09	+4.87	80.45	76.97	80.30	0.780	0.4300	864	839
100M	76.52	+8.30	85.80	79.43	83.51	0.833	0.3200	861	839

Table 2: Primary scaling results. LB#1 is the published EnterpriseRAG baseline;

\Delta

Combined reports absolute percentage-point differences. Promo and Demote count state transitions over 500 queries. FULL is the official 618M-token collection, while Expanded extends it to 1.14B tokens (Appendix A ).

Scale	Memory provided for answering	Answer-input tokens/query	Change (%)	Combined (%) $\uparrow$
10M	Detailed only	46,994	–	82.68
10M	Both representations	49,174	+4.6%	70.60
10M	Distilled only	2,963	-93.7%	51.08
10M	Selective detailed memory ( MemOnDemand )	16,944	-63.9%	82.75
20M	Detailed only	45,784	–	77.91
20M	Both representations	51,078	+11.6%	68.22

Table 3: Token consumption across memory policies at 10M and 20M. Changes are relative to detailed-only loading. Selective-detail quality uses revised prompts, while token counts follow matched packing traces.

Scale	Strategy	Answer-input tokens/query	Combined (%) $\uparrow$	Correct (%) $\uparrow$	DocRcl (%) $\uparrow$
10M	Flat dense RAG	6,001	74.48	74.6	52.91
	BM25 RAG, no promotion	5,726	29.68	34.4	24.43
	Hierarchical, detailed only	46,994	46.80	52.2	70.03
	Direct long-context packing	93,318	44.44	49.4	78.05
	MemOnDemand	16,944	82.75	91.40	92.71
20M	Flat dense RAG	6,018	71.68	71.8	48.09

Table 4: Comparison with direct retrieval strategies. All methods share 500 questions, evaluation models, and embeddings; MemOnDemand uses the revised answer prompt.

Method	DocRcl (%) $\uparrow$	Correct (%) $\uparrow$	Combined (%) $\uparrow$	$\Delta$ vs. Full (%)
System variants
Full system	92.71	91.40	82.75	–
One-step navigation	59.41	79.4	64.52	$-18.23$
No promotion	59.60	82.7	64.20	$-18.55$
25-candidate pool	57.44	78.4	62.64	$-20.11$
Retrieval baselines

Table 5: Ablation and retrieval comparisons at 10M. Variants modify individual components;

\Delta

reports the Combined difference from the full system in %.

Benchmark	Questions	DocRcl (%) $\uparrow$	Evid. F1 $\uparrow$	Correct (%) $\uparrow$	InvDoc $\downarrow$
FinanceBench	150	80.00	0.729	66.0	0.289
HotpotQA	500	49.90	0.549	78.4	0.320
FRAMES	500	88.76	0.617	55.0	0.439

Table 6: External results. Retrieval and answer quality for financial QA, multi-hop reasoning, and fact retrieval.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Scale and source composition of the extension. Approximately 522M generated tokens extend the unchanged 618M-token base to 1.14B tokens. Source-specific generation quotas reproduce the empirical source mixture of the official collection.

Promotion strategy	Wall time
On-demand promotion	1.46 s
Eager promotion of all parent nodes	593.5 s
Wall-time ratio	$407\times$

Appendix

Table 7: Preparation wall time under different promotion strategies at 20M. On-demand promotion avoids eagerly preparing every promotable parent node.

Online quantity	Value
Mean navigation steps per query	2.9
95th-percentile navigation steps	3
Questions activating promotion	470 / 500 (94.0%)

Appendix

Table 8: Promotion activity at 20M. Statistics cover the complete set of 500 evaluation queries.

Operation	API identifier
L0 distillation and key facts	gpt-5.4-mini
Higher-level decision and abstraction	gpt-5.4
Navigation and source selection	gpt-5.4
Answer generation	gpt-5.4
EnterpriseRAG evaluation	gpt-5.4
Semantic indexing	text-embedding-3-small

Appendix

Table 9: API assignment. GPT-5.4 mini is used for L0 memory construction, while GPT-5.4 handles higher-level construction(L1 L2 and L3), retrieval, answering, and evaluation.

Tier	L0 nodes	L1 nodes	L2 nodes
10M	8,927	94	9
20M	17,051	287	9
60M	49,978	223	14
100M	82,779	287	16
150M	124,330	352	18
250M	207,179	455	21

Appendix

Table 10: Realized hierarchy sizes. The dynamic construction procedure determines the number of nodes at each level for every corpus tier. FULL is the unmodified 618M-token benchmark collection; Expanded is the merged 1.14B-token collection described in Section A .

Floor	10M	150M	FULL (618M)
0.1	77.86	72.89	68.69
0.2	75.35	70.99	67.89
0.3	79.58	76.15	66.04
0.4	80.35	74.22	70.24
0.5	74.95	75.16	67.96
0.6	78.71	73.48	67.10

Appendix

Table 11: Promotion-floor sensitivity. Combined (%) is reported at each floor for three collection sizes. The best scores occur at 0.9 for 10M, 0.3 for 150M, and 0.4 for FULL (618M); bold marks the maximum in each column.

Figure 3: Promotion-floor sensitivity at 10M. The highest tested Combined score is 82.75% at a floor of 0.9.

Figure 4: Promotion-floor sensitivity at 150M. The highest tested Combined score is 76.15% at a floor of 0.3.

Figure 5: Promotion-floor sensitivity on the complete 618M-token collection. The highest tested Combined score is 70.24% at a floor of 0.4.

Scale	DocRcl (%) $\uparrow$	Correct (%) $\uparrow$
10M	68.25	81.2
20M	65.19	80.2
60M	55.02	74.2
100M	59.04	74.4
150M	55.30	77.2

Appendix

Table 12: Global source-leaf retrieval across corpus scales. Document Recall and Correct are reported as percentages.

Runtime component	Mean	Median	p95
Retrieval-control input	3,758	3,969	4,270
Retrieval-control output	155	155	186
Answer input	13,629	13,284	19,413
Answer output	142	124	291
Total online tokens	17,683	17,425	23,534

Appendix

Table 13: Online token use at 20M. Columns report the mean, median, and 95th-percentile token count per query. Total tokens include retrieval-control input and output and answer input and output.

Figure 6: Scaling performance and memory activity. MemOnDemand maintains a Combined advantage over LB#1 across all official corpus tiers, including FULL (618M tokens). The Expanded collection (1.14B tokens) further evaluates billion-token scalability, while promotion and demotion track adaptive memory management.

Figure 7: Quality–context configurations at 20M. Complete strategies use their own evidence-loading policies; the horizontal axis is logarithmic.

Figure 8: Quality and answer-input cost. Labels report Combined for the evidence-loading policies in Table 3 ; selective loading reaches 82.75% at 10M and 81.66% at 20M.

Figure 9: Component and configuration sensitivity. Full-system Combined is 82.75%; the three variants reach 64.52%, 64.20%, and 62.64%. Parenthesized labels show absolute score differences from the full system. Promotion removal disables both the current-query bonus and the managed-state update.

Figure 10: Up-front availability at 20M. On-demand preparation takes 1.46 seconds; eager parent preparation is estimated at 593.5 seconds, a

407\times

ratio.

Figure 11: External benchmark profile. Document Recall is 80.00%, 49.90%, and 88.76% on FinanceBench, HotpotQA, and FRAMES, respectively. Evidence F1 is multiplied by 100 for display; all values match Table 6 .

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 792 papers

MegaMem: A Retrieval Solution for Ultra-Large Context Windows
Aug 22, 2026Xinyuan Song, Bowen Zhu, Hasibul Haque +1Retrieval-Augmented Generation Persistent Memory for Language Models
Causal Dynamics Lab, USA · Emory University, USA
Mem++: Non-Destructive Memory for Long-Term Organizational LLM Agents
Oct 1, 2026Ahmad Yehia, Aly O. Abdelkareem, Islam Ahmed +4LLM Agent Memory Persistent Memory for Language Models
The University of Texas at Austin · AIDAChip Inc.
EGCForest: Structured Agent Memory Organization for Enterprise Group Chats
Oct 14, 2026Lingxi Cui, Jinhao Ruan, Wenqing Wei +4Conversational Memory Evidence Retrieval
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Xiaohongshu

Aug 22, 2026J/K move · Enter open · S save

MegaMem: A Retrieval Solution for Ultra-Large Context Windows

Xinyuan Song, Bowen Zhu, Hasibul Haque, Liang Zhao

Causal Dynamics Lab, USA · Emory University, USA

Retrieval-Augmented Generation Persistent Memory for Language Models Information Retrieval

Open paper AskPDF arXiv

Modern language models and agents increasingly require persistent memory for complete codebases, long interaction histories, and heterogeneous enterprise records. The key challenge is to keep hundreds of millions of tokens searchable while passing only bounded source evidence to the answer model. We introduce MegaMem, a source-resolved dual-view retrieval system that separates semantic access from generation evidence. Distilled records and detailed evidence are searched with original and transformed queries; every distilled hit resolves to an immutable source ID before reciprocal-rank fusion, deduplication, and cross-encoder reranking; and only the highest-ranked detailed evidence within a fixed budget supports generation. Post-answer attribution then identifies which loaded sources support the fixed answer. We evaluate MegaMem on EnterpriseRAG-Bench, which contains more than 500,000 heterogeneous enterprise documents and approximately 650M tokens. MegaMem improves Overall from 68.22 to 82.26 and reaches 86.50 Correctness. These results show that MegaMem supports ultra-large persistent memory while preserving strong answer accuracy under a bounded generation context. By separating searchable memory scale from answer-context size, MegaMem provides a practical path toward accurate retrieval over memories ranging from hundreds of millions to one billion tokens. Our code is available at https://github.com/ xfab-xinyuansong/MegaMem.git.

Abstract

Retrieval-Augmented Generation

Explore similar work

MegaMem: A Retrieval Solution for Ultra-Large Context Windows

Mem++: Non-Destructive Memory for Long-Term Organizational LLM Agents

EGCForest: Structured Agent Memory Organization for Enterprise Group Chats