MemCoT: Test-Time Scaling through Memory-Driven Chain-of-Thought

Haodong Lei; Junming Liu; Yirong Chen; Ding Wang; Hongsong Wang

cs.MAApr 9, 2026

MemCoT: Test-Time Scaling through Memory-Driven Chain-of-Thought

Authors: Haodong Lei, Junming Liu, Yirong Chen, Ding Wang, Hongsong Wang

Organizations: Southeast University Nanjing, Jiangsu, China · Shanghai Artificial Intelligence Laboratory Shanghai, China

Abstract

Large Language Models (LLMs) still suffer from severe hallucinations and catastrophic forgetting during causal reasoning over massive, fragmented long contexts. Existing memory mechanisms typically treat retrieval as a static, single-step passive matching process, leading to severe semantic dilution and contextual fragmentation. To overcome these fundamental bottlenecks, we propose MemCoT, a test-time memory scaling framework that redefines the reasoning process by transforming long-context reasoning into an iterative, stateful information search. MemCoT introduces a multi-view long-term memory perception module that enables Zoom-In evidence localization and Zoom-Out contextual expansion, allowing the model to first identify where relevant evidence resides and then reconstruct the surrounding causal structure necessary for reasoning. In addition, MemCoT employs a task-conditioned dual short-term memory system composed of semantic state memory and episodic trajectory memory. This short-term memory records historical search decisions and dynamically guides query decomposition and pruning across iterations. Empirical evaluations demonstrate that MemCoT establishes a state-of-the-art performance. Empowered by MemCoT, several open- and closed-source models achieve SOTA performance on the LoCoMo benchmark and LongMemEval-S benchmark.

Figures & tables

Figure 1. The answer prediction

F_{1}

scores (%) of GPT-4o-mini on the LoCoMo benchmark. MemCoT achieves a state-of-the-art overall

F_{1}

score of 58.03%, significantly outperforming all baseline memory systems across all sub-tasks.

Panel of Figure 2 — Figure 2. Analysis of reasoning and retrieval failures in long-context: (a) A relational diagram illustrating the dependencies between evidence and answers in complex single-hop and multi-hop reasoning scenarios. The red circle $A$ denotes the final answer, while the blue circles $e_{i}$ represent the supporting evidence. (b) A Pareto chart depicting the distribution of chunk distances (ranging from 1 to 880) between the falsely retrieved chunks and the ground-truth answer chunks evaluated on the LoCoMo dataset. (c) A magnified Pareto chart detailing the chunk distances within a narrowed range of 1 to 100 on the LoCoMo dataset.

Figure 3. The overall architecture of the MemCoT framework. It illustrates the scaling test-time compute memory loop, driven by a multi-view memory perception module for long-term memory extraction and a short-term memory evolution module for iterative query $q_{j}$ updates in the $j-$ th iteration. On one hand, the retrieved memory $m_{j}$ provides factual anchors that empower the Judge Agent to construct a more rigorous and grounded CoT for reasoning. On the other hand, the reasoning process explicitly identifies the deficiencies in the retrieved memory $m_{j}$ relative to the current query $q_{j}$ and refines the $q_{j}$ into the next query $q_{j+1}$ to optimize the next retrieved memory $m_{j+1}$ . This bidirectional synergy loop establishes the foundation of our Memory-driven Chain of Thought paradigm.

Model	Method	Single-hop	Multi-hop	Open-domain	Temporal	Overall
GPT-4o-mini ( Cao et al., 2025 )	Full Context	45.89	25.01	15.49	23.83	35.57
	RAG ( Lewis et al., 2020b )	52.19	32.17	23.21	30.77	42.25
	Mem0 ( Chhikara et al., 2025 )	47.65	38.72	28.64	48.93	45.10
	MemoryOS ( Li et al., 2025b )	48.62	35.27	20.02	41.15	42.84
	HippoRAG ( Gutiérrez et al., 2025 )	54.84	33.59	28.59	48.17	47.92
	A-Mem ( Xu et al., 2025b )	44.65	27.02	12.14	45.85	39.65

Table 1. Performance (

F_{1}

score) comparison on the LoCoMo benchmark, covering single-hop, multi-hop, open-domain, and temporal settings. Best results are highlighted in bold , and second-best results are underlined .

Model	Method	Single-Hop	Multi-Hop	Open-domain	Temporal	Overall
Qwen2.5-7B ( Qwen et al., 2025 )	Full Context	10.61	3.39	8.24	3.72	7.38
	RAG ( Lewis et al., 2020b )	21.72	8.05	14.51	4.89	13.88
	A-Mem ( Xu et al., 2025b )	12.73	8.67	13.86	6.28	12.1
	EpMem ( Terranova et al., 2025 )	17.64	6.35	2.66	9.96	6.63
	Mem0 ( Chhikara et al., 2025 )	24.96	20.31	32.74	33.16	28.75
	MemoryOS ( Li et al., 2025b )	29.55	21.03	40.85	26.26	33.74

Table 2. Performance (

F_{1}

score) comparison on the LoCoMo benchmark using the tiny model Qwen2.5-7B model, highlighting MemCoT’s consistent superiority even in smaller-scale architectures.

Model	Method	SSU	MS	SSP	TR	KU	SSA	Overall
GPT-4o-mini ( Cao et al., 2025 )	Full Context	78.6	38.3	6.7	42.1	78.2	89.3	55.0
	RAG ( Lewis et al., 2020b )	88.6	47.4	70.0	63.2	70.5	91.1	67.2
	Mem0 ( Chhikara et al., 2025 )	91.4	66.2	34.0	63.9	74.4	96.4	71.1
	Zep ( Rasmussen et al., 2025 )	92.9	47.4	53.3	54.1	74.4	75.0	63.2
	Nemori ( Nan et al., 2025 )	88.6	51.1	46.7	61.7	61.5	83.9	64.2
	EMem-G ( Zhou and Han, 2025 )	87.0	73.6	32.2	74.8	94.4	87.5	77.9

Table 3. Performance (LLM-as-a-Judge score) comparison on the LongMemEval-S benchmark. Best results are highlighted in bold , and second-best results are underlined .

Figure 4. Effects of different parameters: (a) Impact of the adjacent window size ( $W$ ) on memory retrieval Recall (%) and reasoning $F_{1}$ score (%) using the Qwen2.5-7B model. (b) Impact of the maximum evolution iterations ( $J$ ) on memory retrieval Recall (%) using the Qwen2.5-7B model. (c) Impact of Fine-grained TopK ( $K$ ) on memory retrieval Recall (%) using the Qwen2.5-7B model.

Method	Multi-hop	Temporal	Open	Single-hop	Overall
MemCoT	40.43	60.31	42.67	64.81	58.03
w/o $\psi_{\text{z-o}}$	35.48	55.13	38.58	61.35	53.89
w/o $\psi_{\text{vis}}$	39.21	59.73	39.58	63.82	56.19
w/o $\psi_{\text{z-o+vis}}$	34.78	50.88	41.49	59.62	53.71

Table 4. Ablation study results on the LoCoMo benchmark using Qwen2.5-14B (

F_{1}

scores).

\psi_{\text{z-o}}

and

\psi_{\text{vis}}

denote zoom-out and visual modules, respectively.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	$F_{1}$ scores	Token
GPT-4o-mini ( Cao et al., 2025 )	Full Context	35.57	16,910
	RAG ( Lewis et al., 2020b )	42.25	649
	Mem0 ( Chhikara et al., 2025 )	45.10	1,172
	MemoryOS ( Li et al., 2025b )	42.84	1,589
	A-Mem ( Xu et al., 2025b )	39.65	2,520
	CompassMem ( Hu et al., 2026a )	52.18	20,100

Appendix

Table 5. The analysis of the cost and

F_{1}

scores in the LoCoMo benchmark. Best results are highlighted in bold , and second-best results are underlined .

Model	Average Steps
	MH	TP	OD	SH	Overall
GPT-4o-mini ( Cao et al., 2025 )	1.83	2.41	3.25	1.52	1.87
Qwen2.5-7B ( Qwen et al., 2025 )	1.90	2.69	2.55	1.76	2.03

Appendix

Table 6. The analysis of the average steps by category in the LoCoMo benchmark. The MH, TP, OD, and SH represent Multi-hop, Temporal, Open-domain, and Single-hop, respectively.

Type	Method	Single-hop	Multi-hop	Open	Temporal	Overall
	Full Context	45.89	25.01	15.49	23.83	35.57
	RAG ( Lewis et al., 2020b )	52.19	32.17	23.21	30.77	42.25
Non graph	Mem0 ( Chhikara et al., 2025 )	47.65	38.72	28.64	48.93	45.10
	A-Mem ( Xu et al., 2025b )	44.65	27.02	12.14	45.85	39.65
	MemCoT(w NaïveRag ( Lewis et al., 2020b ) )	65.38	43.85	32.99	56.10	57.73
	MemoryOS ( Li et al., 2025b )	48.62	35.27	20.02	41.15	42.84

Appendix

Table 7. The analysis of different types of retrieval methods in the LoCoMo benchmark. Best results are highlighted in bold . The model is GPT-4o-mini.

Model	Method	Multi-hop	Temporal	Open-domain	Single-hop	Overall	Token
GPT-4.1-mini	Full Context	25.02	12.04	19.05	18.68	18.70	16,910
	MemoryBank ( Zhong et al., 2024 )	5.00	5.94	5.16	5.72	5.46	432
	MemGPT ( Packer et al., 2024 )	17.72	19.44	11.29	25.59	18.51	16,977
	A-Mem ( Xu et al., 2025b )	25.06	51.01	13.22	41.02	32.58	2,520
	Mem0 ( Chhikara et al., 2025 )	30.14	48.91	16.43	41.3	34.20	1,764
	Zep ( Rasmussen et al., 2025 )	26.73	20.97	21.24	39.96	32.40	1,602

Appendix

Table 8. Performance on the LoCoMo benchmark with Advanced Models GPT-4.1-mini.Best results are highlighted in bold , and second-best results are underlined .

Figure 5. The specific comparison of different methods. The orange mark indicates the keywords of the truth label.

Explore similar work

CoEM: Empowering Long-Context Reasoning with Commit-on-Evidence Memory
Sep 29, 2026Jingguang Li, Yebo Wu, Zuyi Guo +7Memory-Augmented Language Models Persistent Memory for Language Models
Korea Advanced Institute of Science & Technology · University of Macau · Massachusetts Institute of Technology
PI-Mem: Pushing Long-Context Reasoning to 3.6M Tokens with Parallel-Iterative Memory
Aug 4, 2026Dawei Liu, Haixu Song, Shuang Cheng +9Memory-Augmented Language Models Long-Context QA
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Tsinghua University +3
ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning
Jul 2, 2026Yanjun Zhao, Ruizhong Qiu, Tianxin Wei +6Long-Context Inference
University of Illinois Urbana-Champaign

Sep 29, 2026J/K move · Enter open · S save

CoEM: Empowering Long-Context Reasoning with Commit-on-Evidence Memory

Jingguang Li, Yebo Wu, Zuyi Guo, Kailang Ma+6

Korea Advanced Institute of Science & Technology · University of Macau · Massachusetts Institute of Technology

Memory-Augmented Language Models Persistent Memory for Language Models LLM Context Management

Open paper AskPDF arXiv

Long-context reasoning is essential for complex and long-horizon tasks, yet the performance of large language models (LLMs) degrades as context length increases. Recent approaches address this by processing input chunk by chunk while maintaining a bounded textual memory in model context. However, premature information compression can discard critical details essential for subsequent reasoning. In this paper, we introduce Commit-on-Evidence Memory (CoEM), which learns when to convert source evidence into compact memory facts. Specifically, under a fixed context-memory budget, CoEM preserves potentially useful source excerpts verbatim in a pending set, allowing subsequent context to clarify their relevance before irreversible compression. As new context arrives, a learned policy revisits each pending excerpt and decides whether to promote it to the committed memory, retain it for further consideration, or discard it. A frozen verifier ensures proposed facts are accepted only if supported by retained excerpts and current context. To further guide effective memory management, we train this policy using reinforcement learning by combining fine-grained, step-level evidence rewards with final answer rewards. Extensive experiments demonstrate that CoEM consistently improves long-context reasoning. When evaluated on 6,400 documents long-context input, CoEM outperforms the strongest memory baseline by 10.4-11.4 F1 points on Qwen3.5-9B. Code repository: https://github.com/benmagnifico/CoEM.

Abstract

Explore similar work

CoEM: Empowering Long-Context Reasoning with Commit-on-Evidence Memory

PI-Mem: Pushing Long-Context Reasoning to 3.6M Tokens with Parallel-Iterative Memory

ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning